FPGA-Neural/docs/FPGA-NeuralNetwork-Engine.md

FPGA Neural Network Engine

Hardware Neural Network Engine based on FPGA + dedicated RAM.

The project implements a parametric hardware accelerator for neural networks, designed to be reusable across different embedded systems and applications.

The fundamental design principle is that the neural-network computation is performed entirely inside the FPGA, while the host system communicates with the engine through a simple hardware-independent interface such as SPI.

---

1. Project Goal

The goal of this project is to develop a reusable Neural Network Engine implemented in FPGA hardware.

The engine is composed of:

• FPGA;
• dedicated RAM connected to the FPGA;
• host interface, initially SPI and potentially Dual SPI.

The host system is not part of the neural-network computational datapath.

Possible host systems include:

• Linux SoCs;
• Raspberry-Pi-like systems;
• ESP32;
• microcontrollers;
• other embedded processors;
• development PCs.

The same Neural Network Engine architecture should therefore be usable in completely different systems.

                    HOST SYSTEM
             ┌─────────────────────────┐
             │                         │
             │ Linux / ESP32 / MCU     │
             │                         │
             │ Configuration           │
             │ Training                │
             │ Control                 │
             └────────────┬────────────┘
                          │
                       SPI / Dual SPI
                          │
                          ▼
             ┌─────────────────────────┐
             │          FPGA           │
             │                         │
             │ Neural Network Engine   │
             │                         │
             │ Compute / Control       │
             │                         │
             └────────────┬────────────┘
                          │
                          │
                    Dedicated RAM

---

2. Architectural Principle

The FPGA is the actual neural-network accelerator.

The RAM required by the neural network is physically associated with the FPGA and is not part of the host system memory.

The host only provides:

• configuration;
• network parameters;
• input data;
• control;
• result retrieval.

The neural-network calculations themselves are executed by the FPGA.

This separation is a fundamental architectural requirement.

---

3. Hardware Configuration vs Network Configuration

An important distinction is made between the hardware architecture of the accelerator and the parameters of the neural network.

3.1 FPGA hardware configuration

The physical architecture of the Neural Network Engine is defined when the FPGA design is synthesized and implemented.

Typical hardware parameters include:

N_INPUTS
N_NEURONS
N_LAYERS
PARALLEL
DATA_WIDTH
ACCUMULATOR_WIDTH

These parameters can therefore be Verilog/SystemVerilog parameters or equivalent synthesis-time configuration values.

For example:

N_INPUTS  = 32
N_NEURONS = 4
PARALLEL  = 8

defines a specific hardware implementation optimized for that architecture.

The resulting FPGA bitstream contains the corresponding datapath.

---

3.2 Neural-network configuration

Once the FPGA has been configured and initialized, the actual neural-network parameters can be loaded through the host interface.

These parameters may include:

• weights;
• biases;
• activation parameters;
• quantization parameters;
• network-specific constants.

These values are stored in the RAM associated with the FPGA.

Therefore:

FPGA BITSTREAM
        │
        │ defines hardware architecture
        ▼
┌─────────────────────┐
│ Neural Network      │
│ Hardware Engine     │
└──────────┬──────────┘
           │
           │ loads
           ▼
┌─────────────────────┐
│ Dedicated RAM       │
│                     │
│ weights             │
│ biases              │
│ parameters          │
│ buffers             │
└─────────────────────┘

This provides an important separation between hardware specialization and network data.

---

4. Application-Specific Neural Networks

The Neural Network Engine is not intended to implement one fixed neural network.

Instead, each application can define its own network.

For example:

Application A
    16 inputs
    8 neurons
    1 output

Application B
    32 inputs
    16 neurons
    4 outputs

Application C
    64 inputs
    multiple layers
    custom parallelism

The FPGA hardware can then be generated specifically for the required architecture.

This allows the design to exploit the FPGA resources efficiently rather than implementing a completely generic and potentially inefficient neural-network processor.

---

5. Parametric Compute Engine

The current implementation contains a parametric neural-network layer.

A validated configuration is:

N_INPUTS  = 32
N_NEURONS = 4
PARALLEL  = 8

The datapath processes the inputs in parallel groups.

Conceptually:

32 inputs
    │
    ├── 8 parallel MACs
    │
    ├── 8 parallel MACs
    │
    ├── 8 parallel MACs
    │
    └── 8 parallel MACs
             │
             ▼
        Accumulation
             │
             ▼
           Bias
             │
             ▼
        Activation
             │
             ▼
          Output

The architecture is intended to scale by changing the synthesis parameters.

---

6. Current Functional Validation

The 32 × 4 / PARALLEL = 8 configuration has been successfully simulated.

Test output:

========================================
PARAMETRIC LAYER TEST
N_INPUTS  = 32
N_NEURONS = 4
PARALLEL  = 8
========================================

PASS N0: 8192
PASS N1: 4096
PASS N2: 0 (ReLU)
PASS N3: 16384

========================================
PARAMETRIC TEST PASSED
========================================

Validated functionality:

• multiple inputs;
• multiple neurons;
• parallel MAC processing;
• accumulation across multiple input groups;
• bias handling;
• independent neuron outputs;
• ReLU activation;
• parametric layer architecture.

The corresponding test has been committed to the repository.

Commit:

test: validate parametric 32x4 layer with parallelism 8

---

7. FPGA Boot and Initialization

The FPGA is configured during system initialization using its normal FPGA configuration mechanism.

The FPGA bitstream defines the hardware architecture of the Neural Network Engine.

Conceptually:

Power-on
   │
   ▼
FPGA configuration
   │
   │ bitstream
   ▼
Neural Network Engine available
   │
   ▼
Host initialization
   │
   │ SPI
   ▼
Load network parameters
   │
   ▼
Load weights / biases
   │
   ▼
Engine ready

This means that the host does not dynamically construct the FPGA datapath during normal operation.

The datapath already exists in hardware.

The host configures the network data that the datapath operates on.

---

8. Host Interface

The primary external interface is intended to be:

SPI

with possible future support for:

Dual SPI

The interface must remain independent of the host operating system.

The same hardware protocol should therefore be usable from:

Linux
ESP32
MCU
PC

The host interface should provide access to:

• control registers;
• status;
• network configuration;
• RAM;
• input data;
• output data;
• start/stop control;
• completion status.

A conceptual command sequence is:

RESET
  │
  ▼
CONFIGURE
  │
  ▼
LOAD NETWORK PARAMETERS
  │
  ▼
LOAD WEIGHTS
  │
  ▼
LOAD BIASES
  │
  ▼
LOAD INPUT
  │
  ▼
START
  │
  ▼
WAIT FOR DONE
  │
  ▼
READ OUTPUT

8.1 SPI Protocol v1 (draft, 2026-09-02)

Concrete opcode-level draft of the section above, written before any Phase 4 RTL. Opcode values and the exact set of commands are illustrative/example at this stage, not frozen — the framing rules (MSB-first, explicit length, sticky STATUS.done) and the two decisions already made (explicit length field over CS-delimited streaming; a runtime READ_CONFIG command) are the parts intended to stick; the opcode table itself is expected to be revised as Phase 4 RTL work starts.

Physical layer: SPI Mode 0 (CPOL=0, CPHA=0), MSB-first, single SPI for v1 (Dual SPI is a future extension per §8, not addressed here). The FPGA is always SPI slave. One command per CS-low period; byte 0 of every transaction is the opcode.

Multi-byte fields are big-endian (most significant byte first). Byte addresses are ADDR_WIDTH-bit (22 bits today, from rtl/neuron_memory.v), carried in a 3-byte field with the top 2 bits reserved as 0.

Length is explicit, not CS-edge-delimited: WRITE_RAM/READ_RAM carry a 2-byte length field, so the SPI controller only needs a byte counter, not CS-edge detection mid-transfer.

Opcode table

Opcode	Name	Payload (host → FPGA)	Response (FPGA → host)	Function
0x00	NOP	—	—	No operation (idle/dummy clocking)
0x01	WRITE_RAM	addr(3B) + len(2B) + len data bytes	—	Write a block into PSRAM (X, weights, bias, network params)
0x02	READ_RAM	addr(3B) + len(2B)	len data bytes	Read a block back from PSRAM
0x0F	RESET	—	—	Synchronous reset pulse to the compute engine (neuron_memory) and clears the STATUS latch below. Does not erase PSRAM contents. Kept as a distinct opcode from NOP.
0x10	SET_BASE	sel(1B) + addr(3B)	—	Sets x_base(sel=0) / w_base(sel=1) / bias_addr(sel=2)
0x20	START	—	—	Pulses start; ignored (no-op) if busy=1
0x21	STATUS	—	1 byte	bit0=busy (live), bit1=done (sticky, clear-on-read), bits7:2 reserved=0
0x22	READ_OUTPUT	—	N_NEURONS bytes	y_bus, neuron-major (byte 0 = neuron 0)
0x30	READ_CONFIG	—	8 bytes	Hardware config record, see below

Why STATUS.done is sticky / clear-on-read: in rtl/neuron_memory.v done is a single-cycle pulse (asserted for exactly one clock in STATE_WAIT_N, deasserted the next cycle). A host polling over SPI — orders of magnitude slower than the FPGA clock — would almost certainly miss a raw one-cycle pulse. The SPI register bank must therefore latch done into a sticky bit on the pulse, and clear it when the host issues STATUS (or RESET), not sample the raw neuron_memory.done signal directly. busy has no such problem (it is level-held for the whole computation) and can be read live.

READ_CONFIG payload (fixed 8 bytes, lets one host firmware build work across different bitstreams without recompiling):

Byte(s)	Field	Source
0	ADDR_WIDTH (bits)	neuron_memory.ADDR_WIDTH
1–2	N_INPUTS (16-bit BE)	neuron_memory.N_INPUTS
3	N_NEURONS	neuron_memory.N_NEURONS
4	PARALLEL	neuron_memory.PARALLEL
5	DATA_WIDTH (bits)	neuron_memory.DATA_WIDTH
6–7	protocol version (16-bit BE)	0x0001 for this spec

Example session (fills in the conceptual sequence above with concrete opcodes):

RESET                 -> 0x0F
READ_CONFIG           -> 0x30           (host learns N_INPUTS/N_NEURONS/...)
WRITE_RAM (weights)   -> 0x01 ...
WRITE_RAM (biases)    -> 0x01 ...
SET_BASE (X/W/BIAS)   -> 0x10 x3
WRITE_RAM (input X)   -> 0x01 ...
START                 -> 0x20
poll STATUS           -> 0x21           (until done bit set; clears on this read)
READ_OUTPUT           -> 0x22

Not yet decided / explicitly out of scope for v1: Dual SPI framing, a CRC/checksum on transfers (SPI is assumed reliable for a board-level trace in v1), and multi-layer sequencing commands (that belongs to Phase 5, once intermediate buffers exist).

---

9. Dedicated FPGA RAM

The RAM is considered part of the Neural Network Engine.

It is not intended to be supplied by the host system.

Depending on the final architecture, RAM may contain:

Weights
Biases
Input buffers
Intermediate layer buffers
Output buffers
Network parameters

The memory architecture must be designed according to:

• number of parallel MAC units;
• data width;
• required bandwidth;
• number of layers;
• buffering requirements;
• FPGA block-RAM resources;
• possible external RAM requirements.

The preferred architecture is that the FPGA directly controls this memory.

---

10. Training

Training and inference are conceptually separated.

The first implementation does not require the FPGA to perform the complete training process.

Training can be performed externally:

PC / Linux / other host
        │
        │ training
        ▼
Network weights
        │
        │ SPI
        ▼
FPGA RAM

The FPGA then performs inference using the resulting parameters.

This approach greatly reduces the complexity of the initial hardware implementation.

However, the architecture should not prevent future implementation of hardware-assisted or fully hardware-based training.

---

11. Inference

During inference, the host only supplies input data and retrieves the result.

              HOST
                │
             Input
                │
                │ SPI
                ▼
        ┌───────────────┐
        │      FPGA     │
        │               │
        │ Neural Network│
        │    Engine     │
        │               │
        └───────┬───────┘
                │
             Output
                │
                │ SPI
                ▼
              HOST

The host is not involved in the individual MAC operations.

This provides:

• deterministic computation;
• reduced host workload;
• hardware parallelism;
• predictable latency;
• independence from the host CPU architecture.

---

12. Multi-Layer Architecture

The current implementation starts from a single parametrized layer.

The intended architecture is eventually:

Input
  │
  ▼
┌──────────────┐
│   Layer 0    │
└──────┬───────┘
       │
       ▼
┌──────────────┐
│   Layer 1    │
└──────┬───────┘
       │
       ▼
┌──────────────┐
│   Layer 2    │
└──────┬───────┘
       │
       ▼
    Output

Intermediate data will be stored in FPGA-controlled memory buffers.

The number and size of layers should ultimately be part of the hardware generation process.

---

13. Reusability

The main purpose of the architecture is reuse.

A future project should be able to use the same general Neural Network Engine architecture with a different hardware configuration.

For example:

Project A
    N_INPUTS  = 32
    N_NEURONS = 8
    PARALLEL  = 8

Project B
    N_INPUTS  = 64
    N_NEURONS = 16
    PARALLEL  = 16

Project C
    N_INPUTS  = 128
    N_NEURONS = 32
    PARALLEL  = 32

The HDL architecture remains conceptually the same while synthesis parameters generate an implementation appropriate for the target application.

---

14. Design Philosophy

The project should be considered a:

Reusable FPGA Neural Network Accelerator Platform

rather than a single neural-network implementation.

The application determines:

Input size
Network topology
Number of layers
Number of neurons
Parallelism
Numerical precision
Activation functions
Memory requirements
Performance requirements

The hardware generator then produces the corresponding FPGA implementation.

---

15. Development Roadmap

Phase 1 — Parametric Layer

• Parametric inputs
• Parametric neurons
• Parametric parallelism
• Accumulation
• Bias
• ReLU
• 32×4 / P=8 functional test

Phase 2 — Parameter Sweep

Validate multiple combinations of:

N_INPUTS
N_NEURONS
PARALLEL

including configurations where the number of inputs is not an exact multiple of the parallelism.

• Exact-multiple sanity configs (32×8, 64×32)
• Non-exact-multiple configs (30×8, 20×16)
• Degenerate config, PARALLEL > N_INPUTS (4×8)
• Sweep testbench: sim/parameter_sweep_tb.v

Findings — FIXED (2026-09-02):

• neuron_parallel.v computed GROUPS = N_INPUTS / PARALLEL with integer division. When N_INPUTS was not an exact multiple of PARALLEL, only the first GROUPS * PARALLEL inputs were ever read by the accumulator — the remainder was silently dropped (no error, no warning). Confirmed for 30×8 → only 24 of 30 inputs summed, and 20×16 → only 16 of 20 inputs summed.
• If PARALLEL > N_INPUTS, GROUPS = 0 and the controller's group_index == GROUPS-1 terminal condition was never satisfied: the neuron entered busy and never asserted done (confirmed hang, 500-cycle watchdog in the sweep bench).
• Both share the same root cause (N_INPUTS % PARALLEL != 0, degenerate PARALLEL > N_INPUTS included) and both are now rejected at elaboration time, in simulation and synthesis alike, by a generate guard added to neuron_parallel.v (instantiates a deliberately undefined module when the parameter combination is invalid — zero cost, zero behavior change for any valid configuration). The validated datapath itself (mac8/mac_unit/accumulation/ReLU/saturation) was not modified. See sim/neuron_parallel_guard_negative_nonmultiple_tb.v and sim/neuron_parallel_guard_negative_degenerate_tb.v for the negative-test proof, and sim/parameter_sweep_tb.v for the updated positive sweep (now valid-configs-only, including PARALLEL=2 and PARALLEL=4, the two best-performing values from docs/FPGA-Neural-Datapatch-Benchmark.md).

Phase 3 — Memory Architecture

Define:

• weight memory;

• bias memory;

• input buffers;

• output buffers;

• intermediate buffers;

• memory addressing;

• bandwidth requirements.

• neuron_memory.v: single-neuron memory integration (N_NEURONS=1)

• neuron_memory.v: multi-neuron memory integration (N_NEURONS>1)

• Intermediate/multi-layer buffers (deferred to Phase 5)

• Bandwidth analysis against PSRAM timing (deferred to Phase 7)

Multi-neuron design (2026-09-02): neuron_memory.v now takes an N_NEURONS parameter and loops over neurons in memory: X is read once (shared input vector), and for each neuron in turn W and bias are re-read from PSRAM and fed to a single, reused neuron_parallel instance — memory-bound by design, one neuron computed at a time, no change to the validated compute datapath. Addressing follows the same neuron-major convention as layer.v: neuron n's weights live at w_base + n*N_INPUTS bytes, its bias at bias_addr + n. Output is now y_bus (packed, DATA_WIDTH*N_NEURONS bits, neuron-major), replacing the old single-neuron y port. Validated end to end through the full memory stack (memory_interface + psram_controller + psram_model) in sim/neuron_memory_multi_tb.v (N_NEURONS=3: scale, larger value, ReLU). sim/neuron_memory_tb.v (N_NEURONS=1) still passes unchanged, confirming backward compatibility.

Phase 4 — SPI Interface

Implement:

• SPI controller;

• register map;

• RAM access;

• configuration protocol;

• input/output protocol;

• status and control.

• Protocol/opcode set drafted — see §8.1 SPI Protocol v1

• SPI controller RTL (physical layer: shift register, CS/clock sync)

• Register bank RTL (SET_BASE, sticky STATUS, READ_CONFIG constants)

• RAM access passthrough RTL (WRITE_RAM/READ_RAM -> memory_interface)

• Testbench (SPI master BFM + full stack, mirroring neuron_memory_tb.v style)

Phase 5 — Multi-Layer Network

Implement:

• multiple layers;
• intermediate buffers;
• layer sequencing;
• configurable activation functions.

Phase 6 — Host Software

Develop host-side drivers for:

• Linux;
• ESP32.

The same FPGA protocol should be usable by both.

Phase 7 — Optimization

Evaluate:

• pipeline depth;
• MAC parallelism;
• memory bandwidth;
• numerical precision;
• FPGA resource utilization;
• latency;
• throughput.

Phase 8 — Optional Hardware Training

Investigate:

• backpropagation;
• gradient calculation;
• weight updates;
• hardware-assisted training.

---

16. Long-Term Vision

The final objective is to create a reusable hardware block that can be integrated into different future MIKILAB projects.

                         APPLICATION
                              │
                ┌─────────────┴─────────────┐
                │                           │
             Linux                       ESP32
                │                           │
                └─────────────┬─────────────┘
                              │
                         SPI / Dual SPI
                              │
                              ▼
                 ┌────────────────────────┐
                 │          FPGA          │
                 │                        │
                 │ Neural Network Engine  │
                 │                        │
                 │ ┌────────────────────┐ │
                 │ │ Control            │ │
                 │ ├────────────────────┤ │
                 │ │ Input Interface    │ │
                 │ ├────────────────────┤ │
                 │ │ NN Compute Core    │ │
                 │ ├────────────────────┤ │
                 │ │ Activation         │ │
                 │ ├────────────────────┤ │
                 │ │ Output Interface   │ │
                 │ └────────────────────┘ │
                 │                        │
                 │ Dedicated RAM          │
                 │                        │
                 └────────────────────────┘

The host platform can change without changing the fundamental Neural Network Engine architecture.

The FPGA becomes a dedicated neural-computation peripheral, analogous to other hardware accelerators, but optimized specifically for the neural-network topology required by each application.

---

17. Current Status

Component	Status
Parametric neuron layer	OK Working
Parametric input count	OK
Parametric neuron count	OK
Parametric parallelism	OK
Accumulation	OK
Bias	OK
ReLU	OK
32×4 / P=8 validation	OK
Dedicated RAM architecture	- Design
SPI interface	Planned
Dual SPI	Future
Multi-layer engine	Planned
Linux host driver	Planned
ESP32 host driver	Planned
Hardware training	Future

---

Core architectural principle

The FPGA implements the neural-network machine.
The FPGA owns its RAM.
The host configures and uses the machine.
The network topology is specialized at FPGA build time, while its trained parameters are loaded into FPGA-local memory at initialization.

This separation is the foundation of the project.