working with memory
This commit is contained in:
@@ -0,0 +1,735 @@
|
||||
# FPGA Neural Network Engine
|
||||
|
||||
Hardware Neural Network Engine based on FPGA + dedicated RAM.
|
||||
|
||||
The project implements a **parametric hardware accelerator for neural networks**, designed to be reusable across different embedded systems and applications.
|
||||
|
||||
The fundamental design principle is that the neural-network computation is performed entirely inside the FPGA, while the host system communicates with the engine through a simple hardware-independent interface such as SPI.
|
||||
|
||||
---
|
||||
|
||||
## 1. Project Goal
|
||||
|
||||
The goal of this project is to develop a reusable **Neural Network Engine implemented in FPGA hardware**.
|
||||
|
||||
The engine is composed of:
|
||||
|
||||
- FPGA;
|
||||
- dedicated RAM connected to the FPGA;
|
||||
- host interface, initially SPI and potentially Dual SPI.
|
||||
|
||||
The host system is not part of the neural-network computational datapath.
|
||||
|
||||
Possible host systems include:
|
||||
|
||||
- Linux SoCs;
|
||||
- Raspberry-Pi-like systems;
|
||||
- ESP32;
|
||||
- microcontrollers;
|
||||
- other embedded processors;
|
||||
- development PCs.
|
||||
|
||||
The same Neural Network Engine architecture should therefore be usable in completely different systems.
|
||||
|
||||
```text
|
||||
HOST SYSTEM
|
||||
┌─────────────────────────┐
|
||||
│ │
|
||||
│ Linux / ESP32 / MCU │
|
||||
│ │
|
||||
│ Configuration │
|
||||
│ Training │
|
||||
│ Control │
|
||||
└────────────┬────────────┘
|
||||
│
|
||||
SPI / Dual SPI
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────┐
|
||||
│ FPGA │
|
||||
│ │
|
||||
│ Neural Network Engine │
|
||||
│ │
|
||||
│ Compute / Control │
|
||||
│ │
|
||||
└────────────┬────────────┘
|
||||
│
|
||||
│
|
||||
Dedicated RAM
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# 2. Architectural Principle
|
||||
|
||||
The FPGA is the actual neural-network accelerator.
|
||||
|
||||
The RAM required by the neural network is physically associated with the FPGA and is **not part of the host system memory**.
|
||||
|
||||
The host only provides:
|
||||
|
||||
- configuration;
|
||||
- network parameters;
|
||||
- input data;
|
||||
- control;
|
||||
- result retrieval.
|
||||
|
||||
The neural-network calculations themselves are executed by the FPGA.
|
||||
|
||||
This separation is a fundamental architectural requirement.
|
||||
|
||||
---
|
||||
|
||||
# 3. Hardware Configuration vs Network Configuration
|
||||
|
||||
An important distinction is made between the **hardware architecture of the accelerator** and the **parameters of the neural network**.
|
||||
|
||||
## 3.1 FPGA hardware configuration
|
||||
|
||||
The physical architecture of the Neural Network Engine is defined when the FPGA design is synthesized and implemented.
|
||||
|
||||
Typical hardware parameters include:
|
||||
|
||||
```text
|
||||
N_INPUTS
|
||||
N_NEURONS
|
||||
N_LAYERS
|
||||
PARALLEL
|
||||
DATA_WIDTH
|
||||
ACCUMULATOR_WIDTH
|
||||
```
|
||||
|
||||
These parameters can therefore be Verilog/SystemVerilog parameters or equivalent synthesis-time configuration values.
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
N_INPUTS = 32
|
||||
N_NEURONS = 4
|
||||
PARALLEL = 8
|
||||
```
|
||||
|
||||
defines a specific hardware implementation optimized for that architecture.
|
||||
|
||||
The resulting FPGA bitstream contains the corresponding datapath.
|
||||
|
||||
---
|
||||
|
||||
## 3.2 Neural-network configuration
|
||||
|
||||
Once the FPGA has been configured and initialized, the actual neural-network parameters can be loaded through the host interface.
|
||||
|
||||
These parameters may include:
|
||||
|
||||
- weights;
|
||||
- biases;
|
||||
- activation parameters;
|
||||
- quantization parameters;
|
||||
- network-specific constants.
|
||||
|
||||
These values are stored in the RAM associated with the FPGA.
|
||||
|
||||
Therefore:
|
||||
|
||||
```text
|
||||
FPGA BITSTREAM
|
||||
│
|
||||
│ defines hardware architecture
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ Neural Network │
|
||||
│ Hardware Engine │
|
||||
└──────────┬──────────┘
|
||||
│
|
||||
│ loads
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ Dedicated RAM │
|
||||
│ │
|
||||
│ weights │
|
||||
│ biases │
|
||||
│ parameters │
|
||||
│ buffers │
|
||||
└─────────────────────┘
|
||||
```
|
||||
|
||||
This provides an important separation between **hardware specialization** and **network data**.
|
||||
|
||||
---
|
||||
|
||||
# 4. Application-Specific Neural Networks
|
||||
|
||||
The Neural Network Engine is not intended to implement one fixed neural network.
|
||||
|
||||
Instead, each application can define its own network.
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
Application A
|
||||
16 inputs
|
||||
8 neurons
|
||||
1 output
|
||||
|
||||
Application B
|
||||
32 inputs
|
||||
16 neurons
|
||||
4 outputs
|
||||
|
||||
Application C
|
||||
64 inputs
|
||||
multiple layers
|
||||
custom parallelism
|
||||
```
|
||||
|
||||
The FPGA hardware can then be generated specifically for the required architecture.
|
||||
|
||||
This allows the design to exploit the FPGA resources efficiently rather than implementing a completely generic and potentially inefficient neural-network processor.
|
||||
|
||||
---
|
||||
|
||||
# 5. Parametric Compute Engine
|
||||
|
||||
The current implementation contains a parametric neural-network layer.
|
||||
|
||||
A validated configuration is:
|
||||
|
||||
```text
|
||||
N_INPUTS = 32
|
||||
N_NEURONS = 4
|
||||
PARALLEL = 8
|
||||
```
|
||||
|
||||
The datapath processes the inputs in parallel groups.
|
||||
|
||||
Conceptually:
|
||||
|
||||
```text
|
||||
32 inputs
|
||||
│
|
||||
├── 8 parallel MACs
|
||||
│
|
||||
├── 8 parallel MACs
|
||||
│
|
||||
├── 8 parallel MACs
|
||||
│
|
||||
└── 8 parallel MACs
|
||||
│
|
||||
▼
|
||||
Accumulation
|
||||
│
|
||||
▼
|
||||
Bias
|
||||
│
|
||||
▼
|
||||
Activation
|
||||
│
|
||||
▼
|
||||
Output
|
||||
```
|
||||
|
||||
The architecture is intended to scale by changing the synthesis parameters.
|
||||
|
||||
---
|
||||
|
||||
# 6. Current Functional Validation
|
||||
|
||||
The `32 × 4 / PARALLEL = 8` configuration has been successfully simulated.
|
||||
|
||||
Test output:
|
||||
|
||||
```text
|
||||
========================================
|
||||
PARAMETRIC LAYER TEST
|
||||
N_INPUTS = 32
|
||||
N_NEURONS = 4
|
||||
PARALLEL = 8
|
||||
========================================
|
||||
|
||||
PASS N0: 8192
|
||||
PASS N1: 4096
|
||||
PASS N2: 0 (ReLU)
|
||||
PASS N3: 16384
|
||||
|
||||
========================================
|
||||
PARAMETRIC TEST PASSED
|
||||
========================================
|
||||
```
|
||||
|
||||
Validated functionality:
|
||||
|
||||
- multiple inputs;
|
||||
- multiple neurons;
|
||||
- parallel MAC processing;
|
||||
- accumulation across multiple input groups;
|
||||
- bias handling;
|
||||
- independent neuron outputs;
|
||||
- ReLU activation;
|
||||
- parametric layer architecture.
|
||||
|
||||
The corresponding test has been committed to the repository.
|
||||
|
||||
Commit:
|
||||
|
||||
```text
|
||||
test: validate parametric 32x4 layer with parallelism 8
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# 7. FPGA Boot and Initialization
|
||||
|
||||
The FPGA is configured during system initialization using its normal FPGA configuration mechanism.
|
||||
|
||||
The FPGA bitstream defines the hardware architecture of the Neural Network Engine.
|
||||
|
||||
Conceptually:
|
||||
|
||||
```text
|
||||
Power-on
|
||||
│
|
||||
▼
|
||||
FPGA configuration
|
||||
│
|
||||
│ bitstream
|
||||
▼
|
||||
Neural Network Engine available
|
||||
│
|
||||
▼
|
||||
Host initialization
|
||||
│
|
||||
│ SPI
|
||||
▼
|
||||
Load network parameters
|
||||
│
|
||||
▼
|
||||
Load weights / biases
|
||||
│
|
||||
▼
|
||||
Engine ready
|
||||
```
|
||||
|
||||
This means that the host does **not dynamically construct the FPGA datapath** during normal operation.
|
||||
|
||||
The datapath already exists in hardware.
|
||||
|
||||
The host configures the network data that the datapath operates on.
|
||||
|
||||
---
|
||||
|
||||
# 8. Host Interface
|
||||
|
||||
The primary external interface is intended to be:
|
||||
|
||||
```text
|
||||
SPI
|
||||
```
|
||||
|
||||
with possible future support for:
|
||||
|
||||
```text
|
||||
Dual SPI
|
||||
```
|
||||
|
||||
The interface must remain independent of the host operating system.
|
||||
|
||||
The same hardware protocol should therefore be usable from:
|
||||
|
||||
```text
|
||||
Linux
|
||||
ESP32
|
||||
MCU
|
||||
PC
|
||||
```
|
||||
|
||||
The host interface should provide access to:
|
||||
|
||||
- control registers;
|
||||
- status;
|
||||
- network configuration;
|
||||
- RAM;
|
||||
- input data;
|
||||
- output data;
|
||||
- start/stop control;
|
||||
- completion status.
|
||||
|
||||
A conceptual command sequence is:
|
||||
|
||||
```text
|
||||
RESET
|
||||
│
|
||||
▼
|
||||
CONFIGURE
|
||||
│
|
||||
▼
|
||||
LOAD NETWORK PARAMETERS
|
||||
│
|
||||
▼
|
||||
LOAD WEIGHTS
|
||||
│
|
||||
▼
|
||||
LOAD BIASES
|
||||
│
|
||||
▼
|
||||
LOAD INPUT
|
||||
│
|
||||
▼
|
||||
START
|
||||
│
|
||||
▼
|
||||
WAIT FOR DONE
|
||||
│
|
||||
▼
|
||||
READ OUTPUT
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# 9. Dedicated FPGA RAM
|
||||
|
||||
The RAM is considered part of the Neural Network Engine.
|
||||
|
||||
It is not intended to be supplied by the host system.
|
||||
|
||||
Depending on the final architecture, RAM may contain:
|
||||
|
||||
```text
|
||||
Weights
|
||||
Biases
|
||||
Input buffers
|
||||
Intermediate layer buffers
|
||||
Output buffers
|
||||
Network parameters
|
||||
```
|
||||
|
||||
The memory architecture must be designed according to:
|
||||
|
||||
- number of parallel MAC units;
|
||||
- data width;
|
||||
- required bandwidth;
|
||||
- number of layers;
|
||||
- buffering requirements;
|
||||
- FPGA block-RAM resources;
|
||||
- possible external RAM requirements.
|
||||
|
||||
The preferred architecture is that the FPGA directly controls this memory.
|
||||
|
||||
---
|
||||
|
||||
# 10. Training
|
||||
|
||||
Training and inference are conceptually separated.
|
||||
|
||||
The first implementation does not require the FPGA to perform the complete training process.
|
||||
|
||||
Training can be performed externally:
|
||||
|
||||
```text
|
||||
PC / Linux / other host
|
||||
│
|
||||
│ training
|
||||
▼
|
||||
Network weights
|
||||
│
|
||||
│ SPI
|
||||
▼
|
||||
FPGA RAM
|
||||
```
|
||||
|
||||
The FPGA then performs inference using the resulting parameters.
|
||||
|
||||
This approach greatly reduces the complexity of the initial hardware implementation.
|
||||
|
||||
However, the architecture should not prevent future implementation of hardware-assisted or fully hardware-based training.
|
||||
|
||||
---
|
||||
|
||||
# 11. Inference
|
||||
|
||||
During inference, the host only supplies input data and retrieves the result.
|
||||
|
||||
```text
|
||||
HOST
|
||||
│
|
||||
Input
|
||||
│
|
||||
│ SPI
|
||||
▼
|
||||
┌───────────────┐
|
||||
│ FPGA │
|
||||
│ │
|
||||
│ Neural Network│
|
||||
│ Engine │
|
||||
│ │
|
||||
└───────┬───────┘
|
||||
│
|
||||
Output
|
||||
│
|
||||
│ SPI
|
||||
▼
|
||||
HOST
|
||||
```
|
||||
|
||||
The host is not involved in the individual MAC operations.
|
||||
|
||||
This provides:
|
||||
|
||||
- deterministic computation;
|
||||
- reduced host workload;
|
||||
- hardware parallelism;
|
||||
- predictable latency;
|
||||
- independence from the host CPU architecture.
|
||||
|
||||
---
|
||||
|
||||
# 12. Multi-Layer Architecture
|
||||
|
||||
The current implementation starts from a single parametrized layer.
|
||||
|
||||
The intended architecture is eventually:
|
||||
|
||||
```text
|
||||
Input
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ Layer 0 │
|
||||
└──────┬───────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ Layer 1 │
|
||||
└──────┬───────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ Layer 2 │
|
||||
└──────┬───────┘
|
||||
│
|
||||
▼
|
||||
Output
|
||||
```
|
||||
|
||||
Intermediate data will be stored in FPGA-controlled memory buffers.
|
||||
|
||||
The number and size of layers should ultimately be part of the hardware generation process.
|
||||
|
||||
---
|
||||
|
||||
# 13. Reusability
|
||||
|
||||
The main purpose of the architecture is reuse.
|
||||
|
||||
A future project should be able to use the same general Neural Network Engine architecture with a different hardware configuration.
|
||||
|
||||
For example:
|
||||
|
||||
```text
|
||||
Project A
|
||||
N_INPUTS = 32
|
||||
N_NEURONS = 8
|
||||
PARALLEL = 8
|
||||
|
||||
Project B
|
||||
N_INPUTS = 64
|
||||
N_NEURONS = 16
|
||||
PARALLEL = 16
|
||||
|
||||
Project C
|
||||
N_INPUTS = 128
|
||||
N_NEURONS = 32
|
||||
PARALLEL = 32
|
||||
```
|
||||
|
||||
The HDL architecture remains conceptually the same while synthesis parameters generate an implementation appropriate for the target application.
|
||||
|
||||
---
|
||||
|
||||
# 14. Design Philosophy
|
||||
|
||||
The project should be considered a:
|
||||
|
||||
> **Reusable FPGA Neural Network Accelerator Platform**
|
||||
|
||||
rather than a single neural-network implementation.
|
||||
|
||||
The application determines:
|
||||
|
||||
```text
|
||||
Input size
|
||||
Network topology
|
||||
Number of layers
|
||||
Number of neurons
|
||||
Parallelism
|
||||
Numerical precision
|
||||
Activation functions
|
||||
Memory requirements
|
||||
Performance requirements
|
||||
```
|
||||
|
||||
The hardware generator then produces the corresponding FPGA implementation.
|
||||
|
||||
---
|
||||
|
||||
# 15. Development Roadmap
|
||||
|
||||
## Phase 1 — Parametric Layer
|
||||
|
||||
- [x] Parametric inputs
|
||||
- [x] Parametric neurons
|
||||
- [x] Parametric parallelism
|
||||
- [x] Accumulation
|
||||
- [x] Bias
|
||||
- [x] ReLU
|
||||
- [x] 32×4 / P=8 functional test
|
||||
|
||||
## Phase 2 — Parameter Sweep
|
||||
|
||||
Validate multiple combinations of:
|
||||
|
||||
```text
|
||||
N_INPUTS
|
||||
N_NEURONS
|
||||
PARALLEL
|
||||
```
|
||||
|
||||
including configurations where the number of inputs is not an exact multiple of the parallelism.
|
||||
|
||||
## Phase 3 — Memory Architecture
|
||||
|
||||
Define:
|
||||
|
||||
- weight memory;
|
||||
- bias memory;
|
||||
- input buffers;
|
||||
- output buffers;
|
||||
- intermediate buffers;
|
||||
- memory addressing;
|
||||
- bandwidth requirements.
|
||||
|
||||
## Phase 4 — SPI Interface
|
||||
|
||||
Implement:
|
||||
|
||||
- SPI controller;
|
||||
- register map;
|
||||
- RAM access;
|
||||
- configuration protocol;
|
||||
- input/output protocol;
|
||||
- status and control.
|
||||
|
||||
## Phase 5 — Multi-Layer Network
|
||||
|
||||
Implement:
|
||||
|
||||
- multiple layers;
|
||||
- intermediate buffers;
|
||||
- layer sequencing;
|
||||
- configurable activation functions.
|
||||
|
||||
## Phase 6 — Host Software
|
||||
|
||||
Develop host-side drivers for:
|
||||
|
||||
- Linux;
|
||||
- ESP32.
|
||||
|
||||
The same FPGA protocol should be usable by both.
|
||||
|
||||
## Phase 7 — Optimization
|
||||
|
||||
Evaluate:
|
||||
|
||||
- pipeline depth;
|
||||
- MAC parallelism;
|
||||
- memory bandwidth;
|
||||
- numerical precision;
|
||||
- FPGA resource utilization;
|
||||
- latency;
|
||||
- throughput.
|
||||
|
||||
## Phase 8 — Optional Hardware Training
|
||||
|
||||
Investigate:
|
||||
|
||||
- backpropagation;
|
||||
- gradient calculation;
|
||||
- weight updates;
|
||||
- hardware-assisted training.
|
||||
|
||||
---
|
||||
|
||||
# 16. Long-Term Vision
|
||||
|
||||
The final objective is to create a reusable hardware block that can be integrated into different future MIKILAB projects.
|
||||
|
||||
```text
|
||||
APPLICATION
|
||||
│
|
||||
┌─────────────┴─────────────┐
|
||||
│ │
|
||||
Linux ESP32
|
||||
│ │
|
||||
└─────────────┬─────────────┘
|
||||
│
|
||||
SPI / Dual SPI
|
||||
│
|
||||
▼
|
||||
┌────────────────────────┐
|
||||
│ FPGA │
|
||||
│ │
|
||||
│ Neural Network Engine │
|
||||
│ │
|
||||
│ ┌────────────────────┐ │
|
||||
│ │ Control │ │
|
||||
│ ├────────────────────┤ │
|
||||
│ │ Input Interface │ │
|
||||
│ ├────────────────────┤ │
|
||||
│ │ NN Compute Core │ │
|
||||
│ ├────────────────────┤ │
|
||||
│ │ Activation │ │
|
||||
│ ├────────────────────┤ │
|
||||
│ │ Output Interface │ │
|
||||
│ └────────────────────┘ │
|
||||
│ │
|
||||
│ Dedicated RAM │
|
||||
│ │
|
||||
└────────────────────────┘
|
||||
```
|
||||
|
||||
The host platform can change without changing the fundamental Neural Network Engine architecture.
|
||||
|
||||
The FPGA becomes a dedicated neural-computation peripheral, analogous to other hardware accelerators, but optimized specifically for the neural-network topology required by each application.
|
||||
|
||||
---
|
||||
|
||||
# 17. Current Status
|
||||
|
||||
| Component | Status |
|
||||
|---|---|
|
||||
| Parametric neuron layer | OK Working |
|
||||
| Parametric input count | OK |
|
||||
| Parametric neuron count | OK |
|
||||
| Parametric parallelism | OK |
|
||||
| Accumulation | OK |
|
||||
| Bias | OK |
|
||||
| ReLU | OK |
|
||||
| 32×4 / P=8 validation | OK |
|
||||
| Dedicated RAM architecture | - Design |
|
||||
| SPI interface | Planned |
|
||||
| Dual SPI | Future |
|
||||
| Multi-layer engine | Planned |
|
||||
| Linux host driver | Planned |
|
||||
| ESP32 host driver | Planned |
|
||||
| Hardware training | Future |
|
||||
|
||||
---
|
||||
|
||||
## Core architectural principle
|
||||
|
||||
**The FPGA implements the neural-network machine.
|
||||
The FPGA owns its RAM.
|
||||
The host configures and uses the machine.
|
||||
The network topology is specialized at FPGA build time, while its trained parameters are loaded into FPGA-local memory at initialization.**
|
||||
|
||||
This separation is the foundation of the project.
|
||||
Reference in New Issue
Block a user