FPGA-Neural/docs/ARCHITECTURE_ANALYSIS.md

# FPGA-Neural V3 — Architecture Analysis: Timing, Bottlenecks, and Recommended Interventions

Scope: the current, real, P&R-verified V3 design (`hardware/v3/`, branch
`v3-artix7`), as of EXP-0079 (real activation-fetch engine, real DDR3 for
both weights and activations, real P&R: WNS +0.030ns). Every number in this
document is either directly measured (real simulation trace, real P&R
report) or a calculation built from directly-measured building blocks — the
two are labeled explicitly throughout. Nothing here is guessed.

---

## 1. Executive summary

The single most important finding of this analysis: **the system is DDR3
memory-bandwidth-bound, not DSP-bound, already at N=1 core** — and this is
true *before* considering any core-count scaling. Adding more compute cores
(N=4/8/16) without first addressing memory bandwidth would not increase real
throughput; it would only add more cores contending for the same saturated
DDR3 channel.

| Metric | Value | Source |
|---|---|---|
| Real DDR3 back-to-back burst bandwidth | **1.24 GB/s** (9.92 Gbps) | measured, real JEDEC trace (§3.1) |
| Real DDR3 bandwidth needed for ONE core at peak DSP throughput | **4.96 GB/s** | calculated from measured DSP rate + real memory layout (§3.2) |
| → DDR3 can sustain at best | **~25%** of one core's peak compute throughput | §3.2 |
| Real P&R timing margin (WNS) | **+0.030 ns** | measured, EXP-0079 real P&R |
| DSP48E1 headroom for scaling | 224/240 free (93%) | measured, real P&R utilization |

The DSP headroom is real and large. The memory-bandwidth ceiling is real and
small. **The highest-leverage next step is fixing the activation-fetch memory
waste (§5.1), not adding cores.** The user's own proposed DDRManager/
prefetch idea (§5.2) is valuable and complementary, but solves a *different*
problem (latency/stalling) than the bandwidth ceiling (§3.2) — both are
covered below, with the distinction made explicit.

---

## 2. Real signoff history (all real P&R runs to date)

| EXP | What changed | WNS (ns) | LUTs | DSP48E1 | Notes |
|---|---|---|---|---|---|
| 0059 | isolated packed core, out-of-context | n/a (isolated) | 507 | 8 | first real P&R, out-of-context only |
| 0074 | first real in-context P&R: DDR3 + pins + register file not yet added | +0.040 | 5140 | 16 | first trustworthy board-accurate number |
| 0076 | + register file, + pin constraints, + SPI physical-layer fix | +0.056 | 5173 | 16 | margin improved slightly (P&R is not perfectly monotonic run to run) |
| 0078 | + config-flash bridge (real STARTUPE2 placement) | +0.013 | 5213 | 16 | margin dropped — real added logic |
| 0079 | + real activation-fetch engine (`act_tile_fetch.v`) | **+0.030** | 5379 | 16 | current, final, trustworthy number |

**Observation**: WNS does not move monotonically with LUT count (0.056 →
0.013 → 0.030 while LUTs only ever grow) — this is normal P&R behavior
(placer/router heuristics find different solutions each run, small logic
changes can shift which path is critical). **Do not extrapolate a trend
line from 3-4 data points** — the only safe practice is a fresh real P&R
after every real change, which this project already does.

**DSP48E1 has stayed at 16 across every real change since EXP-0059's core
design was fixed** — confirms the packed-DSP MAC design (§4.1) is the
efficient, stable part of this architecture; all the margin pressure has
come from *control/glue logic* (arbitration, the flash bridge, the
activation engine's FSM), not from the compute datapath itself.

---

## 3. The real memory-bandwidth bottleneck (the analysis's central finding)

### 3.1 Real measured DDR3 throughput

From the actual `ddr3_model.sv` JEDEC command trace captured during the
EXP-0079 real simulation run (`tb_n2_system_ddr3.v` via `xsim`), two
back-to-back `Read` commands to the same open row:

```
Read  bank 0 col 000 @ 7090615 ps
Read  bank 0 col 000 @ 7103515 ps   (delta = 12900 ps = 12.9 ns)
```

One `BURST_LEN=8` transaction moves 8 × 16 bits = 128 bits. Real measured
throughput for same-row back-to-back bursts:

```
128 bits / 12.9 ns = 9.92 Gbps = 1.24 GB/s
```

This exactly matches the *theoretical* peak for a 16-bit DDR3 interface at
310.078 MHz (16 bits × 2 (DDR) × 310.078 MHz = 9.92 Gbps) — confirming the
real controller achieves its theoretical ceiling for the best case
(same-row, no row switches). This is the **best-case** number; anything
requiring a row change (Activate/Precharge) is real-measured to cost more
(§3.3).

### 3.2 Real compute-side bandwidth requirement

Each packed core (EXP-0059's design, unchanged since) uses 8 DSP48E1, each
computing 2 packed INT8 MACs per cycle (lane A + lane B sharing one resident
weight) = 16 MACs/cycle/core. At the real measured 155.039 MHz compute clock:

```
16 MACs/cycle × 155.039 × 10^6 cycles/s = 2.48 GMAC/s per core (real, from measured Fmax)
```

Each compute cycle consumes 1 activation byte per MAC lane (16 bytes total:
8 for lane A, 8 for lane B). Under the current real memory layout
(`act_tile_fetch.v`, EXP-0079: one tile = one full `BURST_LEN=8` burst = 16
bytes moved, only 8 useful), feeding one compute cycle costs:

```
2 tiles (A+B) × 16 bytes/burst = 32 bytes moved DDR3 traffic → 16 MACs
= 2 real DDR3 bytes moved per MAC operation
```

Bandwidth needed to keep ONE core's DSPs fed at their real peak rate:

```
2.48 GMAC/s × 2 bytes/MAC = 4.96 GB/s
```

### 3.3 The real gap

```
1.24 GB/s available (measured, best case)  vs  4.96 GB/s needed per core (calculated)
→ DDR3 can sustain at most ~25% of one core's peak DSP throughput,
  even in the BEST case (zero row-switch overhead, one core, nothing else
  sharing the bus).
```

This ceiling gets **worse**, not better, with:
- **Row switches**: real measured Activate→Read latency is 16.125 ns
  (5 DDR3 clock cycles at 3.225 ns = real CAS-latency-5 timing, matches the
  MIG's own configured CL=5). A tile fetch that requires a fresh row
  activation costs ~16-30 ns instead of the 12.9 ns same-row case — a
  real, measured 25-130% penalty per row switch.
- **More cores sharing the one DDR3 channel** (N=2 today, N=4/8/16
  proposed): the 1.24 GB/s ceiling is shared across ALL active requesters,
  not per-core. Adding cores divides an already-insufficient budget further.
- **Weight fetching** (amortized, but not free): each job pair's weight
  prefetch (8 bursts for a 128-byte layer) adds real DDR3 traffic on top of
  activation fetching, though this cost is shared across the M reuse
  positions and becomes negligible for large M.

**Conclusion**: this is a genuine architectural ceiling, not a tuning
problem. The 2× byte-overhead from the current "1 tile = 1 full burst"
convention (chosen in EXP-0079 specifically to avoid a runtime-indexed
part-select, given the then-already-thin timing margin) is now confirmed,
with real numbers, to be the single most expensive design decision in the
current memory path. See §5.1 for the recommended fix.

---

## 4. Module-by-module review

### 4.1 Compute core (`mac2_dsp_packed.v`, `neural_processor_packed.v`)

Real, stable, efficient. 8 DSP48E1/core unchanged since EXP-0059. The
2-INT8-MAC-per-DSP48E1 packing technique is the correct choice for this INT8
workload — real utilization (16/240 DSP = 6.67% at N=2) confirms there is no
DSP-side pressure at all; **all scaling headroom is here, and all scaling
risk is elsewhere** (§3, §4.5).

No real change recommended here. This is the part of the design that is
*not* the bottleneck.

### 4.2 Weight-reuse path (`layer_prefetch_ctrl.v`, `layer_weight_buffer.v`,
`weight_tile_gather.v`)

Reused unmodified from V2 (ECP5 era), real and well-verified across many
EXPs (0057/0058/0061/0062 and every integration test since). Correctly
amortizes DDR3 traffic across M reuse positions — this part of the design
already does the "fetch once, use many times" optimization the activation
path currently lacks (§5.1's recommendation follows the SAME philosophy).

No real change recommended; this module is a good template for how the
activation path should evolve.

### 4.3 Activation-fetch path (`act_tile_fetch.v`, EXP-0079)

Real, correct (verified 3 levels deep, §2 of EXP-0079's own log entry), but
— per §3 above — the current real bottleneck. Two real design choices worth
re-examining now that real bandwidth numbers exist:

1. **1 tile = 1 full burst (2× byte overhead)**: chosen to avoid a
   runtime-indexed part-select. §5.1 proposes a way to recover this
   efficiency without reintroducing that risk.
2. **Lane A then lane B, sequential, per tile**: doubles the real number of
   DDR3 transactions (and row-switch risk) versus a design that could fetch
   both lanes in a single wider transaction when they happen to be adjacent
   in memory. Not changed in this analysis pass — flagged for future work
   if §5.1's fix doesn't fully close the gap.

### 4.4 Scheduling (`neural_director_packed.v`) and arbitration
(`sdram_arbiter_n.v`)

Real, correct, and — importantly for §5.2 — **`neural_director_packed.v`
already maintains a real queue of pending jobs** (`QUEUE_DEPTH=8`,
`q_x_base`/`q_w_base`/etc arrays). This is directly relevant to the
DDRManager/prefetch proposal (§5.2): the information needed to "know what's
coming next" already exists in this module, it just isn't currently used for
anything beyond pairing/dispatch decisions.

`sdram_arbiter_n.v`'s combinational-first-grant design (EXP-0066) is real,
proven, and already generalized to N-way (verified at N=3, EXP-0069) — no
real change needed to scale its own requester count for a DDRManager
addition or for N=4/8/16 scaling, though the arbiter's own **fairness
policy** (lowest-index-wins, a deliberate simplicity choice, EXP-0066) would
need re-examination if a DDRManager starts issuing speculative/anticipatory
requests that could starve a real, urgent request — see §5.2's own caveat.

### 4.5 Host interface (`spi_host_bridge_v3.v`, `flash_spi_master.v`,
`host_mem_bridge.v`)

Real, verified, low resource cost, not on any critical performance path
(host commands are inherently much slower than the internal compute/memory
loop). No bottleneck here. Not a scaling concern.

### 4.6 Missing: result-writeback engine

Still genuinely absent (disclosed since `packed_slot.v`'s own original
header, unchanged through EXP-0079). Currently `result_data_a/b` are literal
top-level pins — functional at N=2 (32 pins), but this is the **exact same
class of mistake already caught once** for activation data (EXP-0074: ~360
pins nearly exceeded the whole package's I/O budget). At N=16 this port
alone would need 8 bits × 2 lanes × 16 cores = 256 pins — **a real, hard
blocker for any scaling beyond a handful of cores**, independent of the
memory-bandwidth ceiling in §3. Recommended fix in §5.3.

---

## 5. Recommended interventions, ranked by real leverage

### 5.1 [Highest leverage] Denser activation packing — attack the real bandwidth ceiling directly

**What**: pack 2 tiles (lane A + lane B, or two consecutive tiles of the
same lane) into one `BURST_LEN=8` burst instead of one tile per burst,
halving real DDR3 bytes-per-MAC from 2 to 1. This alone would raise the
real achievable fraction of one core's peak throughput from ~25% to ~50%
(§3.2's math, halved).

**Why this was avoided in EXP-0079**: doing so naively requires a
runtime-indexed part-select (which half of the burst response to use,
selected by a runtime tile-index bit) — the same anti-pattern
`weight_tile_gather.v` (EXP-0061) already flagged as a real Fmax risk, and
the P&R margin was already thin (+0.013ns) when this design decision was
made.

**A safer path to the same efficiency gain** (not yet built, this is a
recommendation): register the tile-index LSB **one cycle ahead of** the
burst response arriving (it's already known at request time, not something
that needs to race the read data) — using it to select a *pre-registered*
mux input rather than gating the read-data path itself keeps the selection
off the critical timing path. This needs a real prototype and a real P&R
check before being trusted — proposed as the next concrete engineering task,
not asserted as already safe.

### 5.2 [Complementary, addresses latency not bandwidth] DDRManager with orchestrator-driven prefetch (user's proposal)

**The real problem this solves**: even within whatever bandwidth ceiling
§5.1 establishes, the CURRENT design only ever requests a tile the moment
`packed_slot.v`'s own FSM reaches `S_TILEREQ` for it — meaning the DSPs
stall waiting for that fetch's real latency (§3.3: 12.9-30+ ns) every
single tile, with no overlap between "fetching tile N+1" and "computing on
tile N". A DDRManager that issues tile N+1's fetch WHILE tile N is still
computing would hide that latency almost entirely (compute time per tile,
1/155.039MHz ≈ 6.4ns per cycle, vs a real fetch latency of 12.9-30+ns —
today's design is very likely stalling the DSPs for the majority of real
time, an real, additional cost on top of §3's raw bandwidth ceiling).

**What it does NOT solve**: §3.2's bandwidth ceiling is a hard physical
limit (bytes/second the DDR3 channel can physically move) — prefetching
earlier doesn't move more bytes per second, it only avoids IDLE gaps where
the channel is free but nothing is queued to use it. **§5.1 and §5.2 are
complementary, not alternatives** — §5.1 reduces bytes needed per MAC, §5.2
ensures the channel is never idle when there's real bandwidth budget
available and useful work queued. Do both, in this order (§5.1 first, since
it raises the ceiling §5.2 will then use more fully).

**Concrete design sketch** (informed by what already exists in this
codebase, not a from-scratch proposal):

- `neural_director_packed.v` already queues up to `QUEUE_DEPTH=8` pending
  jobs, each with a known `x_base`/`w_base`/`n_tiles` — this is exactly the
  "reservation" information a DDRManager needs. No new bookkeeping is
  required at the Director level; a DDRManager would READ this existing
  queue, not need the Director to change its own job-acceptance logic.
- A new module (name suggestion: `ddr_prefetch_mgr.v`) would sit between
  `sdram_arbiter_n.v` and the per-slot `act_tile_fetch.v`/
  `layer_prefetch_ctrl.v` instances, with a small staging buffer per slot
  (double-buffered, matching `layer_weight_buffer.v`'s own already-proven
  double-buffer pattern) — while `packed_slot.v` computes on the CURRENT
  tile, the manager issues the request for the NEXT tile into the "other"
  buffer, swapping on completion.
- **Real caveat, not glossed over**: this adds real arbitration complexity
  — a prefetched-but-not-yet-consumed request competing with another slot's
  genuinely urgent request needs a real priority policy, not just
  `sdram_arbiter_n.v`'s current lowest-index-wins simplicity (§4.4). A
  speculative prefetch that turns out to be wrong (e.g., the Director
  reorders/never dispatches that queued job) also wastes real bandwidth —
  needs a real cancellation/staleness mechanism, not assumed away.
- **Recommended validation before committing engineering time**: build a
  minimal version scoped to ONE slot's OWN activation-tile look-ahead
  (prefetch tile N+1 while computing tile N, using the double-buffer
  pattern above) before attempting the full "reserve across the whole
  Director queue" version — matches this project's own "one variable at a
  time" discipline, and would give a real, measured stall-reduction number
  to justify (or not) the added complexity of the full design.

### 5.3 [Blocking for any real scaling] Result-writeback engine

Must exist before N>2 is even attemptable (§4.6) — result data needs to go
into DDR3 (or through the SPI status/register path for small result sets),
never as N-scaled literal top-level pins again. Same architectural shape as
the weight-fetch path, in reverse (write instead of read) — a reasonable,
bounded scope, and a real prerequisite, not optional polish.

### 5.4 Scaling path recommendation (real numbers, not a guess)

Given §3's real bandwidth ceiling: **scaling core count alone, without
§5.1/§5.2, provides no real additional throughput past whatever N already
saturates the 1.24 GB/s ceiling** — back-of-envelope, using §3.2's numbers,
that's already close to N=1 in the worst case, and at most N≈2 in the best
(zero-row-switch) case. **Building N=4/8/16 today, before §5.1, would very
likely show near-IDENTICAL real throughput to N=2 in a real P&R'd, real-
DDR3-simulated test** — a real, wasted engineering cycle the analysis
recommends avoiding.

**Recommended real order of work**:
1. §5.3 (result-writeback) — genuine blocker, bounded scope.
2. §5.1 (denser activation packing) — highest real leverage on the actual
   ceiling, needs a real P&R check given the thin margin.
3. Re-measure real achievable throughput at N=2 with §5.1 in place (real
   simulation, real cycle counts) — THIS number, not a projection, should
   decide whether N=4 is worth building next.
4. §5.2 (DDRManager/prefetch) — real, valuable, but its benefit is easier to
   quantify and justify once §5.1 has already raised the ceiling it's
   filling.
5. Only then: N=4/8/16, each with its own real P&R signoff (the margin is
   thin, §2 — do not assume a prior N's timing closure predicts the next).

---

## 6. Summary table: what's real vs. what's a calculation

| Claim | Status |
|---|---|
| WNS/WHS/LUT/DSP numbers throughout | **Measured** (real Vivado P&R reports) |
| DDR3 back-to-back burst throughput (1.24 GB/s) | **Measured** (real `ddr3_model.sv` JEDEC trace) |
| Real Activate→Read latency (16.125 ns) | **Measured** (same trace) |
| Per-core compute throughput (2.48 GMAC/s) | **Calculated** from measured Fmax (155.039MHz) + known, fixed DSP-packing factor |
| Bandwidth needed per core (4.96 GB/s) | **Calculated** from the above + the real, as-built memory layout convention |
| "~25% of peak sustainable" | **Calculated** ratio of the two measured/calculated numbers above |
| Row-switch penalty as a fraction of real workloads | **Not measured** — depends on host-chosen memory layout, flagged as an open question, not asserted |
| DDRManager's real stall-reduction benefit | **Not measured** — no prototype exists yet; §5.2 recommends building a minimal version specifically to get this real number |