FPGA-Neural/docs/ARCHITECTURE_ANALYSIS.md

# FPGA-Neural V3 — Architecture Analysis: Timing, Bottlenecks, and Recommended Interventions

Scope: the current, real, P&R-verified V3 design (`hardware/v3/`, branch
`v3-artix7`), updated through EXP-0095/0096 (N=8 hybrid systolic, real,
closed timing — the current, definitive real deployment target, chosen
by the user after a real, measured N=4/8/16 timing curve). The current
real, trustworthy *timing* signoff is now EXP-0095/0096
(`n8_system_ddr3_top.v`, 32-bit DDR3, 155.039MHz, WNS 0.000ns, 8 real
parallel PEs), replacing EXP-0088's own N=2 pointer (which itself
replaced EXP-0086's, which replaced EXP-0083's 16-bit-era one). See §5.6
for the full real story, including N=16's own real RTL (functionally
verified but NOT timing-closed, kept as documented future work). Every
number in this document is either
directly measured (real simulation trace, real P&R report) or a calculation
built from directly-measured building blocks — the two are labeled
explicitly throughout. Nothing here is guessed.

**Status note (post EXP-0087)**: §5.1 (denser activation packing) is
**DONE and real-P&R-verified** (EXP-0081/0082); §5.2 (DDRManager) phase 1
is **DONE, built, and RE-MEASURED against the closed 32-bit channel**
(EXP-0083's original modest ~2.9% real benefit was measured only at the
OLD 16-bit signoff — real re-measurement, EXP-0087, found the benefit is
now ~0%/a wash at 32-bit, kept wired in for correctness/timing-neutrality
only, not performance); §5.4 (32-bit widening)
is **fully DONE — real-verified AND real timing-closed** (EXP-0084
functional + EXP-0086 timing closure; honest intermediate story: the first
timing attempt failed because of a SEPARATE, stacked clock speedup, not the
width change itself — see §5.4.1). See the "DONE" markers in those
sections. The document originally analyzed the pre-fix state; it's kept
below (marked historical) because the comparison is itself informative,
then updated with the real post-fix numbers throughout.

---

## 1. Executive summary

The single most important finding of this analysis: **the system is DDR3
memory-bandwidth-bound, not DSP-bound, already at N=1 core** — and this is
true *before* considering any core-count scaling. Adding more compute cores
(N=4/8/16) without first addressing memory bandwidth would not increase real
throughput; it would only add more cores contending for the same saturated
DDR3 channel.

| Metric | Value | Source |
|---|---|---|
| Real DDR3 back-to-back burst bandwidth (physical channel) | **1.24 GB/s** (9.92 Gbps) | measured, real JEDEC trace (§3.1) — unchanged by packing, this is a physical-channel limit |
| Real DDR3 bandwidth needed for ONE core at peak DSP throughput | **4.96 GB/s** | calculated from measured DSP rate + memory layout (§3.2) |
| → DDR3 can sustain, pre-EXP-0081 packing (1 tile/burst) | **~25%** of one core's peak compute throughput | §3.2, historical |
| → DDR3 can sustain, post-EXP-0081/0082 packing (2 tiles/burst, DONE) | **~50%** of one core's peak compute throughput | §3.2, current, real |
| Real P&R timing margin (WNS), current trustworthy (32-bit, CLOSED) | **+0.096 ns** | measured, EXP-0086 real P&R |
| Real DDR3 physical bandwidth ceiling, 32-bit channel (closed timing) | **~2.48 GB/s** | measured, real P&R signoff, EXP-0086 — 2× EXP-0083's 16-bit ~1.24 GB/s, real not projected |
| DDRManager phase 1 (single-slot look-ahead prefetch) real benefit | **~0% (0.0064% regression)** at the closed 32-bit channel | measured, real xsim A/B on `tb_n2_system_ddr3.v` (§5.2, EXP-0087) — the old 16-bit-only 2.86% figure (EXP-0083) does not survive the wider channel; kept wired in (timing-neutral, real P&R already signs off with it) but no longer a real performance win |
| DSP48E1 headroom for scaling | 224/240 free (93%) | measured, real P&R utilization |

The DSP headroom is real and large. The memory-bandwidth ceiling is real,
and **denser activation packing (§5.1) has already doubled the real
achievable fraction of it** — from ~25% to ~50% of one core's peak DSP
throughput, with the real P&R margin *improving*, not degrading, as a side
effect. This was free leverage and it's now banked.

Even at ~50%, DDR3 is still the limiting resource, not DSP count. The 32-bit
channel widening (§5.4) is now **fully real, verified, AND timing-closed**
(real xsim against the real 2-chip DDR3 model, plus a real closed P&R at
WNS=+0.096ns, EXP-0086) — width alone delivers the full intended 2×
physical bandwidth gain (1.24 → ~2.48 GB/s), with real, closed timing, not
a projection. The honest intermediate story: the first real P&R attempt
stacked an ADDITIONAL clock speedup (`ui_clk` +11.2%) in the same wizard
session, which an unrelated, pre-existing compute-pipeline path couldn't
absorb (§5.4.1) — width and clock rate turned out to be separable levers,
and reverting just the clock period closed timing with the width gain
intact. The DDRManager (§5.2) was re-measured against this now-closed,
wider channel (EXP-0087): its real benefit is gone (~0%, a 0.0064%
regression) — the wider channel's own lower per-tile latency already
absorbed the gap the look-ahead used to hide.

---

## 2. Real signoff history (all real P&R runs to date)

| EXP | What changed | WNS (ns) | LUTs | DSP48E1 | Notes |
|---|---|---|---|---|---|
| 0059 | isolated packed core, out-of-context | n/a (isolated) | 507 | 8 | first real P&R, out-of-context only |
| 0074 | first real in-context P&R: DDR3 + pins + register file not yet added | +0.040 | 5140 | 16 | first trustworthy board-accurate number |
| 0076 | + register file, + pin constraints, + SPI physical-layer fix | +0.056 | 5173 | 16 | margin improved slightly (P&R is not perfectly monotonic run to run) |
| 0078 | + config-flash bridge (real STARTUPE2 placement) | +0.013 | 5213 | 16 | margin dropped — real added logic |
| 0079 | + real activation-fetch engine (`act_tile_fetch.v`) | +0.030 | 5379 | 16 | pre-packing baseline |
| 0082 | + denser activation packing (2 tiles/burst, EXP-0081) | +0.068 | 5437 | 16 | pre-DDRManager baseline |
| 0083 | + DDRManager phase 1 (`ddr_prefetch_mgr.v`, single-slot look-ahead prefetch) | +0.073 | 5644 | 16 | 16-bit/155.039MHz — superseded by EXP-0086 below |
| 0084 | 32-bit DDR3 widening (2 chips) + clock speedup to 172.414MHz | -0.618 (FAILED) | 6418 | 16 | functionally verified (real xsim), real timing NOT closed — root cause is the clock speedup, not the width; see §5.4.1. |
| 0086 | 32-bit DDR3 widening, clock period reverted to 3225ps/155.039MHz | +0.096 (CLOSED) | 6382 | 16 | real, closed timing, full 2× real bandwidth (§5.4.1). Also uncovered and fixed a real recurrence of the stale-import bug (whole-tree, triggered by MIG regeneration) — see CLAUDE.md. Superseded by 0088 below. |
| 0088 | + result-writeback engine (`result_writeback.v`, removes the last hard N-scaling pin blocker, §4.6/§5.3) | **+0.100 (CLOSED)** | 6642 | 16 | **current, trustworthy signoff** — essentially unchanged margin, zero real timing cost for closing the pin-scaling blocker. |

**Observation**: WNS does not move monotonically with LUT count (0.056 →
0.013 → 0.030 → 0.068 → 0.073 while LUTs only ever grow) — this is normal P&R behavior
(placer/router heuristics find different solutions each run, small logic
changes can shift which path is critical). **Do not extrapolate a trend
line from 3-4 data points** — the only safe practice is a fresh real P&R
after every real change, which this project already does.

**DSP48E1 has stayed at 16 across every real change since EXP-0059's core
design was fixed** — confirms the packed-DSP MAC design (§4.1) is the
efficient, stable part of this architecture; all the margin pressure has
come from *control/glue logic* (arbitration, the flash bridge, the
activation engine's FSM), not from the compute datapath itself.

---

## 3. The real memory-bandwidth bottleneck (the analysis's central finding)

### 3.1 Real measured DDR3 throughput

From the actual `ddr3_model.sv` JEDEC command trace captured during the
EXP-0079 real simulation run (`tb_n2_system_ddr3.v` via `xsim`), two
back-to-back `Read` commands to the same open row:

```
Read  bank 0 col 000 @ 7090615 ps
Read  bank 0 col 000 @ 7103515 ps   (delta = 12900 ps = 12.9 ns)
```

One `BURST_LEN=8` transaction moves 8 × 16 bits = 128 bits. Real measured
throughput for same-row back-to-back bursts:

```
128 bits / 12.9 ns = 9.92 Gbps = 1.24 GB/s
```

This exactly matches the *theoretical* peak for a 16-bit DDR3 interface at
310.078 MHz (16 bits × 2 (DDR) × 310.078 MHz = 9.92 Gbps) — confirming the
real controller achieves its theoretical ceiling for the best case
(same-row, no row switches). This is the **best-case** number; anything
requiring a row change (Activate/Precharge) is real-measured to cost more
(§3.3).

### 3.2 Real compute-side bandwidth requirement

Each packed core (EXP-0059's design, unchanged since) uses 8 DSP48E1, each
computing 2 packed INT8 MACs per cycle (lane A + lane B sharing one resident
weight) = 16 MACs/cycle/core. At the real measured 155.039 MHz compute clock:

```
16 MACs/cycle × 155.039 × 10^6 cycles/s = 2.48 GMAC/s per core (real, from measured Fmax)
```

Each compute cycle consumes 1 activation byte per MAC lane (16 bytes total:
8 for lane A, 8 for lane B).

**Historical (EXP-0079, "1 tile = 1 burst")**: each tile fetch moved a full
16-byte burst for only 8 useful bytes:

```
2 tiles (A+B) × 16 bytes/burst = 32 bytes moved DDR3 traffic → 16 MACs
= 2 real DDR3 bytes moved per MAC operation
2.48 GMAC/s × 2 bytes/MAC = 4.96 GB/s needed per core
```

**Current (EXP-0081/0082, "2 tiles = 1 burst", DONE and real-P&R-verified)**:
two tiles now share one 16-byte burst, halving the moved-bytes-per-useful-byte
ratio:

```
2 tiles (A+B) × 16 bytes/burst ÷ 2 tiles-per-burst = 16 bytes moved DDR3 traffic → 16 MACs
= 1 real DDR3 byte moved per MAC operation (halved)
2.48 GMAC/s × 1 byte/MAC = 2.48 GB/s needed per core (halved)
```

### 3.3 The real gap

```
Historical:  1.24 GB/s available  vs  4.96 GB/s needed per core → ~25% sustainable
Current:     1.24 GB/s available  vs  2.48 GB/s needed per core → ~50% sustainable
```

The physical channel ceiling (1.24 GB/s, §3.1) did **not** change — §5.1's
fix reduced waste against a fixed ceiling, it did not raise the ceiling
itself. Even at ~50%, DDR3 remains the binding constraint, not DSP count,
even in the BEST case (zero row-switch overhead, one core, nothing else
sharing the bus). Raising the *physical* ceiling requires a wider channel
(§5.5) or a second channel — both analyzed below.

This ceiling gets **worse**, not better, with:
- **Row switches**: real measured Activate→Read latency is 16.125 ns
  (5 DDR3 clock cycles at 3.225 ns = real CAS-latency-5 timing, matches the
  MIG's own configured CL=5). A tile fetch that requires a fresh row
  activation costs ~16-30 ns instead of the 12.9 ns same-row case — a
  real, measured 25-130% penalty per row switch.
- **More cores sharing the one DDR3 channel** (N=2 today, N=4/8/16
  proposed): the 1.24 GB/s ceiling is shared across ALL active requesters,
  not per-core. Adding cores divides an already-insufficient budget further.
- **Weight fetching** (amortized, but not free): each job pair's weight
  prefetch (8 bursts for a 128-byte layer) adds real DDR3 traffic on top of
  activation fetching, though this cost is shared across the M reuse
  positions and becomes negligible for large M.

**Conclusion**: this was a genuine architectural ceiling, not a tuning
problem, and half of it has now been recovered for free. The 2× byte-
overhead from the original "1 tile = 1 full burst" convention (chosen in
EXP-0079 specifically to avoid a runtime-indexed part-select, given the
then-already-thin timing margin) was confirmed, with real numbers, to be the
single most expensive design decision in the memory path — and has since
been fixed (§5.1, DONE, EXP-0081/0082) by registering the select bit at
request time instead of avoiding the select entirely. The remaining gap
(~50% sustainable, not 100%) is now a *physical channel width* problem, not
a packing-waste problem — see §5.5.

---

## 4. Module-by-module review

### 4.1 Compute core (`mac2_dsp_packed.v`, `neural_processor_packed.v`)

Real, stable, efficient. 8 DSP48E1/core unchanged since EXP-0059. The
2-INT8-MAC-per-DSP48E1 packing technique is the correct choice for this INT8
workload — real utilization (16/240 DSP = 6.67% at N=2) confirms there is no
DSP-side pressure at all; **all scaling headroom is here, and all scaling
risk is elsewhere** (§3, §4.5).

No real change recommended here. This is the part of the design that is
*not* the bottleneck.

### 4.2 Weight-reuse path (`layer_prefetch_ctrl.v`, `layer_weight_buffer.v`,
`weight_tile_gather.v`)

Reused unmodified from V2 (ECP5 era), real and well-verified across many
EXPs (0057/0058/0061/0062 and every integration test since). Correctly
amortizes DDR3 traffic across M reuse positions — this part of the design
already does the "fetch once, use many times" optimization the activation
path currently lacks (§5.1's recommendation follows the SAME philosophy).

No real change recommended; this module is a good template for how the
activation path should evolve.

### 4.3 Activation-fetch path (`act_tile_fetch.v`, EXP-0079, updated EXP-0081/0082)

Real, correct (verified 3 levels deep, §2 of EXP-0079's own log entry; the
EXP-0081 packing change re-verified at all 3 levels again — `tb_act_tile_fetch.v`
8/8, `tb_packed_slot.v` 9/9, `tb_n2_system_ddr3.v` 8/8, plus real P&R). Two
real design choices identified in this analysis:

1. **1 tile = 1 full burst (2× byte overhead)** — **FIXED (EXP-0081/0082,
   DONE)**: now 2 tiles share 1 burst via a request-time-registered select
   bit (`sel_lat`), avoiding the runtime-indexed-part-select Fmax risk while
   still halving DDR3 waste. Real P&R confirms margin improved, not
   degraded (+0.030ns → +0.068ns). See §5.1.
2. **Lane A then lane B, sequential, per tile**: still doubles the real
   number of DDR3 transactions (and row-switch risk) versus a design that
   could fetch both lanes in a single wider transaction when they happen to
   be adjacent in memory. Not changed — flagged for future work; the
   DDRManager (§5.2) and 32-bit channel widening (§5.5) are higher-leverage
   and were prioritized first per the user's explicit direction.

### 4.4 Scheduling (`neural_director_packed.v`) and arbitration
(`sdram_arbiter_n.v`)

Real, correct, and — importantly for §5.2 — **`neural_director_packed.v`
already maintains a real queue of pending jobs** (`QUEUE_DEPTH=8`,
`q_x_base`/`q_w_base`/etc arrays). This is directly relevant to the
DDRManager/prefetch proposal (§5.2): the information needed to "know what's
coming next" already exists in this module, it just isn't currently used for
anything beyond pairing/dispatch decisions.

`sdram_arbiter_n.v`'s combinational-first-grant design (EXP-0066) is real,
proven, and already generalized to N-way (verified at N=3, EXP-0069) — no
real change needed to scale its own requester count for a DDRManager
addition or for N=4/8/16 scaling, though the arbiter's own **fairness
policy** (lowest-index-wins, a deliberate simplicity choice, EXP-0066) would
need re-examination if a DDRManager starts issuing speculative/anticipatory
requests that could starve a real, urgent request — see §5.2's own caveat.

### 4.5 Host interface (`spi_host_bridge_v3.v`, `flash_spi_master.v`,
`host_mem_bridge.v`)

Real, verified, low resource cost, not on any critical performance path
(host commands are inherently much slower than the internal compute/memory
loop). No bottleneck here. Not a scaling concern.

### 4.6 [DONE, EXP-0088] Result-writeback engine

**No longer absent.** `result_writeback.v` (new module) is instantiated
inside each `packed_slot.v`, sharing that slot's own ctrl port with
`layer_prefetch_ctrl.v`/`ddr_prefetch_mgr.v` exactly the way those two
already share it with each other (mutually exclusive in time by FSM
construction — writeback only starts in `S_RESULT`, strictly after the
tile loop has finished). Each completed job's result is written directly
into DDR3 at the job's own `result_addr_a`/`result_addr_b` — the SAME
architectural shape as the weight-fetch path, in reverse, exactly as this
section used to recommend before it was built. `job_done` now means "the
result is durably in DDR3", not merely "captured in a register only a
literal top-level pin could see". `n2_system_ddr3_top.v`'s own
`s0_result_data_a/b`/`s1_result_data_a/b` top-level PACKAGE PINS are
**removed** — the real, hard N-scaling blocker this section used to flag
(8 bits × 2 lanes × N cores → 256 pins at N=16) no longer exists at any N.
See §5.3 for the full real design/verification story.

---

## 5. Recommended interventions, ranked by real leverage

### 5.1 [Highest leverage] Denser activation packing — attack the real bandwidth ceiling directly — **DONE (EXP-0081/0082)**

**What was built**: 2 tiles (even/odd tile index) now share one
`BURST_LEN=8` burst instead of one tile per burst, halving real DDR3
bytes-per-MAC from 2 to 1. Real, measured effect: the achievable fraction
of one core's peak DSP throughput rose from ~25% to ~50% (§3.2/§3.3,
current numbers).

**Why this was avoided in EXP-0079**: doing so naively requires a
runtime-indexed part-select (which half of the burst response to use,
selected by a runtime tile-index bit) — the same anti-pattern
`weight_tile_gather.v` (EXP-0061) already flagged as a real Fmax risk, and
the P&R margin was already thin (+0.013ns) when that original design
decision was made.

**The safe path that was actually built**: the tile-index LSB is registered
into `sel_lat` at *request* time (`S_IDLE`, the same cycle `tcnt` is
latched) — many `ui_clk` cycles before the real DDR3 round-trip completes
and `ctrl_rdata` becomes valid. The eventual data-select mux therefore
selects on an already-long-stable registered bit, never one racing the read
data. **Real P&R confirms this is genuinely timing-safe, not just
functionally correct**: margin *improved* from +0.030ns to +0.068ns
(EXP-0082), despite the added mux logic (LUTs 5379→5437). Full detail:
`hardware/v3/rtl/act_tile_fetch.v` header, `docs/PHYSICAL_REALIZATION.md` §4,
`hardware/v2/logs/experiments.log` EXP-0081/EXP-0082.

**Verification**: `tb_act_tile_fetch.v` (8/8 PASS, covers even/odd-in-same-
burst, new-burst crossing, back-to-back alternation), `tb_packed_slot.v`
(9/9 PASS, bit-identical numeric results to the pre-change run),
`tb_n2_system_ddr3.v` (8/8 PASS, real xsim against real `ddr3_model.sv`,
JEDEC trace confirmed to show no more half-burst zero-padding).

### 5.2 [Complementary, addresses latency not bandwidth] DDRManager with orchestrator-driven prefetch (user's proposal) — **phase 1 DONE (EXP-0083); real benefit at the closed 32-bit channel is now ZERO (EXP-0087)**

**The original hypothesis** (written before building anything, now corrected
by real measurement — kept here so the correction is visible, not silently
edited away): the CURRENT design only ever requests a tile the moment
`packed_slot.v`'s own FSM reaches `S_TILEREQ` for it, so a DDRManager that
issues tile N+1's fetch WHILE tile N is still being consumed should hide
"today's design is very likely stalling ... for the majority of real time".

**What the real EXP-0083 measurement actually found**: this hypothesis
overstated the achievable benefit, for a specific, now-confirmed reason —
`neural_processor_packed.v`'s own pipeline accepts one operand **per cycle**
whenever it's in `NP_WAIT_OPERANDS` (`operand_ready` is state-only, not
gated on any internal pipeline stall). The old design's real "dead time"
between one tile's fetch completing and the next one's fetch being issued
was therefore only the ~2-cycle request/consume handshake overhead
(`S_TILEREQ` + `S_OPERAND`), not a large compute-bound stall — and that
small overhead is what look-ahead prefetch can actually remove, not the
DDR3 fetch latency itself (which is dominated by row activation/precharge,
§3.3, and look-ahead cannot make a single fetch faster, only start it
earlier).

**Real, measured result at the OLD 16-bit channel** (`ddr_prefetch_mgr.v`,
real P&R WNS +0.073ns, up from EXP-0082's +0.068ns, LUTs 5644, DSP48E1 16
unchanged):
- Real apples-to-apples comparison on the **real DDR3 backend**
  (`tb_n2_system_ddr3.v` via real xsim, same N=2/8-position workload,
  before vs after, same `ddr3_model.sv`): **2.86% reduction in total real
  simulated time** (108370.88ns → 105268.43ns) — EXP-0083's own headline
  number, but see EXP-0087 below: this figure does NOT survive the 32-bit
  channel and should not be quoted as the current real benefit.
- On the fast SDR placeholder backend (used for isolated glue-logic
  testing, `tb_ddr_prefetch_mgr.v`): 0.9% reduction in a row-switch-heavy
  scenario, and -1.4% (i.e. not faster) in an isolated same-row best case —
  that placeholder model's own per-fetch cost turned out to be dominated by
  a near-fixed protocol cost regardless of address locality, so it doesn't
  cleanly isolate the mechanism the real DDR3 backend's own row/bank timing
  does. Full detail: EXP-0083 in `hardware/v2/logs/experiments.log`.

**Real RE-measurement at the closed 32-bit channel (EXP-0087) — the
benefit is now GONE**: EXP-0083's 2.86% figure was only ever measured
against the OLD 16-bit/155MHz channel. Once the 32-bit channel's own real
timing closed (EXP-0086), this project's own standing plan
("re-measure once the wider channel's timing actually closes") was
carried out: a real, fair A/B on the SAME 32-bit/3225ps config, comparing
the current, committed `packed_slot.v` (with `ddr_prefetch_mgr.v`) against
a measurement-only fork reproducing the pre-EXP-0083 direct-fetch
sequencing (`hardware/v3/sim/packed_slot_noprefetch.v`, not part of the
real synthesis target). Real result, both runs 8/8 PASS, 0 errors,
identical golden data:
- WITH `ddr_prefetch_mgr.v`: $finish at 100663.1335 ns
- WITHOUT `ddr_prefetch_mgr.v`: $finish at 100656.6835 ns
- **WITH is 6.45ns SLOWER — a 0.0064% real regression**, statistically a
  wash, definitively NOT a benefit any more.

**Real, honest interpretation**: the 32-bit widening already halves the
real per-tile DDR3 round-trip latency. EXP-0083's own real finding
already showed the achievable benefit was capped by
`neural_processor_packed.v`'s fixed one-operand-per-cycle consumption
rate, not DDR3 latency, even at 16-bit — the wider channel shrinks the
real per-tile wait below whatever small gap the look-ahead could still
hide, leaving effectively nothing left to overlap. `ddr_prefetch_mgr.v`
stays wired into the real, committed `packed_slot.v` (real P&R already
signs off with it included, EXP-0086, and it causes zero real harm) —
but its justification is now "real, correct, timing-neutral", not "real
performance win". Building the larger multi-slot scheduler sketched below
is **not justified** by this real result.

**What it does NOT solve** (this part of the original reasoning holds):
§3.2's bandwidth ceiling is a hard physical limit (bytes/second the DDR3
channel can physically move) — prefetching earlier doesn't move more bytes
per second, it only avoids idle gaps. §5.1 (done) reduced bytes needed per
MAC; §5.4 (32-bit widening, decided, pending) raises the physical ceiling
itself; this phase-1 DDRManager only removes a small, now-quantified,
per-tile dead-time — real, free (zero timing cost, margin still improving),
but genuinely modest, not the larger win a first-principles estimate
suggested before it was actually built and measured.

**What was built** (`ddr_prefetch_mgr.v`, real RTL, not a sketch): wraps
`act_tile_fetch.v` (unmodified) with a depth-2 ping-pong buffer scoped to
ONE slot's own activation-tile look-ahead, exactly the validated,
scoped-first approach recommended below before this experiment ran. Bank
selection uses a registered index bit at both fill and read time, same
"known long before the data it gates" discipline as `act_tile_fetch.v`'s
own EXP-0081 layout — confirmed timing-safe by real P&R, not asserted.

**Full multi-slot / whole-Director-queue scheduler — still NOT built, and
now real-disconfirmed, not just deferred**: the re-examination against the
32-bit channel's real numbers (EXP-0087, above) has now happened, and the
result is that phase 1's own benefit is gone entirely at this core count
— there is no real cost/benefit case left for the larger design below
until a NEW real bottleneck reintroduces DDR3-latency-bound behavior
(e.g. a higher core count where the shared bus is contended again, or a
workload with heavier row-switching than this experiment's own test
pattern). Revisit only if/when N=4/8/16 scaling (§5.5) reveals DDR3
latency, not `neural_processor_packed.v`'s own consumption rate, as the
real limiting factor again.

**Concrete design sketch for the full version** (informed by what already
exists in this codebase; kept for when it's revisited):

- `neural_director_packed.v` already queues up to `QUEUE_DEPTH=8` pending
  jobs, each with a known `x_base`/`w_base`/`n_tiles` — this is exactly the
  "reservation" information a full DDRManager needs. No new bookkeeping is
  required at the Director level; it would READ this existing queue, not
  need the Director to change its own job-acceptance logic.
- A cross-slot manager would sit between `sdram_arbiter_n.v` and each
  slot's own `ddr_prefetch_mgr.v`/`layer_prefetch_ctrl.v` instances,
  scheduling across slots (not just within one slot's own tile loop as
  phase 1 does) — e.g. prioritizing requests that share an already-open
  DDR3 row across DIFFERENT slots, which phase 1 cannot see or exploit.
- **Real caveat, not glossed over**: this adds real arbitration complexity
  — a prefetched-but-not-yet-consumed request competing with another slot's
  genuinely urgent request needs a real priority policy, not just
  `sdram_arbiter_n.v`'s current lowest-index-wins simplicity (§4.4). A
  speculative prefetch that turns out to be wrong (e.g., the Director
  reorders/never dispatches that queued job) also wastes real bandwidth —
  needs a real cancellation/staleness mechanism, not assumed away.

### 5.3 [DONE, EXP-0088] Result-writeback engine

**Built and real-verified.** New module `result_writeback.v`, one instance
per `packed_slot.v` (matching how `layer_prefetch_ctrl.v`/`act_tile_
fetch.v`/`ddr_prefetch_mgr.v` are already one-per-slot, not a new
arbiter-requester count as N scales). On job completion (`S_RESULT`), it
writes BOTH lanes' results into DDR3 at the job's own `result_addr_a`/
`result_addr_b` and only THEN asserts `job_done` — real, correctness-first
sequencing (`job_done` now means "durably in DDR3", not "captured in a
register" the way it used to).

**Real addressing** (verified against `act_tile_fetch.v`'s/`layer_
prefetch_ctrl.v`'s own real address-computation code, not guessed):
`result_addr_a/b` arrive in `packed_slot.v`'s own `JOB_ADDR_WIDTH=26`-bit
convention; the low `ADDR_WIDTH=25` bits (dropping the unused top/MSB
headroom bit) are used directly as a ctrl-bus-native 32-bit-word address
— the exact same address space `x_base_a`/`w_base` already live in. One
full 32-bit ctrl-word is written per lane: `{node_id[15:0], 8'h00,
result_data[7:0]}`. The host reads results back via the ALREADY-EXISTING
`READ_MEM` (0x02) SPI opcode — no new protocol. Real, disclosed
host-firmware implication (not yet built, same class of gap as this
project's other disclosed firmware work, e.g. JTAG bit-banging): reading
a result back needs `mem_addr = result_addr[24:0]*2` for the value and
`+1` for node_id (2 host reads per lane), since `READ_MEM`'s own
`mem_addr` is 16-bit-word-granular while this engine writes a native
32-bit ctrl-word — the same real halving `host_mem_bridge.v`'s own header
already discloses for the debug raw-access path.

**Shared-bus discipline** (mirrors `act_tile_fetch.v`'s own real,
proven pattern, not reinvented): waits for `mem_grant` before ever
issuing `ctrl_req` (EXP-0066's own established rule — an early/blind
request on a shared bus can lose the request permanently); a real
`S_GAP` state waits for `!ctrl_busy` between lane A's write and lane
B's own, since `mig_native_adapter.v`'s own `busy` stays asserted one
cycle past `ctrl_ready`. `wmask` polarity matches `host_mem_bridge.v`'s
own real, already-working convention exactly (0 = write this byte, 1 =
masked, the same DQM-style polarity this project's whole memory stack
already uses end to end).

**Real, hard scaling blocker removed**: `n2_system_ddr3_top.v`'s own
`s0_result_data_a/b`/`s1_result_data_a/b` top-level PACKAGE PINS are
gone — each slot's `packed_slot.v` still exposes `result_data_a/b` etc.
as plain output ports (for debug/testbench visibility, unchanged), but
these are no longer wired to literal FPGA package pins at any N.

**Verification**: `tb_packed_slot.v` extended with a new real
read-after-write check (`verify_writeback` task) — after each job's
`job_done`, the testbench independently reads back the exact DDR3
location `result_writeback.v` should have written and confirms both the
result value AND node_id match, closing the loop (not just checking
`job_done` eventually pulses). **9/9 PASS, 0 errors**, real Icarus xsim
against `burst_mem_model32.v`. Also re-verified at the full N=2 system
level (`tb_n2_system_ddr3.v`, real xsim against the real, closed
32-bit/3225ps DDR3 model) to confirm the writeback engine behaves
correctly under real shared-bus arbitration contention between 2 slots
— **8/8 PASS, 0 errors**.

**Real P&R signoff**: **WNS = +0.099962ns, WHS = +0.036275ns**, 0
failing endpoints, "All user specified timing constraints are met" —
essentially unchanged from EXP-0086's own +0.095707ns (real, closed,
not a regression). 6642 LUTs (+260 vs EXP-0086, expected), 16 DSP48E1
(unchanged). This is now the current, trustworthy P&R signoff,
superseding EXP-0086/0087. See `hardware/v2/logs/experiments.log`
EXP-0088 for the full real result.

### 5.4 [DONE] Widening the physical DDR3 channel: 32-bit single channel vs. a second independent 16-bit channel — **functionally complete AND real timing closed (EXP-0084 → EXP-0086)**

**The real question**: §5.1 halved *waste* against a fixed 1.24 GB/s
physical ceiling; it did not raise the ceiling itself. Getting past ~50%
sustained DSP utilization requires more physical bytes/second, which means
either (a) widening the existing channel from 16-bit to 32-bit data width,
or (b) adding a second, independent 16-bit DDR3 channel. Both roughly
double the real 1.24 GB/s ceiling to ~2.48 GB/s. The user asked for an
honest comparison, not a diplomatically-balanced non-answer — here it is,
based on **real device data**, not guessed.

**Real device data** (queried directly from the actual Vivado part database
for this exact part/package, XC7A100T-**CSG324**): this package has only
**5 total I/O banks** — 14 (56 pins), 15 (56 pins), 16 (11 pins), 34 (56
pins), 35 (56 pins). All report `BANK_TYPE=BT_HIGH_RANGE` (Artix-7 has no
separate "HP" bank class the way some other families do). Banks **14 and
15** each expose **8 DQS-capable pin pairs** — the same memory-PHY
signature already used by the real, placed DDR3 controller on banks 34/35.
This means a second, independent DDR3 channel is *physically plausible* on
this package (the DQS-capable pins exist), but:

| Factor | 32-bit single channel | Second independent 16-bit channel |
|---|---|---|
| Real ceiling gain | ~2× (1.24 → ~2.48 GB/s) | ~2× (aggregate, same total) |
| Pin cost | Reuses/extends the existing MIG's own bank(s); no new bank claimed | Would claim banks 14 **and/or** 15 (the only banks with free DQS-capable pins) |
| **Conflict with already-placed I/O** | None | **Real, direct**: the management-SPI bus (bank 15: A15/B16/B17/A16) and the config-flash bus (bank 14: K17/K18/L13) are already placed in exactly the banks that would need to host a second channel. Only bank 16 (11 pins) would remain free — not enough margin for either SPI bus, let alone both. |
| Controller logic cost | One MIG instance, wider data path (mostly automatic — the MIG wizard regenerates CAS/CWL/MMCM ratios for the new width) | A full second MIG instance: second calibration sequence, second `ui_clk` domain, and critically the **DDRManager/arbiter would need to become channel-aware**, not just requester-aware — real added complexity on top of §5.2's own design, not a simplification |
| Real risk given current thin margin (+0.073ns) | Lower — one controller, one clock domain, incremental change to an already-proven design | Higher — two independent PHYs, two calibration state machines, cross-channel coordination logic, all new |
| PCB impact | **Corrected 2026-09-20 (real, caught mid-MIG-wizard-session, not assumed away)**: the selected DDR3 part, MT41J128M16JT-125:K, is a **x16 component** — the MIG wizard's "Data Width" field is the *controller's* aggregate width, not the chip's own width, so Data Width=32 with this part means **two physical MT41J128M16 chips wired in parallel**, each supplying half the data bus (standard DDR3 ganging — address/command fan out to both chips, only DQ/DQS/DM roughly double). This is a real BOM/footprint/routing addition, **not** "no PCB change" as an earlier draft of this document incorrectly claimed. Whether the extra DQ/DQS pins still fit within banks 34/35's existing allocation (likely, since address/command isn't duplicated) or require reaching into bank 14/15 must be confirmed from the real regenerated pinout, not assumed. | Also requires a second physical DDR3 chip (or a second x16-wide subsystem) for its own full independent controller, **plus** claims bank 14/15 for that second channel's own address/command/data — the SPI/flash pin conflict is additional to, not instead of, the same per-chip PCB cost as the 32-bit option. |

**Recommendation (honest, not deferential, as requested — and corrected
once, honestly, mid-session)**: **32-bit single-channel widening**, not a
second independent channel — this part of the recommendation still holds.
What does **not** hold, and was a real mistake in this document's earlier
version: the claim that widening needs "no PCB changes". It does — a second
physical DDR3 chip, confirmed by the MIG wizard itself (Memory Details:
x16, single component) when Data Width was set to 32. The real, remaining
differentiator versus a second independent channel is narrower than
originally stated but still real: 32-bit widening very likely stays within
the ALREADY-USED banks 34/35 (only data/strobe pins roughly double, address/
command is shared), while a second independent channel definitely needs its
OWN address/command/data pins, conflicting with the already-placed SPI
management bus (bank 15) and config-flash bus (bank 14). **User confirmed
they already understood the second-chip requirement and decided to proceed
with 32-bit widening regardless** (2026-09-20) — this remains the decided
path, now on the correct, fully-disclosed premise.

**What this required**: the real Xilinx MIG "Customize IP" wizard, re-run
interactively by the user (Data Width 16→32 AND Input Clock Period changed
in the same wizard session, since both require the wizard's own JEDEC/PLL
calculator — never hand-edited, per this project's own established
discipline). **Done, 2026-09-20.**

#### 5.4.1 Real outcome (EXP-0084 → EXP-0086): functionally complete AND timing closed — the width/clock-speed decoupling

**Confirmed correct, as predicted above**: the widened DQ/DQS/DM pins
stayed within banks 34/35 (no bank 14/15 conflict for the DDR3 signals
themselves) — real, verified from the actual routed pinout, not assumed.
One real prediction from this section did **not** hold, though, and is
corrected here rather than hidden: the differential **reference clock**
(`clk_ref_p/n`, a signal this section didn't analyze, chosen later in the
same wizard session) landed in **bank 14** by the wizard's own real UG586
placement rules — directly conflicting with the already-placed config-flash
SPI bus there (LVCMOS33 vs LVDS_25, incompatible VCCO in one bank). Real,
observed as an actual `place_design` failure, not hypothetical — fixed by
moving the flash bus to bank 16 (see `docs/PHYSICAL_REALIZATION.md` §2.3).

**The real, more consequential finding**: the SAME wizard session paired
the Data Width change with an Input Clock Period change (3225ps→2900ps,
chosen to keep the PHY:Controller ratio at 2:1 rather than falling back to
4:1, which would have halved `ui_clk` instead of speeding it up). This
clock speedup pushed `ui_clk` from 155.039MHz to 172.414MHz (+11.2%) — and
real P&R shows this specific increase breaks timing in
`neural_processor_packed.v`'s own packed-MAC accumulation tree (unchanged
since EXP-0059, real positive margin at the old rate, real -0.618ns
violation at the new one). **This is unrelated to the 32-bit width change
itself** — every module touched for the widening is real-verified
functionally correct (real xsim, real 2-chip DDR3 model, all tests PASS).

**The honest, useful conclusion**: width and clock rate are separable
levers, and this experiment shows they don't have to be exercised together.
Bandwidth = width × clock — 32-bit width **alone**, even at the OLD,
already-timing-proven 155.039MHz, already delivers the full intended 2×
bandwidth gain (1.24 → ~2.48 GB/s). The clock speedup was free upside *if*
the compute datapath could absorb it; real measurement showed it couldn't
without a real, separate re-pipelining of that accumulation tree — so it
was reverted rather than pursued, keeping the task scoped to channel width.

**Real closure (EXP-0086)**: the user reverted Input Clock Period back to
3225ps (keeping Data Width=32) via a second real MIG wizard session. This
regeneration itself triggered a real, wholesale recurrence of the
stale-import bug — Vivado silently re-imported the entire v3 RTL tree (9
files) plus the top XDC back to their pre-EXP-0084 state, reproducing the
already-fixed VCCO/PROHIBIT failures until root-caused (via the actual P&R
log, not guessed) and fixed again by re-establishing direct-reference
sources (see CLAUDE.md's stale-import lesson, now extended to cover this).
Once fixed, the real P&R closed cleanly: **WNS=+0.095707ns, WHS=+0.036275ns,
0 failing endpoints, "All user specified timing constraints are met."** The
`clk_pll_i`/155.039MHz domain — the exact domain that failed at -0.618ns in
the 2900ps attempt — closes at +0.096ns across 24522 endpoints, matching
EXP-0083's own 16-bit-era margin (+0.073ns) closely, confirming the width
change itself was never the real problem.

**Current real status**: fully DONE — RTL functionally verified (real xsim)
AND real P&R timing-closed (EXP-0086). This is now the project's current,
trustworthy signoff, replacing EXP-0083. `data_ready_n` (EXP-0085) also
confirmed placed and routed cleanly (D14/LVCMOS33) in this same run. See
`hardware/v2/logs/experiments.log` EXP-0084 and EXP-0086 for the complete
story, including all real P&R/XDC bugs found and fixed along the way.

### 5.5 Scaling path recommendation (real numbers, not a guess) — updated per user's final directive

Given §3's real bandwidth ceiling: **scaling core count alone, without
§5.2/§5.4, provides no real additional throughput past whatever N already
saturates the (post-§5.1) ~2.48 GB/s-equivalent demand ceiling** —
back-of-envelope, using §3.2's current numbers (2.48 GB/s needed per core
at peak, 1.24 GB/s physically available today pre-widening), that's already
close to N≈1 in the worst case and at most N≈2 in the best (zero-row-switch)
case on the current 16-bit channel. **Building N=4/8/16 before §5.2/§5.4 are
in place would very likely show near-IDENTICAL real throughput to N=2** — a
real, wasted engineering cycle the analysis recommends avoiding.

**Decided real order of work** (§5.1 and §5.2-phase-1 already done; this
reflects the user's own explicit final direction — 32-bit widening, then
the DDRManager, then N=2/4/8/16 tests, real target N=8, N=16 built
specifically to document where/how it breaks rather than to succeed):
1. ~~§5.3 (result-writeback)~~ / ~~§5.1 (denser activation packing)~~ — §5.1
   **DONE** (EXP-0081/0082). §5.3 remains a genuine blocker for N>2 and must
   land before any scaling test that needs real result data out of more than
   2 cores' worth of pins.
2. §5.4 (32-bit channel widening) — **DONE** (EXP-0084 functional +
   EXP-0086 real timing closure, WNS=+0.096ns). Doubles the physical
   ceiling itself (real, closed, ~2.48 GB/s), which §5.1 alone could not do.
3. §5.2 (DDRManager) — phase 1 (single-slot look-ahead prefetch) is
   **DONE and RE-MEASURED** (EXP-0083's original ~2.9% benefit was
   measured on the OLD 16-bit channel; EXP-0087 re-measured the same real
   A/B against the now-closed 32-bit channel and found the benefit is
   gone, ~0%/a 0.0064% regression — the wider channel's lower per-tile
   latency already absorbed the gap the look-ahead used to hide). Kept
   wired in for correctness/timing-neutrality; the larger multi-slot
   scheduler is **not justified** by this real result — revisit only if
   N=4/8/16 scaling below reintroduces DDR3-latency-bound behavior.
4. Real N=2/4/8/16 tests, each with its own real P&R signoff (margin is
   thin, §2 — do not assume a prior N's timing closure predicts the next).
   N=8 is the real target configuration; N=16 is expected to expose real
   bus/arbitration/timing limits and is built specifically to document that
   breakdown, not to be a viable production configuration. With DDR3
   latency no longer the real bottleneck at N=2 (EXP-0087), this is now
   the more promising real lever to pursue next.

---

### 5.6 [N=8 IS THE REAL, DEFINITIVE DEPLOYMENT TARGET — `n8_system_ddr3_top.v`, EXP-0089…0096] Hybrid systolic scaling: up to 4 groups × 4-PE weight-stationary chains

Captured from a 2026-09-20 brainstorming session as a purely exploratory
idea; the same day, per the user's own explicit reprioritization, the real
open design question below (shared-weight broadcast vs. a literal PE-to-PE
systolic shift register) was resolved with the user directly (not guessed).
**Real progress so far**:
- `systolic_group.v` + `packed_pe.v` (EXP-0089) — one real group of 4 PEs
  sharing a single broadcast weight fetch, real barrier-synchronized, real
  xsim-verified (8/8 PASS across 2 consecutive group jobs).
- Real out-of-context synthesis of one group (EXP-0090): **32 DSP48E1**
  (13.3%), confirming the original brainstorm's own quantified DSP
  projection exactly (8 DSP/PE × 4 PEs = 32).
- `neural_director_grouped.v` (EXP-0090) — real Director extension
  dispatching 8-position octets to free groups, a direct extension of
  `neural_director_packed.v`'s own already-proven 2-position pairing
  discipline. Real, deliberate finding: the host-facing SPI/`WRITE_JOB`
  submission protocol needs **zero changes** — the host just submits 8
  jobs sharing a weight base instead of 2. Real xsim-verified (4/4 PASS:
  correct octet dispatch + per-PE addressing, correct stall-not-mis-
  dispatch on a mismatched octet, correct queue wraparound). Also
  surfaced a real, generalizable testbench-race lesson (tight back-to-
  back stimulus pulses on the same edge the DUT samples on — fixed with
  `@(negedge clk)` stimulus — see CLAUDE.md).
- `n16_system_ddr3_top.v` (EXP-0091) — real N=16 top-level, directly
  adapted from `n2_system_ddr3_top.v`'s own proven structure (same real
  MIG, `spi_host_bridge_v3.v`, `flash_spi_master.v`, `host_mem_bridge.v`,
  all completely unmodified — confirms the zero-protocol-change finding
  above holds at full scale too), 4× `systolic_group.v` + `neural_
  director_grouped.v` + a new 21-way arbiter (4 group weight-fetch + 16
  PE activation/writeback + 1 host_mem_bridge). **Real synthesis-only
  result: 0 Errors, 128 DSP48E1/240 (53.33%)** — an exact real match to
  this section's own original DSP projection, now confirmed by real
  synthesis rather than estimated.
- **Real functional verification (EXP-0092)**: `tb_n16_system_ddr3.v`,
  directly adapted from `tb_n2_system_ddr3.v`'s own real DDR3-model
  methodology (same real 2-chip 32-bit DDR3 model, same real
  `mig_7series_0_mig`, real Vivado xsim — not the Icarus-with-stub-
  primitives check EXP-0091 itself used, which cannot instantiate the
  real MIG/DDR3 models at all). One shared layer across 32 positions,
  filling all 4 groups × 4 PEs × 2 lanes exactly once. **Real result:
  32/32 PASS, 0 errors**, `$finish` at 190718.0335ns — confirms the
  grouped Director's octet dispatch, the new 21-way→20-way (test-scope,
  no host slot needed) arbiter's slot map, and the shared-weight-
  broadcast barrier all really work wired together at full N=16 scale,
  not just in isolated unit tests.

- **Real, full P&R (EXP-0093)**: real `place_design`/`route_design`
  against the real XC7A100T-CSG324-2 part, same real Vivado project
  EXP-0086/0088's own N=2 signoffs used. Real post-route utilization
  confirmed: 19751 LUTs (31.15%), 34877 registers (27.51%), **128
  DSP48E1/240 (53.33%)** — holds through real place+route, not just
  synthesis. **Real timing result: WNS=−0.913ns, WHS=+0.029ns,
  TNS=−690.085ns, 3021 failing setup endpoints — TIMING CONSTRAINTS
  ARE NOT MET** on the real `clk_pll_i` (155.039MHz) domain, the same
  clock N=2's own real EXP-0088 signoff closed at +0.099962ns.

**Real, honest root cause (traced via the actual worst violated path,
not guessed)**: the critical path runs from one of the 16 real per-PE
`act_tile_fetch.v` FSM state registers, through 11 real logic levels, into
`mig_native_adapter.v`'s own `wdata_lat_reg`. That register is fed by
`sdram_arbiter_n.v`'s own `req_wdata` select mux, which grew from a
3-way select at N=2 to a real **20/21-way** select at N=16 over the
same 256-bit-wide bus — a real, substantial combinational fan-in
increase on the one shared resource every PE's DDR3 write must pass
through.

**Current real status**: N=16 is **functionally correct (EXP-0092)
but NOT YET timing-closed (EXP-0093)** — honestly not ready for real
hardware at the target clock. This does not invalidate the functional/
connectivity results; timing closure is a genuinely separate gate. N=2
(EXP-0088) remains the real, trustworthy, deployable signoff.

- **Real, hierarchical 2-level arbiter (EXP-0094)**: `sdram_arbiter_
  hier.v`, reusing `sdram_arbiter_n.v` unmodified, twice — 4 real LEAF
  instances (NUM_REQ=5, one per group, physically local to their own
  PEs) + 1 real TOP instance (NUM_REQ=5: 4 groups' own pipelined output
  + 1 host, host bypassed/unpipelined since it was never the real
  bottleneck), with a real pipeline register stage between the two
  levels — the direct, targeted fix for EXP-0093's own real, traced
  route-delay-dominated critical path. Isolated verification (new
  `tb_sdram_arbiter_hier.v`, real `burst_mem_model32.v`): **23/23
  PASS**, including cross-group and mixed host+group worst-case
  contention. Two real bugs found and fixed via signal tracing before
  a trustworthy result was possible: a testbench helper not waiting for
  grant before firing req (fixed to match `act_tile_fetch.v`'s own real
  S_MEMWAIT discipline), and a genuine RTL lost-pulse bug at the
  leaf-to-top boundary (a transient one-shot request pulse could be
  dropped if the top level was still busy with a different group —
  fixed with a real, sticky per-group `pending_req_r` latch). Wired
  into `n16_system_ddr3_top.v` as a drop-in replacement.
- **Real re-verification with the new arbiter**: functional xsim
  (`tb_n16_system_ddr3.v`) still **32/32 PASS, 0 errors**. Real, full
  P&R re-run: utilization essentially unchanged (128 DSP48E1/53.33%).
  **Real timing: WNS improved from −0.913ns to −0.646ns, TNS from
  −690.085ns to −97.541ns, failing endpoints from 3021 to 771** — a
  real, substantial, measured improvement, confirming the arbiter was
  correctly root-caused and fixed (the bottleneck demonstrably moved
  elsewhere). **Timing is still not fully met.**

**Real, honest current bottleneck (traced via the actual new worst
violated path, not guessed)**: the new critical path is inside
`neural_processor_packed.v`'s own DSP48E1 MAC datapath (`GEN_MAC_
PACKED`→`prodb1_reg`), logic-dominated (79%), not route — a
pre-existing module, unchanged since N=2, where it closed with an
already razor-thin real margin (WNS=+0.099962ns, EXP-0088). Real,
coherent interpretation: N=16's real overall die utilization (31% LUT)
increases general placement congestion enough, on its own, to erode
that already-thin margin — a different, more diffuse problem than the
arbiter's own single structural bottleneck, without an equally obvious
single-point fix.

**Current real status**: N=16 is **functionally correct (EXP-0092,
re-confirmed EXP-0094) with a real, substantially improved but still
NOT fully closed timing result (EXP-0094)** — honestly not yet ready
for real hardware at the target clock. N=2 (EXP-0088) remains the
real, trustworthy, deployable signoff.

- **Real P&R strategy tuning (EXP-0094, continued)**: same RTL,
  `opt_design -directive Explore` + `place_design -directive
  ExtraNetDelay_high` + a new `phys_opt_design -directive
  AggressiveExplore` step + `route_design -directive
  AggressiveExplore` — zero RTL risk. **Real result: WNS improved
  further to −0.338ns, TNS to −5.299ns, failing endpoints to 60** —
  another real, substantial, measured improvement (recovered roughly
  half of the remaining gap). Still not fully met. The new worst path
  is the SAME structural class (a different PE instance, confirming a
  recurring per-PE issue) but now measured at 84% logic delay (up from
  79%) — real evidence the recoverable route-delay slack is largely
  exhausted; what remains is dominated by the FPGA's own intrinsic
  DSP48E1→CARRY4 interconnect delay, unlikely to shrink further via
  more P&R strategy tuning alone.

**Real, cumulative progress**: two safe, real improvements (the
hierarchical arbiter + P&R directive tuning), neither touching
`neural_processor_packed.v`, together close ~89% of the original TNS
gap (−690ns → −5.3ns) and ~63% of the original WNS gap (−0.913ns →
−0.338ns). **Deliberate stop here**: the remaining gap needs the
shared, load-bearing `neural_processor_packed.v` MAC datapath itself
touched to close via RTL — a meaningfully larger, more careful step
(must be verified against both N=2's own signoff and N=16) than
anything else this session, better started with explicit direction
than pushed further autonomously.

**Not yet done, real and disclosed, real options for closing the
remaining N=16 gap**: (1) real, careful pipelining inside
`neural_processor_packed.v`'s own MAC datapath at the specific
`GEN_MAC_PACKED`/`prodb1_reg` boundary (fork-before-promote, N=2's own
signoff must stay protected, verify both). (2) a real, measured lower
target clock for the N=16 variant specifically (unquantified
throughput trade-off against N=2). See EXP-0094's own `next_action`.

**Real intermediate-N timing curve (EXP-0095)** — the user's own
explicit request ("cerchiamo dove fallisce"): `N_GROUPS` is a real,
already-existing top-level parameter, so `synth_design -generic
N_GROUPS=<n>` against the exact same RTL/arbiter (EXP-0094's own
hierarchical arbiter already wired in) gave a real, measured curve
without any new top-level files:

| N | N_GROUPS | DSP48E1 | LUT% | real WNS | real failing endpoints |
|---|---|---|---|---|---|
| 2 (flat, EXP-0088) | — | 16 | — | **+0.099962ns** | 0 (closed) |
| 4 (systolic) | 1 | 32 (13.3%) | 13.87% | −0.005ns | 2 |
| 8 (systolic) | 2 | 64 (26.7%) | 19.77% | **0.000ns** | **0 (REALLY CLOSED)** |
| 16 (systolic, EXP-0094) | 4 | 128 (53.3%) | 31.39% | −0.338ns | 60 |

**N=8 is a real, new, closed P&R signoff** — a real post-route
`report_timing_summary` result, 0 failing endpoints, same real part
and clock domain, not a projection. N=4 is a hair's breadth away
(N=8's own better WNS despite 2× the logic is real run-to-run
placer/router variance, not a contradiction). **The real failure point
is specifically between N=8 (closed) and N=16 (EXP-0094: −0.338ns)** —
N=12 (N_GROUPS=3) has not yet been measured and would narrow this
further. Real bug found and fixed first: `neural_director_grouped.v`'s
own bare `$clog2(N_GROUPS)` was invalid (`[-1:0]`) for the
never-before-tested N_GROUPS=1 case — fixed with the same real
`SELW`-style guard pattern `sdram_arbiter_n.v` already established.

**N=8 promoted to the real, definitive deployment target (EXP-0096)**
— the user's own explicit decision ("creiamo una versione funzionante
completamente per N=8, la pushiamo come versione definitiva"). New
`hardware/v3/rtl/n8_system_ddr3_top.v` — a real, permanent, named
top-level (not a build-time override), byte-for-byte the same RTL as
`n16_system_ddr3_top.v` with `N_GROUPS` defaulting to 2. Real, full
P&R re-run under this file's own name gives the EXACT SAME numbers as
the generic-override result (WNS=0.000ns, WHS=+0.017ns, 0 failing
endpoints, 64 DSP48E1/26.7%, 12535 LUTs/19.77%) — confirms the file
split is correct. New `tb_n8_system_ddr3.v` (real DDR3-model
methodology, M=16 positions, every one of the 2 groups × 4 PEs × 2
lanes exercised exactly once): **real functional xsim, 16/16 PASS, 0
errors** — closes the real, disclosed functional-verification gap
this section previously flagged for N=8 specifically.

**Current real status: N=8 (`n8_system_ddr3_top.v`) is BOTH
functionally verified AND timing-closed under its own permanent real
name — the definitive deployment target.** N=2 (`n2_system_ddr3_top.v`,
EXP-0088) remains documented as a real, valid, simpler fallback. N=16
(`n16_system_ddr3_top.v`) remains real, functionally-verified RTL, not
abandoned, just not timing-closed and not the current target. Real,
disclosed caveat carried forward: N=8's WNS=0.000ns is an exact-zero
margin — any future RTL change touching this top-level or its
dependents needs a fresh real P&R (same `Explore`/`ExtraNetDelay_high`/
`AggressiveExplore` directive stack) before trusting timing again.

**The problem it targets**: plain N=16 independent cores (§5.5's own
"documentary, expected to break" framing) means 16 independent DDR3
requesters contending for one arbitrated channel — real congestion that
neither §5.1 (packing) nor §5.4 (32-bit widening) alone removes, since both
attack bytes-per-MAC or raw bandwidth, not the *number of independent
consumers*.

**The idea**: instead of 16 flat, independent `packed_slot.v` instances,
group them into **4 systolic chains of 4 PEs each**. Within a chain: the
weight tile stays resident (loaded once, same "weight-stationary" pattern
`layer_prefetch_ctrl.v`/`layer_weight_buffer.v` already implement for the
existing A/B lane reuse — this is a direct extension to 4 positions instead
of 2, not a new mechanism), and activation data streams through the chain
position by position, fetched from DDR3 once per chain rather than once per
PE. Between the 4 chains (groups), full task-level parallelism is
preserved — each group can run an independent job, same as today's model.

**Why weight-stationary specifically (not activation-broadcast)**: chosen
because it generalizes to any layer type (FC, conv-via-im2col, attention —
anything reducible to "same weight matrix, many activation vectors") without
assuming a specific model's channel count or convolution overlap pattern —
important since this is a general-purpose accelerator, not built for one
fixed network.

**Real, quantifiable rationale** (order-of-magnitude, not yet measured —
flagged explicitly as a projection):
- DSP48E1: 16 cores × 8 DSP/core = 128/240 (53%) — real, fits with margin.
- DDR3 requesters: drops from 16 (flat) to 4 (one per chain) — a real 4×
  reduction in the number of independent contenders for the arbitrated
  channel, on top of (not instead of) §5.1's packing and §5.4's widening.
- Combined with §5.4's 32-bit widening (2× physical ceiling), the
  available-bandwidth-to-demand ratio improves by roughly 8× versus the
  naive flat-N=16 baseline — a real, meaningful, but NOT a full solve by
  itself; still needs real measurement once anything is built.

**Real open question, RESOLVED (EXP-0089)**: intra-chain dataflow — the
user was asked directly (`AskUserQuestion`, concrete side-by-side preview
of both real topologies) rather than guessed, given how different their
real risk/complexity profiles are. Chosen: **shared-weight broadcast** (one
real weight fetch per group of 4 PEs, no inter-PE result/data propagation,
each PE computes its own independent activation positions in parallel) —
**not** a literal PE-to-PE systolic shift register. This achieves the real,
quantified rationale above (4× reduction in redundant weight-fetch DDR3
traffic per group) without the real complexity/risk of genuine pipeline
fill/drain at chain boundaries, which the literal-systolic alternative
would have required. Real result-writeback stays exactly as §5.3 already
built it — `packed_pe.v` reuses `result_writeback.v` completely unmodified,
one real DDR3 write per PE, not a new "4 chain-output events" model (that
framing, from the original brainstorm, doesn't apply to the broadcast
design actually built). Arbitration: confirmed real — `sdram_arbiter_n.v`'s
own `NUM_REQ` parameter generalized to a 5-way arbiter (1 group weight-fetch
+ 4 PE activation/writeback) in `tb_systolic_group.v` with zero changes to
that module.

**Decision**: first real step DONE (EXP-0089) — build and verify the
isolated group mechanism before any Director/SPI/top-level integration,
per this project's own "one variable at a time" discipline. Full N=16
integration (Director-level group job dispatch, SPI protocol extension,
real top-level, real P&R) is real, disclosed, deliberately deferred work,
not yet done — see EXP-0089's own `next_action` in
`hardware/v2/logs/experiments.log`.

#### 5.6.1 [EXPLORATORY] Opportunistic BRAM cache for activation tiles

Smaller and more incremental than §5.6's systolic restructuring — doesn't
require knowing anything about the target network's structure in advance.
Artix-7 100T's Block RAM is real and currently **0% utilized** (§3, every
real P&R signoff to date) — real, free, unused capacity.

**The idea**: a small direct-mapped or low-associativity cache, in BRAM,
remembering the last few activation tiles fetched from DDR3 (address +
data). Before `act_tile_fetch.v` (or `ddr_prefetch_mgr.v`, EXP-0083) issues
a real DDR3 request, check the cache first — on a hit (e.g. two jobs, on
the same or different slots, requesting overlapping/adjacent tiles, common
in convolution with sliding-window overlap), skip the DDR3 round-trip
entirely.

**Why it's attractive**: catches real reuse the design doesn't have to
predict or assume in advance — unlike §5.6's systolic chains (which commit
to a specific reuse *pattern*, weight-stationary), a cache opportunistically
exploits WHATEVER locality the real workload happens to have, including
patterns nobody designed for. Composable with everything else already
built or decided (§5.1 packing, §5.4 widening, §5.6 systolic groups if that
direction is taken) — it's a cache in front of the existing fetch path, not
a replacement for it.

**Real open questions**: cache size vs. real hit rate is workload-dependent
and NOT measured — would need a real trace-driven estimate (or a real
simulation with representative test data) before sizing it, not guessed.
Coherency is simple here (activation data in DDR3 is written once by the
host before a job runs and never modified during compute, per the current
protocol) — no cache-invalidation problem to solve, a real simplification
versus a general-purpose cache design.

#### 5.6.2 [EXPLORATORY] Host-side (ESP32) job-queue optimization — a "software DDRManager"

A different kind of lever than anything else in this section: instead of
adding hardware intelligence inside the FPGA, exploit the fact that the
**ESP32 already has full visibility of the whole job queue before
submitting it** — `neural_director_packed.v`'s own `QUEUE_DEPTH=8` hardware
queue only sees jobs one at a time as they're written over SPI; the ESP32
firmware, upstream of that, could see and reorder ALL pending jobs at once.

**The idea**: the ESP32's own job-submission firmware groups/reorders jobs
before writing them over the management SPI bus (§2.2 of
`docs/PHYSICAL_REALIZATION.md`), so that jobs whose DDR3 addresses are
close together (same or adjacent rows) are submitted close together in
time — directly reducing the real row-switch cost (§3.3) that dominates
per-tile latency, WITHOUT any new RTL at all. A real, software-only
"DDRManager" living in ESP32 firmware, upstream of and complementary to
`ddr_prefetch_mgr.v` (EXP-0083, which only looks ahead within one already-
submitted job).

**Why this is worth capturing seriously, not just as a curiosity**: it's
the cheapest possible lever in this whole list — zero RTL, zero real P&R
risk, zero timing-margin cost (the project's real margin is thin, §2, and
every RTL addition risks it; this doesn't touch RTL at all) — and it's far
faster to iterate on than Verilog (the user's own established preference
for where complexity is easiest to absorb). It doesn't compete with any
other idea in this section — it can be built independently, at any time,
by whoever writes the ESP32-side firmware, and composes with all of them.

**Real open question**: requires the host firmware to know DDR3 addresses
well enough to group by row locality (`ROW_BITS`/`COL_BITS`/`BANK_BITS`
convention, §2 of `docs/PHYSICAL_REALIZATION.md`) — a real firmware-side
design task, not yet scoped, and out of this repository's own RTL scope
(ESP32 firmware isn't part of `hardware/v3/`).

---

## 6. Summary table: what's real vs. what's a calculation

| Claim | Status |
|---|---|
| WNS/WHS/LUT/DSP numbers throughout | **Measured** (real Vivado P&R reports, current: EXP-0086) |
| DDR3 back-to-back burst throughput (1.24 GB/s) | **Measured** (real `ddr3_model.sv` JEDEC trace) — physical channel limit, unchanged by §5.1's packing fix |
| Real Activate→Read latency (16.125 ns) | **Measured** (same trace) |
| Per-core compute throughput (2.48 GMAC/s) | **Calculated** from measured Fmax (155.039MHz) + known, fixed DSP-packing factor |
| Bandwidth needed per core, pre-packing (4.96 GB/s) | **Calculated**, historical (EXP-0079 layout) |
| Bandwidth needed per core, post-packing (2.48 GB/s) | **Calculated** from the real, as-built EXP-0081/0082 memory layout — current |
| "~25% of peak sustainable" (pre-packing) / "~50%" (post-packing, current) | **Calculated** ratios; post-packing figure re-verified against real P&R (EXP-0082) and real simulation (§5.1) |
| I/O bank/DQS pin counts for XC7A100T-CSG324 (banks 14/15/16/34/35) | **Measured** — queried directly from the real Vivado part database for this exact part/package, used in §5.4's dual-channel-vs-widening analysis |
| Row-switch penalty as a fraction of real workloads | **Not measured** — depends on host-chosen memory layout, flagged as an open question, not asserted |
| DDRManager phase-1 real stall-reduction benefit | **Measured** — real xsim A/B on `tb_n2_system_ddr3.v`, real DDR3 backend, before vs after `ddr_prefetch_mgr.v`. **2.86%** at the OLD 16-bit channel (EXP-0083); **~0% (0.0064% regression)** re-measured at the closed 32-bit channel (EXP-0087) — the benefit does not survive the wider channel, real and honestly reported, not oversold either direction. |
| Full multi-slot DDRManager's real benefit | **Not measured, not built, and now real-disconfirmed as a priority** — phase 1's own re-measurement against the closed 32-bit channel (EXP-0087) found ~0% benefit; §5.2 recommends against building the larger version unless N=4/8/16 scaling reintroduces DDR3-latency-bound behavior |
| 32-bit widening's real post-change bandwidth/timing numbers | **Measured** — real, closed P&R signoff (EXP-0086): WNS=+0.096ns, WHS=+0.036ns, 0 failing endpoints. The ~2.48 GB/s figure is now a real, closed-timing result, not a projection. |