# NMS STEP13 — Batch/Continuous Neural Processor: Final Report

Full data: `hardware/v2/nms/reports/batch_processor_sweep.csv`. Full
narrative: `hardware/v2/logs/experiments.log` (EXP-0025 through
EXP-0028), `decisions.log` (DEC-0024, DEC-0025). Architecture:
`hardware/v2/docs/architecture/neural_processor_batch.md`.

## What was actually built (and why it differs from the initial brief)

The governing spec framed the problem as "batch K neurons per
dispatch" to amortize the ~68.5-cycles/neuron non-memory floor
EXP-0024 measured for N_SLOTS=2. **Step 1's own mandatory RTL trace
overturned that framing before any RTL was written.**

Tracing `nms_memory_manager_pf.v` + `neural_processor.v` with a
zero-real-memory-latency isolated testbench (EXP-0025) showed the
floor is **93.4% explained by a 4-cycles/tile serialization bug**
inside the memory manager's own `ST_RUN` state (a strictly sequential
`read_issued → read_ready → present → consumed` chain with zero
overlap between tiles), not by per-job dispatch overhead (only 6.6% of
the floor). Neither the local SRAMs (1-cycle read latency) nor
`neural_processor.v` (designed for continuous 1-tile/cycle acceptance)
require this — it's purely an artifact of the manager's own
un-pipelined FSM.

**The fix built is therefore a continuous per-tile streaming redesign
of the memory manager** (`nms_memory_manager_stream.v`), not a
neuron-batching scheme. `neural_processor.v` was not modified — it was
never the bottleneck.

## Results

1. **The fix works exactly as designed** (EXP-0026): a read-ahead
   pointer + 1-deep skid buffer decouples "issue next tile's read"
   from "current tile consumed," confirmed cycle-by-cycle in
   simulation.
2. **It immediately exposed a second, previously-hidden bottleneck**
   at the *same* numeric value: `weight_prefetch_engine.v`'s own
   word-fetch rate is capped at 4 cycles/tile (P_IN=8 bytes ÷ 16-bit
   real PSRAM bus = 4 word-transactions/tile, 1 cycle/word minimum
   even at zero real latency) — a physical bus-**width** ceiling, not
   a latency or scheduling ceiling.
3. **Net real-system benefit at today's bandwidth: ~0%** (EXP-0028:
   185270 vs. 185398 cycles at N=2/PFD=8, -0.07%, noise-level). Bit-exact
   PASS, 256/256 neurons.
4. **The fix genuinely removes a 4× architectural ceiling** — proven
   directly (EXP-0027, a control experiment with weight-fetch
   bypassed): the new design achieves a clean **1 cycle/tile** (100%
   of `neural_processor.v`'s own theoretical per-tile acceptance
   rate), where the old design is hard-capped at 4 cycles/tile (25%)
   **regardless of bandwidth**. This ceiling is real and was
   previously invisible because a coincidentally-equal bandwidth
   ceiling was masking it.
5. **Resource/Fmax cost is small and in the expected direction**
   (EXP-0028): N=1 Fmax +3.7% (142.92 vs 137.76 MHz), N=2 Fmax -2.8%
   (92.57 vs 95.25 MHz, still comfortably above the 80 MHz target);
   LUT4/FF/CCU2C all within ±6%. No combinational-controller blowup.
6. **N=4 fails the 80 MHz target** (55.22 MHz) — but for a
   *different*, already-documented reason (`nms_activation_fill_ctrl.v`'s
   own priority-scan Fmax regression, first found in EXP-0022),
   unrelated to and unaffected by this STEP's own fix.

## The nine final-decision questions, answered directly

1. **Is per-neuron dispatch still acceptable?** Yes — it was never the
   dominant cost (only 6.6% of the floor). Batching neurons was not
   pursued; it would not have addressed the real bottleneck.
2. **What batch/stream granularity is optimal?** Per-*tile* streaming
   within a job (1-deep read-ahead), not per-neuron batching.
3. **What is the new fixed overhead?** With weight-fetch not the
   limiter, steady-state cost drops to 1 cycle/tile (down from 4) —
   the fixed per-job overhead (~17 cycles: entry, pipeline drain,
   write-back) is essentially unchanged and now the dominant residual
   cost per job.
4. **New asymptotic utilization ceiling?** For the control-plane
   component alone: 100% (removed entirely). For the *whole system*
   at today's real bandwidth: unchanged from EXP-0024's ~11.674%
   ceiling — the weight-fetch bus-width ceiling is now the sole real
   limiter.
5. **Does N=4 become viable?** No — blocked by a separate, pre-existing
   Fmax regression in the activation fill controller (55.22 MHz vs. 80
   MHz target), not by anything this STEP addressed.
6. **Does N=8 become viable?** No, for the same reason (N=4 already
   fails; N=8 was not synthesized given N=4's own failure).
7. **What memory bandwidth is actually required after this fix?** The
   same requirement EXP-0024 quantified (~36-82× today's real
   bandwidth for 90-99% targets) — this STEP's fix is *necessary but
   not sufficient*: without it, a future bandwidth increase would
   immediately hit the old 4-cycles/tile FSM ceiling and only realize
   25% of its potential benefit; with it, a future bandwidth increase
   can translate into up to 100% of its potential benefit.
8. **Is `neural_processor.v` reusable?** Fully reusable, unmodified —
   confirmed by direct RTL trace to already support the required
   continuous 1-tile/cycle acceptance; it was never the bottleneck.
9. **Next architectural step?** Real external memory bandwidth
   (wider bus, multiple independent PSRAM banks, or a redesigned
   weight-fetch protocol able to move more than one 16-bit word per
   cycle) — and, independently, the already-documented
   `nms_activation_fill_ctrl.v` Fmax regression that blocks N=4/N=8
   regardless of memory bandwidth. Both are flagged as future work,
   not undertaken this round.

## Final decision

**Outcome B** (STEP13's own framework: "batching/continuous execution
helps, but another bottleneck appears"). The executed fix — continuous
per-tile streaming inside the memory manager, not neuron-batching — is
real, correct (bit-exact), resource-neutral, and **removes a genuine,
previously-hidden 4× architectural ceiling** (proven via a direct
control experiment). It delivers **zero measurable benefit today**
because a second, independent, currently-co-dominant bottleneck
(weight-fetch bus width) already caps the system at the identical
rate. `nms_memory_manager_stream.v` is **adopted** as the new reference
NMS configuration: it is required groundwork for any future bandwidth
increase to actually pay off, and has no measured downside today. The
old `nms_memory_manager_pf.v` and `nms_memory_manager.v` remain
preserved for A/B/C reference. N=4/N=8 viability is now blocked by two
*separate* issues (external bandwidth, and the activation fill
controller's own Fmax regression) — neither of which this STEP could
or should redesign without further dedicated evidence, per the
project's own "no major redesign without evidence" discipline.