V2.0.0 hardware freeze - single SDRAM

FASE #1 hardware freeze for FPGA-Neural V2, N4/P8, single external
SDRAM (Alliance Memory AS4C4M16SA-6TIN) serving weights, activations,
and results through one physical sdram_controller.v instance. Removes
the PSRAM dependency (hardware/v1/rtl/psram_controller.v +
memory_interface.v) from the V2 physical path entirely -- V1 itself
remains fully unmodified, the golden reference.

New RTL: sdram_unified_backend.v (2-way W/AR arbitration over one
SDRAM controller, real per-byte DQM write masking added to
sdram_controller.v for correct single-byte result writes with no
read-modify-write), nms_neural_multiprocessor_sdram_unified.v (the
frozen top-level). Two real bugs found and fixed via full-system
testing before being accepted (ERR-0023): a deadlock and an off-by-one
data-shift bug in the new arbitration logic.

Real results: N=4 and N=2 D-Stress bit-exact (256/256 neurons), 40
real AUTO REFRESH events interleaved with zero corruption, real
Yosys+nextpnr-ecp5 synthesis/P&R for LFE5U-45F-8CABGA381 (149/245
TRELLIS_IO, a real 45-pin reduction from the prior dual-memory
design). Timing is MARGINAL (1/8 P&R seeds >=80MHz), reported honestly
rather than masked by the best seed.

Real, sourced ball-level pinout for the SDRAM bus + clk/rst (39/149
signals, P&R-verified) using the official Lattice ECP5U-45 pinout CSV
found on disk during this step's own pre-commit review -- corrects an
earlier draft that wrongly assumed no real pinout data was available.

Chip readiness: NO. Real, disclosed blockers remain (no physical host
interface exists yet -- the RTL's own reg_* ports are a 110-pin raw
test-harness bus; clock source/PLL decision; power/configuration
component selection) -- see hardware/v2/docs/{HARDWARE_FREEZE,
CHIP_READINESS,OPEN_ITEMS}.md for the complete, itemized status.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013xXuuRUWZScuo1DeYJxs3v
This commit is contained in:
2026-09-06 13:39:55 +02:00
co-authored by Claude Sonnet 5
parent 5c9ec618d3
commit 8e014d8d49
208 changed files with 3000390 additions and 0 deletions
+43
View File
@@ -12,3 +12,46 @@ NP_ERROR as an isolated fault (report + exclude from scheduling) rather
than a system-wide halt, once the Director (M5) exists to observe it.
No shared/global reset or shared bus signal is driven by any single
processor's error state at M1.
[2026-09-05] NMS STEP2 -- closed-form traffic model (activation/weight/
result bytes/cycle), cross-validated against EXP-0017's real simulated
floor.
Per slot, at steady-state 1 tile/cycle compute consumption (P_IN=8,
DATA_WIDTH=8):
WEIGHT traffic: never shared across neurons (each neuron owns its
weight vector) -- exactly P_IN=8 bytes/cycle/slot, always, with no
possible amortization.
ACTIVATION traffic: WORST CASE (no sharing, e.g. a cache miss every
tile, or genuinely different x_base per concurrent request) is
also P_IN=8 bytes/cycle/slot -- identical to weight traffic, since
both halves of a tile are the same size. BEST CASE (many neurons
of one layer sharing one x_base, fetched once and served from an
on-chip buffer thereafter -- exactly what
hardware/v2/rtl/activation_cache.v already exploits, DEC-0016) is
close to 0 bytes/cycle/slot amortized across the whole layer after
the first fetch.
RESULT traffic: exactly 1 byte per completed neuron, spread over
that neuron's own n_tiles cycles -- 1/n_tiles bytes/cycle/slot,
negligible next to activation/weight for any realistic n_tiles
(e.g. 1/16 for a 16-tile neuron) -- a low-bandwidth, low-priority
background stream, matching the LOW-priority scheduler tier the
user's own NMS spec calls for.
TOTAL per-slot demand: 8 B/cycle/slot (weight only, activation fully
amortized -- the aggressive/ideal NMS target) to 16 B/cycle/slot
(weight + activation, no sharing exploited -- the SAFE worst-case
floor). The 16 B/cycle/slot worst case is EXACTLY EXP-0017's own
measured simulated floor (TILE_BYTES=2*P_IN modeled combined,
unshared activation+weight per tile) -- a clean cross-validation of
the simulated result against the closed-form model, not a
coincidence: EXP-0017's harness never modeled activation sharing at
all (every tile issues its own fresh 16-byte fetch), i.e. it directly
measured the worst-case floor this note derives analytically.
IMPLICATION for STEP3 (bank/bandwidth architectural sweep): the NMS's
own Activation SRAM (built correctly -- banked/pipelined, not the
current single-tag, Fmax-fragile activation_cache.v) is what
determines whether the system sits near the 8 B/cycle/slot ideal or
the 16 B/cycle/slot worst case for a given workload's real
producer/consumer locality -- the Weight SRAM's own bandwidth
requirement is fixed and non-negotiable at P_IN bytes/cycle/slot
regardless of any caching scheme (weights are never shared, per
§6/§11 of the user's own NMS spec).
File diff suppressed because it is too large Load Diff
+124
View File
@@ -363,3 +363,127 @@ decision: vedi decisions.log DEC-0016 -- vittoria reale netta
next_action: nessuna ulteriore richiesta esplicitamente dall'utente per
questo giro. Datasheet V2 professionale, ora con dati definitivi
post-ottimizzazione da includere.
=== Sessione: NMS (Neural Memory System) -- avvio, STEP1 ===
timestamp: 2026-09-05
session: v2-NMS-STEP1
Avviato il nuovo progetto NMS (Neural Memory System), richiesto
dall'utente per superare il collo di bottiglia PSRAM condiviso emerso
dalla campagna finale V2 (91% utilizzo, N_SLOTS>=2 senza scaling
reale). Obiettivo: separare COMPUTE DOMAIN (Neural Processor x N) da
MEMORY DOMAIN (nuovo NMS: scheduler/prefetch/DMA/bank manager) da
BACKING STORAGE (PSRAM V1, invariata). Sviluppato interamente sotto
hardware/v2/nms/ (rtl/, sim/, docs/, reports/), senza toccare
hardware/v2/rtl/ esistente (baseline V2 congelata e sempre
confrontabile).
Seguito l'ordine di lavoro esplicitamente richiesto dall'utente (NON
scrivere RTL complesso prima di avere dati): completato STEP1
(bandwidth requirement study) -- vedi experiments.log EXP-0017,
errors.log ERR-0011 (3 bug reali trovati e risolti nel NUOVO harness
di simulazione stesso, non nel V2 congelato), decisions.log DEC-0017.
Nuovi file: hardware/v2/nms/rtl/ideal_memory_model.v (modello di
memoria idealizzato, SOLO simulazione, mai sintetizzato),
hardware/v2/nms/sim/tb_bandwidth_study.v (harness di sweep, riusa
hardware/v2/rtl/neural_processor.v reale e invariato per il lato
compute).
next_action: STEP2 (analisi matematica del traffico, in gran parte
derivabile a costo zero dai dati EXP-0017) poi STEP3 (sweep
architetturale bank/bandwidth in simulazione, usando i floor di
EXP-0017/DEC-0017 come target di progetto, non come ipotesi).
=== Sessione: NMS STEP3 (bank contention sweep) ===
timestamp: 2026-09-05
session: v2-NMS-STEP3
Completato STEP3 (sweep architetturale bank/bandwidth in simulazione).
Nuovi file: hardware/v2/nms/rtl/ideal_banked_activation.v (modello di
riferimento a N_BANKS fisso), hardware/v2/nms/sim/tb_bank_contention.v
(harness di sweep runtime su N_BANKS/stagger). Risultato chiave: con
N_BANKS=N_SLOTS (banking + broadcast su stesso indirizzo + arbitraggio
round-robin sui conflitti), lo scaling parallelo aggregato torna quasi
lineare (N=8 -> 7.87x vs N=1) anche con stagger di dispatch realistico
-- vedi EXP-0018/DEC-0018. 2 bug reali trovati e risolti nel nuovo
harness (ERR-0012): priorita' fissa -> starvation permanente; latch
job_started su job_ready invece che sull'handshake reale job_valid&&
job_ready.
next_action: STEP4 -- selezionare 2-3 architetture candidate per la
sintesi reale (Yosys+nextpnr-ecp5), usando N_BANKS=N_SLOTS come
candidato guida ma confrontando anche N_BANKS<N_SLOTS per il costo
reale BRAM/Fmax.
=== Sessione: NMS STEP4/5/6 (candidati Activation + Weight SRAM, sintesi reale) ===
timestamp: 2026-09-06
session: v2-NMS-STEP4-STEP5-STEP6
Completati STEP4 (selezione candidati), STEP5 (sintesi reale) e STEP6
(place&route reale) per entrambe le meta' del sottosistema memoria NMS:
Activation SRAM: 2 candidati reali (nms_activation_replicated.v,
nms_activation_banked.v), entrambi verificati bit-exact in simulazione
(broadcast, no-contesa, contesa forzata sullo stesso banco), poi
sintetizzati per N_SLOTS=2/4/8 x MAX_TILES=16/256 -- vedi EXP-0019.
Vinto: replicated (Fmax 2-4x piu' alto, ~24x meno LUT a N_SLOTS=8,
nessun arbitraggio -> nessuna possibile starvation), costo reale BRAM
accettabile (8 DP16KD, 7% del chip, a MAX_TILES=256/N_SLOTS=8) --
DEC-0019.
Weight SRAM: 2 candidati reali (nms_weight_direct.v, nms_weight_packed.v),
entrambi verificati bit-exact, sintetizzati stesso sweep -- vedi
EXP-0020. Vinto: packed (stessa BRAM di direct, ~2x meno LUT/FF a
N_SLOTS=8/MAX_TILES=256, Fmax comparabile) -- DEC-0020.
Costo reale combinato: 16 DP16KD (14.8% del chip) a N_SLOTS=8 con
vettori profondi realistici -- onestamente accettabile.
next_action: STEP7 e' concluso per la parte memoria. STEP8 --
integrazione NMS completa (prefetch engine, DMA, scheduler,
forwarding, interfaccia verso Neural Processor) -- ancora da fare.
=== Sessione: NMS STEP8 (integrazione completa, bit-exact) ===
timestamp: 2026-09-06
session: v2-NMS-STEP8
Completata l'integrazione STEP8: nms_dataflow_core.v (Dependency
Manager + Neural Director riusati invariati + N_SLOTS x
nms_memory_manager/neural_processor + nms_activation_fill_ctrl
condiviso, su nms_activation_replicated.v + nms_weight_packed.v).
2 bug reali trovati e risolti (ERR-0013): mancava un secondo stadio
di pipeline sulla lettura SRAM (risultato calcolato = 0 invece del
valore reale, causa lettura di un ciclo troppo presto); race nel
riavvio del prefetch peso privato che duplicava/saltava tile alterni
per job con n_tiles>1 (mai testato prima, dato che i primi test
usavano tutti n_tiles=1). 7/7 test PASS bit-exact (stesso DAG di
dataflow_core.v M7, test di broadcast su x_base condiviso, nuovo test
n_tiles=4) -- EXP-0021/DEC-0021.
next_action: STEP9 (benchmark end-to-end con Fmax reale da sintesi) e
STEP10 (tabella di confronto Current V2 vs NMS).
=== Sessione: NMS STEP9/STEP10 (benchmark reale + confronto finale) -- ROADMAP CHIUSA ===
timestamp: 2026-09-06
session: v2-NMS-STEP9-STEP10
Completati STEP9 (benchmark end-to-end reale, Fmax da sintesi reale) e
STEP10 (confronto Current V2 vs NMS), chiudendo l'intera roadmap NMS
(STEP1-STEP10).
Nuovo file: nms_neural_multiprocessor.v (top-level reale, rispecchia
neural_multiprocessor.v: nms_dataflow_core.v + slot_mem_arbiter.v reale
+ catena PSRAM V1 reale). Sintesi reale N_SLOTS=1/2/4/8. Benchmark
D-Stress reale (256 neuroni, 16 tile, stesso workload della campagna
V2) via psram_model.v reale, adattando tb_benchmark_suite.v.
4 bug reali trovati e risolti (ERR-0014), tutti la stessa causa radice:
un contatore che deve rappresentare il VALORE MAX_TILES stesso (non
solo indicizzarlo 0..MAX_TILES-1) dichiarato con la larghezza sbagliata
in 4 punti diversi (nms_memory_manager.v x2, nms_activation_fill_ctrl.v,
e il wire di collegamento in nms_dataflow_core.v) -- mai esposto prima
perche' nessun test STEP8 usava n_tiles=MAX_TILES esattamente.
Risultato finale (EXP-0022/DEC-0022): a N_SLOTS=2 (la config
raccomandata), NMS batte V2 attuale del 6.0% in wall-clock reale
(93.10MHz vs 87.72MHz Fmax reale) usando il 55% in meno di LUT4 e il
10% in meno di FF, a parita' di DSP/BRAM, bit-exact sullo stesso
workload D-Stress. A N_SLOTS=4, invece, V2 attuale vince ancora
(NMS -12.9%) per un problema di Fmax reale nel fill controller
condiviso dell'activation (stesso tipo di problema gia' documentato
per activation_cache.v in DEC-0016) -- pipeline del suo scan di
priorita' e' il fix concreto, non ancora tentato.
next_action: pipeline nms_activation_fill_ctrl.v per recuperare
N_SLOTS=4/8; completare N_SLOTS=1/8 su D-Stress per parita' con la
tabella V2. Nessun'altra richiesta esplicita dall'utente al momento.
+713
View File
@@ -385,3 +385,716 @@ DIAGNOSIS METHOD (item 1): periodic $display cycle-by-cycle tracing of
own owner/grant state, added temporarily to tb_memory_manager.v and
removed once the bug was isolated and fixed -- the same "trace real
signals, don't guess" discipline used throughout this project.
ERR-0011
timestamp: 2026-09-05T21:45:00Z
git_commit: 63cac6a7e5e0126bb13bea89dfe21966d5c9a1a5 (+ uncommitted NMS STEP1 work)
session: v2-NMS-STEP1-bandwidth-study
module: hardware/v2/nms/rtl/ideal_memory_model.v, hardware/v2/nms/sim/tb_bandwidth_study.v
context: building the NMS (Neural Memory System) STEP1 bandwidth
requirement study harness (EXP-0017) -- a NEW, simulation-only
idealized backing-store model + testbench, not part of the frozen
V2 datapath. Three real bugs were found and fixed in this new
harness itself before its output could be trusted, via the same
"trace real signals, don't guess" discipline as ERR-0006..0010.
1. REGISTERED-GRANT DOUBLE ADMISSION (ideal_memory_model.v, first
revision):
SYMPTOM: the very first sweep combination hung indefinitely (no
CSV row produced after 2+ minutes of wall-clock simulation,
manually killed).
ROOT CAUSE: req_ready was a REGISTERED output (visible one cycle
after the arbiter's own grant decision), so a requester holding
req_valid high (standard valid/ready) would still see req_valid
asserted for a second cycle after being granted -- the arbiter's
round-robin scan would find that same still-high req_valid and
grant it AGAIN, double-admitting one logical request (or worse,
re-granting indefinitely under some timing, causing the hang).
FIX: made req_ready purely COMBINATIONAL (same cycle as req_valid,
ordinary same-cycle valid/ready), computed in a separate
always @* block; the clocked state-update logic reuses that same
cycle's combinational grant decision via non-blocking assignment.
2. SINGLE-TRANSFER-AT-A-TIME SERIALIZATION CAP (ideal_memory_model.v,
second revision, after fixing item 1):
SYMPTOM: measured utilization collapsed to exactly 1/N_SLOTS at
every N_SLOTS>1 configuration regardless of configured bandwidth
(even cfg_bw_bytes=64, far above TILE_BYTES=16) -- a
suspiciously exact, BW-independent ratio, not a physically
motivated curve.
ROOT CAUSE: the model's "busy port" served only ONE TILE_BYTES-
sized transfer at a time regardless of cfg_bw_bytes, capping
aggregate system throughput at exactly 1 transfer/cycle once
bandwidth exceeded TILE_BYTES -- bandwidth above the per-transfer
floor was silently discarded instead of enabling several
concurrent transfers to drain in the same cycle.
FIX: rewrote the drain stage as a shared per-cycle byte BUDGET
(cfg_bw_bytes) applied against the queue head, completing as
many whole TILE_BYTES-sized transfers as the budget allows in one
cycle (bounded unrolled loop, MAX_ITERS=16) before carrying any
leftover partial progress to the next cycle -- correctly
separates aggregate BANDWIDTH from per-transfer LATENCY.
3. SINGLE-FLAG ISSUANCE THROTTLE (tb_bandwidth_study.v):
SYMPTOM: even after fixing items 1-2, utilization plateaued at
~50% regardless of configured bandwidth (>=16) or
PREFETCH_DEPTH (tested up to 8) at latency=0 -- again a
suspiciously round, BW/PFD-independent ceiling.
ROOT CAUSE: each per-slot feeder tracked outstanding fetches with a
single registered `fetch_pending` flag (set, wait a cycle, clear
on grant, re-evaluate `want_fetch` only the FOLLOWING cycle) --
a serial ~2-cycle round trip per request REGARDLESS of
PREFETCH_DEPTH, an artificial testbench-side issuance-rate cap
unrelated to the memory bandwidth actually being studied.
FIX: tied `mem_req_valid` COMBINATIONALLY to `want_fetch` (safe
since ideal_memory_model's own admission is unconstrained,
req_ready mirrors req_valid every cycle) -- removed the
fetch_pending latch entirely, allowing issuance at up to
1 request/cycle/requester, matching real achievable demand.
DIAGNOSIS METHOD: for item 1, direct hang observation + manual kill.
For items 2-3, isolated single-requester debug harnesses
(/tmp/tb_debug.v, /tmp/tb_debug2.v, not committed -- throwaway)
with cycle-by-cycle $display tracing of ideal_memory_model's own
internal head_active/head_remain/inq_count/lat_count via
hierarchical reference from the testbench, comparing the isolated
harness's hand-derivable expected rate (TILE_BYTES/cfg_bw_bytes
cycles/transfer) against the full sweep's reported utilization to
locate exactly which stage diverged from the physically-expected
value -- each bug was caught specifically because the buggy result
was a suspiciously ROUND, BW/PFD-INDEPENDENT ratio (1/N_SLOTS, then
0.5) rather than a smoothly-varying, physically-motivated curve.
VERIFICATION: after all three fixes, minimum aggregate bandwidth for
>=90/95/99% of compute-only throughput scales EXACTLY linearly with
N_SLOTS at 16 bytes/cycle/slot (matching the raw P_IN=8 tile
demand, 2*P_IN bytes/tile at 1 tile/cycle) -- see EXP-0017.
ERR-0012
timestamp: 2026-09-05T22:10:00Z
git_commit: 63cac6a7e5e0126bb13bea89dfe21966d5c9a1a5 (+ uncommitted NMS STEP3 work)
session: v2-NMS-STEP3-bank-contention
module: hardware/v2/nms/sim/tb_bank_contention.v
context: building EXP-0018's bank-contention harness. Two real bugs
found via literal simulation hangs, same "trace real signals, don't
guess" discipline as ERR-0006..0011.
1. FIXED-PRIORITY STARVATION:
SYMPTOM: every N_SLOTS>=2 run hung indefinitely at the very first
sweep combination (N_BANKS=1, stagger=0).
ROOT CAUSE: the bank arbiter picked the LOWEST-INDEX requester in a
bank every cycle, unconditionally -- with N_BANKS=1 and 2+ slots
wanting genuinely different tile indices, the lowest-index slot
always won every tie, permanently starving every other slot
(whose tile_idx never advances, so its done_flag never sets, so
the sweep driver's own "while(!all_done)" loop never exits).
FIX: added a per-bank ROUND-ROBIN priority pointer, rotated to
(last winner's index + 1) after every cycle it wins -- guarantees
every requester is eventually served even under permanent
contention. Directly matches the user's own NMS spec requirement
("Evita starvation", §10).
2. STAGE-BOOKKEEPING/DUT MISMATCH (job_started latched on job_ready
alone):
SYMPTOM: after fixing item 1, every combo with STAGGER>0 still hung
on its LAST slot.
ROOT CAUSE: `if (!job_started[s] && job_ready_s[s]) job_started[s]
<= 1;` gated only on job_ready_s[s], which neural_processor.v
asserts whenever it is idle REGARDLESS of job_valid -- for a
staggered slot whose job_valid gate hadn't opened yet, job_ready
was already high (idle) at cycle 0, so job_started[s] latched
immediately and incorrectly, permanently deasserting job_valid_s[s]
(`gate && !job_started[s]`) before the real, staggered job_valid
pulse was ever generated -- the testbench's own bookkeeping
believed the job had started while the real DUT (neural_processor.v)
sat in NP_IDLE forever, never having seen a real job_valid&&
job_ready handshake.
FIX: gated the latch on the ACTUAL accepted handshake,
`job_valid_s[s] && job_ready_s[s]`, not job_ready_s[s] alone.
DIAGNOSIS METHOD: cycle-by-cycle $display tracing (tile_idx, np_state,
req/ack, done flags per slot) added to a throwaway debug copy of the
testbench, with a hard cycle-count bailout to avoid an actual
infinite wait -- isolated item 2 specifically by noticing np_state
for the stalled slot stayed at NP_IDLE (0) forever even though this
testbench's own act_req_valid for that slot was already high,
meaning the DUT and the testbench bookkeeping disagreed about
whether the job had started.
VERIFICATION: after both fixes, all 4 N_SLOTS x 20 combos (80 total)
complete and produce the clean, monotonic-in-N_BANKS,
monotonic-in-stagger results reported in EXP-0018.
ERR-0013
timestamp: 2026-09-06T01:45:00Z
git_commit: 5c9ec61 (+ uncommitted NMS STEP8 work)
session: v2-NMS-STEP8-integration
module: hardware/v2/nms/rtl/nms_memory_manager.v
context: building/verifying nms_dataflow_core.v (STEP8 full NMS
integration: Dependency Manager -> Neural Director -> N_SLOTS x
(nms_memory_manager + neural_processor), backed by
nms_activation_replicated.v/nms_activation_fill_ctrl.v/
nms_weight_packed.v). Two real bugs found and fixed before trusting
correctness, via the same "trace real signals, don't guess"
discipline as ERR-0006..0012.
1. MISSING SECOND PIPELINE STAGE ON SRAM READS:
SYMPTOM: node0's own first (and only) tile computed result=0
instead of the expected 48 (2*3*8) -- a real, deterministic
wrong-answer bug, not a hang.
ROOT CAUSE: both nms_activation_replicated.v and
nms_weight_packed.v register rd_en THEN register the memory read
off that same registered rd_en (`if (rd_en) rd_data_reg <=
mem[addr];`) -- asserting rd_en at cycle T only becomes visible
to the SRAM's own always block at cycle T+1, which THEN schedules
the actual read for T+2. nms_memory_manager.v's own read-issue
logic used a SINGLE `read_issued` flag, capturing act_rd_data/
wgt_rd_data into input_data/weight_data exactly the cycle
`read_issued` first became visible (T+1) -- one full cycle too
early, grabbing the SRAM's stale (pre-read) output. Traced via
cycle-by-cycle tracing of act_rd_en/act_rd_data/wgt_rd_en/
wgt_rd_data/operand_valid/input_data/weight_data: at the exact
cycle op_valid first asserted, act_rd_data/wgt_rd_data already
held the CORRECT values (2 and 3) but input_data/weight_data
(captured the SAME cycle via non-blocking assignment) still read
as their PREVIOUS value (0), one cycle stale.
FIX: added a second flag (`read_ready`), turning the single-stage
read_issued -> capture sequence into a genuine 2-stage pipeline
(read_issued: SRAM now computing; read_ready: SRAM output now
valid, capture NOW) matching the SRAM's own real 2-cycle
rd_en-to-data latency.
2. STALE-VALUE RACE IN THE PRIVATE WEIGHT PREFETCH RESTART LOGIC:
SYMPTOM: not caught by any n_tiles=1 test (all of this file's first
3 test groups used n_tiles=1) -- would have silently corrupted
every n_tiles>1 job, duplicating/skipping alternating tiles.
ROOT CAUSE: `if (job_active_reg && !pf_busy && !pf_start &&
(wgt_fetched < n_tiles_reg[...])) pf_start<=1; pf_w_addr<=
w_base_reg + wgt_fetched*P_IN;` fires the SAME cycle pf_done
completes a fetch (prefetch_engine.v's own ST_DONE clears
fetch_busy the same cycle it pulses fetch_done), reading the OLD
(pre-increment) wgt_fetched -- re-issuing a fetch for the tile
that JUST completed instead of the next one; that duplicate
fetch's own later completion then writes into the NEXT tile's
SRAM slot using the WRONG (duplicated) source data. Same bug
CLASS as ERR-0006 (gating solely on a signal with its own
same-cycle side effect).
FIX: added `&& !pf_done` to the restart condition.
DIAGNOSIS METHOD: cycle-by-cycle $display tracing (act_rd_en/
act_rd_addr/act_rd_data/wgt_rd_en/wgt_rd_addr/wgt_rd_data/
operand_valid/input_data/weight_data, and separately mm_state/
np_state/result_valid/result_ready/result_data) added temporarily to
tb_nms_dataflow_core.v, gated by $time ranges around the specific
job windows of interest, removed once each bug was isolated and
fixed. Item 2 was only found AFTER item 1's fix, once a new
n_tiles=4 test (added specifically because every prior test used
n_tiles=1 and could not have exercised this path) was written and
initially failed.
VERIFICATION: after both fixes, tb_nms_dataflow_core.v passes 7/7
(same DAG-dependency shape as tb_dataflow_core.v's own M7 test;
shared-x_base broadcast-fill test across 2 concurrently-dispatched
slots; new n_tiles=4 multi-tile test) at N_SLOTS=2, bit-exact.
ERR-0014
timestamp: 2026-09-06T03:00:00Z
git_commit: 5c9ec61 (+ uncommitted NMS STEP9 work)
session: v2-NMS-STEP9-real-benchmark
module: hardware/v2/nms/rtl/nms_memory_manager.v,
hardware/v2/nms/rtl/nms_activation_fill_ctrl.v,
hardware/v2/nms/rtl/nms_dataflow_core.v
context: running a REAL D-Stress-scale benchmark (256 neurons, 16
tiles each -- exactly MAX_TILES) through the real V1 PSRAM chain for
STEP9. Every prior NMS test (tb_nms_dataflow_core.v, STEP8) used
n_tiles in {1,4}, never n_tiles==MAX_TILES exactly -- this one real,
necessary scale change (16-tile neurons, matching the project's own
realistic dense-layer workloads) exposed THREE compounding real bugs,
all instances of the SAME root cause, found via a real hang (0/8
neurons completed) traced cycle-by-cycle across three separate fix
attempts.
ROOT CAUSE (all three items): TIW=$clog2(MAX_TILES) sizes an SRAM
ADDRESS field (0..MAX_TILES-1), but several signals in this new
architecture are COUNTERS whose real value must reach MAX_TILES
itself (e.g. wgt_fetched/tile_idx/resident_count when a job's own
n_tiles equals MAX_TILES exactly) -- one bit short of what a counter
needs, though exactly right for an address.
1. nms_memory_manager.v: `n_tiles_reg[TIW-1:0]` truncated the 16-bit
n_tiles value (16) down to 4 bits, wrapping it to 0 -- every
"< n_tiles" check was permanently false, so wgt_fetched never even
STARTED advancing (stuck at 0 forever). Fixed: compare the full
16-bit n_tiles_reg directly (zero-extending the narrower counter),
never truncate n_tiles_reg itself.
2. Same modules: tile_idx/wgt_fetched/resident_count were themselves
declared TIW-bit (0..15 max), so even after fixing item 1's
comparison, incrementing wgt_fetched from 15 wrapped it back to 0
instead of reaching 16 -- an infinite re-fetch loop (real PSRAM
utilization visibly rose to 90%+ with zero forward progress).
Fixed: introduced a separate CNTW=$clog2(MAX_TILES+1) width for
these specific counters (one bit wider than TIW), keeping TIW only
for the SRAM address ports these counters occasionally drive
(explicit truncation at those specific assignment sites, always
safe since addressing only ever happens while the counter is
provably < MAX_TILES).
3. Introduced WHILE fixing item 1: `can_present`'s own second term was
mistakenly rewritten to compare wgt_fetched against n_tiles_reg
(checking "is the fetch done" instead of the actually-needed "is
THIS tile's weight available", i.e. tile_idx < wgt_fetched) -- once
wgt_fetched legitimately reached n_tiles (16), that comparison went
permanently false and deadlocked consumption of the LAST tile
forever even though every tile was genuinely ready (real activation
resident_count kept climbing normally the whole time, isolating
this as a logic error, not a data/timing problem).
4. Introduced WHILE fixing item 2: nms_dataflow_core.v's own top-level
wire connecting nms_activation_fill_ctrl.v's resident_count output
to nms_memory_manager.v's act_resident_count input was declared
`[TIW-1:0]` (4 bits), NOT widened to CNTW (5 bits) alongside the two
modules it connects -- a real value of 16 was silently truncated to
0 AT THE WIRE ITSELF, even though both endpoint modules' own
internal logic was by then already correct. Found by tracing
act_resident_count jumping from 15 directly to 0 (skipping 16)
exactly the cycle the real fetch completed pf_done=1.
DIAGNOSIS METHOD: cycle-by-cycle $display tracing added to a scaled-
down repro (8 neurons instead of 256, same real PSRAM model/n_tiles
=16) at each of 3 successive fix attempts, isolating each remaining
symptom (wgt_fetched stuck at 0 -> wgt_fetched wrapping 15->0
forever -> resident_count itself jumping 15->0 instead of reaching
16) before moving to the next. One self-inflicted tracing mistake
along the way (re-copying the clean testbench source over an
already-instrumented copy, silently producing zero debug output
until noticed and corrected) cost one extra iteration but found
nothing new.
VERIFICATION: after all four fixes, the scaled-down 8-neuron repro
passes bit-exact (6465 cycles), and the REAL, full-scale D-Stress
workload (256 neurons, 16 tiles each, through the real V1 PSRAM
chain) passes 256/256 bit-exact vs the golden model -- see EXP-0022.
ERR-0015
timestamp: 2026-09-05T23:41:08Z
module: hardware/v2/nms/rtl/weight_prefetch_engine.v
severity: CRITICAL (real, reproducible full deadlock -- 0/256 neurons
ever completed, watchdog timeout at 2,000,000 cycles, 0.0% PSRAM
utilization for the entire run)
found_during: STEP11's own real, full-scale N_SLOTS=1 D-Stress
PREFETCH_DISTANCE sweep (PFD in {1,2,4,8,16,32}) through
tb_nms_dstress_pf.v -- required by the STEP11 spec's own explicit
instruction not to assume PFD=32 is simply "unnecessary" but to
actually measure it ("larger [PFD], if necessary").
symptom: at PFD=32 (MAX_TILES=16, so CNTW=$clog2(MAX_TILES+1)=5 bits),
every one of the 256 D-Stress neurons hung forever -- the engine
never issued a single mem_req for the entire run.
root_cause: window_limit was computed as
`consumed_count + PREFETCH_DISTANCE[CNTW-1:0]` -- i.e. the
PREFETCH_DISTANCE PARAMETER ITSELF (not a tile counter, unlike every
prior instance of this bug class) was truncated down to CNTW bits
before the addition. For MAX_TILES=16, CNTW=5 bits (0..31), so
PFD=32 (6'b100000) truncates to 5'b00000 = 0. window_limit then
equals consumed_count exactly, forever, so `fetch_tile < window_limit`
is never true (fetch_tile >= consumed_count always holds by
construction) -- more_to_fetch is permanently false, and the engine
never fetches a single word.
Same ROOT CAUSE CLASS as ERR-0014 (a value that must be able to
reach/exceed a bound gets silently wrapped by a too-narrow field),
but this time the truncated value is a user-supplied, unbounded
build-time PARAMETER (PREFETCH_DISTANCE) rather than an internal
tile counter -- a genuinely new instance, not a recurrence of the
same fixed bug, since PREFETCH_DISTANCE did not exist before this
STEP.
fix: widened window_limit (and the two `fetch_tile [+1] < window_limit`
comparisons that use it) to a fixed 32-bit width, computed as
`{{(32-CNTW){1'b0}}, consumed_count} + PREFETCH_DISTANCE` (the
parameter itself, un-truncated) compared against a zero-extended
fetch_tile -- see weight_prefetch_engine.v's own updated comment at
the window_limit declaration.
verification: added a dedicated ERR-0015 regression case to
tb_weight_prefetch.v (`err0015_large_pfd_test`, gated `if (PFD>=32)`
so the file stays valid for every PFD it's compiled with) running a
full MAX_TILES=16 job at PFD=32 -- PASSES bit-exact post-fix
(confirmed: "PASS n_tiles=16 PFD=32 EXTRA_WAIT=0: ready_count=16,
all tiles bit-exact"). Re-ran the full tb_weight_prefetch.v suite at
PFD in {1,2,4,8,32}: 10/10 (9/9 at PFD=32, since the PFD/PFD+1 job-
size edge case is n/a once PFD>=MAX_TILES) tests PASS, 0 errors, at
every PFD tested.
secondary testbench-only issue found and fixed alongside (NOT an RTL
bug): tb_weight_prefetch.v's own `run_job(base, PFD, ...)` /
`run_job(base, PFD+1, ...)` edge-case calls and its windowing-cap
test's expected value both silently assumed PFD < MAX_TILES (asking
for a PFD+1-tile job when PFD+1 > MAX_TILES is asking the weight
SRAM to hold more tiles than it has, and the windowing cap can never
reach PFD once n_tiles/MAX_TILES itself is the tighter bound) --
fixed by gating those two run_job calls on `PFD < MAX_TILES` and
changing the windowing-cap expectation to `min(PFD, MAX_TILES)`.
impact_on_step11_conclusions: none -- PFD=32 was never the STEP11
benchmark's real focus (the D-Stress MAX_TILES=16 workload makes any
PFD>=16 equivalent to "unbounded"); this bug would only have
surfaced if a user tried MAX_TILES>32 with a correspondingly large
PFD. Fixed regardless, since a build-time parameter silently
deadlocking the whole design outside its narrow tested range is a
real correctness defect, not a documentation footnote.
ERR-0016
timestamp: 2026-09-06T06:08:18Z
module: hardware/v2/nms/rtl/sdram_controller.v
severity: CRITICAL (real, reproducible bank-open-forever condition;
every READ/WRITE after the first corrupted the next transaction)
found_during: STEP16 Phase 3 isolated regression (tb_sdram_controller.v,
BURST_LEN=4, CLK_FREQ_MHZ=166) -- first full run produced 0/460 tests
passed, 460 errors, with the real timing-checked sdram_model.v itself
flagging "ACTIVATE to bank 0 while already active" on nearly every
transaction.
symptom: every ACTIVATE after the very first one hit a real protocol
violation in the behavioral model (bank never precharged/closed by
the previous transaction), and every read returned corrupted/shifted
data as a side effect of the row/bank state being wrong.
root_cause: the READ/WRITE command's own address word was built as
`sdram_a <= {3'b000, 1'b1, req_col_reg}` with req_col_reg 8 bits wide
-- 3+1+8=12 bits total (correct width), but the concatenation placed
the auto-precharge '1' bit at position a[8], not a[10] where the
device's mode register / command decode actually expects it. A10 was
therefore always 0 on every real READ/WRITE -- auto-precharge was
silently never requested, so every opened row/bank stayed active
forever, and the next ACTIVATE (to a different row) hit a real
already-active bank.
found_by: static/hand trace of the FSM against the model's own
violation messages -- not assumed, root-caused from the actual
simulation log.
fix: `sdram_a <= {4'b0100, req_col_reg};` -- places the '1' at bit 10
exactly, with a[11]/a[9]/a[8] correctly tied 0.
verification: re-ran tb_sdram_controller.v (BURST_LEN=4, 166MHz) --
the "ACTIVATE while already active" violation disappeared completely;
see ERR-0017/ERR-0018 for the remaining data-corruption bugs found
after this fix, and EXP-0040 for the final clean 460/460 PASS run.
ERR-0017
timestamp: 2026-09-06T06:08:18Z
module: hardware/v2/nms/sim/sdram_model.v
severity: HIGH (every WRITE burst's first word was silently dropped)
found_during: STEP16 Phase 3 regression, same debugging session as
ERR-0016 (found immediately after fixing the A10 bug, since data
corruption persisted with the ACTIVATE violations gone).
symptom: after ERR-0016's fix, reads still came back shifted -- the
first data word of every burst was missing/zero, later words shifted
down by one burst position.
root_cause: the model captured write data into `mem[]` only from the
registered `wr_burst_active`-gated block, which does not become true
until the cycle AFTER the WRITE command is decoded. But real SDR
SDRAM presents the FIRST write word on DQ CONCURRENTLY with the
WRITE command itself, not one cycle later -- so the model's own
capture logic was structurally one cycle too late for word 0, and
silently never wrote it anywhere.
fix: in the `cmd_write` branch (command-decode cycle itself), capture
word 0 directly into `mem[ba,bank_row[ba],a[7:0]]` right there, and
initialize `wr_col`/`wr_remaining` to already point at the SECOND
word/column for the (unchanged) registered continuation logic to
handle words 1..N-1. Special-cased burst_len==1 (single-word write,
no continuation needed).
verification: re-ran the regression -- write side now matches the
controller's own per-cycle DQ timing exactly (confirmed by hand
cycle-trace before re-running); see ERR-0018 for the matching
read-side bug found next, and EXP-0040 for the final clean run.
ERR-0018
timestamp: 2026-09-06T06:08:18Z
module: hardware/v2/nms/sim/sdram_model.v
severity: HIGH (every READ burst arrived corrupted/shifted by two
cycles' worth of latency, in two separate compounding sub-bugs)
found_during: STEP16 Phase 3 regression, immediately after ERR-0017's
fix (write side alone did not clear the corruption -- read side had
its own, structurally identical class of bug, discovered by the same
cycle-exact hand trace method used for ERR-0016/ERR-0017).
symptom: reads still returned data shifted by 1-2 burst positions with
leading zeros, even though the underlying stored memory content was
now correct (per ERR-0017's fix).
root_cause (two distinct, compounding one-cycle-latency bugs):
(a) the model's read-side pipe insertion was gated by the registered
`rd_burst_active`, which -- exactly like ERR-0017's write-side bug --
becomes true only the cycle AFTER the READ command is decoded, adding
a spurious extra pipeline cycle before the first word ever entered
the CAS-latency shift register.
(b) independently, the model's DQ output was driven through its own
registered NBA stage (`dq_out <= rd_pipe[0]` inside the same
always block), adding ANOTHER cycle of latency beyond the shift
register's own CAS_LATENCY depth -- real SDR SDRAM's output register
is already accounted for inside the published CAS_LATENCY spec value,
so this was a redundant, uncounted-for extra stage unique to how the
model was built (a 16-entry shift array PLUS a separate registered
output buffer), not something the controller's own CAS_LATENCY-cycle
wait_cnt derivation could have anticipated.
fix: (a) mirrored the ERR-0017 write-side fix -- in the `cmd_read`
branch (command-decode cycle itself), insert word 0 directly into
`rd_pipe[cas_latency-1]` and initialize `rd_col`/`rd_remaining` to
already point at the second word for the registered continuation
logic (again special-cased burst_len==1). (b) converted `dq_out`/
`dq_out_en` from `reg` (NBA-driven) to `wire` (combinational,
`= rd_pipe[0]` / `= rd_valid_pipe[0]`), removing the extra output
register stage entirely.
verification: re-ran tb_sdram_controller.v (BURST_LEN=4, 166MHz) after
both fixes -- 460/460 tests PASS, 0 errors, bit-exact across every
Phase 3 scenario (A/B/E/F/H/I/G). See EXP-0040 for the full result.
ERR-0019
timestamp: 2026-09-06T06:08:18Z
module: hardware/v2/nms/rtl/sdram_controller.v
severity: CRITICAL (real, reproducible full deadlock -- caller waits
forever for `ready` that never comes)
found_during: STEP16 Phase 4 real 100/133/166MHz x BURST_LEN{1,4,8}
measurement sweep -- 8/9 configurations passed cleanly, but
BURST_LEN=1 @ CLK_FREQ_MHZ=133 hung indefinitely (confirmed via a
backgrounded run: >50s of continuous 100% CPU with zero output,
killed and root-caused rather than assumed to be a slow-but-finite
run).
symptom: tb_sdram_controller.v's do_transaction task deadlocks inside
`while (!ready) @(posedge clk);` for one specific transaction in the
400-iteration Test G sequence, at this one specific frequency/burst
combination (BURST_LEN=4 and BURST_LEN=8 at the same 133MHz did NOT
hang, confirmed by isolated re-run).
root_cause: this project's own established mem_req convention (per
weight_prefetch_engine.v's header comment) is a SINGLE-CYCLE req
pulse, not a held/latched signal. sdram_controller.v's S_IDLE state
gave periodic AUTO REFRESH unconditional priority over a same-cycle
`req` ("if (refresh_timer==0) ... else if (req) ..."), and did
nothing to capture the request's fields when refresh won -- so if a
test vector's req pulse happened to land on the exact cycle
refresh_timer reached 0, the request was silently discarded: the
caller's req had already gone low again on the next cycle (per the
single-pulse convention), the controller served the refresh with no
memory of the missed request, and no `ready` ever followed. This is
a general protocol race, not something inherent to 133MHz or
BURST_LEN=1 specifically -- it only reproduced there because that is
where this test sequence's absolute cycle counts happened to align
the two events; the SAME race is latent at any frequency/burst
length whenever a real caller's timing lines up unluckily.
fix: added a `req_pending` register that latches a fresh req's
wr/bank/row/col/wdata fields in S_IDLE the instant req is seen,
regardless of whether refresh wins arbitration that same cycle.
Added `eff_wr/eff_bank/eff_row/eff_col/eff_wdata` combinational
wires (`req ? <live input> : <latched req_*_reg>`) so the one real
S_IDLE branch that starts a transaction handles both "req arriving
fresh this cycle" and "req deferred behind a just-finished refresh"
uniformly, without duplicating the ACTIVATE-issuing code path.
`req_pending` resets to 0 under `rst`.
verification: re-ran the full 3x3 (100/133/166MHz x BURST_LEN 1/4/8)
sweep after the fix -- all 9 configurations now PASS 460/460, 0
errors, including the previously-hanging BURST_LEN=1/133MHz case
(re-confirmed finishing in <1s wall time, not merely "eventually").
See EXP-0041 for the full sweep result and the real measured
cycles/transaction used for Phase 4's throughput calculations.
ERR-0020
timestamp: 2026-09-06T06:08:18Z
module: hardware/v2/nms/rtl/sdram_controller.v
severity: CRITICAL (real, reproducible full-system deadlock -- 28/256
neurons never completed, both slots of an N=2 run permanently
starved; the isolated single-requester regression NEVER exercised
this because it always issues req from S_IDLE, never overlapping a
real arbiter's own outstanding-transaction bookkeeping)
found_during: STEP16 Phase 5 real FPGA-Neural integration (tb_nms_
dstress_sdram.v, N_SLOTS_CFG=2, D-Stress 256-neuron workload) -- the
SAME workload at N_SLOTS_CFG=4 passed cleanly (49430 cycles, 256/256
bit-exact), but N_SLOTS_CFG=2 hung at 228/256 neurons after 2,000,000
watchdog cycles (40x more cycles than N=4's entire run took to
finish all 256).
symptom: added temporary hierarchical debug tracing (both slots' own
nms_memory_manager_stream_wide state, plus the SDRAM controller's and
slot_mem_arbiter_wide's own internal state) to the watchdog's
periodic $display. Trace showed: both slots permanently stuck at
tile_idx=0/wgt_ready_count=0 (ST_RUN, waiting forever for the first
weight tile), while simultaneously sdram_controller.v itself sat
completely idle (req=0, busy=0, state=S_IDLE, req_pending=0) AND
slot_mem_arbiter_wide.v's own `owner` register was permanently
locked non-zero (a grant it believed was still in flight) with a
SECOND port's request stuck in `pending`, unable to ever be granted
because the arbiter was waiting for an `m_ready` that would never
come from a controller that had already forgotten the request.
root_cause: ERR-0019's fix only latched a fresh `req` into
`req_pending` from WITHIN the S_IDLE case-statement branch. The real
arbiter (slot_mem_arbiter_wide.v) pulses its own `m_req` for exactly
one cycle the instant it grants a new owner -- which can be ANY
cycle, including one where sdram_controller.v happens to be mid-
refresh (S_REFRESH_WAIT) or finishing a PREVIOUS transaction's own
PRECHARGE_WAIT tail, i.e. NOT executing the S_IDLE branch that
cycle. In that case the old fix's own req-latch code never even ran,
so the request was silently dropped exactly as it was before
ERR-0019's fix -- just for a DIFFERENT interfering state (a real
case the isolated regression's own single-requester testbench could
never reach, since it always waits for `busy==0` -- itself only
cleared in S_IDLE -- before ever asserting req, so its req pulses
are always S_IDLE-synchronous by construction).
fix: moved the req-latching code out of the S_IDLE branch entirely and
into the always-executed section that runs every cycle regardless of
`state` (alongside the existing unconditional refresh_timer
decrement) -- so a req pulse arriving during ANY state is captured
into req_pending/req_*_reg, not just one arriving while already in
S_IDLE. S_IDLE's own servicing logic (`req || req_pending`, `eff_*`
wires) is unchanged.
verification: re-ran the full Phase 3 isolated regression (9/9
frequency/burst-length configurations, 460/460 tests each) --
unaffected, still 100% PASS, confirming the fix is a pure superset
(catches a strictly larger set of req-arrival cycles, changes no
existing passing behavior). Re-ran the STEP16 Phase 5 N=2 D-Stress
benchmark -- now PASSES cleanly, 256/256 neurons bit-exact, 52161
total cycles (previously hung indefinitely). Re-ran N=4 -- unchanged,
49430 cycles (confirms the fix does not alter already-correct
behavior, only recovers previously-lost requests). See EXP-0042 for
the full Phase 5 N=2/N=4 integration results.
ERR-0021
timestamp: 2026-09-06T10:22:22Z
module: hardware/v2/rtl/dependency_manager.v
severity: HIGH (real, reproducible bit-exact regression -- a node
dispatched twice; caught before being kept, via a standalone
regression, NOT shipped)
found_during: STEP17 Part A's own "minimum fix experiment" -- an
attempt to pipeline the first_ready_idx/any_ready priority-encoder
scan (EXP-0044's own identified critical path) by one register
stage, to shorten the scan-to-array-read combinational chain that
limits N=4 Fmax to 81.55MHz.
symptom: after the fix, the full N=2/N=4 D-Stress regression showed
jobs_allocated=266 (N=2) / 268 (N=4), both > the expected 256 --
and roughly half the 256 neurons' results read back as 0 instead of
their real golden value. A minimal standalone unit test (16
independent always-ready nodes, `ready_ready` held high, isolated
from the rest of the system) reproduced the exact mechanism: node 0
is DISPATCHED TWICE, at cycles 20ns apart (exactly one extra pipeline
cycle).
root_cause: the scan (`first_ready_idx`/`any_ready`) is combinational
over node_state's PRE-edge value. On the exact cycle a dispatch
COMMITS (`ready_valid && ready_ready`, which schedules node_state
[ready_node_id] <= ST_DISPATCHED for the following cycle), the scan
THAT SAME cycle still sees the dispatching node as READY (its state
has not committed yet) -- an unconditional one-cycle register of the
scan's output would capture this still-READY snapshot, and one cycle
later (once node_state has actually committed and ready_valid has
dropped), the dispatch branch would fire again using the now-stale
snapshot, re-dispatching the just-dispatched node. A first attempted
repair -- gating the register update by `!ready_valid` -- did NOT fix
it: traced further (via the same standalone unit test) and found the
gate blocks the register from updating during the COMMIT cycle
itself (correct), but the register had ALREADY captured the stale
"node is READY" snapshot ONE CYCLE EARLIER (at the cycle the dispatch
was merely INITIATED, before ready_valid even reads 1), and that
stale value survives frozen through the commit cycle (blocked by the
gate) and is consumed immediately after, still stale by exactly one
cycle relative to the real commit.
fix: NONE SHIPPED. The change was reverted in full (`git checkout --
hardware/v2/rtl/dependency_manager.v`), restoring the exact STEP16-
validated RTL. Re-confirmed via the full N=2/N=4 D-Stress regression
that reverting restores the original correct behavior (256/256
bit-exact, 49430/52161 cycles, matching STEP16 exactly).
impact: none on the shipped design -- caught entirely within STEP17's
own investigation, before any P&R or benchmark result relying on the
broken version was reported as final. Documented here so a future
attempt at pipelining this scan does not repeat the same two failed
approaches; a fully correct version (if attempted again) will need
to reconcile the snapshot-capture cycle with the commit cycle more
carefully than a simple `!ready_valid` gate -- e.g. by explicitly
invalidating the registered snapshot when its own named node is the
one just committed, not merely suppressing updates during the commit
cycle.
ERR-0022
timestamp: 2026-09-06T12:30:00Z
module: hardware/v2/nms/rtl/sdram_weight_backend_pack128.v (STEP18,
first draft)
severity: HIGH (real, measured throughput REGRESSION -- not a crash or
data-corruption bug, but a real performance defect that would have
shipped as a "fix" if not benchmarked end-to-end before accepting it)
found_during: STEP18 Part C/D "weight packing" experiment -- a new
BURST_LEN=8 (128-bit/16-byte, 2-tile) SDRAM weight-fetch wrapper,
designed to halve the number of real SDRAM transactions needed per
P8 weight tile by caching the "other half" of each 128-bit fetch for
the next sequential request.
symptom: isolated unit-test regression (tb_sdram_weight_backend_pack128
.v, single requester, sequential access) passed 20/20, showing the
intended ~50% cache-hit rate (10 real fetches @16 cycles + 10 cache
hits @1 cycle for 20 requests). But the FULL N=4 D-Stress integration
benchmark got WORSE, not better: 74,004 cycles vs the STEP16 baseline
's 49,430 (+49.7%), despite remaining bit-exact correct.
root_cause: the first draft used a SINGLE cache entry. In the real
system, N_SLOTS=4 independent memory managers share this ONE
physical backend through slot_mem_arbiter_wide.v, which interleaves
their requests round-robin. A single cache entry holding slot A's
own "other half" gets overwritten by slot B's own "other half" (a
DIFFERENT address) before slot A's memory manager ever comes back
around to request its own paired tile -- so nearly EVERY access
became a real 16-cycle miss instead of the intended ~50% instant
hit, and 16 cycles/miss is WORSE than the STEP16 baseline's 10-cycle
BURST_LEN=4 transactions. This was found by actually running the
full integrated benchmark, not by trusting the isolated single-
requester unit test's own PASS result -- exactly the class of gap
the governing spec's own "this hypothesis MUST be tested, not
assumed" instruction anticipates.
fix: widened the cache from 1 entry to N_ENTRIES (parameter, sized to
the real N_SLOTS in the system, default 4) with fully-associative
address-tag lookup and simple round-robin eviction. This is SAFE
regardless of sizing accuracy (an evicted-too-early entry only costs
an extra real fetch -- a performance effect, never incorrect data,
since a cache MISS always falls back to a real, address-exact
fetch) and bounded correctly by construction: each slot has at most
one outstanding request at a time (the existing memory manager's own
design), so at most N_SLOTS "pending other halves" can exist
simultaneously in real traffic -- never more than N_ENTRIES=N_SLOTS.
verification: re-ran the full N=2/N=4 D-Stress regression -- now a
REAL improvement: N=4 44,935 cycles (-9.1% vs the 49,430 baseline),
N=2 47,399 cycles (-9.1% vs the 52,161 baseline), both 256/256
bit-exact. See EXP-0046 for the full before/after comparison.
ERR-0023
timestamp: 2026-09-06T11:26:46Z
module: hardware/v2/nms/rtl/sdram_unified_backend.v (STEP19, new module)
severity: CRITICAL (real, reproducible full-system deadlock -- 0/256
neurons completed; found immediately on the first real N=4 D-Stress
integration run, before being shipped)
found_during: STEP19 -- consolidating the two previously-separate
physical memory paths (weight-fetch SDRAM + activation/result PSRAM)
into ONE physical SDRAM chip serving all three traffic classes
through a single sdram_controller.v instance, per this step's own
explicit "single external SDRAM only" mandate.
symptom: the isolated unit regression (tb_sdram_unified_backend.v,
single requester on each of the W and AR ports, no contention)
passed 40/40. The first full N=4 D-Stress integration run deadlocked
completely: jobs_allocated stuck at 12, tiles_delivered=0,
neurons_completed=0/256, watchdog timeout at 2,000,000 cycles, AR
arbiter-side utilization pegged at 100% (permanently stuck).
root_cause: this is the SAME bug class as ERR-0019/ERR-0020 in sdram_
controller.v itself, recurring one layer up. sdram_unified_backend.v
's own top-level S_IDLE arbitration between its W (weight) and AR
(activation/result) ports only checked the LIVE w_req/ar_req signals
-- a single-cycle req pulse (this project's own established
convention) arriving on a cycle this backend happened to be busy
servicing the OTHER port was silently dropped, and the caller
(slot_mem_arbiter.v or slot_mem_arbiter_wide.v) waited forever for a
`ready` that would never come. Under real N=4 contention (weight
prefetch traffic and the very first activation-fill read racing for
the same physical port from the start), this raced and deadlocked
immediately -- the isolated single-requester test structurally could
not reach this case (no contention exists there).
fix (round 1): added req_pending latches for both W and AR ports,
mirroring sdram_controller.v's own corrected ERR-0020 fix (latch
unconditionally, every cycle, regardless of current state). This
introduced a SECOND, different real bug (see below) before the full
fix was correct.
fix (round 2, the actual bug this ID documents): the unconditional
latch fires whenever w_req is seen, INCLUDING the cycle a request is
ALSO being fully serviced via the W-port's own single-cycle cache-hit
fast path (w_cache_hit, reused from sdram_weight_backend_pack128.v's
own STEP18 caching logic) -- that fast path does not go through the
same-cycle "new transaction" branch and therefore never clears the
just-set w_req_pending latch. The result: after every cache-hit
service, w_req_pending is left spuriously set, and on the FOLLOWING
idle cycle the backend issues a bogus extra real SDRAM fetch using
the STALE latched address from the already-completed request --
which then delivers ITS OWN (stale, wrong) data as if it were the
response to the NEXT genuinely new request, shifting every
subsequent W-port response by one position. Found via the isolated
regression itself regressing from 40/40 to 26/40 after round-1's own
fix, with a clear "off by one iteration" failure signature (got ==
previous iteration's own expected value) -- traced by hand-cycle
tracing the interaction between the unconditional latch and the
case-statement's own NBA ordering (last assignment in the same
always block, same time step, wins).
fix (final): the w_cache_hit branch now explicitly clears
w_req_pending itself (`w_req_pending <= 1'b0`), overriding the
spurious set from the SAME cycle's unconditional latch (textually
later in the same always block, so it wins per standard Verilog NBA
ordering) -- cancelling the latch precisely when (and only when) the
SAME cycle's request was ALREADY fully handled via the fast path.
verification: re-ran the isolated unit regression (40/40 PASS,
restored) and the full N=4 AND N=2 D-Stress integration benchmark --
both now PASS bit-exact (256/256 neurons), with real, sustained
multi-hundred-refresh-interval operation and zero deadlock/timeout/
dropped/duplicated results. See EXP-0048 for the full integration
results.
File diff suppressed because it is too large Load Diff