micheleandClaude Sonnet 5
8890b0abc2
Real Alliance Memory AS4C4M16SA-6TIN datasheet (Rev 5.0, Table 17) specifies tMRD as a fixed 2-tCK cycle count, not an ns value. sdram_controller.v modeled it via ns_to_cycles(12), which rounded to 2 cycles by coincidence at every previously-tested frequency (100/133/166MHz) but rounds to only 1 cycle at the real 64MHz board target -- an under-provisioned one-time init sequence. Fixed by hardcoding T_MRD=2, matching how CAS_LATENCY is already modeled. Verified zero regression: full 9-config legacy sweep + a new dedicated 64MHz config (461/461 PASS each), N=2/N=4 D-Stress (identical cycle counts), board-level smoke test (11/11 PASS). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013xXuuRUWZScuo1DeYJxs3v
1291 lines
76 KiB
Plaintext
1291 lines
76 KiB
Plaintext
# V2 errors log -- solo append, mai troncato/sovrascritto (vedi README.md)
|
|
# Nessuna entry ancora -- popolato incrementalmente man mano che avanza lo sviluppo V2.
|
|
|
|
ERR-0001 (Icarus Verilog v13.0 toolchain bug, TASK/SCOPE-ENTRY DESYNC)
|
|
DATE: 2026-09-05
|
|
MODULE: hardware/v2/sim/tb_neural_processor.v (M1 testbench development)
|
|
SYMPTOM: a task (or any named `begin:label` block) whose FIRST executable
|
|
statement is a blocking assignment to a signal read by another module's
|
|
`always @(posedge clk)`, when the task/block is entered immediately after
|
|
a time-consuming statement in the caller with no intervening
|
|
`@(posedge clk)`, can make that FIRST assignment invisible to the DUT at
|
|
the very next clock edge (the DUT's own always block behaves as if the
|
|
signal never changed). Confirmed at V1's own frozen `neuron_parallel.v`
|
|
(unmodified, already certified) via a minimal 3-statement task
|
|
(`v1_start=1; @(posedge clk); v1_start=0;`) -- `busy` never asserted.
|
|
REPRODUCTION: /tmp/mt6.v-style repro (not committed, transient scratch
|
|
file) -- see conversation record for the exact minimal case.
|
|
DIAGNOSIS METHOD: bisected from the full dual-DUT testbench down to a
|
|
standalone ~15-line repro, ruling out RTL, port connections, and
|
|
operator precedence one at a time.
|
|
WORKAROUND: always begin such a task with an explicit `@(posedge clk);`
|
|
before its first assignment (matches the pre-existing convention in
|
|
hardware/v1/sim's own tasks, e.g. neuron_parallel_tb.v's run_neuron,
|
|
which is presumably why V1's own test suite was never affected).
|
|
STATUS: WORKAROUND APPLIED in hardware/v2/sim/tb_neural_processor.v's
|
|
run_case. NOT reported upstream (out of scope for this session). See
|
|
ERR-0004 for the broader consequence of this finding.
|
|
|
|
ERR-0002 (Icarus Verilog v13.0 toolchain bug, SPURIOUS CONDITION EVALUATION)
|
|
DATE: 2026-09-05
|
|
MODULE: hardware/v2/rtl/neural_processor.v (protocol-violation guard,
|
|
removed -- see decisions.log DEC-0003)
|
|
SYMPTOM: `if (operand_valid && !operand_ready && (state-is-one-of-four))`
|
|
inside `always @(posedge clk)` evaluated TRUE at an edge where
|
|
`operand_valid` was independently confirmed (via $display in the same
|
|
timestep, and via the testbench's own port-connected signal) to be 0.
|
|
Bisected term-by-term: even `if (operand_valid && !operand_ready)`
|
|
alone, and even `if (operand_valid)` alone with explicit `== 1'b1`
|
|
comparisons, still fired spuriously. Confirmed NOT a precedence issue
|
|
(parens are unambiguous) and NOT specific to this exact expression
|
|
shape (multiple simplified variants all reproduced it).
|
|
CROSS-CHECK: root-caused further via a minimal 2-state FSM
|
|
(`if (go) st<=B;`) with NO relation to the removed guard -- Icarus
|
|
failed to transition on an ODD-numbered testbench clock edge
|
|
(`repeat(3)` before the pulse) but succeeded on an EVEN-numbered one
|
|
(`repeat(4)`), reproduced identically with both `always #5 clk=~clk`
|
|
and `initial ... forever #5 clk=~clk` clock generators. VERILATOR
|
|
5.050 gives the CORRECT result for the same repro in both cases.
|
|
This suggests ERR-0001 and ERR-0002 are two symptoms of the same
|
|
underlying VVP scheduling defect (edge-count/thread-parity dependent),
|
|
not two unrelated bugs.
|
|
STATUS: the offending RTL block (protocol-violation detection) was
|
|
REMOVED rather than chased further -- see DEC-0003. Root cause not
|
|
fully isolated (documented honestly, not overclaimed).
|
|
|
|
ERR-0003 (real RTL bug in hardware/v2/rtl/neural_processor.v, FOUND AND FIXED)
|
|
DATE: 2026-09-05
|
|
MODULE: hardware/v2/rtl/neural_processor.v, stage 0 (input
|
|
alignment/register)
|
|
SYMPTOM: back-to-back single-tile jobs (and some multi-tile jobs)
|
|
produced result_data=0 instead of the correct value, while the
|
|
internal `y7` register (one stage upstream of the FSM's capture)
|
|
showed the CORRECT value one cycle later than `valid7` first asserted.
|
|
ROOT CAUSE: `last0 <= tile_last;` was unconditional, while
|
|
`valid0 <= operand_valid && operand_ready;` was correctly gated. A
|
|
master asserting `tile_last` before `operand_ready` rises (legal
|
|
valid-before-ready behavior) let a "last" tag propagate through the
|
|
pipeline (last1, last_tree[], last5, last6) with NO corresponding
|
|
valid tile behind it, arriving at stage 7 one cycle ahead of the
|
|
real valid/data pair and causing the FSM to capture a stale/wrong
|
|
`y7`.
|
|
EVIDENCE: isolated to a single-DUT, no-task, no-V1 repro
|
|
(hardware/v2/sim/tb_neural_processor.v run under Verilator, with a
|
|
cycle-by-cycle dump of valid5/last5/valid6/last6/valid7/y7) --
|
|
`last5=1` while `valid5=0` on the same cycle, confirmed the
|
|
desync's exact origin at stage 0.
|
|
FIX: `last0 <= (operand_valid && operand_ready) ? tile_last : 1'b0;`
|
|
-- last0 is now gated identically to valid0.
|
|
VERIFICATION: full 7-test bit-exact-vs-V1 regression
|
|
(hardware/v2/sim/tb_neural_processor.v under Verilator) -- 7/7 PASS
|
|
after the fix, including the back-to-back and single-tile-after-
|
|
multi-tile cases that exposed it.
|
|
STATUS: FIXED, verified.
|
|
|
|
ERR-0004 (methodology consequence of ERR-0001/ERR-0002)
|
|
DATE: 2026-09-05
|
|
NOTE: this session's V1 certification campaign (docs/validation/,
|
|
hardware/v1/docs/validation/) was verified exclusively with Icarus
|
|
Verilog v13.0, the ONLY simulator available on this machine at the
|
|
time. ERR-0001/ERR-0002 show that v13.0 has at least one real,
|
|
reproducible scheduling defect around clock-edge/task-entry timing.
|
|
V1's own testbenches were NOT observed to trigger it in this session
|
|
(V1's `run_neuron`-style tasks already begin with `@(posedge clk)`,
|
|
which incidentally avoids ERR-0001's trigger condition), and V1
|
|
remains frozen/untouched regardless. This is flagged here for
|
|
honesty, not to imply V1's certification is wrong -- re-verifying
|
|
the full V1 suite under Verilator was explicitly OUT OF SCOPE for
|
|
this V2-kickoff session (V1 is frozen, not to be touched) and was
|
|
not performed. See decisions.log DEC-0004.
|
|
STATUS: OPEN CAVEAT, not actioned in this session by design.
|
|
|
|
ERR-0005 (synthesis measurement artifact, WORKED AROUND, not an RTL bug)
|
|
DATE: 2026-09-05
|
|
MODULE: hardware/v2/rtl/neural_processor_array.v
|
|
SYMPTOM: synthesizing neural_processor_array as a bare top-level
|
|
module (every per-processor job/operand/result field exposed as a
|
|
real TRELLIS_IO pin) works at N_PROCESSORS=1 but fails place&route
|
|
at N_PROCESSORS=2 with "Unable to place cell ...$tr_io, no BELs
|
|
remaining to implement cell type 'TRELLIS_IO'".
|
|
ROOT CAUSE: not a logic/timing limit -- the LFE5U-45F-8BG381 package
|
|
has 245 TRELLIS_IO pins total; the array's wide per-processor buses
|
|
(input_data/weight_data alone are DATA_WIDTH*P_IN*N_PROCESSORS bits)
|
|
exceed that budget once N_PROCESSORS>=2, purely because these ports
|
|
have no on-chip consumer yet (the Memory Manager/M4 and Neural
|
|
Director/M5 that will drive them in the real system don't exist
|
|
yet).
|
|
WORKAROUND: hardware/v2/synthesis/harness_neural_processor_array.v --
|
|
a synthesis-only wrapper (NOT part of rtl/, not a functional
|
|
deliverable) that drives all wide buses from an internal free-
|
|
running LFSR and reduces outputs to a small checksum, keeping only
|
|
clk/rst/seed/checksum as real top-level pins. See its own header
|
|
comment and experiments.log EXP-0003 for the resulting real
|
|
resource/Fmax numbers.
|
|
STATUS: WORKED AROUND. Will become moot once M4/M5 exist and the array
|
|
is synthesized as part of a larger design with on-chip ports instead
|
|
of a bare top-level module.
|
|
|
|
ERR-0006 (real RTL bugs in hardware/v2/rtl/memory_manager.v, FOUND AND FIXED)
|
|
DATE: 2026-09-05
|
|
MODULE: hardware/v2/rtl/memory_manager.v
|
|
SYMPTOM: end-to-end M4 testbench (real V1 PSRAM backend + real M1
|
|
neural_processor) hung permanently partway through the first
|
|
multi-tile job -- bank_ready for the "current" bank never became 1,
|
|
even though the byte-level backend clearly kept completing read
|
|
transactions (observed via cycle-by-cycle hierarchical tracing of
|
|
memory_manager/prefetch_engine internal state).
|
|
ROOT CAUSES (three, found together during the same debugging session):
|
|
1. The tile-N+2 prefetch request, queued on tile-N's handoff, could
|
|
be issued (pf_start asserted) while prefetch_engine was STILL
|
|
mid-fetch for tile-N+1 -- there was no single-in-flight-request
|
|
discipline at all in the first draft. Fixed by adding a
|
|
single-entry pf_pending register: requests are queued, not
|
|
issued directly, and a dedicated rule launches the queued
|
|
request only once the engine reports free.
|
|
2. Even with that queue, pf_busy does not read 1 until the cycle
|
|
AFTER pf_start was first observed by prefetch_engine (its own
|
|
fetch_busy<=1 lags its own fetch_start sampling by one clock) --
|
|
checking only `!pf_busy` left a genuine one-cycle window where a
|
|
second queued request would fire on top of the one just
|
|
launched, silently overwriting pf_target_bank (and pf_x_addr/
|
|
pf_w_addr) for the fetch already in flight. The address corruption
|
|
was harmless (prefetch_engine had already latched the correct
|
|
address into its own state that same edge), but pf_target_bank
|
|
corruption meant the eventually-completed fetch's real data got
|
|
filed into the WRONG bank's bank_ready/bank_x/bank_w, permanently
|
|
starving the bank actually needed next. Fixed by gating the issue
|
|
rule on `!pf_busy && !pf_start` (the extra term closes exactly
|
|
this one-cycle window).
|
|
3. A combinational mux selecting between prefetch_engine's own
|
|
backend wires and the result-write-back FSM's wires was gated on
|
|
`state == MM_WRITE_RESULT`, but wr_mem_req (asserted while
|
|
state==MM_WRITE_RESULT) only becomes valid the FOLLOWING cycle,
|
|
i.e. while state==MM_DONE -- the mux therefore selected the wrong
|
|
source for the one cycle the write request pulse was actually
|
|
high, silently dropping the PSRAM write entirely. Fixed by
|
|
widening the mux's select condition to cover both states.
|
|
DIAGNOSIS METHOD: cycle-by-cycle hierarchical signal dumps (mm.state,
|
|
tile_idx, bank_ready, pf_busy, pf_pending, pf_target_bank,
|
|
prefetch_engine's own state) under Verilator, printed only on
|
|
signal-change to keep the trace readable, comparing against the
|
|
hand-derived expected sequence of events for a 3-tile job.
|
|
VERIFICATION: hardware/v2/sim/tb_memory_manager.v -- 3/3 tests PASS
|
|
after all three fixes, including a 5-tile job (steady-state
|
|
double-buffer swap across more than 2 tiles) and independent
|
|
PSRAM read-back of the written-back result byte (not just internal
|
|
signal inspection).
|
|
STATUS: FIXED, verified end-to-end with the real (unmodified) V1
|
|
PSRAM backend chain and a real M1 neural_processor.
|
|
|
|
ERR-0007 (Yosys usage quirk, WORKED AROUND, not an RTL bug)
|
|
DATE: 2026-09-05
|
|
MODULE: hardware/v2/synthesis/harness_dataflow_core.v (build script)
|
|
SYMPTOM: `chparam -set N_SLOTS 2 dataflow_core` (setting the parameter
|
|
directly on the NON-top child module, before running `synth_ecp5
|
|
-top harness_dataflow_core`) synthesizes with no visible error from
|
|
the chparam/hierarchy commands themselves, but `synth_ecp5` then
|
|
fails with "Module `\dataflow_core' referenced in module
|
|
`\harness_dataflow_core' in cell `\dut' is not part of the design" --
|
|
even though a standalone `hierarchy -top harness_dataflow_core` run
|
|
(no synth_ecp5) with the exact same chparam succeeds.
|
|
ROOT CAUSE: harness_dataflow_core.v's own instantiation of
|
|
dataflow_core explicitly overrides N_SLOTS via its own local
|
|
parameter (`.N_SLOTS(N_SLOTS)`) -- chparam on the child module's
|
|
DEFAULT is therefore always shadowed at that instantiation site
|
|
regardless of its value, and synth_ecp5's own internal re-hierarchy
|
|
pass (distinct from a standalone `hierarchy` call) does not
|
|
reconcile a chparam'd-but-never-actually-used child default the
|
|
same way, dropping the generic module reference instead.
|
|
WORKAROUND: set the parameter on the TOP module being synthesized
|
|
instead (`chparam -set N_SLOTS 2 harness_dataflow_core`), letting
|
|
its own instantiation forward the value down to dataflow_core as
|
|
designed. Confirmed working for both N_SLOTS=2 and N_SLOTS=4.
|
|
STATUS: WORKED AROUND. A build-script ordering detail, not a defect in
|
|
dataflow_core.v or harness_dataflow_core.v themselves -- noted here
|
|
so a future N_SLOTS sweep (M9/M10) does not re-trip over it.
|
|
|
|
ERR-0008 (real RTL bug in the first draft of hardware/v2/rtl/slot_mem_arbiter.v, FOUND AND FIXED)
|
|
DATE: 2026-09-05
|
|
MODULE: hardware/v2/rtl/slot_mem_arbiter.v (M8, new)
|
|
SYMPTOM: hardware/v2/sim/tb_neural_multiprocessor.v -- node0 (slot 0)
|
|
completes correctly (result=48), but node1 (slot 1, dispatched
|
|
concurrently to node0) never completes; its byte-level backend
|
|
request appears to simply vanish, and the slot hangs forever
|
|
(watchdog timeout at 20000 cycles, result stays 0).
|
|
ROOT CAUSE: memory_manager.v/prefetch_engine.v's own byte-level
|
|
backend protocol (mem_req/mem_wr/mem_addr/mem_wdata/mem_rdata/
|
|
mem_ready) is FIRE-AND-FORGET: mem_req is asserted for exactly ONE
|
|
clock cycle per byte transaction, with no separate "request
|
|
accepted" acknowledgment -- only `mem_ready` (transaction
|
|
COMPLETION) exists. M4's own testbench (tb_memory_manager.v) never
|
|
exposed this because it connects exactly ONE memory_manager directly
|
|
to int8_memory_access, which is always idle and therefore always
|
|
able to accept that single pulse the instant it fires. The first
|
|
arbiter draft only granted a port while its s_req was LIVE that same
|
|
cycle -- if slot 1's one-cycle pulse arrived on a cycle where the
|
|
arbiter was already owned by slot 0, the pulse was gone the very
|
|
next cycle with no record of it ever having happened, and slot 1's
|
|
prefetch_engine sat in ST_READ_X/ST_READ_W waiting forever for a
|
|
mem_ready that could never arrive (its request never reached
|
|
int8_memory_access at all).
|
|
DIAGNOSIS METHOD: ran the M8 testbench, observed node0 (whichever slot
|
|
the Director happened to grant the shared bus to first) complete
|
|
while node1 (the other, concurrently-dispatched slot) hung; traced
|
|
the fire-and-forget nature of mem_req directly in
|
|
prefetch_engine.v's own state machine (`mem_req <= 1'b1;` appearing
|
|
only inside single-cycle state-transition branches, unconditionally
|
|
cleared to 0 every other cycle) -- confirmed the arbiter's naive
|
|
"grant only while req is live" logic could not possibly catch a
|
|
pulse arriving during contention.
|
|
FIX: every incoming s_req pulse is now LATCHED into a per-port
|
|
`pending` register (capturing wr/addr/wdata the same cycle),
|
|
regardless of arbiter state -- the same single-entry "queue, don't
|
|
drop the request" idiom already used by memory_manager's own
|
|
pf_pending register (ERR-0006 fix #1). Grants are drawn from
|
|
`pending`, never from a live s_req directly. This adds a uniform
|
|
minimum 1-cycle latency to every byte transaction (a real, honestly
|
|
measured cost of sharing one PSRAM port across N_SLOTS -- see
|
|
timing.log/benchmark.log EXP-0009), but never drops a request
|
|
regardless of contention.
|
|
VERIFICATION: hardware/v2/sim/tb_neural_multiprocessor.v -- 4/4 PASS
|
|
after the fix (444 cycles end-to-end, vs the buggy draft's 20000-
|
|
cycle watchdog timeout). hardware/v2/sim/tb_memory_manager.v (M4,
|
|
untouched) re-run unchanged -- still 3/3 PASS, confirming the fix is
|
|
entirely contained inside the new arbiter module.
|
|
STATUS: FIXED, verified end-to-end with the real (unmodified) V1
|
|
PSRAM backend chain and real concurrent multi-slot contention.
|
|
|
|
ERR-0009 (one real RTL bug + two testbench bugs, all FOUND AND FIXED
|
|
during the post-M10 final benchmark campaign -- hardware/v2/sim/
|
|
tb_benchmark_suite.v)
|
|
DATE: 2026-09-05
|
|
|
|
1. REAL RTL BUG in hardware/v2/rtl/neural_director.v (M5, previously
|
|
committed/synthesized, never before exercised at N_SLOTS=1):
|
|
SYMPTOM: Verilator compile error building neural_multiprocessor at
|
|
N_SLOTS=1 -- "%Error-ZEROREPL: Replication value of 0 is only
|
|
legal under a concatenation" at three sites.
|
|
ROOT CAUSE: at N_SLOTS=1, $clog2(1)=0, making
|
|
`{$clog2(N_SLOTS){1'b0}}` a ZERO-width replication (illegal
|
|
outside a concatenation, IEEE 1800 11.4.12.1). Every prior
|
|
milestone (M5-M10) only ever built/simulated/synthesized
|
|
neural_director at N_SLOTS=2/4/8 -- N_SLOTS=1 was never actually
|
|
exercised until this benchmark campaign asked for it as the
|
|
baseline for parallel-scaling measurement.
|
|
FIX: replaced the three `{$clog2(N_SLOTS){1'b0}}` reset/default
|
|
expressions with the width-agnostic `'0` literal, which self-sizes
|
|
correctly for any width including 0. No functional change for
|
|
N_SLOTS>1 (same reset value).
|
|
VERIFICATION: hardware/v2/sim/tb_neural_director.v (M5's own
|
|
testbench, N_SLOTS=2) re-run unchanged -- still 4/4 PASS. All 6
|
|
benchmark-suite workloads then verified bit-exact at N_SLOTS=1
|
|
through the real full neural_multiprocessor + real V1 PSRAM chain.
|
|
STATUS: FIXED, verified at both N_SLOTS=1 (newly working) and
|
|
N_SLOTS=2 (no regression).
|
|
|
|
2. TESTBENCH BUG (tb_benchmark_suite.v, not RTL): psram_model's own
|
|
DEPTH parameter (524288... originally 131072 words = 256KB) was
|
|
smaller than the byte address range some workloads actually use
|
|
(workload C-Large's own result region alone needs word address
|
|
~0x24000, beyond a 131072-word/0x20000 DEPTH) -- a silent
|
|
out-of-bounds array access, same bug CLASS already documented once
|
|
before (an M5 testbench bug, sim_byte_mem's too-small DEPTH).
|
|
SYMPTOM: every C-Large neuron read back real=0 (poison value never
|
|
overwritten) despite the RTL reporting all 128 jobs completed.
|
|
FIX: DEPTH raised to 524288 words (1MB byte-addressable), computed
|
|
to safely exceed the highest byte address used by any of the six
|
|
workloads' regions (~0xB2006, workload F).
|
|
VERIFICATION: C-Large re-run bit-exact PASS after the fix.
|
|
|
|
3. TESTBENCH BUG (tb_benchmark_suite.v, not RTL): N_NODES=512 was
|
|
smaller than the highest node_id actually used -- workload D-Stress
|
|
(node_base=400, 256 neurons) reaches node_id 655, which silently
|
|
WRAPS at the 9-bit node_id width (512 -> truncates to 0), colliding
|
|
with workload A's node_id 0, already permanently ST_DISPATCHED (M6's
|
|
own design never reclaims dispatched node slots, decisions.log
|
|
DEC-0008). register_node's blocking `while(!reg_ready)` wait then
|
|
deadlocks forever (reg_ready never returns true for an already-
|
|
occupied, non-EMPTY node_id).
|
|
SYMPTOM: simulation appeared to hang indefinitely partway through
|
|
D-Stress's node registration (confirmed via periodic progress
|
|
`$display` instrumentation added specifically to localize this --
|
|
registration silently stopped advancing past neuron index 112).
|
|
FIX: N_NODES raised to 1024, comfortably exceeding every workload's
|
|
own node_id range.
|
|
VERIFICATION: D-Stress re-run bit-exact PASS (256/256 neurons) after
|
|
the fix, no further hangs at any N_SLOTS configuration (1/2/4/8).
|
|
NOTE: this is a REAL, honest consequence of DEC-0008's own design
|
|
choice (no node-slot reclamation) -- a long-running system that
|
|
keeps registering new nodes without ever reusing old (DISPATCHED)
|
|
ids will eventually exhaust its node_id space and deadlock exactly
|
|
this way. Flagged in the final benchmark report's Limitations
|
|
section, not just fixed and forgotten.
|
|
|
|
DIAGNOSIS METHOD (bug 3): periodic `$display` progress heartbeats
|
|
added to the registration loop and the completion watchdog loop,
|
|
run under `stdbuf -oL` to force line-buffered (not block-buffered)
|
|
output for real-time visibility, isolating the exact neuron index
|
|
where progress stopped advancing -- the same "trace real signals,
|
|
don't guess" discipline used throughout this whole project.
|
|
|
|
ERR-0010 (real RTL bugs found and fixed during activation_cache.v
|
|
implementation, DEC-0016)
|
|
DATE: 2026-09-05
|
|
|
|
1. Same bug CLASS as ERR-0006 (pf_target_bank/pf_pending_bank), NEW
|
|
instance, memory_manager.v's activation-cache side:
|
|
SYMPTOM: hardware/v2/sim/tb_memory_manager.v -- job xb=4096 (3
|
|
tiles) hung/produced wrong results after adding the shared
|
|
activation_cache request path; cycle-by-cycle tracing (temporary
|
|
$display instrumentation, later removed) showed a cache ack for
|
|
tile 1 (queued for bank 1) instead applying its data to bank 0.
|
|
ROOT CAUSE: `xc_target_bank` (which bank an ack's data should be
|
|
written into) was being written DIRECTLY by the queueing logic
|
|
(MM_IDLE/MM_PREFETCH_FIRST/MM_STREAM), the same register used to
|
|
resolve an ack that might still be OUTSTANDING from an EARLIER
|
|
queued request. Real PSRAM miss latency can exceed one
|
|
neural_processor tile's own compute time, so a LATER handoff can
|
|
queue a NEW request (targeting a DIFFERENT bank) in the same or a
|
|
following cycle, before the EARLIER request's ack has arrived --
|
|
with only one `xc_target_bank` register, NBA "last write in
|
|
program order wins" semantics silently overwrote which bank the
|
|
EARLIER, already-in-flight request's eventual ack gets applied
|
|
to. This is the EXACT bug ERR-0006 already found and fixed once
|
|
for pf_target_bank/pf_pending_bank (which already used a correct
|
|
two-register pattern: a `_pending_bank` staging register written
|
|
at queueing time, and the real `_target_bank` written ONLY by the
|
|
issue rule at the moment the request actually fires) -- this
|
|
module's newly-added activation-cache side did not follow that
|
|
already-established pattern, until now.
|
|
FIX: introduced `xc_pending_bank` (written at queueing time) and
|
|
changed `xc_target_bank` to be written ONLY by the issue rule
|
|
(`xc_target_bank <= xc_pending_bank;`, the same cycle xc_req
|
|
fires), mirroring pf_target_bank/pf_pending_bank exactly.
|
|
VERIFICATION: tb_memory_manager.v 3/3 PASS bit-exact after the fix.
|
|
Full final-benchmark campaign (24/24 workload/config
|
|
combinations) re-verified bit-exact.
|
|
|
|
2. Same bug CLASS as ERR-0009 item 1 (neural_director.v's N_SLOTS=1
|
|
zero-width replication), NEW instance, activation_cache.v:
|
|
SYMPTOM: Verilator compile error building the M4-level regression
|
|
testbench (activation_cache instantiated with N_SLOTS=1 there,
|
|
to serve a single memory_manager instance) -- same
|
|
"%Error-ZEROREPL: Replication value of 0 is only legal under a
|
|
concatenation" as ERR-0009.
|
|
ROOT CAUSE: `miss_idx = {$clog2(N_SLOTS){1'b0}};` -- identical root
|
|
cause to ERR-0009 item 1 ($clog2(1)=0 at N_SLOTS=1).
|
|
FIX: replaced with the same width-agnostic `'0` literal used in
|
|
neural_director.v's own fix.
|
|
VERIFICATION: tb_memory_manager.v (N_SLOTS=1 activation_cache) and
|
|
the full campaign (N_SLOTS=1/2/4/8) all build and pass.
|
|
|
|
DIAGNOSIS METHOD (item 1): periodic $display cycle-by-cycle tracing of
|
|
memory_manager's own internal state (xc_req/xc_ack/xc_outstanding/
|
|
xc_pending/bank_x_ready/bank_w_ready) and the 2-port test arbiter's
|
|
own owner/grant state, added temporarily to tb_memory_manager.v and
|
|
removed once the bug was isolated and fixed -- the same "trace real
|
|
signals, don't guess" discipline used throughout this project.
|
|
|
|
ERR-0011
|
|
timestamp: 2026-09-05T21:45:00Z
|
|
git_commit: 63cac6a7e5e0126bb13bea89dfe21966d5c9a1a5 (+ uncommitted NMS STEP1 work)
|
|
session: v2-NMS-STEP1-bandwidth-study
|
|
module: hardware/v2/nms/rtl/ideal_memory_model.v, hardware/v2/nms/sim/tb_bandwidth_study.v
|
|
context: building the NMS (Neural Memory System) STEP1 bandwidth
|
|
requirement study harness (EXP-0017) -- a NEW, simulation-only
|
|
idealized backing-store model + testbench, not part of the frozen
|
|
V2 datapath. Three real bugs were found and fixed in this new
|
|
harness itself before its output could be trusted, via the same
|
|
"trace real signals, don't guess" discipline as ERR-0006..0010.
|
|
|
|
1. REGISTERED-GRANT DOUBLE ADMISSION (ideal_memory_model.v, first
|
|
revision):
|
|
SYMPTOM: the very first sweep combination hung indefinitely (no
|
|
CSV row produced after 2+ minutes of wall-clock simulation,
|
|
manually killed).
|
|
ROOT CAUSE: req_ready was a REGISTERED output (visible one cycle
|
|
after the arbiter's own grant decision), so a requester holding
|
|
req_valid high (standard valid/ready) would still see req_valid
|
|
asserted for a second cycle after being granted -- the arbiter's
|
|
round-robin scan would find that same still-high req_valid and
|
|
grant it AGAIN, double-admitting one logical request (or worse,
|
|
re-granting indefinitely under some timing, causing the hang).
|
|
FIX: made req_ready purely COMBINATIONAL (same cycle as req_valid,
|
|
ordinary same-cycle valid/ready), computed in a separate
|
|
always @* block; the clocked state-update logic reuses that same
|
|
cycle's combinational grant decision via non-blocking assignment.
|
|
|
|
2. SINGLE-TRANSFER-AT-A-TIME SERIALIZATION CAP (ideal_memory_model.v,
|
|
second revision, after fixing item 1):
|
|
SYMPTOM: measured utilization collapsed to exactly 1/N_SLOTS at
|
|
every N_SLOTS>1 configuration regardless of configured bandwidth
|
|
(even cfg_bw_bytes=64, far above TILE_BYTES=16) -- a
|
|
suspiciously exact, BW-independent ratio, not a physically
|
|
motivated curve.
|
|
ROOT CAUSE: the model's "busy port" served only ONE TILE_BYTES-
|
|
sized transfer at a time regardless of cfg_bw_bytes, capping
|
|
aggregate system throughput at exactly 1 transfer/cycle once
|
|
bandwidth exceeded TILE_BYTES -- bandwidth above the per-transfer
|
|
floor was silently discarded instead of enabling several
|
|
concurrent transfers to drain in the same cycle.
|
|
FIX: rewrote the drain stage as a shared per-cycle byte BUDGET
|
|
(cfg_bw_bytes) applied against the queue head, completing as
|
|
many whole TILE_BYTES-sized transfers as the budget allows in one
|
|
cycle (bounded unrolled loop, MAX_ITERS=16) before carrying any
|
|
leftover partial progress to the next cycle -- correctly
|
|
separates aggregate BANDWIDTH from per-transfer LATENCY.
|
|
|
|
3. SINGLE-FLAG ISSUANCE THROTTLE (tb_bandwidth_study.v):
|
|
SYMPTOM: even after fixing items 1-2, utilization plateaued at
|
|
~50% regardless of configured bandwidth (>=16) or
|
|
PREFETCH_DEPTH (tested up to 8) at latency=0 -- again a
|
|
suspiciously round, BW/PFD-independent ceiling.
|
|
ROOT CAUSE: each per-slot feeder tracked outstanding fetches with a
|
|
single registered `fetch_pending` flag (set, wait a cycle, clear
|
|
on grant, re-evaluate `want_fetch` only the FOLLOWING cycle) --
|
|
a serial ~2-cycle round trip per request REGARDLESS of
|
|
PREFETCH_DEPTH, an artificial testbench-side issuance-rate cap
|
|
unrelated to the memory bandwidth actually being studied.
|
|
FIX: tied `mem_req_valid` COMBINATIONALLY to `want_fetch` (safe
|
|
since ideal_memory_model's own admission is unconstrained,
|
|
req_ready mirrors req_valid every cycle) -- removed the
|
|
fetch_pending latch entirely, allowing issuance at up to
|
|
1 request/cycle/requester, matching real achievable demand.
|
|
|
|
DIAGNOSIS METHOD: for item 1, direct hang observation + manual kill.
|
|
For items 2-3, isolated single-requester debug harnesses
|
|
(/tmp/tb_debug.v, /tmp/tb_debug2.v, not committed -- throwaway)
|
|
with cycle-by-cycle $display tracing of ideal_memory_model's own
|
|
internal head_active/head_remain/inq_count/lat_count via
|
|
hierarchical reference from the testbench, comparing the isolated
|
|
harness's hand-derivable expected rate (TILE_BYTES/cfg_bw_bytes
|
|
cycles/transfer) against the full sweep's reported utilization to
|
|
locate exactly which stage diverged from the physically-expected
|
|
value -- each bug was caught specifically because the buggy result
|
|
was a suspiciously ROUND, BW/PFD-INDEPENDENT ratio (1/N_SLOTS, then
|
|
0.5) rather than a smoothly-varying, physically-motivated curve.
|
|
VERIFICATION: after all three fixes, minimum aggregate bandwidth for
|
|
>=90/95/99% of compute-only throughput scales EXACTLY linearly with
|
|
N_SLOTS at 16 bytes/cycle/slot (matching the raw P_IN=8 tile
|
|
demand, 2*P_IN bytes/tile at 1 tile/cycle) -- see EXP-0017.
|
|
|
|
ERR-0012
|
|
timestamp: 2026-09-05T22:10:00Z
|
|
git_commit: 63cac6a7e5e0126bb13bea89dfe21966d5c9a1a5 (+ uncommitted NMS STEP3 work)
|
|
session: v2-NMS-STEP3-bank-contention
|
|
module: hardware/v2/nms/sim/tb_bank_contention.v
|
|
context: building EXP-0018's bank-contention harness. Two real bugs
|
|
found via literal simulation hangs, same "trace real signals, don't
|
|
guess" discipline as ERR-0006..0011.
|
|
|
|
1. FIXED-PRIORITY STARVATION:
|
|
SYMPTOM: every N_SLOTS>=2 run hung indefinitely at the very first
|
|
sweep combination (N_BANKS=1, stagger=0).
|
|
ROOT CAUSE: the bank arbiter picked the LOWEST-INDEX requester in a
|
|
bank every cycle, unconditionally -- with N_BANKS=1 and 2+ slots
|
|
wanting genuinely different tile indices, the lowest-index slot
|
|
always won every tie, permanently starving every other slot
|
|
(whose tile_idx never advances, so its done_flag never sets, so
|
|
the sweep driver's own "while(!all_done)" loop never exits).
|
|
FIX: added a per-bank ROUND-ROBIN priority pointer, rotated to
|
|
(last winner's index + 1) after every cycle it wins -- guarantees
|
|
every requester is eventually served even under permanent
|
|
contention. Directly matches the user's own NMS spec requirement
|
|
("Evita starvation", §10).
|
|
2. STAGE-BOOKKEEPING/DUT MISMATCH (job_started latched on job_ready
|
|
alone):
|
|
SYMPTOM: after fixing item 1, every combo with STAGGER>0 still hung
|
|
on its LAST slot.
|
|
ROOT CAUSE: `if (!job_started[s] && job_ready_s[s]) job_started[s]
|
|
<= 1;` gated only on job_ready_s[s], which neural_processor.v
|
|
asserts whenever it is idle REGARDLESS of job_valid -- for a
|
|
staggered slot whose job_valid gate hadn't opened yet, job_ready
|
|
was already high (idle) at cycle 0, so job_started[s] latched
|
|
immediately and incorrectly, permanently deasserting job_valid_s[s]
|
|
(`gate && !job_started[s]`) before the real, staggered job_valid
|
|
pulse was ever generated -- the testbench's own bookkeeping
|
|
believed the job had started while the real DUT (neural_processor.v)
|
|
sat in NP_IDLE forever, never having seen a real job_valid&&
|
|
job_ready handshake.
|
|
FIX: gated the latch on the ACTUAL accepted handshake,
|
|
`job_valid_s[s] && job_ready_s[s]`, not job_ready_s[s] alone.
|
|
DIAGNOSIS METHOD: cycle-by-cycle $display tracing (tile_idx, np_state,
|
|
req/ack, done flags per slot) added to a throwaway debug copy of the
|
|
testbench, with a hard cycle-count bailout to avoid an actual
|
|
infinite wait -- isolated item 2 specifically by noticing np_state
|
|
for the stalled slot stayed at NP_IDLE (0) forever even though this
|
|
testbench's own act_req_valid for that slot was already high,
|
|
meaning the DUT and the testbench bookkeeping disagreed about
|
|
whether the job had started.
|
|
VERIFICATION: after both fixes, all 4 N_SLOTS x 20 combos (80 total)
|
|
complete and produce the clean, monotonic-in-N_BANKS,
|
|
monotonic-in-stagger results reported in EXP-0018.
|
|
|
|
ERR-0013
|
|
timestamp: 2026-09-06T01:45:00Z
|
|
git_commit: 5c9ec61 (+ uncommitted NMS STEP8 work)
|
|
session: v2-NMS-STEP8-integration
|
|
module: hardware/v2/nms/rtl/nms_memory_manager.v
|
|
context: building/verifying nms_dataflow_core.v (STEP8 full NMS
|
|
integration: Dependency Manager -> Neural Director -> N_SLOTS x
|
|
(nms_memory_manager + neural_processor), backed by
|
|
nms_activation_replicated.v/nms_activation_fill_ctrl.v/
|
|
nms_weight_packed.v). Two real bugs found and fixed before trusting
|
|
correctness, via the same "trace real signals, don't guess"
|
|
discipline as ERR-0006..0012.
|
|
|
|
1. MISSING SECOND PIPELINE STAGE ON SRAM READS:
|
|
SYMPTOM: node0's own first (and only) tile computed result=0
|
|
instead of the expected 48 (2*3*8) -- a real, deterministic
|
|
wrong-answer bug, not a hang.
|
|
ROOT CAUSE: both nms_activation_replicated.v and
|
|
nms_weight_packed.v register rd_en THEN register the memory read
|
|
off that same registered rd_en (`if (rd_en) rd_data_reg <=
|
|
mem[addr];`) -- asserting rd_en at cycle T only becomes visible
|
|
to the SRAM's own always block at cycle T+1, which THEN schedules
|
|
the actual read for T+2. nms_memory_manager.v's own read-issue
|
|
logic used a SINGLE `read_issued` flag, capturing act_rd_data/
|
|
wgt_rd_data into input_data/weight_data exactly the cycle
|
|
`read_issued` first became visible (T+1) -- one full cycle too
|
|
early, grabbing the SRAM's stale (pre-read) output. Traced via
|
|
cycle-by-cycle tracing of act_rd_en/act_rd_data/wgt_rd_en/
|
|
wgt_rd_data/operand_valid/input_data/weight_data: at the exact
|
|
cycle op_valid first asserted, act_rd_data/wgt_rd_data already
|
|
held the CORRECT values (2 and 3) but input_data/weight_data
|
|
(captured the SAME cycle via non-blocking assignment) still read
|
|
as their PREVIOUS value (0), one cycle stale.
|
|
FIX: added a second flag (`read_ready`), turning the single-stage
|
|
read_issued -> capture sequence into a genuine 2-stage pipeline
|
|
(read_issued: SRAM now computing; read_ready: SRAM output now
|
|
valid, capture NOW) matching the SRAM's own real 2-cycle
|
|
rd_en-to-data latency.
|
|
2. STALE-VALUE RACE IN THE PRIVATE WEIGHT PREFETCH RESTART LOGIC:
|
|
SYMPTOM: not caught by any n_tiles=1 test (all of this file's first
|
|
3 test groups used n_tiles=1) -- would have silently corrupted
|
|
every n_tiles>1 job, duplicating/skipping alternating tiles.
|
|
ROOT CAUSE: `if (job_active_reg && !pf_busy && !pf_start &&
|
|
(wgt_fetched < n_tiles_reg[...])) pf_start<=1; pf_w_addr<=
|
|
w_base_reg + wgt_fetched*P_IN;` fires the SAME cycle pf_done
|
|
completes a fetch (prefetch_engine.v's own ST_DONE clears
|
|
fetch_busy the same cycle it pulses fetch_done), reading the OLD
|
|
(pre-increment) wgt_fetched -- re-issuing a fetch for the tile
|
|
that JUST completed instead of the next one; that duplicate
|
|
fetch's own later completion then writes into the NEXT tile's
|
|
SRAM slot using the WRONG (duplicated) source data. Same bug
|
|
CLASS as ERR-0006 (gating solely on a signal with its own
|
|
same-cycle side effect).
|
|
FIX: added `&& !pf_done` to the restart condition.
|
|
DIAGNOSIS METHOD: cycle-by-cycle $display tracing (act_rd_en/
|
|
act_rd_addr/act_rd_data/wgt_rd_en/wgt_rd_addr/wgt_rd_data/
|
|
operand_valid/input_data/weight_data, and separately mm_state/
|
|
np_state/result_valid/result_ready/result_data) added temporarily to
|
|
tb_nms_dataflow_core.v, gated by $time ranges around the specific
|
|
job windows of interest, removed once each bug was isolated and
|
|
fixed. Item 2 was only found AFTER item 1's fix, once a new
|
|
n_tiles=4 test (added specifically because every prior test used
|
|
n_tiles=1 and could not have exercised this path) was written and
|
|
initially failed.
|
|
VERIFICATION: after both fixes, tb_nms_dataflow_core.v passes 7/7
|
|
(same DAG-dependency shape as tb_dataflow_core.v's own M7 test;
|
|
shared-x_base broadcast-fill test across 2 concurrently-dispatched
|
|
slots; new n_tiles=4 multi-tile test) at N_SLOTS=2, bit-exact.
|
|
|
|
ERR-0014
|
|
timestamp: 2026-09-06T03:00:00Z
|
|
git_commit: 5c9ec61 (+ uncommitted NMS STEP9 work)
|
|
session: v2-NMS-STEP9-real-benchmark
|
|
module: hardware/v2/nms/rtl/nms_memory_manager.v,
|
|
hardware/v2/nms/rtl/nms_activation_fill_ctrl.v,
|
|
hardware/v2/nms/rtl/nms_dataflow_core.v
|
|
context: running a REAL D-Stress-scale benchmark (256 neurons, 16
|
|
tiles each -- exactly MAX_TILES) through the real V1 PSRAM chain for
|
|
STEP9. Every prior NMS test (tb_nms_dataflow_core.v, STEP8) used
|
|
n_tiles in {1,4}, never n_tiles==MAX_TILES exactly -- this one real,
|
|
necessary scale change (16-tile neurons, matching the project's own
|
|
realistic dense-layer workloads) exposed THREE compounding real bugs,
|
|
all instances of the SAME root cause, found via a real hang (0/8
|
|
neurons completed) traced cycle-by-cycle across three separate fix
|
|
attempts.
|
|
ROOT CAUSE (all three items): TIW=$clog2(MAX_TILES) sizes an SRAM
|
|
ADDRESS field (0..MAX_TILES-1), but several signals in this new
|
|
architecture are COUNTERS whose real value must reach MAX_TILES
|
|
itself (e.g. wgt_fetched/tile_idx/resident_count when a job's own
|
|
n_tiles equals MAX_TILES exactly) -- one bit short of what a counter
|
|
needs, though exactly right for an address.
|
|
1. nms_memory_manager.v: `n_tiles_reg[TIW-1:0]` truncated the 16-bit
|
|
n_tiles value (16) down to 4 bits, wrapping it to 0 -- every
|
|
"< n_tiles" check was permanently false, so wgt_fetched never even
|
|
STARTED advancing (stuck at 0 forever). Fixed: compare the full
|
|
16-bit n_tiles_reg directly (zero-extending the narrower counter),
|
|
never truncate n_tiles_reg itself.
|
|
2. Same modules: tile_idx/wgt_fetched/resident_count were themselves
|
|
declared TIW-bit (0..15 max), so even after fixing item 1's
|
|
comparison, incrementing wgt_fetched from 15 wrapped it back to 0
|
|
instead of reaching 16 -- an infinite re-fetch loop (real PSRAM
|
|
utilization visibly rose to 90%+ with zero forward progress).
|
|
Fixed: introduced a separate CNTW=$clog2(MAX_TILES+1) width for
|
|
these specific counters (one bit wider than TIW), keeping TIW only
|
|
for the SRAM address ports these counters occasionally drive
|
|
(explicit truncation at those specific assignment sites, always
|
|
safe since addressing only ever happens while the counter is
|
|
provably < MAX_TILES).
|
|
3. Introduced WHILE fixing item 1: `can_present`'s own second term was
|
|
mistakenly rewritten to compare wgt_fetched against n_tiles_reg
|
|
(checking "is the fetch done" instead of the actually-needed "is
|
|
THIS tile's weight available", i.e. tile_idx < wgt_fetched) -- once
|
|
wgt_fetched legitimately reached n_tiles (16), that comparison went
|
|
permanently false and deadlocked consumption of the LAST tile
|
|
forever even though every tile was genuinely ready (real activation
|
|
resident_count kept climbing normally the whole time, isolating
|
|
this as a logic error, not a data/timing problem).
|
|
4. Introduced WHILE fixing item 2: nms_dataflow_core.v's own top-level
|
|
wire connecting nms_activation_fill_ctrl.v's resident_count output
|
|
to nms_memory_manager.v's act_resident_count input was declared
|
|
`[TIW-1:0]` (4 bits), NOT widened to CNTW (5 bits) alongside the two
|
|
modules it connects -- a real value of 16 was silently truncated to
|
|
0 AT THE WIRE ITSELF, even though both endpoint modules' own
|
|
internal logic was by then already correct. Found by tracing
|
|
act_resident_count jumping from 15 directly to 0 (skipping 16)
|
|
exactly the cycle the real fetch completed pf_done=1.
|
|
DIAGNOSIS METHOD: cycle-by-cycle $display tracing added to a scaled-
|
|
down repro (8 neurons instead of 256, same real PSRAM model/n_tiles
|
|
=16) at each of 3 successive fix attempts, isolating each remaining
|
|
symptom (wgt_fetched stuck at 0 -> wgt_fetched wrapping 15->0
|
|
forever -> resident_count itself jumping 15->0 instead of reaching
|
|
16) before moving to the next. One self-inflicted tracing mistake
|
|
along the way (re-copying the clean testbench source over an
|
|
already-instrumented copy, silently producing zero debug output
|
|
until noticed and corrected) cost one extra iteration but found
|
|
nothing new.
|
|
VERIFICATION: after all four fixes, the scaled-down 8-neuron repro
|
|
passes bit-exact (6465 cycles), and the REAL, full-scale D-Stress
|
|
workload (256 neurons, 16 tiles each, through the real V1 PSRAM
|
|
chain) passes 256/256 bit-exact vs the golden model -- see EXP-0022.
|
|
|
|
ERR-0015
|
|
timestamp: 2026-09-05T23:41:08Z
|
|
module: hardware/v2/nms/rtl/weight_prefetch_engine.v
|
|
severity: CRITICAL (real, reproducible full deadlock -- 0/256 neurons
|
|
ever completed, watchdog timeout at 2,000,000 cycles, 0.0% PSRAM
|
|
utilization for the entire run)
|
|
found_during: STEP11's own real, full-scale N_SLOTS=1 D-Stress
|
|
PREFETCH_DISTANCE sweep (PFD in {1,2,4,8,16,32}) through
|
|
tb_nms_dstress_pf.v -- required by the STEP11 spec's own explicit
|
|
instruction not to assume PFD=32 is simply "unnecessary" but to
|
|
actually measure it ("larger [PFD], if necessary").
|
|
symptom: at PFD=32 (MAX_TILES=16, so CNTW=$clog2(MAX_TILES+1)=5 bits),
|
|
every one of the 256 D-Stress neurons hung forever -- the engine
|
|
never issued a single mem_req for the entire run.
|
|
root_cause: window_limit was computed as
|
|
`consumed_count + PREFETCH_DISTANCE[CNTW-1:0]` -- i.e. the
|
|
PREFETCH_DISTANCE PARAMETER ITSELF (not a tile counter, unlike every
|
|
prior instance of this bug class) was truncated down to CNTW bits
|
|
before the addition. For MAX_TILES=16, CNTW=5 bits (0..31), so
|
|
PFD=32 (6'b100000) truncates to 5'b00000 = 0. window_limit then
|
|
equals consumed_count exactly, forever, so `fetch_tile < window_limit`
|
|
is never true (fetch_tile >= consumed_count always holds by
|
|
construction) -- more_to_fetch is permanently false, and the engine
|
|
never fetches a single word.
|
|
Same ROOT CAUSE CLASS as ERR-0014 (a value that must be able to
|
|
reach/exceed a bound gets silently wrapped by a too-narrow field),
|
|
but this time the truncated value is a user-supplied, unbounded
|
|
build-time PARAMETER (PREFETCH_DISTANCE) rather than an internal
|
|
tile counter -- a genuinely new instance, not a recurrence of the
|
|
same fixed bug, since PREFETCH_DISTANCE did not exist before this
|
|
STEP.
|
|
fix: widened window_limit (and the two `fetch_tile [+1] < window_limit`
|
|
comparisons that use it) to a fixed 32-bit width, computed as
|
|
`{{(32-CNTW){1'b0}}, consumed_count} + PREFETCH_DISTANCE` (the
|
|
parameter itself, un-truncated) compared against a zero-extended
|
|
fetch_tile -- see weight_prefetch_engine.v's own updated comment at
|
|
the window_limit declaration.
|
|
verification: added a dedicated ERR-0015 regression case to
|
|
tb_weight_prefetch.v (`err0015_large_pfd_test`, gated `if (PFD>=32)`
|
|
so the file stays valid for every PFD it's compiled with) running a
|
|
full MAX_TILES=16 job at PFD=32 -- PASSES bit-exact post-fix
|
|
(confirmed: "PASS n_tiles=16 PFD=32 EXTRA_WAIT=0: ready_count=16,
|
|
all tiles bit-exact"). Re-ran the full tb_weight_prefetch.v suite at
|
|
PFD in {1,2,4,8,32}: 10/10 (9/9 at PFD=32, since the PFD/PFD+1 job-
|
|
size edge case is n/a once PFD>=MAX_TILES) tests PASS, 0 errors, at
|
|
every PFD tested.
|
|
secondary testbench-only issue found and fixed alongside (NOT an RTL
|
|
bug): tb_weight_prefetch.v's own `run_job(base, PFD, ...)` /
|
|
`run_job(base, PFD+1, ...)` edge-case calls and its windowing-cap
|
|
test's expected value both silently assumed PFD < MAX_TILES (asking
|
|
for a PFD+1-tile job when PFD+1 > MAX_TILES is asking the weight
|
|
SRAM to hold more tiles than it has, and the windowing cap can never
|
|
reach PFD once n_tiles/MAX_TILES itself is the tighter bound) --
|
|
fixed by gating those two run_job calls on `PFD < MAX_TILES` and
|
|
changing the windowing-cap expectation to `min(PFD, MAX_TILES)`.
|
|
impact_on_step11_conclusions: none -- PFD=32 was never the STEP11
|
|
benchmark's real focus (the D-Stress MAX_TILES=16 workload makes any
|
|
PFD>=16 equivalent to "unbounded"); this bug would only have
|
|
surfaced if a user tried MAX_TILES>32 with a correspondingly large
|
|
PFD. Fixed regardless, since a build-time parameter silently
|
|
deadlocking the whole design outside its narrow tested range is a
|
|
real correctness defect, not a documentation footnote.
|
|
|
|
ERR-0016
|
|
timestamp: 2026-09-06T06:08:18Z
|
|
module: hardware/v2/nms/rtl/sdram_controller.v
|
|
severity: CRITICAL (real, reproducible bank-open-forever condition;
|
|
every READ/WRITE after the first corrupted the next transaction)
|
|
found_during: STEP16 Phase 3 isolated regression (tb_sdram_controller.v,
|
|
BURST_LEN=4, CLK_FREQ_MHZ=166) -- first full run produced 0/460 tests
|
|
passed, 460 errors, with the real timing-checked sdram_model.v itself
|
|
flagging "ACTIVATE to bank 0 while already active" on nearly every
|
|
transaction.
|
|
symptom: every ACTIVATE after the very first one hit a real protocol
|
|
violation in the behavioral model (bank never precharged/closed by
|
|
the previous transaction), and every read returned corrupted/shifted
|
|
data as a side effect of the row/bank state being wrong.
|
|
root_cause: the READ/WRITE command's own address word was built as
|
|
`sdram_a <= {3'b000, 1'b1, req_col_reg}` with req_col_reg 8 bits wide
|
|
-- 3+1+8=12 bits total (correct width), but the concatenation placed
|
|
the auto-precharge '1' bit at position a[8], not a[10] where the
|
|
device's mode register / command decode actually expects it. A10 was
|
|
therefore always 0 on every real READ/WRITE -- auto-precharge was
|
|
silently never requested, so every opened row/bank stayed active
|
|
forever, and the next ACTIVATE (to a different row) hit a real
|
|
already-active bank.
|
|
found_by: static/hand trace of the FSM against the model's own
|
|
violation messages -- not assumed, root-caused from the actual
|
|
simulation log.
|
|
fix: `sdram_a <= {4'b0100, req_col_reg};` -- places the '1' at bit 10
|
|
exactly, with a[11]/a[9]/a[8] correctly tied 0.
|
|
verification: re-ran tb_sdram_controller.v (BURST_LEN=4, 166MHz) --
|
|
the "ACTIVATE while already active" violation disappeared completely;
|
|
see ERR-0017/ERR-0018 for the remaining data-corruption bugs found
|
|
after this fix, and EXP-0040 for the final clean 460/460 PASS run.
|
|
|
|
ERR-0017
|
|
timestamp: 2026-09-06T06:08:18Z
|
|
module: hardware/v2/nms/sim/sdram_model.v
|
|
severity: HIGH (every WRITE burst's first word was silently dropped)
|
|
found_during: STEP16 Phase 3 regression, same debugging session as
|
|
ERR-0016 (found immediately after fixing the A10 bug, since data
|
|
corruption persisted with the ACTIVATE violations gone).
|
|
symptom: after ERR-0016's fix, reads still came back shifted -- the
|
|
first data word of every burst was missing/zero, later words shifted
|
|
down by one burst position.
|
|
root_cause: the model captured write data into `mem[]` only from the
|
|
registered `wr_burst_active`-gated block, which does not become true
|
|
until the cycle AFTER the WRITE command is decoded. But real SDR
|
|
SDRAM presents the FIRST write word on DQ CONCURRENTLY with the
|
|
WRITE command itself, not one cycle later -- so the model's own
|
|
capture logic was structurally one cycle too late for word 0, and
|
|
silently never wrote it anywhere.
|
|
fix: in the `cmd_write` branch (command-decode cycle itself), capture
|
|
word 0 directly into `mem[ba,bank_row[ba],a[7:0]]` right there, and
|
|
initialize `wr_col`/`wr_remaining` to already point at the SECOND
|
|
word/column for the (unchanged) registered continuation logic to
|
|
handle words 1..N-1. Special-cased burst_len==1 (single-word write,
|
|
no continuation needed).
|
|
verification: re-ran the regression -- write side now matches the
|
|
controller's own per-cycle DQ timing exactly (confirmed by hand
|
|
cycle-trace before re-running); see ERR-0018 for the matching
|
|
read-side bug found next, and EXP-0040 for the final clean run.
|
|
|
|
ERR-0018
|
|
timestamp: 2026-09-06T06:08:18Z
|
|
module: hardware/v2/nms/sim/sdram_model.v
|
|
severity: HIGH (every READ burst arrived corrupted/shifted by two
|
|
cycles' worth of latency, in two separate compounding sub-bugs)
|
|
found_during: STEP16 Phase 3 regression, immediately after ERR-0017's
|
|
fix (write side alone did not clear the corruption -- read side had
|
|
its own, structurally identical class of bug, discovered by the same
|
|
cycle-exact hand trace method used for ERR-0016/ERR-0017).
|
|
symptom: reads still returned data shifted by 1-2 burst positions with
|
|
leading zeros, even though the underlying stored memory content was
|
|
now correct (per ERR-0017's fix).
|
|
root_cause (two distinct, compounding one-cycle-latency bugs):
|
|
(a) the model's read-side pipe insertion was gated by the registered
|
|
`rd_burst_active`, which -- exactly like ERR-0017's write-side bug --
|
|
becomes true only the cycle AFTER the READ command is decoded, adding
|
|
a spurious extra pipeline cycle before the first word ever entered
|
|
the CAS-latency shift register.
|
|
(b) independently, the model's DQ output was driven through its own
|
|
registered NBA stage (`dq_out <= rd_pipe[0]` inside the same
|
|
always block), adding ANOTHER cycle of latency beyond the shift
|
|
register's own CAS_LATENCY depth -- real SDR SDRAM's output register
|
|
is already accounted for inside the published CAS_LATENCY spec value,
|
|
so this was a redundant, uncounted-for extra stage unique to how the
|
|
model was built (a 16-entry shift array PLUS a separate registered
|
|
output buffer), not something the controller's own CAS_LATENCY-cycle
|
|
wait_cnt derivation could have anticipated.
|
|
fix: (a) mirrored the ERR-0017 write-side fix -- in the `cmd_read`
|
|
branch (command-decode cycle itself), insert word 0 directly into
|
|
`rd_pipe[cas_latency-1]` and initialize `rd_col`/`rd_remaining` to
|
|
already point at the second word for the registered continuation
|
|
logic (again special-cased burst_len==1). (b) converted `dq_out`/
|
|
`dq_out_en` from `reg` (NBA-driven) to `wire` (combinational,
|
|
`= rd_pipe[0]` / `= rd_valid_pipe[0]`), removing the extra output
|
|
register stage entirely.
|
|
verification: re-ran tb_sdram_controller.v (BURST_LEN=4, 166MHz) after
|
|
both fixes -- 460/460 tests PASS, 0 errors, bit-exact across every
|
|
Phase 3 scenario (A/B/E/F/H/I/G). See EXP-0040 for the full result.
|
|
|
|
ERR-0019
|
|
timestamp: 2026-09-06T06:08:18Z
|
|
module: hardware/v2/nms/rtl/sdram_controller.v
|
|
severity: CRITICAL (real, reproducible full deadlock -- caller waits
|
|
forever for `ready` that never comes)
|
|
found_during: STEP16 Phase 4 real 100/133/166MHz x BURST_LEN{1,4,8}
|
|
measurement sweep -- 8/9 configurations passed cleanly, but
|
|
BURST_LEN=1 @ CLK_FREQ_MHZ=133 hung indefinitely (confirmed via a
|
|
backgrounded run: >50s of continuous 100% CPU with zero output,
|
|
killed and root-caused rather than assumed to be a slow-but-finite
|
|
run).
|
|
symptom: tb_sdram_controller.v's do_transaction task deadlocks inside
|
|
`while (!ready) @(posedge clk);` for one specific transaction in the
|
|
400-iteration Test G sequence, at this one specific frequency/burst
|
|
combination (BURST_LEN=4 and BURST_LEN=8 at the same 133MHz did NOT
|
|
hang, confirmed by isolated re-run).
|
|
root_cause: this project's own established mem_req convention (per
|
|
weight_prefetch_engine.v's header comment) is a SINGLE-CYCLE req
|
|
pulse, not a held/latched signal. sdram_controller.v's S_IDLE state
|
|
gave periodic AUTO REFRESH unconditional priority over a same-cycle
|
|
`req` ("if (refresh_timer==0) ... else if (req) ..."), and did
|
|
nothing to capture the request's fields when refresh won -- so if a
|
|
test vector's req pulse happened to land on the exact cycle
|
|
refresh_timer reached 0, the request was silently discarded: the
|
|
caller's req had already gone low again on the next cycle (per the
|
|
single-pulse convention), the controller served the refresh with no
|
|
memory of the missed request, and no `ready` ever followed. This is
|
|
a general protocol race, not something inherent to 133MHz or
|
|
BURST_LEN=1 specifically -- it only reproduced there because that is
|
|
where this test sequence's absolute cycle counts happened to align
|
|
the two events; the SAME race is latent at any frequency/burst
|
|
length whenever a real caller's timing lines up unluckily.
|
|
fix: added a `req_pending` register that latches a fresh req's
|
|
wr/bank/row/col/wdata fields in S_IDLE the instant req is seen,
|
|
regardless of whether refresh wins arbitration that same cycle.
|
|
Added `eff_wr/eff_bank/eff_row/eff_col/eff_wdata` combinational
|
|
wires (`req ? <live input> : <latched req_*_reg>`) so the one real
|
|
S_IDLE branch that starts a transaction handles both "req arriving
|
|
fresh this cycle" and "req deferred behind a just-finished refresh"
|
|
uniformly, without duplicating the ACTIVATE-issuing code path.
|
|
`req_pending` resets to 0 under `rst`.
|
|
verification: re-ran the full 3x3 (100/133/166MHz x BURST_LEN 1/4/8)
|
|
sweep after the fix -- all 9 configurations now PASS 460/460, 0
|
|
errors, including the previously-hanging BURST_LEN=1/133MHz case
|
|
(re-confirmed finishing in <1s wall time, not merely "eventually").
|
|
See EXP-0041 for the full sweep result and the real measured
|
|
cycles/transaction used for Phase 4's throughput calculations.
|
|
|
|
ERR-0020
|
|
timestamp: 2026-09-06T06:08:18Z
|
|
module: hardware/v2/nms/rtl/sdram_controller.v
|
|
severity: CRITICAL (real, reproducible full-system deadlock -- 28/256
|
|
neurons never completed, both slots of an N=2 run permanently
|
|
starved; the isolated single-requester regression NEVER exercised
|
|
this because it always issues req from S_IDLE, never overlapping a
|
|
real arbiter's own outstanding-transaction bookkeeping)
|
|
found_during: STEP16 Phase 5 real FPGA-Neural integration (tb_nms_
|
|
dstress_sdram.v, N_SLOTS_CFG=2, D-Stress 256-neuron workload) -- the
|
|
SAME workload at N_SLOTS_CFG=4 passed cleanly (49430 cycles, 256/256
|
|
bit-exact), but N_SLOTS_CFG=2 hung at 228/256 neurons after 2,000,000
|
|
watchdog cycles (40x more cycles than N=4's entire run took to
|
|
finish all 256).
|
|
symptom: added temporary hierarchical debug tracing (both slots' own
|
|
nms_memory_manager_stream_wide state, plus the SDRAM controller's and
|
|
slot_mem_arbiter_wide's own internal state) to the watchdog's
|
|
periodic $display. Trace showed: both slots permanently stuck at
|
|
tile_idx=0/wgt_ready_count=0 (ST_RUN, waiting forever for the first
|
|
weight tile), while simultaneously sdram_controller.v itself sat
|
|
completely idle (req=0, busy=0, state=S_IDLE, req_pending=0) AND
|
|
slot_mem_arbiter_wide.v's own `owner` register was permanently
|
|
locked non-zero (a grant it believed was still in flight) with a
|
|
SECOND port's request stuck in `pending`, unable to ever be granted
|
|
because the arbiter was waiting for an `m_ready` that would never
|
|
come from a controller that had already forgotten the request.
|
|
root_cause: ERR-0019's fix only latched a fresh `req` into
|
|
`req_pending` from WITHIN the S_IDLE case-statement branch. The real
|
|
arbiter (slot_mem_arbiter_wide.v) pulses its own `m_req` for exactly
|
|
one cycle the instant it grants a new owner -- which can be ANY
|
|
cycle, including one where sdram_controller.v happens to be mid-
|
|
refresh (S_REFRESH_WAIT) or finishing a PREVIOUS transaction's own
|
|
PRECHARGE_WAIT tail, i.e. NOT executing the S_IDLE branch that
|
|
cycle. In that case the old fix's own req-latch code never even ran,
|
|
so the request was silently dropped exactly as it was before
|
|
ERR-0019's fix -- just for a DIFFERENT interfering state (a real
|
|
case the isolated regression's own single-requester testbench could
|
|
never reach, since it always waits for `busy==0` -- itself only
|
|
cleared in S_IDLE -- before ever asserting req, so its req pulses
|
|
are always S_IDLE-synchronous by construction).
|
|
fix: moved the req-latching code out of the S_IDLE branch entirely and
|
|
into the always-executed section that runs every cycle regardless of
|
|
`state` (alongside the existing unconditional refresh_timer
|
|
decrement) -- so a req pulse arriving during ANY state is captured
|
|
into req_pending/req_*_reg, not just one arriving while already in
|
|
S_IDLE. S_IDLE's own servicing logic (`req || req_pending`, `eff_*`
|
|
wires) is unchanged.
|
|
verification: re-ran the full Phase 3 isolated regression (9/9
|
|
frequency/burst-length configurations, 460/460 tests each) --
|
|
unaffected, still 100% PASS, confirming the fix is a pure superset
|
|
(catches a strictly larger set of req-arrival cycles, changes no
|
|
existing passing behavior). Re-ran the STEP16 Phase 5 N=2 D-Stress
|
|
benchmark -- now PASSES cleanly, 256/256 neurons bit-exact, 52161
|
|
total cycles (previously hung indefinitely). Re-ran N=4 -- unchanged,
|
|
49430 cycles (confirms the fix does not alter already-correct
|
|
behavior, only recovers previously-lost requests). See EXP-0042 for
|
|
the full Phase 5 N=2/N=4 integration results.
|
|
|
|
ERR-0021
|
|
timestamp: 2026-09-06T10:22:22Z
|
|
module: hardware/v2/rtl/dependency_manager.v
|
|
severity: HIGH (real, reproducible bit-exact regression -- a node
|
|
dispatched twice; caught before being kept, via a standalone
|
|
regression, NOT shipped)
|
|
found_during: STEP17 Part A's own "minimum fix experiment" -- an
|
|
attempt to pipeline the first_ready_idx/any_ready priority-encoder
|
|
scan (EXP-0044's own identified critical path) by one register
|
|
stage, to shorten the scan-to-array-read combinational chain that
|
|
limits N=4 Fmax to 81.55MHz.
|
|
symptom: after the fix, the full N=2/N=4 D-Stress regression showed
|
|
jobs_allocated=266 (N=2) / 268 (N=4), both > the expected 256 --
|
|
and roughly half the 256 neurons' results read back as 0 instead of
|
|
their real golden value. A minimal standalone unit test (16
|
|
independent always-ready nodes, `ready_ready` held high, isolated
|
|
from the rest of the system) reproduced the exact mechanism: node 0
|
|
is DISPATCHED TWICE, at cycles 20ns apart (exactly one extra pipeline
|
|
cycle).
|
|
root_cause: the scan (`first_ready_idx`/`any_ready`) is combinational
|
|
over node_state's PRE-edge value. On the exact cycle a dispatch
|
|
COMMITS (`ready_valid && ready_ready`, which schedules node_state
|
|
[ready_node_id] <= ST_DISPATCHED for the following cycle), the scan
|
|
THAT SAME cycle still sees the dispatching node as READY (its state
|
|
has not committed yet) -- an unconditional one-cycle register of the
|
|
scan's output would capture this still-READY snapshot, and one cycle
|
|
later (once node_state has actually committed and ready_valid has
|
|
dropped), the dispatch branch would fire again using the now-stale
|
|
snapshot, re-dispatching the just-dispatched node. A first attempted
|
|
repair -- gating the register update by `!ready_valid` -- did NOT fix
|
|
it: traced further (via the same standalone unit test) and found the
|
|
gate blocks the register from updating during the COMMIT cycle
|
|
itself (correct), but the register had ALREADY captured the stale
|
|
"node is READY" snapshot ONE CYCLE EARLIER (at the cycle the dispatch
|
|
was merely INITIATED, before ready_valid even reads 1), and that
|
|
stale value survives frozen through the commit cycle (blocked by the
|
|
gate) and is consumed immediately after, still stale by exactly one
|
|
cycle relative to the real commit.
|
|
fix: NONE SHIPPED. The change was reverted in full (`git checkout --
|
|
hardware/v2/rtl/dependency_manager.v`), restoring the exact STEP16-
|
|
validated RTL. Re-confirmed via the full N=2/N=4 D-Stress regression
|
|
that reverting restores the original correct behavior (256/256
|
|
bit-exact, 49430/52161 cycles, matching STEP16 exactly).
|
|
impact: none on the shipped design -- caught entirely within STEP17's
|
|
own investigation, before any P&R or benchmark result relying on the
|
|
broken version was reported as final. Documented here so a future
|
|
attempt at pipelining this scan does not repeat the same two failed
|
|
approaches; a fully correct version (if attempted again) will need
|
|
to reconcile the snapshot-capture cycle with the commit cycle more
|
|
carefully than a simple `!ready_valid` gate -- e.g. by explicitly
|
|
invalidating the registered snapshot when its own named node is the
|
|
one just committed, not merely suppressing updates during the commit
|
|
cycle.
|
|
|
|
ERR-0022
|
|
timestamp: 2026-09-06T12:30:00Z
|
|
module: hardware/v2/nms/rtl/sdram_weight_backend_pack128.v (STEP18,
|
|
first draft)
|
|
severity: HIGH (real, measured throughput REGRESSION -- not a crash or
|
|
data-corruption bug, but a real performance defect that would have
|
|
shipped as a "fix" if not benchmarked end-to-end before accepting it)
|
|
found_during: STEP18 Part C/D "weight packing" experiment -- a new
|
|
BURST_LEN=8 (128-bit/16-byte, 2-tile) SDRAM weight-fetch wrapper,
|
|
designed to halve the number of real SDRAM transactions needed per
|
|
P8 weight tile by caching the "other half" of each 128-bit fetch for
|
|
the next sequential request.
|
|
symptom: isolated unit-test regression (tb_sdram_weight_backend_pack128
|
|
.v, single requester, sequential access) passed 20/20, showing the
|
|
intended ~50% cache-hit rate (10 real fetches @16 cycles + 10 cache
|
|
hits @1 cycle for 20 requests). But the FULL N=4 D-Stress integration
|
|
benchmark got WORSE, not better: 74,004 cycles vs the STEP16 baseline
|
|
's 49,430 (+49.7%), despite remaining bit-exact correct.
|
|
root_cause: the first draft used a SINGLE cache entry. In the real
|
|
system, N_SLOTS=4 independent memory managers share this ONE
|
|
physical backend through slot_mem_arbiter_wide.v, which interleaves
|
|
their requests round-robin. A single cache entry holding slot A's
|
|
own "other half" gets overwritten by slot B's own "other half" (a
|
|
DIFFERENT address) before slot A's memory manager ever comes back
|
|
around to request its own paired tile -- so nearly EVERY access
|
|
became a real 16-cycle miss instead of the intended ~50% instant
|
|
hit, and 16 cycles/miss is WORSE than the STEP16 baseline's 10-cycle
|
|
BURST_LEN=4 transactions. This was found by actually running the
|
|
full integrated benchmark, not by trusting the isolated single-
|
|
requester unit test's own PASS result -- exactly the class of gap
|
|
the governing spec's own "this hypothesis MUST be tested, not
|
|
assumed" instruction anticipates.
|
|
fix: widened the cache from 1 entry to N_ENTRIES (parameter, sized to
|
|
the real N_SLOTS in the system, default 4) with fully-associative
|
|
address-tag lookup and simple round-robin eviction. This is SAFE
|
|
regardless of sizing accuracy (an evicted-too-early entry only costs
|
|
an extra real fetch -- a performance effect, never incorrect data,
|
|
since a cache MISS always falls back to a real, address-exact
|
|
fetch) and bounded correctly by construction: each slot has at most
|
|
one outstanding request at a time (the existing memory manager's own
|
|
design), so at most N_SLOTS "pending other halves" can exist
|
|
simultaneously in real traffic -- never more than N_ENTRIES=N_SLOTS.
|
|
verification: re-ran the full N=2/N=4 D-Stress regression -- now a
|
|
REAL improvement: N=4 44,935 cycles (-9.1% vs the 49,430 baseline),
|
|
N=2 47,399 cycles (-9.1% vs the 52,161 baseline), both 256/256
|
|
bit-exact. See EXP-0046 for the full before/after comparison.
|
|
|
|
ERR-0023
|
|
timestamp: 2026-09-06T11:26:46Z
|
|
module: hardware/v2/nms/rtl/sdram_unified_backend.v (STEP19, new module)
|
|
severity: CRITICAL (real, reproducible full-system deadlock -- 0/256
|
|
neurons completed; found immediately on the first real N=4 D-Stress
|
|
integration run, before being shipped)
|
|
found_during: STEP19 -- consolidating the two previously-separate
|
|
physical memory paths (weight-fetch SDRAM + activation/result PSRAM)
|
|
into ONE physical SDRAM chip serving all three traffic classes
|
|
through a single sdram_controller.v instance, per this step's own
|
|
explicit "single external SDRAM only" mandate.
|
|
symptom: the isolated unit regression (tb_sdram_unified_backend.v,
|
|
single requester on each of the W and AR ports, no contention)
|
|
passed 40/40. The first full N=4 D-Stress integration run deadlocked
|
|
completely: jobs_allocated stuck at 12, tiles_delivered=0,
|
|
neurons_completed=0/256, watchdog timeout at 2,000,000 cycles, AR
|
|
arbiter-side utilization pegged at 100% (permanently stuck).
|
|
root_cause: this is the SAME bug class as ERR-0019/ERR-0020 in sdram_
|
|
controller.v itself, recurring one layer up. sdram_unified_backend.v
|
|
's own top-level S_IDLE arbitration between its W (weight) and AR
|
|
(activation/result) ports only checked the LIVE w_req/ar_req signals
|
|
-- a single-cycle req pulse (this project's own established
|
|
convention) arriving on a cycle this backend happened to be busy
|
|
servicing the OTHER port was silently dropped, and the caller
|
|
(slot_mem_arbiter.v or slot_mem_arbiter_wide.v) waited forever for a
|
|
`ready` that would never come. Under real N=4 contention (weight
|
|
prefetch traffic and the very first activation-fill read racing for
|
|
the same physical port from the start), this raced and deadlocked
|
|
immediately -- the isolated single-requester test structurally could
|
|
not reach this case (no contention exists there).
|
|
fix (round 1): added req_pending latches for both W and AR ports,
|
|
mirroring sdram_controller.v's own corrected ERR-0020 fix (latch
|
|
unconditionally, every cycle, regardless of current state). This
|
|
introduced a SECOND, different real bug (see below) before the full
|
|
fix was correct.
|
|
fix (round 2, the actual bug this ID documents): the unconditional
|
|
latch fires whenever w_req is seen, INCLUDING the cycle a request is
|
|
ALSO being fully serviced via the W-port's own single-cycle cache-hit
|
|
fast path (w_cache_hit, reused from sdram_weight_backend_pack128.v's
|
|
own STEP18 caching logic) -- that fast path does not go through the
|
|
same-cycle "new transaction" branch and therefore never clears the
|
|
just-set w_req_pending latch. The result: after every cache-hit
|
|
service, w_req_pending is left spuriously set, and on the FOLLOWING
|
|
idle cycle the backend issues a bogus extra real SDRAM fetch using
|
|
the STALE latched address from the already-completed request --
|
|
which then delivers ITS OWN (stale, wrong) data as if it were the
|
|
response to the NEXT genuinely new request, shifting every
|
|
subsequent W-port response by one position. Found via the isolated
|
|
regression itself regressing from 40/40 to 26/40 after round-1's own
|
|
fix, with a clear "off by one iteration" failure signature (got ==
|
|
previous iteration's own expected value) -- traced by hand-cycle
|
|
tracing the interaction between the unconditional latch and the
|
|
case-statement's own NBA ordering (last assignment in the same
|
|
always block, same time step, wins).
|
|
fix (final): the w_cache_hit branch now explicitly clears
|
|
w_req_pending itself (`w_req_pending <= 1'b0`), overriding the
|
|
spurious set from the SAME cycle's unconditional latch (textually
|
|
later in the same always block, so it wins per standard Verilog NBA
|
|
ordering) -- cancelling the latch precisely when (and only when) the
|
|
SAME cycle's request was ALREADY fully handled via the fast path.
|
|
verification: re-ran the isolated unit regression (40/40 PASS,
|
|
restored) and the full N=4 AND N=2 D-Stress integration benchmark --
|
|
both now PASS bit-exact (256/256 neurons), with real, sustained
|
|
multi-hundred-refresh-interval operation and zero deadlock/timeout/
|
|
dropped/duplicated results. See EXP-0048 for the full integration
|
|
results.
|
|
|
|
ERR-0024 (Icarus Verilog v13.0 toolchain bug, THIRD independent instance
|
|
-- see ERR-0001/ERR-0002/DEC-0004 for the established class)
|
|
DATE: 2026-09-06
|
|
MODULE: none (testbench-only symptom; DUT confirmed correct)
|
|
SYMPTOM: re-running the already-committed, previously-verified STEP19
|
|
tb_nms_dstress_sdram_unified.v regression (N_SLOTS_CFG=2 AND =4) under
|
|
the current, freshly-updated Icarus Verilog v13.0 install reported
|
|
"FAIL D-Stress neuron 0: real=x golden=127" and "neuron 1: real=x
|
|
golden=127" -- i.e. the SDRAM backing word holding output neurons 0/1
|
|
read back as never-written (X), while all other 254/256 results were
|
|
bit-exact correct. Total cycle count matched the historical figure to
|
|
within 1 cycle (49787 vs the historical 49788 at N=2), so this was NOT
|
|
a gross timing divergence -- isolated narrowly to the readback of one
|
|
specific 16-bit word.
|
|
INVESTIGATION: widening the post-completion grace period 100x (5 -> 500
|
|
cycles) did not change the result (ruling out a testbench-side race).
|
|
Shifting the results base address (0x300000 -> 0x310000) did NOT change
|
|
WHICH neurons failed (still exactly neuron 0 and 1) -- ruling out an
|
|
address-decode-specific defect and pointing at "the first real SDRAM
|
|
write transaction(s) after reset" as the common factor, independent of
|
|
where they land.
|
|
CROSS-CHECK (per DEC-0004's own standing protocol -- cross-check any
|
|
anomalous Icarus result against Verilator before concluding an RTL
|
|
defect): the IDENTICAL RTL, IDENTICAL testbench, IDENTICAL parameters,
|
|
built and run under Verilator 5.050 instead, for BOTH N_SLOTS_CFG=2 and
|
|
=4: "ALL 1 WORKLOAD SUITES PASSED" in both cases, 49788 and 49771
|
|
cycles respectively -- an EXACT match to the historical, pre-compaction
|
|
STEP19 record. CONFIRMS this is a fourth-ish real, reproducible Icarus
|
|
Verilog v13.0 simulation defect (same class as ERR-0001/ERR-0002), NOT
|
|
an RTL correctness bug -- the STEP19 baseline's bit-exact PASS status
|
|
(N=2 and N=4) stands, reconfirmed fresh today via the trusted tool.
|
|
STATUS: NOT reported upstream (out of scope). Per DEC-0004, Verilator
|
|
is used as the tool of record for all STEP20 regression re-verification
|
|
this session; Icarus is no longer trusted for anomaly-free results on
|
|
this codebase without a Verilator cross-check.
|
|
|
|
ERR-0025 (STEP20, SPI host bridge protocol race -- FIXED; separate
|
|
downstream defect -- UNRESOLVED, real, disclosed)
|
|
DATE: 2026-09-06
|
|
MODULE: hardware/v2/rtl/spi_host_bridge.v (fixed); downstream location
|
|
NOT YET root-caused (see below).
|
|
PART A -- FIXED: spi_host_bridge.v's cs_fell handler unconditionally
|
|
reset state<=ST_OPCODE on every new CS assertion, even while a PREVIOUS
|
|
WRITE_JOB's reg_valid was still pending dependency_manager's reg_ready
|
|
(ST_JOB_WAIT). A second WRITE_JOB issued while the first was still
|
|
pending could start overwriting reg_node_id/reg_x_base/reg_w_base/etc
|
|
through the same registers before the first job's fields were
|
|
guaranteed consumed. Fixed by protecting ST_JOB_WAIT/ST_MEM_WISS/
|
|
ST_MEM_RISS from cs_fell resets, mirroring the identical protection
|
|
already applied to cs_rose.
|
|
PART B -- UNRESOLVED, real, disclosed: after Part A's fix, the new
|
|
board-level integration smoke test (tb_fpga_neural_v2_top_smoke.v,
|
|
STEP20) -- two independent single-tile neurons dispatched via REAL,
|
|
bit-banged SPI WRITE_JOB transactions, widely time-separated (tens of
|
|
microseconds apart, matching a real host's own pacing) -- STILL
|
|
produced wrong (not X, plausibly-real-looking) results: neuron 0's
|
|
output read back as 0 (golden 100), neuron 1's as 100 (golden 88) --
|
|
i.e. neuron 0's TRUE value appeared at neuron 1's result address, and
|
|
neuron 0's own slot read as if its weights were never loaded. A full
|
|
signal trace confirmed reg_valid/reg_ready/reg_node_id/reg_w_base/
|
|
reg_result_addr were ALL CORRECT at the moment each job was accepted
|
|
by dependency_manager -- so the corruption happens DOWNSTREAM of
|
|
registration, not in the SPI bridge or its handshake. The already-
|
|
verified STEP19 tb_nms_dstress_sdram_unified.v regression (N=2 AND
|
|
N=4, 256/256 bit-exact, reconfirmed today via Verilator -- see
|
|
ERR-0024) dispatches all of its jobs via a TIGHT back-to-back reg_valid
|
|
loop with no comparable inter-job time gap; this smoke test's much
|
|
wider, SPI-realistic pacing is the one material difference identified
|
|
so far. Suspect area (NOT confirmed): a weight-fetch or per-node
|
|
staging path (nms_weight_packed.v / weight_prefetch_engine_wide.v /
|
|
nms_activation_fill_ctrl_v3.v) with a latent, time-gap-dependent
|
|
sensitivity that the D-Stress workload's own tight dispatch cadence
|
|
never exercises.
|
|
STATUS: UNRESOLVED. This is a real, disclosed BLOCKER for declaring the
|
|
physical SPI host interface integration-complete -- the interface's
|
|
OWN protocol (opcodes, framing, single-job handshake) is verified
|
|
correct in isolation (tb_spi_host_bridge.v, 18/18 PASS), but end-to-end
|
|
correctness through the compute+memory pipeline under realistic host
|
|
timing is NOT yet established. Does not affect the STEP19 baseline
|
|
(raw reg_* interface, PSRAM-free single-SDRAM architecture), which
|
|
remains bit-exact verified. Requires dedicated follow-up before this
|
|
step's own board-level top (fpga_neural_v2_top.v) can be considered
|
|
hardware-release-ready.
|
|
|
|
ERR-0025 Part B -- RESOLUTION (STEP20, ERR-0025 Part B closed)
|
|
DATE: 2026-09-06
|
|
ROOT CAUSE: nms_weight_packed.v and nms_activation_replicated.v both
|
|
used a REGISTERED read port (`rd_data_reg <= mem[addr]`, gated by
|
|
rd_en -- a full clock cycle of latency from address/enable to valid
|
|
output), but nms_memory_manager_stream_wide.v's own pipelined read-
|
|
ahead consumer (`rd_pending`) is designed around a COMBINATIONAL read
|
|
(issue this cycle -> already-valid data captured next cycle). Traced
|
|
via a full internal signal walk (dependency_manager -> neural_director
|
|
-> per-slot memory_manager -> weight_prefetch_engine_wide/nms_weight_
|
|
packed -> nms_activation_fill_ctrl_v3/nms_activation_replicated):
|
|
job registration, slot dispatch, and per-job context capture (w_base_
|
|
reg/result_addr_reg) were ALL confirmed correct at every stage: the
|
|
corruption traced all the way down to the exact cycle where `buf_
|
|
weight`/`buf_input` are captured, which used the PRE-edge (stale)
|
|
value of the SRAM's own registered output -- one real cycle before
|
|
that register's own update (from the read issued the SAME cycle)
|
|
actually committed. A busy, multi-tile job (e.g. STEP19's D-Stress,
|
|
128 inputs/16 tiles per neuron) never exposes this: its own weight/
|
|
activation prefetch runs far enough ahead (PREFETCH_DISTANCE=8) that
|
|
by the time any given tile is actually consumed, that tile's data has
|
|
already been sitting stable in the SRAM for many cycles, masking the
|
|
extra latency completely. An uncontested single-tile job (STEP20's own
|
|
board-level SPI smoke test, tb_fpga_neural_v2_top_smoke.v) has ZERO
|
|
such margin: its one-and-only tile's read fires on the exact edge the
|
|
data nominally becomes "ready", landing squarely on the missing cycle
|
|
-- permanently latching stale/zero weight and activation data. This
|
|
explains BOTH observed symptoms exactly: neuron 0 (first job, first
|
|
real weight fetch of the whole session) read back 0 (all-zero SRAM
|
|
reset content); neuron 1 (second job, reusing the same physical slot)
|
|
read back 100 -- neuron 0's own TRUE value, one full cycle "behind"
|
|
where it should have been.
|
|
FIX: made both SRAMs' read ports combinational (`assign rd_data = mem
|
|
[addr]`, replacing the registered `always @(posedge clk) rd_data_reg
|
|
<= mem[addr]`), with an explicit same-cycle fill/read address-match
|
|
bypass (forwarding fill_data directly) for the one hazard a plain
|
|
combinational read alone would still miss -- a fill and a read to the
|
|
IDENTICAL address landing on the IDENTICAL edge, where mem[] itself
|
|
would not yet reflect that same-edge write. Files changed: hardware/
|
|
v2/nms/rtl/nms_weight_packed.v, hardware/v2/nms/rtl/nms_activation_
|
|
replicated.v. No change to any FSM, arbiter, SDRAM controller, or
|
|
dependency-tracking logic -- this is a pure, minimal, two-file SRAM
|
|
read-timing fix.
|
|
VERIFICATION (all via Verilator, the trusted tool per DEC-0004):
|
|
- tb_fpga_neural_v2_top_smoke.v: 11/11 PASS -- single job alone,
|
|
two jobs back-to-back, two jobs with a ~85us realistic SPI-paced
|
|
gap, and a parametric sweep of inter-job gaps (100ns/5000ns/
|
|
50000ns), all bit-exact.
|
|
- tb_nms_dstress_sdram_unified.v N=2: 49788 cycles, 256/256
|
|
bit-exact -- IDENTICAL cycle count to before this fix (zero
|
|
regression).
|
|
- tb_nms_dstress_sdram_unified.v N=4: 49771 cycles, 256/256
|
|
bit-exact -- IDENTICAL cycle count to before this fix (zero
|
|
regression).
|
|
- tb_sdram_unified_backend.v: 40/40 PASS (unaffected, unrelated file).
|
|
- tb_spi_host_bridge.v: 18/18 PASS (unaffected, confirms Part A's own
|
|
fix still holds).
|
|
STATUS: ERR-0025 (Parts A and B) fully RESOLVED. The physical SPI host
|
|
interface is now verified correct end-to-end (SPI -> dependency_
|
|
manager -> compute -> SDRAM -> result) under both tight and realistic-
|
|
gap job pacing, with zero regression to the STEP19 baseline.
|
|
|
|
ERR-0026 -- T_MRD datasheet-unit mismatch (SDRAM init sequence, 64MHz)
|
|
|
|
DATE: 2026-09-06
|
|
FOUND DURING: PRE-PCB VERIFICATION FREEZE, section-8 SDRAM datasheet-
|
|
level audit (real Alliance Memory AS4C4M16SA-6TIN datasheet, Rev 5.0,
|
|
Oct 2018, Table 17 "Electrical Characteristics/AC Operating
|
|
Conditions", -6 speed grade).
|
|
ROOT CAUSE: hardware/v2/nms/rtl/sdram_controller.v modeled tMRD (LOAD
|
|
MODE REGISTER -> any command spacing) as `ns_to_cycles(12)`, i.e. as
|
|
if it were a nanosecond-based timing spec like tRCD/tRP. The real
|
|
datasheet specifies tMRD as a FIXED CYCLE COUNT, "2 tCK", independent
|
|
of clock frequency -- the same category of spec as CAS_LATENCY, which
|
|
this same file already correctly models as a fixed value two lines
|
|
below. `ns_to_cycles(12)` rounds to exactly 2 cycles at every
|
|
frequency this design had previously been tested at (100/133/166MHz),
|
|
so the wrong unit model was silently masked by coincidence. At the
|
|
real V2 board's own 64MHz operating point it rounds to only 1 cycle --
|
|
one cycle short of the real, fixed 2-tCK minimum -- a genuine,
|
|
datasheet-violating under-provisioning of the one-time SDRAM power-up/
|
|
mode-register-set sequence. Confirmed NOT a bug in tRCD/tRP (correctly
|
|
ns-based, correctly use ns_to_cycles()), tRAS (satisfied by construction:
|
|
the fixed tRCD+CAS_LATENCY+BURST_LEN dispatch sequence is always >=6
|
|
cycles, 93.75ns >= the real 42ns minimum at 64MHz), or tWR (folded in
|
|
conservatively via T_RP+1, giving 3 cycles >= the real 2-tCK minimum).
|
|
FIX: hardcoded `localparam T_MRD = 2;` (matching how CAS_LATENCY is
|
|
already modeled), replacing `localparam T_MRD = ns_to_cycles(12);`.
|
|
Only affects the one-time SDRAM init sequence, not per-transaction
|
|
timing.
|
|
VERIFICATION (all via Verilator, the trusted tool per DEC-0004):
|
|
- tb_sdram_controller.v: full 9-config legacy sweep (100/133/166MHz
|
|
x BURST_LEN 1/4/8), 461/461 PASS each config, zero regression.
|
|
- tb_sdram_controller.v: NEW dedicated 64MHz/BURST_LEN=4 config (the
|
|
real board target, not covered by the legacy sweep) -- 461/461
|
|
PASS, confirming the fix is correct at the frequency where the
|
|
bug actually manifested.
|
|
- tb_nms_dstress_sdram_unified.v N=2: 49788 cycles, 256/256
|
|
bit-exact -- IDENTICAL to pre-fix (expected: T_MRD only affects
|
|
one-time init, not steady-state per-transaction timing).
|
|
- tb_nms_dstress_sdram_unified.v N=4: 49771 cycles, 256/256
|
|
bit-exact -- IDENTICAL to pre-fix.
|
|
- tb_fpga_neural_v2_top_smoke.v: 11/11 PASS, unaffected.
|
|
STATUS: RESOLVED. Files changed: hardware/v2/nms/rtl/sdram_
|
|
controller.v (single localparam, plus explanatory comment).
|