micheleandClaude Sonnet 5
685a4d6cfe
New brief (N=8 timing closure, LFE5U-85F retarget, 4/8/16 x 1/2-bank SDRAM sweep). Phase 0: no RTL changes, only measure the current baseline. ERR-0030: constraints/v2_unified.lpf's FREQUENCY PORT "clk" was still 80MHz, a leftover from the STEP19 freeze, never updated to the project's real 64MHz target -- fixed (LPF only, zero RTL/datapath effect). ERR-0031 (bigger one): the first two synthesis attempts targeted nms_neural_multiprocessor_sdram_unified.v, which is NOT the real board-level top -- it's an obsolete wrapper only exercised by one testbench now. The real target is fpga_neural_v2_top.v (adds the real PLL, reset_sync, spi_host_bridge, and a second arbitration level), which is what actually goes through synthesis+P&R for hardware. Re-targeted correctly, matched against constraints/v2_board_top.lpf (all 17 ports real-ball-assigned). An N_SLOTS=8 P&R attempt against the WRONG (wrapper) target ran for 2h42m without converging on a single seed; discarded rather than trusted. N_SLOTS=8 baseline deferred by explicit user request until N_SLOTS=4 is fully understood -- re-attempt against the correct fpga_neural_v2_top target with an agreed time budget. Result (EXP-0049, fresh 8-seed nextpnr-ecp5 P&R, real pins): N_SLOTS=4 8/8 PASS at 64MHz, worst-seed 81.20MHz, mean 91.05MHz. Higher than DEC-0042's historical worst/best (64.55/77.21MHz) despite identical RTL -- disclosed, unresolved (leading hypothesis: nextpnr-ecp5 build/version difference), adopted as the operative baseline for this session's toolchain going forward. Full writeup in errors.log/timing.log/ experiments.log (EXP-0049). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1557 lines
92 KiB
Plaintext
1557 lines
92 KiB
Plaintext
# V2 errors log -- solo append, mai troncato/sovrascritto (vedi README.md)
|
|
# Nessuna entry ancora -- popolato incrementalmente man mano che avanza lo sviluppo V2.
|
|
|
|
ERR-0001 (Icarus Verilog v13.0 toolchain bug, TASK/SCOPE-ENTRY DESYNC)
|
|
DATE: 2026-09-05
|
|
MODULE: hardware/v2/sim/tb_neural_processor.v (M1 testbench development)
|
|
SYMPTOM: a task (or any named `begin:label` block) whose FIRST executable
|
|
statement is a blocking assignment to a signal read by another module's
|
|
`always @(posedge clk)`, when the task/block is entered immediately after
|
|
a time-consuming statement in the caller with no intervening
|
|
`@(posedge clk)`, can make that FIRST assignment invisible to the DUT at
|
|
the very next clock edge (the DUT's own always block behaves as if the
|
|
signal never changed). Confirmed at V1's own frozen `neuron_parallel.v`
|
|
(unmodified, already certified) via a minimal 3-statement task
|
|
(`v1_start=1; @(posedge clk); v1_start=0;`) -- `busy` never asserted.
|
|
REPRODUCTION: /tmp/mt6.v-style repro (not committed, transient scratch
|
|
file) -- see conversation record for the exact minimal case.
|
|
DIAGNOSIS METHOD: bisected from the full dual-DUT testbench down to a
|
|
standalone ~15-line repro, ruling out RTL, port connections, and
|
|
operator precedence one at a time.
|
|
WORKAROUND: always begin such a task with an explicit `@(posedge clk);`
|
|
before its first assignment (matches the pre-existing convention in
|
|
hardware/v1/sim's own tasks, e.g. neuron_parallel_tb.v's run_neuron,
|
|
which is presumably why V1's own test suite was never affected).
|
|
STATUS: WORKAROUND APPLIED in hardware/v2/sim/tb_neural_processor.v's
|
|
run_case. NOT reported upstream (out of scope for this session). See
|
|
ERR-0004 for the broader consequence of this finding.
|
|
|
|
ERR-0002 (Icarus Verilog v13.0 toolchain bug, SPURIOUS CONDITION EVALUATION)
|
|
DATE: 2026-09-05
|
|
MODULE: hardware/v2/rtl/neural_processor.v (protocol-violation guard,
|
|
removed -- see decisions.log DEC-0003)
|
|
SYMPTOM: `if (operand_valid && !operand_ready && (state-is-one-of-four))`
|
|
inside `always @(posedge clk)` evaluated TRUE at an edge where
|
|
`operand_valid` was independently confirmed (via $display in the same
|
|
timestep, and via the testbench's own port-connected signal) to be 0.
|
|
Bisected term-by-term: even `if (operand_valid && !operand_ready)`
|
|
alone, and even `if (operand_valid)` alone with explicit `== 1'b1`
|
|
comparisons, still fired spuriously. Confirmed NOT a precedence issue
|
|
(parens are unambiguous) and NOT specific to this exact expression
|
|
shape (multiple simplified variants all reproduced it).
|
|
CROSS-CHECK: root-caused further via a minimal 2-state FSM
|
|
(`if (go) st<=B;`) with NO relation to the removed guard -- Icarus
|
|
failed to transition on an ODD-numbered testbench clock edge
|
|
(`repeat(3)` before the pulse) but succeeded on an EVEN-numbered one
|
|
(`repeat(4)`), reproduced identically with both `always #5 clk=~clk`
|
|
and `initial ... forever #5 clk=~clk` clock generators. VERILATOR
|
|
5.050 gives the CORRECT result for the same repro in both cases.
|
|
This suggests ERR-0001 and ERR-0002 are two symptoms of the same
|
|
underlying VVP scheduling defect (edge-count/thread-parity dependent),
|
|
not two unrelated bugs.
|
|
STATUS: the offending RTL block (protocol-violation detection) was
|
|
REMOVED rather than chased further -- see DEC-0003. Root cause not
|
|
fully isolated (documented honestly, not overclaimed).
|
|
|
|
ERR-0003 (real RTL bug in hardware/v2/rtl/neural_processor.v, FOUND AND FIXED)
|
|
DATE: 2026-09-05
|
|
MODULE: hardware/v2/rtl/neural_processor.v, stage 0 (input
|
|
alignment/register)
|
|
SYMPTOM: back-to-back single-tile jobs (and some multi-tile jobs)
|
|
produced result_data=0 instead of the correct value, while the
|
|
internal `y7` register (one stage upstream of the FSM's capture)
|
|
showed the CORRECT value one cycle later than `valid7` first asserted.
|
|
ROOT CAUSE: `last0 <= tile_last;` was unconditional, while
|
|
`valid0 <= operand_valid && operand_ready;` was correctly gated. A
|
|
master asserting `tile_last` before `operand_ready` rises (legal
|
|
valid-before-ready behavior) let a "last" tag propagate through the
|
|
pipeline (last1, last_tree[], last5, last6) with NO corresponding
|
|
valid tile behind it, arriving at stage 7 one cycle ahead of the
|
|
real valid/data pair and causing the FSM to capture a stale/wrong
|
|
`y7`.
|
|
EVIDENCE: isolated to a single-DUT, no-task, no-V1 repro
|
|
(hardware/v2/sim/tb_neural_processor.v run under Verilator, with a
|
|
cycle-by-cycle dump of valid5/last5/valid6/last6/valid7/y7) --
|
|
`last5=1` while `valid5=0` on the same cycle, confirmed the
|
|
desync's exact origin at stage 0.
|
|
FIX: `last0 <= (operand_valid && operand_ready) ? tile_last : 1'b0;`
|
|
-- last0 is now gated identically to valid0.
|
|
VERIFICATION: full 7-test bit-exact-vs-V1 regression
|
|
(hardware/v2/sim/tb_neural_processor.v under Verilator) -- 7/7 PASS
|
|
after the fix, including the back-to-back and single-tile-after-
|
|
multi-tile cases that exposed it.
|
|
STATUS: FIXED, verified.
|
|
|
|
ERR-0004 (methodology consequence of ERR-0001/ERR-0002)
|
|
DATE: 2026-09-05
|
|
NOTE: this session's V1 certification campaign (docs/validation/,
|
|
hardware/v1/docs/validation/) was verified exclusively with Icarus
|
|
Verilog v13.0, the ONLY simulator available on this machine at the
|
|
time. ERR-0001/ERR-0002 show that v13.0 has at least one real,
|
|
reproducible scheduling defect around clock-edge/task-entry timing.
|
|
V1's own testbenches were NOT observed to trigger it in this session
|
|
(V1's `run_neuron`-style tasks already begin with `@(posedge clk)`,
|
|
which incidentally avoids ERR-0001's trigger condition), and V1
|
|
remains frozen/untouched regardless. This is flagged here for
|
|
honesty, not to imply V1's certification is wrong -- re-verifying
|
|
the full V1 suite under Verilator was explicitly OUT OF SCOPE for
|
|
this V2-kickoff session (V1 is frozen, not to be touched) and was
|
|
not performed. See decisions.log DEC-0004.
|
|
STATUS: OPEN CAVEAT, not actioned in this session by design.
|
|
|
|
ERR-0005 (synthesis measurement artifact, WORKED AROUND, not an RTL bug)
|
|
DATE: 2026-09-05
|
|
MODULE: hardware/v2/rtl/neural_processor_array.v
|
|
SYMPTOM: synthesizing neural_processor_array as a bare top-level
|
|
module (every per-processor job/operand/result field exposed as a
|
|
real TRELLIS_IO pin) works at N_PROCESSORS=1 but fails place&route
|
|
at N_PROCESSORS=2 with "Unable to place cell ...$tr_io, no BELs
|
|
remaining to implement cell type 'TRELLIS_IO'".
|
|
ROOT CAUSE: not a logic/timing limit -- the LFE5U-45F-8BG381 package
|
|
has 245 TRELLIS_IO pins total; the array's wide per-processor buses
|
|
(input_data/weight_data alone are DATA_WIDTH*P_IN*N_PROCESSORS bits)
|
|
exceed that budget once N_PROCESSORS>=2, purely because these ports
|
|
have no on-chip consumer yet (the Memory Manager/M4 and Neural
|
|
Director/M5 that will drive them in the real system don't exist
|
|
yet).
|
|
WORKAROUND: hardware/v2/synthesis/harness_neural_processor_array.v --
|
|
a synthesis-only wrapper (NOT part of rtl/, not a functional
|
|
deliverable) that drives all wide buses from an internal free-
|
|
running LFSR and reduces outputs to a small checksum, keeping only
|
|
clk/rst/seed/checksum as real top-level pins. See its own header
|
|
comment and experiments.log EXP-0003 for the resulting real
|
|
resource/Fmax numbers.
|
|
STATUS: WORKED AROUND. Will become moot once M4/M5 exist and the array
|
|
is synthesized as part of a larger design with on-chip ports instead
|
|
of a bare top-level module.
|
|
|
|
ERR-0006 (real RTL bugs in hardware/v2/rtl/memory_manager.v, FOUND AND FIXED)
|
|
DATE: 2026-09-05
|
|
MODULE: hardware/v2/rtl/memory_manager.v
|
|
SYMPTOM: end-to-end M4 testbench (real V1 PSRAM backend + real M1
|
|
neural_processor) hung permanently partway through the first
|
|
multi-tile job -- bank_ready for the "current" bank never became 1,
|
|
even though the byte-level backend clearly kept completing read
|
|
transactions (observed via cycle-by-cycle hierarchical tracing of
|
|
memory_manager/prefetch_engine internal state).
|
|
ROOT CAUSES (three, found together during the same debugging session):
|
|
1. The tile-N+2 prefetch request, queued on tile-N's handoff, could
|
|
be issued (pf_start asserted) while prefetch_engine was STILL
|
|
mid-fetch for tile-N+1 -- there was no single-in-flight-request
|
|
discipline at all in the first draft. Fixed by adding a
|
|
single-entry pf_pending register: requests are queued, not
|
|
issued directly, and a dedicated rule launches the queued
|
|
request only once the engine reports free.
|
|
2. Even with that queue, pf_busy does not read 1 until the cycle
|
|
AFTER pf_start was first observed by prefetch_engine (its own
|
|
fetch_busy<=1 lags its own fetch_start sampling by one clock) --
|
|
checking only `!pf_busy` left a genuine one-cycle window where a
|
|
second queued request would fire on top of the one just
|
|
launched, silently overwriting pf_target_bank (and pf_x_addr/
|
|
pf_w_addr) for the fetch already in flight. The address corruption
|
|
was harmless (prefetch_engine had already latched the correct
|
|
address into its own state that same edge), but pf_target_bank
|
|
corruption meant the eventually-completed fetch's real data got
|
|
filed into the WRONG bank's bank_ready/bank_x/bank_w, permanently
|
|
starving the bank actually needed next. Fixed by gating the issue
|
|
rule on `!pf_busy && !pf_start` (the extra term closes exactly
|
|
this one-cycle window).
|
|
3. A combinational mux selecting between prefetch_engine's own
|
|
backend wires and the result-write-back FSM's wires was gated on
|
|
`state == MM_WRITE_RESULT`, but wr_mem_req (asserted while
|
|
state==MM_WRITE_RESULT) only becomes valid the FOLLOWING cycle,
|
|
i.e. while state==MM_DONE -- the mux therefore selected the wrong
|
|
source for the one cycle the write request pulse was actually
|
|
high, silently dropping the PSRAM write entirely. Fixed by
|
|
widening the mux's select condition to cover both states.
|
|
DIAGNOSIS METHOD: cycle-by-cycle hierarchical signal dumps (mm.state,
|
|
tile_idx, bank_ready, pf_busy, pf_pending, pf_target_bank,
|
|
prefetch_engine's own state) under Verilator, printed only on
|
|
signal-change to keep the trace readable, comparing against the
|
|
hand-derived expected sequence of events for a 3-tile job.
|
|
VERIFICATION: hardware/v2/sim/tb_memory_manager.v -- 3/3 tests PASS
|
|
after all three fixes, including a 5-tile job (steady-state
|
|
double-buffer swap across more than 2 tiles) and independent
|
|
PSRAM read-back of the written-back result byte (not just internal
|
|
signal inspection).
|
|
STATUS: FIXED, verified end-to-end with the real (unmodified) V1
|
|
PSRAM backend chain and a real M1 neural_processor.
|
|
|
|
ERR-0007 (Yosys usage quirk, WORKED AROUND, not an RTL bug)
|
|
DATE: 2026-09-05
|
|
MODULE: hardware/v2/synthesis/harness_dataflow_core.v (build script)
|
|
SYMPTOM: `chparam -set N_SLOTS 2 dataflow_core` (setting the parameter
|
|
directly on the NON-top child module, before running `synth_ecp5
|
|
-top harness_dataflow_core`) synthesizes with no visible error from
|
|
the chparam/hierarchy commands themselves, but `synth_ecp5` then
|
|
fails with "Module `\dataflow_core' referenced in module
|
|
`\harness_dataflow_core' in cell `\dut' is not part of the design" --
|
|
even though a standalone `hierarchy -top harness_dataflow_core` run
|
|
(no synth_ecp5) with the exact same chparam succeeds.
|
|
ROOT CAUSE: harness_dataflow_core.v's own instantiation of
|
|
dataflow_core explicitly overrides N_SLOTS via its own local
|
|
parameter (`.N_SLOTS(N_SLOTS)`) -- chparam on the child module's
|
|
DEFAULT is therefore always shadowed at that instantiation site
|
|
regardless of its value, and synth_ecp5's own internal re-hierarchy
|
|
pass (distinct from a standalone `hierarchy` call) does not
|
|
reconcile a chparam'd-but-never-actually-used child default the
|
|
same way, dropping the generic module reference instead.
|
|
WORKAROUND: set the parameter on the TOP module being synthesized
|
|
instead (`chparam -set N_SLOTS 2 harness_dataflow_core`), letting
|
|
its own instantiation forward the value down to dataflow_core as
|
|
designed. Confirmed working for both N_SLOTS=2 and N_SLOTS=4.
|
|
STATUS: WORKED AROUND. A build-script ordering detail, not a defect in
|
|
dataflow_core.v or harness_dataflow_core.v themselves -- noted here
|
|
so a future N_SLOTS sweep (M9/M10) does not re-trip over it.
|
|
|
|
ERR-0008 (real RTL bug in the first draft of hardware/v2/rtl/slot_mem_arbiter.v, FOUND AND FIXED)
|
|
DATE: 2026-09-05
|
|
MODULE: hardware/v2/rtl/slot_mem_arbiter.v (M8, new)
|
|
SYMPTOM: hardware/v2/sim/tb_neural_multiprocessor.v -- node0 (slot 0)
|
|
completes correctly (result=48), but node1 (slot 1, dispatched
|
|
concurrently to node0) never completes; its byte-level backend
|
|
request appears to simply vanish, and the slot hangs forever
|
|
(watchdog timeout at 20000 cycles, result stays 0).
|
|
ROOT CAUSE: memory_manager.v/prefetch_engine.v's own byte-level
|
|
backend protocol (mem_req/mem_wr/mem_addr/mem_wdata/mem_rdata/
|
|
mem_ready) is FIRE-AND-FORGET: mem_req is asserted for exactly ONE
|
|
clock cycle per byte transaction, with no separate "request
|
|
accepted" acknowledgment -- only `mem_ready` (transaction
|
|
COMPLETION) exists. M4's own testbench (tb_memory_manager.v) never
|
|
exposed this because it connects exactly ONE memory_manager directly
|
|
to int8_memory_access, which is always idle and therefore always
|
|
able to accept that single pulse the instant it fires. The first
|
|
arbiter draft only granted a port while its s_req was LIVE that same
|
|
cycle -- if slot 1's one-cycle pulse arrived on a cycle where the
|
|
arbiter was already owned by slot 0, the pulse was gone the very
|
|
next cycle with no record of it ever having happened, and slot 1's
|
|
prefetch_engine sat in ST_READ_X/ST_READ_W waiting forever for a
|
|
mem_ready that could never arrive (its request never reached
|
|
int8_memory_access at all).
|
|
DIAGNOSIS METHOD: ran the M8 testbench, observed node0 (whichever slot
|
|
the Director happened to grant the shared bus to first) complete
|
|
while node1 (the other, concurrently-dispatched slot) hung; traced
|
|
the fire-and-forget nature of mem_req directly in
|
|
prefetch_engine.v's own state machine (`mem_req <= 1'b1;` appearing
|
|
only inside single-cycle state-transition branches, unconditionally
|
|
cleared to 0 every other cycle) -- confirmed the arbiter's naive
|
|
"grant only while req is live" logic could not possibly catch a
|
|
pulse arriving during contention.
|
|
FIX: every incoming s_req pulse is now LATCHED into a per-port
|
|
`pending` register (capturing wr/addr/wdata the same cycle),
|
|
regardless of arbiter state -- the same single-entry "queue, don't
|
|
drop the request" idiom already used by memory_manager's own
|
|
pf_pending register (ERR-0006 fix #1). Grants are drawn from
|
|
`pending`, never from a live s_req directly. This adds a uniform
|
|
minimum 1-cycle latency to every byte transaction (a real, honestly
|
|
measured cost of sharing one PSRAM port across N_SLOTS -- see
|
|
timing.log/benchmark.log EXP-0009), but never drops a request
|
|
regardless of contention.
|
|
VERIFICATION: hardware/v2/sim/tb_neural_multiprocessor.v -- 4/4 PASS
|
|
after the fix (444 cycles end-to-end, vs the buggy draft's 20000-
|
|
cycle watchdog timeout). hardware/v2/sim/tb_memory_manager.v (M4,
|
|
untouched) re-run unchanged -- still 3/3 PASS, confirming the fix is
|
|
entirely contained inside the new arbiter module.
|
|
STATUS: FIXED, verified end-to-end with the real (unmodified) V1
|
|
PSRAM backend chain and real concurrent multi-slot contention.
|
|
|
|
ERR-0009 (one real RTL bug + two testbench bugs, all FOUND AND FIXED
|
|
during the post-M10 final benchmark campaign -- hardware/v2/sim/
|
|
tb_benchmark_suite.v)
|
|
DATE: 2026-09-05
|
|
|
|
1. REAL RTL BUG in hardware/v2/rtl/neural_director.v (M5, previously
|
|
committed/synthesized, never before exercised at N_SLOTS=1):
|
|
SYMPTOM: Verilator compile error building neural_multiprocessor at
|
|
N_SLOTS=1 -- "%Error-ZEROREPL: Replication value of 0 is only
|
|
legal under a concatenation" at three sites.
|
|
ROOT CAUSE: at N_SLOTS=1, $clog2(1)=0, making
|
|
`{$clog2(N_SLOTS){1'b0}}` a ZERO-width replication (illegal
|
|
outside a concatenation, IEEE 1800 11.4.12.1). Every prior
|
|
milestone (M5-M10) only ever built/simulated/synthesized
|
|
neural_director at N_SLOTS=2/4/8 -- N_SLOTS=1 was never actually
|
|
exercised until this benchmark campaign asked for it as the
|
|
baseline for parallel-scaling measurement.
|
|
FIX: replaced the three `{$clog2(N_SLOTS){1'b0}}` reset/default
|
|
expressions with the width-agnostic `'0` literal, which self-sizes
|
|
correctly for any width including 0. No functional change for
|
|
N_SLOTS>1 (same reset value).
|
|
VERIFICATION: hardware/v2/sim/tb_neural_director.v (M5's own
|
|
testbench, N_SLOTS=2) re-run unchanged -- still 4/4 PASS. All 6
|
|
benchmark-suite workloads then verified bit-exact at N_SLOTS=1
|
|
through the real full neural_multiprocessor + real V1 PSRAM chain.
|
|
STATUS: FIXED, verified at both N_SLOTS=1 (newly working) and
|
|
N_SLOTS=2 (no regression).
|
|
|
|
2. TESTBENCH BUG (tb_benchmark_suite.v, not RTL): psram_model's own
|
|
DEPTH parameter (524288... originally 131072 words = 256KB) was
|
|
smaller than the byte address range some workloads actually use
|
|
(workload C-Large's own result region alone needs word address
|
|
~0x24000, beyond a 131072-word/0x20000 DEPTH) -- a silent
|
|
out-of-bounds array access, same bug CLASS already documented once
|
|
before (an M5 testbench bug, sim_byte_mem's too-small DEPTH).
|
|
SYMPTOM: every C-Large neuron read back real=0 (poison value never
|
|
overwritten) despite the RTL reporting all 128 jobs completed.
|
|
FIX: DEPTH raised to 524288 words (1MB byte-addressable), computed
|
|
to safely exceed the highest byte address used by any of the six
|
|
workloads' regions (~0xB2006, workload F).
|
|
VERIFICATION: C-Large re-run bit-exact PASS after the fix.
|
|
|
|
3. TESTBENCH BUG (tb_benchmark_suite.v, not RTL): N_NODES=512 was
|
|
smaller than the highest node_id actually used -- workload D-Stress
|
|
(node_base=400, 256 neurons) reaches node_id 655, which silently
|
|
WRAPS at the 9-bit node_id width (512 -> truncates to 0), colliding
|
|
with workload A's node_id 0, already permanently ST_DISPATCHED (M6's
|
|
own design never reclaims dispatched node slots, decisions.log
|
|
DEC-0008). register_node's blocking `while(!reg_ready)` wait then
|
|
deadlocks forever (reg_ready never returns true for an already-
|
|
occupied, non-EMPTY node_id).
|
|
SYMPTOM: simulation appeared to hang indefinitely partway through
|
|
D-Stress's node registration (confirmed via periodic progress
|
|
`$display` instrumentation added specifically to localize this --
|
|
registration silently stopped advancing past neuron index 112).
|
|
FIX: N_NODES raised to 1024, comfortably exceeding every workload's
|
|
own node_id range.
|
|
VERIFICATION: D-Stress re-run bit-exact PASS (256/256 neurons) after
|
|
the fix, no further hangs at any N_SLOTS configuration (1/2/4/8).
|
|
NOTE: this is a REAL, honest consequence of DEC-0008's own design
|
|
choice (no node-slot reclamation) -- a long-running system that
|
|
keeps registering new nodes without ever reusing old (DISPATCHED)
|
|
ids will eventually exhaust its node_id space and deadlock exactly
|
|
this way. Flagged in the final benchmark report's Limitations
|
|
section, not just fixed and forgotten.
|
|
|
|
DIAGNOSIS METHOD (bug 3): periodic `$display` progress heartbeats
|
|
added to the registration loop and the completion watchdog loop,
|
|
run under `stdbuf -oL` to force line-buffered (not block-buffered)
|
|
output for real-time visibility, isolating the exact neuron index
|
|
where progress stopped advancing -- the same "trace real signals,
|
|
don't guess" discipline used throughout this whole project.
|
|
|
|
ERR-0010 (real RTL bugs found and fixed during activation_cache.v
|
|
implementation, DEC-0016)
|
|
DATE: 2026-09-05
|
|
|
|
1. Same bug CLASS as ERR-0006 (pf_target_bank/pf_pending_bank), NEW
|
|
instance, memory_manager.v's activation-cache side:
|
|
SYMPTOM: hardware/v2/sim/tb_memory_manager.v -- job xb=4096 (3
|
|
tiles) hung/produced wrong results after adding the shared
|
|
activation_cache request path; cycle-by-cycle tracing (temporary
|
|
$display instrumentation, later removed) showed a cache ack for
|
|
tile 1 (queued for bank 1) instead applying its data to bank 0.
|
|
ROOT CAUSE: `xc_target_bank` (which bank an ack's data should be
|
|
written into) was being written DIRECTLY by the queueing logic
|
|
(MM_IDLE/MM_PREFETCH_FIRST/MM_STREAM), the same register used to
|
|
resolve an ack that might still be OUTSTANDING from an EARLIER
|
|
queued request. Real PSRAM miss latency can exceed one
|
|
neural_processor tile's own compute time, so a LATER handoff can
|
|
queue a NEW request (targeting a DIFFERENT bank) in the same or a
|
|
following cycle, before the EARLIER request's ack has arrived --
|
|
with only one `xc_target_bank` register, NBA "last write in
|
|
program order wins" semantics silently overwrote which bank the
|
|
EARLIER, already-in-flight request's eventual ack gets applied
|
|
to. This is the EXACT bug ERR-0006 already found and fixed once
|
|
for pf_target_bank/pf_pending_bank (which already used a correct
|
|
two-register pattern: a `_pending_bank` staging register written
|
|
at queueing time, and the real `_target_bank` written ONLY by the
|
|
issue rule at the moment the request actually fires) -- this
|
|
module's newly-added activation-cache side did not follow that
|
|
already-established pattern, until now.
|
|
FIX: introduced `xc_pending_bank` (written at queueing time) and
|
|
changed `xc_target_bank` to be written ONLY by the issue rule
|
|
(`xc_target_bank <= xc_pending_bank;`, the same cycle xc_req
|
|
fires), mirroring pf_target_bank/pf_pending_bank exactly.
|
|
VERIFICATION: tb_memory_manager.v 3/3 PASS bit-exact after the fix.
|
|
Full final-benchmark campaign (24/24 workload/config
|
|
combinations) re-verified bit-exact.
|
|
|
|
2. Same bug CLASS as ERR-0009 item 1 (neural_director.v's N_SLOTS=1
|
|
zero-width replication), NEW instance, activation_cache.v:
|
|
SYMPTOM: Verilator compile error building the M4-level regression
|
|
testbench (activation_cache instantiated with N_SLOTS=1 there,
|
|
to serve a single memory_manager instance) -- same
|
|
"%Error-ZEROREPL: Replication value of 0 is only legal under a
|
|
concatenation" as ERR-0009.
|
|
ROOT CAUSE: `miss_idx = {$clog2(N_SLOTS){1'b0}};` -- identical root
|
|
cause to ERR-0009 item 1 ($clog2(1)=0 at N_SLOTS=1).
|
|
FIX: replaced with the same width-agnostic `'0` literal used in
|
|
neural_director.v's own fix.
|
|
VERIFICATION: tb_memory_manager.v (N_SLOTS=1 activation_cache) and
|
|
the full campaign (N_SLOTS=1/2/4/8) all build and pass.
|
|
|
|
DIAGNOSIS METHOD (item 1): periodic $display cycle-by-cycle tracing of
|
|
memory_manager's own internal state (xc_req/xc_ack/xc_outstanding/
|
|
xc_pending/bank_x_ready/bank_w_ready) and the 2-port test arbiter's
|
|
own owner/grant state, added temporarily to tb_memory_manager.v and
|
|
removed once the bug was isolated and fixed -- the same "trace real
|
|
signals, don't guess" discipline used throughout this project.
|
|
|
|
ERR-0011
|
|
timestamp: 2026-09-05T21:45:00Z
|
|
git_commit: 63cac6a7e5e0126bb13bea89dfe21966d5c9a1a5 (+ uncommitted NMS STEP1 work)
|
|
session: v2-NMS-STEP1-bandwidth-study
|
|
module: hardware/v2/nms/rtl/ideal_memory_model.v, hardware/v2/nms/sim/tb_bandwidth_study.v
|
|
context: building the NMS (Neural Memory System) STEP1 bandwidth
|
|
requirement study harness (EXP-0017) -- a NEW, simulation-only
|
|
idealized backing-store model + testbench, not part of the frozen
|
|
V2 datapath. Three real bugs were found and fixed in this new
|
|
harness itself before its output could be trusted, via the same
|
|
"trace real signals, don't guess" discipline as ERR-0006..0010.
|
|
|
|
1. REGISTERED-GRANT DOUBLE ADMISSION (ideal_memory_model.v, first
|
|
revision):
|
|
SYMPTOM: the very first sweep combination hung indefinitely (no
|
|
CSV row produced after 2+ minutes of wall-clock simulation,
|
|
manually killed).
|
|
ROOT CAUSE: req_ready was a REGISTERED output (visible one cycle
|
|
after the arbiter's own grant decision), so a requester holding
|
|
req_valid high (standard valid/ready) would still see req_valid
|
|
asserted for a second cycle after being granted -- the arbiter's
|
|
round-robin scan would find that same still-high req_valid and
|
|
grant it AGAIN, double-admitting one logical request (or worse,
|
|
re-granting indefinitely under some timing, causing the hang).
|
|
FIX: made req_ready purely COMBINATIONAL (same cycle as req_valid,
|
|
ordinary same-cycle valid/ready), computed in a separate
|
|
always @* block; the clocked state-update logic reuses that same
|
|
cycle's combinational grant decision via non-blocking assignment.
|
|
|
|
2. SINGLE-TRANSFER-AT-A-TIME SERIALIZATION CAP (ideal_memory_model.v,
|
|
second revision, after fixing item 1):
|
|
SYMPTOM: measured utilization collapsed to exactly 1/N_SLOTS at
|
|
every N_SLOTS>1 configuration regardless of configured bandwidth
|
|
(even cfg_bw_bytes=64, far above TILE_BYTES=16) -- a
|
|
suspiciously exact, BW-independent ratio, not a physically
|
|
motivated curve.
|
|
ROOT CAUSE: the model's "busy port" served only ONE TILE_BYTES-
|
|
sized transfer at a time regardless of cfg_bw_bytes, capping
|
|
aggregate system throughput at exactly 1 transfer/cycle once
|
|
bandwidth exceeded TILE_BYTES -- bandwidth above the per-transfer
|
|
floor was silently discarded instead of enabling several
|
|
concurrent transfers to drain in the same cycle.
|
|
FIX: rewrote the drain stage as a shared per-cycle byte BUDGET
|
|
(cfg_bw_bytes) applied against the queue head, completing as
|
|
many whole TILE_BYTES-sized transfers as the budget allows in one
|
|
cycle (bounded unrolled loop, MAX_ITERS=16) before carrying any
|
|
leftover partial progress to the next cycle -- correctly
|
|
separates aggregate BANDWIDTH from per-transfer LATENCY.
|
|
|
|
3. SINGLE-FLAG ISSUANCE THROTTLE (tb_bandwidth_study.v):
|
|
SYMPTOM: even after fixing items 1-2, utilization plateaued at
|
|
~50% regardless of configured bandwidth (>=16) or
|
|
PREFETCH_DEPTH (tested up to 8) at latency=0 -- again a
|
|
suspiciously round, BW/PFD-independent ceiling.
|
|
ROOT CAUSE: each per-slot feeder tracked outstanding fetches with a
|
|
single registered `fetch_pending` flag (set, wait a cycle, clear
|
|
on grant, re-evaluate `want_fetch` only the FOLLOWING cycle) --
|
|
a serial ~2-cycle round trip per request REGARDLESS of
|
|
PREFETCH_DEPTH, an artificial testbench-side issuance-rate cap
|
|
unrelated to the memory bandwidth actually being studied.
|
|
FIX: tied `mem_req_valid` COMBINATIONALLY to `want_fetch` (safe
|
|
since ideal_memory_model's own admission is unconstrained,
|
|
req_ready mirrors req_valid every cycle) -- removed the
|
|
fetch_pending latch entirely, allowing issuance at up to
|
|
1 request/cycle/requester, matching real achievable demand.
|
|
|
|
DIAGNOSIS METHOD: for item 1, direct hang observation + manual kill.
|
|
For items 2-3, isolated single-requester debug harnesses
|
|
(/tmp/tb_debug.v, /tmp/tb_debug2.v, not committed -- throwaway)
|
|
with cycle-by-cycle $display tracing of ideal_memory_model's own
|
|
internal head_active/head_remain/inq_count/lat_count via
|
|
hierarchical reference from the testbench, comparing the isolated
|
|
harness's hand-derivable expected rate (TILE_BYTES/cfg_bw_bytes
|
|
cycles/transfer) against the full sweep's reported utilization to
|
|
locate exactly which stage diverged from the physically-expected
|
|
value -- each bug was caught specifically because the buggy result
|
|
was a suspiciously ROUND, BW/PFD-INDEPENDENT ratio (1/N_SLOTS, then
|
|
0.5) rather than a smoothly-varying, physically-motivated curve.
|
|
VERIFICATION: after all three fixes, minimum aggregate bandwidth for
|
|
>=90/95/99% of compute-only throughput scales EXACTLY linearly with
|
|
N_SLOTS at 16 bytes/cycle/slot (matching the raw P_IN=8 tile
|
|
demand, 2*P_IN bytes/tile at 1 tile/cycle) -- see EXP-0017.
|
|
|
|
ERR-0012
|
|
timestamp: 2026-09-05T22:10:00Z
|
|
git_commit: 63cac6a7e5e0126bb13bea89dfe21966d5c9a1a5 (+ uncommitted NMS STEP3 work)
|
|
session: v2-NMS-STEP3-bank-contention
|
|
module: hardware/v2/nms/sim/tb_bank_contention.v
|
|
context: building EXP-0018's bank-contention harness. Two real bugs
|
|
found via literal simulation hangs, same "trace real signals, don't
|
|
guess" discipline as ERR-0006..0011.
|
|
|
|
1. FIXED-PRIORITY STARVATION:
|
|
SYMPTOM: every N_SLOTS>=2 run hung indefinitely at the very first
|
|
sweep combination (N_BANKS=1, stagger=0).
|
|
ROOT CAUSE: the bank arbiter picked the LOWEST-INDEX requester in a
|
|
bank every cycle, unconditionally -- with N_BANKS=1 and 2+ slots
|
|
wanting genuinely different tile indices, the lowest-index slot
|
|
always won every tie, permanently starving every other slot
|
|
(whose tile_idx never advances, so its done_flag never sets, so
|
|
the sweep driver's own "while(!all_done)" loop never exits).
|
|
FIX: added a per-bank ROUND-ROBIN priority pointer, rotated to
|
|
(last winner's index + 1) after every cycle it wins -- guarantees
|
|
every requester is eventually served even under permanent
|
|
contention. Directly matches the user's own NMS spec requirement
|
|
("Evita starvation", §10).
|
|
2. STAGE-BOOKKEEPING/DUT MISMATCH (job_started latched on job_ready
|
|
alone):
|
|
SYMPTOM: after fixing item 1, every combo with STAGGER>0 still hung
|
|
on its LAST slot.
|
|
ROOT CAUSE: `if (!job_started[s] && job_ready_s[s]) job_started[s]
|
|
<= 1;` gated only on job_ready_s[s], which neural_processor.v
|
|
asserts whenever it is idle REGARDLESS of job_valid -- for a
|
|
staggered slot whose job_valid gate hadn't opened yet, job_ready
|
|
was already high (idle) at cycle 0, so job_started[s] latched
|
|
immediately and incorrectly, permanently deasserting job_valid_s[s]
|
|
(`gate && !job_started[s]`) before the real, staggered job_valid
|
|
pulse was ever generated -- the testbench's own bookkeeping
|
|
believed the job had started while the real DUT (neural_processor.v)
|
|
sat in NP_IDLE forever, never having seen a real job_valid&&
|
|
job_ready handshake.
|
|
FIX: gated the latch on the ACTUAL accepted handshake,
|
|
`job_valid_s[s] && job_ready_s[s]`, not job_ready_s[s] alone.
|
|
DIAGNOSIS METHOD: cycle-by-cycle $display tracing (tile_idx, np_state,
|
|
req/ack, done flags per slot) added to a throwaway debug copy of the
|
|
testbench, with a hard cycle-count bailout to avoid an actual
|
|
infinite wait -- isolated item 2 specifically by noticing np_state
|
|
for the stalled slot stayed at NP_IDLE (0) forever even though this
|
|
testbench's own act_req_valid for that slot was already high,
|
|
meaning the DUT and the testbench bookkeeping disagreed about
|
|
whether the job had started.
|
|
VERIFICATION: after both fixes, all 4 N_SLOTS x 20 combos (80 total)
|
|
complete and produce the clean, monotonic-in-N_BANKS,
|
|
monotonic-in-stagger results reported in EXP-0018.
|
|
|
|
ERR-0013
|
|
timestamp: 2026-09-06T01:45:00Z
|
|
git_commit: 5c9ec61 (+ uncommitted NMS STEP8 work)
|
|
session: v2-NMS-STEP8-integration
|
|
module: hardware/v2/nms/rtl/nms_memory_manager.v
|
|
context: building/verifying nms_dataflow_core.v (STEP8 full NMS
|
|
integration: Dependency Manager -> Neural Director -> N_SLOTS x
|
|
(nms_memory_manager + neural_processor), backed by
|
|
nms_activation_replicated.v/nms_activation_fill_ctrl.v/
|
|
nms_weight_packed.v). Two real bugs found and fixed before trusting
|
|
correctness, via the same "trace real signals, don't guess"
|
|
discipline as ERR-0006..0012.
|
|
|
|
1. MISSING SECOND PIPELINE STAGE ON SRAM READS:
|
|
SYMPTOM: node0's own first (and only) tile computed result=0
|
|
instead of the expected 48 (2*3*8) -- a real, deterministic
|
|
wrong-answer bug, not a hang.
|
|
ROOT CAUSE: both nms_activation_replicated.v and
|
|
nms_weight_packed.v register rd_en THEN register the memory read
|
|
off that same registered rd_en (`if (rd_en) rd_data_reg <=
|
|
mem[addr];`) -- asserting rd_en at cycle T only becomes visible
|
|
to the SRAM's own always block at cycle T+1, which THEN schedules
|
|
the actual read for T+2. nms_memory_manager.v's own read-issue
|
|
logic used a SINGLE `read_issued` flag, capturing act_rd_data/
|
|
wgt_rd_data into input_data/weight_data exactly the cycle
|
|
`read_issued` first became visible (T+1) -- one full cycle too
|
|
early, grabbing the SRAM's stale (pre-read) output. Traced via
|
|
cycle-by-cycle tracing of act_rd_en/act_rd_data/wgt_rd_en/
|
|
wgt_rd_data/operand_valid/input_data/weight_data: at the exact
|
|
cycle op_valid first asserted, act_rd_data/wgt_rd_data already
|
|
held the CORRECT values (2 and 3) but input_data/weight_data
|
|
(captured the SAME cycle via non-blocking assignment) still read
|
|
as their PREVIOUS value (0), one cycle stale.
|
|
FIX: added a second flag (`read_ready`), turning the single-stage
|
|
read_issued -> capture sequence into a genuine 2-stage pipeline
|
|
(read_issued: SRAM now computing; read_ready: SRAM output now
|
|
valid, capture NOW) matching the SRAM's own real 2-cycle
|
|
rd_en-to-data latency.
|
|
2. STALE-VALUE RACE IN THE PRIVATE WEIGHT PREFETCH RESTART LOGIC:
|
|
SYMPTOM: not caught by any n_tiles=1 test (all of this file's first
|
|
3 test groups used n_tiles=1) -- would have silently corrupted
|
|
every n_tiles>1 job, duplicating/skipping alternating tiles.
|
|
ROOT CAUSE: `if (job_active_reg && !pf_busy && !pf_start &&
|
|
(wgt_fetched < n_tiles_reg[...])) pf_start<=1; pf_w_addr<=
|
|
w_base_reg + wgt_fetched*P_IN;` fires the SAME cycle pf_done
|
|
completes a fetch (prefetch_engine.v's own ST_DONE clears
|
|
fetch_busy the same cycle it pulses fetch_done), reading the OLD
|
|
(pre-increment) wgt_fetched -- re-issuing a fetch for the tile
|
|
that JUST completed instead of the next one; that duplicate
|
|
fetch's own later completion then writes into the NEXT tile's
|
|
SRAM slot using the WRONG (duplicated) source data. Same bug
|
|
CLASS as ERR-0006 (gating solely on a signal with its own
|
|
same-cycle side effect).
|
|
FIX: added `&& !pf_done` to the restart condition.
|
|
DIAGNOSIS METHOD: cycle-by-cycle $display tracing (act_rd_en/
|
|
act_rd_addr/act_rd_data/wgt_rd_en/wgt_rd_addr/wgt_rd_data/
|
|
operand_valid/input_data/weight_data, and separately mm_state/
|
|
np_state/result_valid/result_ready/result_data) added temporarily to
|
|
tb_nms_dataflow_core.v, gated by $time ranges around the specific
|
|
job windows of interest, removed once each bug was isolated and
|
|
fixed. Item 2 was only found AFTER item 1's fix, once a new
|
|
n_tiles=4 test (added specifically because every prior test used
|
|
n_tiles=1 and could not have exercised this path) was written and
|
|
initially failed.
|
|
VERIFICATION: after both fixes, tb_nms_dataflow_core.v passes 7/7
|
|
(same DAG-dependency shape as tb_dataflow_core.v's own M7 test;
|
|
shared-x_base broadcast-fill test across 2 concurrently-dispatched
|
|
slots; new n_tiles=4 multi-tile test) at N_SLOTS=2, bit-exact.
|
|
|
|
ERR-0014
|
|
timestamp: 2026-09-06T03:00:00Z
|
|
git_commit: 5c9ec61 (+ uncommitted NMS STEP9 work)
|
|
session: v2-NMS-STEP9-real-benchmark
|
|
module: hardware/v2/nms/rtl/nms_memory_manager.v,
|
|
hardware/v2/nms/rtl/nms_activation_fill_ctrl.v,
|
|
hardware/v2/nms/rtl/nms_dataflow_core.v
|
|
context: running a REAL D-Stress-scale benchmark (256 neurons, 16
|
|
tiles each -- exactly MAX_TILES) through the real V1 PSRAM chain for
|
|
STEP9. Every prior NMS test (tb_nms_dataflow_core.v, STEP8) used
|
|
n_tiles in {1,4}, never n_tiles==MAX_TILES exactly -- this one real,
|
|
necessary scale change (16-tile neurons, matching the project's own
|
|
realistic dense-layer workloads) exposed THREE compounding real bugs,
|
|
all instances of the SAME root cause, found via a real hang (0/8
|
|
neurons completed) traced cycle-by-cycle across three separate fix
|
|
attempts.
|
|
ROOT CAUSE (all three items): TIW=$clog2(MAX_TILES) sizes an SRAM
|
|
ADDRESS field (0..MAX_TILES-1), but several signals in this new
|
|
architecture are COUNTERS whose real value must reach MAX_TILES
|
|
itself (e.g. wgt_fetched/tile_idx/resident_count when a job's own
|
|
n_tiles equals MAX_TILES exactly) -- one bit short of what a counter
|
|
needs, though exactly right for an address.
|
|
1. nms_memory_manager.v: `n_tiles_reg[TIW-1:0]` truncated the 16-bit
|
|
n_tiles value (16) down to 4 bits, wrapping it to 0 -- every
|
|
"< n_tiles" check was permanently false, so wgt_fetched never even
|
|
STARTED advancing (stuck at 0 forever). Fixed: compare the full
|
|
16-bit n_tiles_reg directly (zero-extending the narrower counter),
|
|
never truncate n_tiles_reg itself.
|
|
2. Same modules: tile_idx/wgt_fetched/resident_count were themselves
|
|
declared TIW-bit (0..15 max), so even after fixing item 1's
|
|
comparison, incrementing wgt_fetched from 15 wrapped it back to 0
|
|
instead of reaching 16 -- an infinite re-fetch loop (real PSRAM
|
|
utilization visibly rose to 90%+ with zero forward progress).
|
|
Fixed: introduced a separate CNTW=$clog2(MAX_TILES+1) width for
|
|
these specific counters (one bit wider than TIW), keeping TIW only
|
|
for the SRAM address ports these counters occasionally drive
|
|
(explicit truncation at those specific assignment sites, always
|
|
safe since addressing only ever happens while the counter is
|
|
provably < MAX_TILES).
|
|
3. Introduced WHILE fixing item 1: `can_present`'s own second term was
|
|
mistakenly rewritten to compare wgt_fetched against n_tiles_reg
|
|
(checking "is the fetch done" instead of the actually-needed "is
|
|
THIS tile's weight available", i.e. tile_idx < wgt_fetched) -- once
|
|
wgt_fetched legitimately reached n_tiles (16), that comparison went
|
|
permanently false and deadlocked consumption of the LAST tile
|
|
forever even though every tile was genuinely ready (real activation
|
|
resident_count kept climbing normally the whole time, isolating
|
|
this as a logic error, not a data/timing problem).
|
|
4. Introduced WHILE fixing item 2: nms_dataflow_core.v's own top-level
|
|
wire connecting nms_activation_fill_ctrl.v's resident_count output
|
|
to nms_memory_manager.v's act_resident_count input was declared
|
|
`[TIW-1:0]` (4 bits), NOT widened to CNTW (5 bits) alongside the two
|
|
modules it connects -- a real value of 16 was silently truncated to
|
|
0 AT THE WIRE ITSELF, even though both endpoint modules' own
|
|
internal logic was by then already correct. Found by tracing
|
|
act_resident_count jumping from 15 directly to 0 (skipping 16)
|
|
exactly the cycle the real fetch completed pf_done=1.
|
|
DIAGNOSIS METHOD: cycle-by-cycle $display tracing added to a scaled-
|
|
down repro (8 neurons instead of 256, same real PSRAM model/n_tiles
|
|
=16) at each of 3 successive fix attempts, isolating each remaining
|
|
symptom (wgt_fetched stuck at 0 -> wgt_fetched wrapping 15->0
|
|
forever -> resident_count itself jumping 15->0 instead of reaching
|
|
16) before moving to the next. One self-inflicted tracing mistake
|
|
along the way (re-copying the clean testbench source over an
|
|
already-instrumented copy, silently producing zero debug output
|
|
until noticed and corrected) cost one extra iteration but found
|
|
nothing new.
|
|
VERIFICATION: after all four fixes, the scaled-down 8-neuron repro
|
|
passes bit-exact (6465 cycles), and the REAL, full-scale D-Stress
|
|
workload (256 neurons, 16 tiles each, through the real V1 PSRAM
|
|
chain) passes 256/256 bit-exact vs the golden model -- see EXP-0022.
|
|
|
|
ERR-0015
|
|
timestamp: 2026-09-05T23:41:08Z
|
|
module: hardware/v2/nms/rtl/weight_prefetch_engine.v
|
|
severity: CRITICAL (real, reproducible full deadlock -- 0/256 neurons
|
|
ever completed, watchdog timeout at 2,000,000 cycles, 0.0% PSRAM
|
|
utilization for the entire run)
|
|
found_during: STEP11's own real, full-scale N_SLOTS=1 D-Stress
|
|
PREFETCH_DISTANCE sweep (PFD in {1,2,4,8,16,32}) through
|
|
tb_nms_dstress_pf.v -- required by the STEP11 spec's own explicit
|
|
instruction not to assume PFD=32 is simply "unnecessary" but to
|
|
actually measure it ("larger [PFD], if necessary").
|
|
symptom: at PFD=32 (MAX_TILES=16, so CNTW=$clog2(MAX_TILES+1)=5 bits),
|
|
every one of the 256 D-Stress neurons hung forever -- the engine
|
|
never issued a single mem_req for the entire run.
|
|
root_cause: window_limit was computed as
|
|
`consumed_count + PREFETCH_DISTANCE[CNTW-1:0]` -- i.e. the
|
|
PREFETCH_DISTANCE PARAMETER ITSELF (not a tile counter, unlike every
|
|
prior instance of this bug class) was truncated down to CNTW bits
|
|
before the addition. For MAX_TILES=16, CNTW=5 bits (0..31), so
|
|
PFD=32 (6'b100000) truncates to 5'b00000 = 0. window_limit then
|
|
equals consumed_count exactly, forever, so `fetch_tile < window_limit`
|
|
is never true (fetch_tile >= consumed_count always holds by
|
|
construction) -- more_to_fetch is permanently false, and the engine
|
|
never fetches a single word.
|
|
Same ROOT CAUSE CLASS as ERR-0014 (a value that must be able to
|
|
reach/exceed a bound gets silently wrapped by a too-narrow field),
|
|
but this time the truncated value is a user-supplied, unbounded
|
|
build-time PARAMETER (PREFETCH_DISTANCE) rather than an internal
|
|
tile counter -- a genuinely new instance, not a recurrence of the
|
|
same fixed bug, since PREFETCH_DISTANCE did not exist before this
|
|
STEP.
|
|
fix: widened window_limit (and the two `fetch_tile [+1] < window_limit`
|
|
comparisons that use it) to a fixed 32-bit width, computed as
|
|
`{{(32-CNTW){1'b0}}, consumed_count} + PREFETCH_DISTANCE` (the
|
|
parameter itself, un-truncated) compared against a zero-extended
|
|
fetch_tile -- see weight_prefetch_engine.v's own updated comment at
|
|
the window_limit declaration.
|
|
verification: added a dedicated ERR-0015 regression case to
|
|
tb_weight_prefetch.v (`err0015_large_pfd_test`, gated `if (PFD>=32)`
|
|
so the file stays valid for every PFD it's compiled with) running a
|
|
full MAX_TILES=16 job at PFD=32 -- PASSES bit-exact post-fix
|
|
(confirmed: "PASS n_tiles=16 PFD=32 EXTRA_WAIT=0: ready_count=16,
|
|
all tiles bit-exact"). Re-ran the full tb_weight_prefetch.v suite at
|
|
PFD in {1,2,4,8,32}: 10/10 (9/9 at PFD=32, since the PFD/PFD+1 job-
|
|
size edge case is n/a once PFD>=MAX_TILES) tests PASS, 0 errors, at
|
|
every PFD tested.
|
|
secondary testbench-only issue found and fixed alongside (NOT an RTL
|
|
bug): tb_weight_prefetch.v's own `run_job(base, PFD, ...)` /
|
|
`run_job(base, PFD+1, ...)` edge-case calls and its windowing-cap
|
|
test's expected value both silently assumed PFD < MAX_TILES (asking
|
|
for a PFD+1-tile job when PFD+1 > MAX_TILES is asking the weight
|
|
SRAM to hold more tiles than it has, and the windowing cap can never
|
|
reach PFD once n_tiles/MAX_TILES itself is the tighter bound) --
|
|
fixed by gating those two run_job calls on `PFD < MAX_TILES` and
|
|
changing the windowing-cap expectation to `min(PFD, MAX_TILES)`.
|
|
impact_on_step11_conclusions: none -- PFD=32 was never the STEP11
|
|
benchmark's real focus (the D-Stress MAX_TILES=16 workload makes any
|
|
PFD>=16 equivalent to "unbounded"); this bug would only have
|
|
surfaced if a user tried MAX_TILES>32 with a correspondingly large
|
|
PFD. Fixed regardless, since a build-time parameter silently
|
|
deadlocking the whole design outside its narrow tested range is a
|
|
real correctness defect, not a documentation footnote.
|
|
|
|
ERR-0016
|
|
timestamp: 2026-09-06T06:08:18Z
|
|
module: hardware/v2/nms/rtl/sdram_controller.v
|
|
severity: CRITICAL (real, reproducible bank-open-forever condition;
|
|
every READ/WRITE after the first corrupted the next transaction)
|
|
found_during: STEP16 Phase 3 isolated regression (tb_sdram_controller.v,
|
|
BURST_LEN=4, CLK_FREQ_MHZ=166) -- first full run produced 0/460 tests
|
|
passed, 460 errors, with the real timing-checked sdram_model.v itself
|
|
flagging "ACTIVATE to bank 0 while already active" on nearly every
|
|
transaction.
|
|
symptom: every ACTIVATE after the very first one hit a real protocol
|
|
violation in the behavioral model (bank never precharged/closed by
|
|
the previous transaction), and every read returned corrupted/shifted
|
|
data as a side effect of the row/bank state being wrong.
|
|
root_cause: the READ/WRITE command's own address word was built as
|
|
`sdram_a <= {3'b000, 1'b1, req_col_reg}` with req_col_reg 8 bits wide
|
|
-- 3+1+8=12 bits total (correct width), but the concatenation placed
|
|
the auto-precharge '1' bit at position a[8], not a[10] where the
|
|
device's mode register / command decode actually expects it. A10 was
|
|
therefore always 0 on every real READ/WRITE -- auto-precharge was
|
|
silently never requested, so every opened row/bank stayed active
|
|
forever, and the next ACTIVATE (to a different row) hit a real
|
|
already-active bank.
|
|
found_by: static/hand trace of the FSM against the model's own
|
|
violation messages -- not assumed, root-caused from the actual
|
|
simulation log.
|
|
fix: `sdram_a <= {4'b0100, req_col_reg};` -- places the '1' at bit 10
|
|
exactly, with a[11]/a[9]/a[8] correctly tied 0.
|
|
verification: re-ran tb_sdram_controller.v (BURST_LEN=4, 166MHz) --
|
|
the "ACTIVATE while already active" violation disappeared completely;
|
|
see ERR-0017/ERR-0018 for the remaining data-corruption bugs found
|
|
after this fix, and EXP-0040 for the final clean 460/460 PASS run.
|
|
|
|
ERR-0017
|
|
timestamp: 2026-09-06T06:08:18Z
|
|
module: hardware/v2/nms/sim/sdram_model.v
|
|
severity: HIGH (every WRITE burst's first word was silently dropped)
|
|
found_during: STEP16 Phase 3 regression, same debugging session as
|
|
ERR-0016 (found immediately after fixing the A10 bug, since data
|
|
corruption persisted with the ACTIVATE violations gone).
|
|
symptom: after ERR-0016's fix, reads still came back shifted -- the
|
|
first data word of every burst was missing/zero, later words shifted
|
|
down by one burst position.
|
|
root_cause: the model captured write data into `mem[]` only from the
|
|
registered `wr_burst_active`-gated block, which does not become true
|
|
until the cycle AFTER the WRITE command is decoded. But real SDR
|
|
SDRAM presents the FIRST write word on DQ CONCURRENTLY with the
|
|
WRITE command itself, not one cycle later -- so the model's own
|
|
capture logic was structurally one cycle too late for word 0, and
|
|
silently never wrote it anywhere.
|
|
fix: in the `cmd_write` branch (command-decode cycle itself), capture
|
|
word 0 directly into `mem[ba,bank_row[ba],a[7:0]]` right there, and
|
|
initialize `wr_col`/`wr_remaining` to already point at the SECOND
|
|
word/column for the (unchanged) registered continuation logic to
|
|
handle words 1..N-1. Special-cased burst_len==1 (single-word write,
|
|
no continuation needed).
|
|
verification: re-ran the regression -- write side now matches the
|
|
controller's own per-cycle DQ timing exactly (confirmed by hand
|
|
cycle-trace before re-running); see ERR-0018 for the matching
|
|
read-side bug found next, and EXP-0040 for the final clean run.
|
|
|
|
ERR-0018
|
|
timestamp: 2026-09-06T06:08:18Z
|
|
module: hardware/v2/nms/sim/sdram_model.v
|
|
severity: HIGH (every READ burst arrived corrupted/shifted by two
|
|
cycles' worth of latency, in two separate compounding sub-bugs)
|
|
found_during: STEP16 Phase 3 regression, immediately after ERR-0017's
|
|
fix (write side alone did not clear the corruption -- read side had
|
|
its own, structurally identical class of bug, discovered by the same
|
|
cycle-exact hand trace method used for ERR-0016/ERR-0017).
|
|
symptom: reads still returned data shifted by 1-2 burst positions with
|
|
leading zeros, even though the underlying stored memory content was
|
|
now correct (per ERR-0017's fix).
|
|
root_cause (two distinct, compounding one-cycle-latency bugs):
|
|
(a) the model's read-side pipe insertion was gated by the registered
|
|
`rd_burst_active`, which -- exactly like ERR-0017's write-side bug --
|
|
becomes true only the cycle AFTER the READ command is decoded, adding
|
|
a spurious extra pipeline cycle before the first word ever entered
|
|
the CAS-latency shift register.
|
|
(b) independently, the model's DQ output was driven through its own
|
|
registered NBA stage (`dq_out <= rd_pipe[0]` inside the same
|
|
always block), adding ANOTHER cycle of latency beyond the shift
|
|
register's own CAS_LATENCY depth -- real SDR SDRAM's output register
|
|
is already accounted for inside the published CAS_LATENCY spec value,
|
|
so this was a redundant, uncounted-for extra stage unique to how the
|
|
model was built (a 16-entry shift array PLUS a separate registered
|
|
output buffer), not something the controller's own CAS_LATENCY-cycle
|
|
wait_cnt derivation could have anticipated.
|
|
fix: (a) mirrored the ERR-0017 write-side fix -- in the `cmd_read`
|
|
branch (command-decode cycle itself), insert word 0 directly into
|
|
`rd_pipe[cas_latency-1]` and initialize `rd_col`/`rd_remaining` to
|
|
already point at the second word for the registered continuation
|
|
logic (again special-cased burst_len==1). (b) converted `dq_out`/
|
|
`dq_out_en` from `reg` (NBA-driven) to `wire` (combinational,
|
|
`= rd_pipe[0]` / `= rd_valid_pipe[0]`), removing the extra output
|
|
register stage entirely.
|
|
verification: re-ran tb_sdram_controller.v (BURST_LEN=4, 166MHz) after
|
|
both fixes -- 460/460 tests PASS, 0 errors, bit-exact across every
|
|
Phase 3 scenario (A/B/E/F/H/I/G). See EXP-0040 for the full result.
|
|
|
|
ERR-0019
|
|
timestamp: 2026-09-06T06:08:18Z
|
|
module: hardware/v2/nms/rtl/sdram_controller.v
|
|
severity: CRITICAL (real, reproducible full deadlock -- caller waits
|
|
forever for `ready` that never comes)
|
|
found_during: STEP16 Phase 4 real 100/133/166MHz x BURST_LEN{1,4,8}
|
|
measurement sweep -- 8/9 configurations passed cleanly, but
|
|
BURST_LEN=1 @ CLK_FREQ_MHZ=133 hung indefinitely (confirmed via a
|
|
backgrounded run: >50s of continuous 100% CPU with zero output,
|
|
killed and root-caused rather than assumed to be a slow-but-finite
|
|
run).
|
|
symptom: tb_sdram_controller.v's do_transaction task deadlocks inside
|
|
`while (!ready) @(posedge clk);` for one specific transaction in the
|
|
400-iteration Test G sequence, at this one specific frequency/burst
|
|
combination (BURST_LEN=4 and BURST_LEN=8 at the same 133MHz did NOT
|
|
hang, confirmed by isolated re-run).
|
|
root_cause: this project's own established mem_req convention (per
|
|
weight_prefetch_engine.v's header comment) is a SINGLE-CYCLE req
|
|
pulse, not a held/latched signal. sdram_controller.v's S_IDLE state
|
|
gave periodic AUTO REFRESH unconditional priority over a same-cycle
|
|
`req` ("if (refresh_timer==0) ... else if (req) ..."), and did
|
|
nothing to capture the request's fields when refresh won -- so if a
|
|
test vector's req pulse happened to land on the exact cycle
|
|
refresh_timer reached 0, the request was silently discarded: the
|
|
caller's req had already gone low again on the next cycle (per the
|
|
single-pulse convention), the controller served the refresh with no
|
|
memory of the missed request, and no `ready` ever followed. This is
|
|
a general protocol race, not something inherent to 133MHz or
|
|
BURST_LEN=1 specifically -- it only reproduced there because that is
|
|
where this test sequence's absolute cycle counts happened to align
|
|
the two events; the SAME race is latent at any frequency/burst
|
|
length whenever a real caller's timing lines up unluckily.
|
|
fix: added a `req_pending` register that latches a fresh req's
|
|
wr/bank/row/col/wdata fields in S_IDLE the instant req is seen,
|
|
regardless of whether refresh wins arbitration that same cycle.
|
|
Added `eff_wr/eff_bank/eff_row/eff_col/eff_wdata` combinational
|
|
wires (`req ? <live input> : <latched req_*_reg>`) so the one real
|
|
S_IDLE branch that starts a transaction handles both "req arriving
|
|
fresh this cycle" and "req deferred behind a just-finished refresh"
|
|
uniformly, without duplicating the ACTIVATE-issuing code path.
|
|
`req_pending` resets to 0 under `rst`.
|
|
verification: re-ran the full 3x3 (100/133/166MHz x BURST_LEN 1/4/8)
|
|
sweep after the fix -- all 9 configurations now PASS 460/460, 0
|
|
errors, including the previously-hanging BURST_LEN=1/133MHz case
|
|
(re-confirmed finishing in <1s wall time, not merely "eventually").
|
|
See EXP-0041 for the full sweep result and the real measured
|
|
cycles/transaction used for Phase 4's throughput calculations.
|
|
|
|
ERR-0020
|
|
timestamp: 2026-09-06T06:08:18Z
|
|
module: hardware/v2/nms/rtl/sdram_controller.v
|
|
severity: CRITICAL (real, reproducible full-system deadlock -- 28/256
|
|
neurons never completed, both slots of an N=2 run permanently
|
|
starved; the isolated single-requester regression NEVER exercised
|
|
this because it always issues req from S_IDLE, never overlapping a
|
|
real arbiter's own outstanding-transaction bookkeeping)
|
|
found_during: STEP16 Phase 5 real FPGA-Neural integration (tb_nms_
|
|
dstress_sdram.v, N_SLOTS_CFG=2, D-Stress 256-neuron workload) -- the
|
|
SAME workload at N_SLOTS_CFG=4 passed cleanly (49430 cycles, 256/256
|
|
bit-exact), but N_SLOTS_CFG=2 hung at 228/256 neurons after 2,000,000
|
|
watchdog cycles (40x more cycles than N=4's entire run took to
|
|
finish all 256).
|
|
symptom: added temporary hierarchical debug tracing (both slots' own
|
|
nms_memory_manager_stream_wide state, plus the SDRAM controller's and
|
|
slot_mem_arbiter_wide's own internal state) to the watchdog's
|
|
periodic $display. Trace showed: both slots permanently stuck at
|
|
tile_idx=0/wgt_ready_count=0 (ST_RUN, waiting forever for the first
|
|
weight tile), while simultaneously sdram_controller.v itself sat
|
|
completely idle (req=0, busy=0, state=S_IDLE, req_pending=0) AND
|
|
slot_mem_arbiter_wide.v's own `owner` register was permanently
|
|
locked non-zero (a grant it believed was still in flight) with a
|
|
SECOND port's request stuck in `pending`, unable to ever be granted
|
|
because the arbiter was waiting for an `m_ready` that would never
|
|
come from a controller that had already forgotten the request.
|
|
root_cause: ERR-0019's fix only latched a fresh `req` into
|
|
`req_pending` from WITHIN the S_IDLE case-statement branch. The real
|
|
arbiter (slot_mem_arbiter_wide.v) pulses its own `m_req` for exactly
|
|
one cycle the instant it grants a new owner -- which can be ANY
|
|
cycle, including one where sdram_controller.v happens to be mid-
|
|
refresh (S_REFRESH_WAIT) or finishing a PREVIOUS transaction's own
|
|
PRECHARGE_WAIT tail, i.e. NOT executing the S_IDLE branch that
|
|
cycle. In that case the old fix's own req-latch code never even ran,
|
|
so the request was silently dropped exactly as it was before
|
|
ERR-0019's fix -- just for a DIFFERENT interfering state (a real
|
|
case the isolated regression's own single-requester testbench could
|
|
never reach, since it always waits for `busy==0` -- itself only
|
|
cleared in S_IDLE -- before ever asserting req, so its req pulses
|
|
are always S_IDLE-synchronous by construction).
|
|
fix: moved the req-latching code out of the S_IDLE branch entirely and
|
|
into the always-executed section that runs every cycle regardless of
|
|
`state` (alongside the existing unconditional refresh_timer
|
|
decrement) -- so a req pulse arriving during ANY state is captured
|
|
into req_pending/req_*_reg, not just one arriving while already in
|
|
S_IDLE. S_IDLE's own servicing logic (`req || req_pending`, `eff_*`
|
|
wires) is unchanged.
|
|
verification: re-ran the full Phase 3 isolated regression (9/9
|
|
frequency/burst-length configurations, 460/460 tests each) --
|
|
unaffected, still 100% PASS, confirming the fix is a pure superset
|
|
(catches a strictly larger set of req-arrival cycles, changes no
|
|
existing passing behavior). Re-ran the STEP16 Phase 5 N=2 D-Stress
|
|
benchmark -- now PASSES cleanly, 256/256 neurons bit-exact, 52161
|
|
total cycles (previously hung indefinitely). Re-ran N=4 -- unchanged,
|
|
49430 cycles (confirms the fix does not alter already-correct
|
|
behavior, only recovers previously-lost requests). See EXP-0042 for
|
|
the full Phase 5 N=2/N=4 integration results.
|
|
|
|
ERR-0021
|
|
timestamp: 2026-09-06T10:22:22Z
|
|
module: hardware/v2/rtl/dependency_manager.v
|
|
severity: HIGH (real, reproducible bit-exact regression -- a node
|
|
dispatched twice; caught before being kept, via a standalone
|
|
regression, NOT shipped)
|
|
found_during: STEP17 Part A's own "minimum fix experiment" -- an
|
|
attempt to pipeline the first_ready_idx/any_ready priority-encoder
|
|
scan (EXP-0044's own identified critical path) by one register
|
|
stage, to shorten the scan-to-array-read combinational chain that
|
|
limits N=4 Fmax to 81.55MHz.
|
|
symptom: after the fix, the full N=2/N=4 D-Stress regression showed
|
|
jobs_allocated=266 (N=2) / 268 (N=4), both > the expected 256 --
|
|
and roughly half the 256 neurons' results read back as 0 instead of
|
|
their real golden value. A minimal standalone unit test (16
|
|
independent always-ready nodes, `ready_ready` held high, isolated
|
|
from the rest of the system) reproduced the exact mechanism: node 0
|
|
is DISPATCHED TWICE, at cycles 20ns apart (exactly one extra pipeline
|
|
cycle).
|
|
root_cause: the scan (`first_ready_idx`/`any_ready`) is combinational
|
|
over node_state's PRE-edge value. On the exact cycle a dispatch
|
|
COMMITS (`ready_valid && ready_ready`, which schedules node_state
|
|
[ready_node_id] <= ST_DISPATCHED for the following cycle), the scan
|
|
THAT SAME cycle still sees the dispatching node as READY (its state
|
|
has not committed yet) -- an unconditional one-cycle register of the
|
|
scan's output would capture this still-READY snapshot, and one cycle
|
|
later (once node_state has actually committed and ready_valid has
|
|
dropped), the dispatch branch would fire again using the now-stale
|
|
snapshot, re-dispatching the just-dispatched node. A first attempted
|
|
repair -- gating the register update by `!ready_valid` -- did NOT fix
|
|
it: traced further (via the same standalone unit test) and found the
|
|
gate blocks the register from updating during the COMMIT cycle
|
|
itself (correct), but the register had ALREADY captured the stale
|
|
"node is READY" snapshot ONE CYCLE EARLIER (at the cycle the dispatch
|
|
was merely INITIATED, before ready_valid even reads 1), and that
|
|
stale value survives frozen through the commit cycle (blocked by the
|
|
gate) and is consumed immediately after, still stale by exactly one
|
|
cycle relative to the real commit.
|
|
fix: NONE SHIPPED. The change was reverted in full (`git checkout --
|
|
hardware/v2/rtl/dependency_manager.v`), restoring the exact STEP16-
|
|
validated RTL. Re-confirmed via the full N=2/N=4 D-Stress regression
|
|
that reverting restores the original correct behavior (256/256
|
|
bit-exact, 49430/52161 cycles, matching STEP16 exactly).
|
|
impact: none on the shipped design -- caught entirely within STEP17's
|
|
own investigation, before any P&R or benchmark result relying on the
|
|
broken version was reported as final. Documented here so a future
|
|
attempt at pipelining this scan does not repeat the same two failed
|
|
approaches; a fully correct version (if attempted again) will need
|
|
to reconcile the snapshot-capture cycle with the commit cycle more
|
|
carefully than a simple `!ready_valid` gate -- e.g. by explicitly
|
|
invalidating the registered snapshot when its own named node is the
|
|
one just committed, not merely suppressing updates during the commit
|
|
cycle.
|
|
|
|
ERR-0022
|
|
timestamp: 2026-09-06T12:30:00Z
|
|
module: hardware/v2/nms/rtl/sdram_weight_backend_pack128.v (STEP18,
|
|
first draft)
|
|
severity: HIGH (real, measured throughput REGRESSION -- not a crash or
|
|
data-corruption bug, but a real performance defect that would have
|
|
shipped as a "fix" if not benchmarked end-to-end before accepting it)
|
|
found_during: STEP18 Part C/D "weight packing" experiment -- a new
|
|
BURST_LEN=8 (128-bit/16-byte, 2-tile) SDRAM weight-fetch wrapper,
|
|
designed to halve the number of real SDRAM transactions needed per
|
|
P8 weight tile by caching the "other half" of each 128-bit fetch for
|
|
the next sequential request.
|
|
symptom: isolated unit-test regression (tb_sdram_weight_backend_pack128
|
|
.v, single requester, sequential access) passed 20/20, showing the
|
|
intended ~50% cache-hit rate (10 real fetches @16 cycles + 10 cache
|
|
hits @1 cycle for 20 requests). But the FULL N=4 D-Stress integration
|
|
benchmark got WORSE, not better: 74,004 cycles vs the STEP16 baseline
|
|
's 49,430 (+49.7%), despite remaining bit-exact correct.
|
|
root_cause: the first draft used a SINGLE cache entry. In the real
|
|
system, N_SLOTS=4 independent memory managers share this ONE
|
|
physical backend through slot_mem_arbiter_wide.v, which interleaves
|
|
their requests round-robin. A single cache entry holding slot A's
|
|
own "other half" gets overwritten by slot B's own "other half" (a
|
|
DIFFERENT address) before slot A's memory manager ever comes back
|
|
around to request its own paired tile -- so nearly EVERY access
|
|
became a real 16-cycle miss instead of the intended ~50% instant
|
|
hit, and 16 cycles/miss is WORSE than the STEP16 baseline's 10-cycle
|
|
BURST_LEN=4 transactions. This was found by actually running the
|
|
full integrated benchmark, not by trusting the isolated single-
|
|
requester unit test's own PASS result -- exactly the class of gap
|
|
the governing spec's own "this hypothesis MUST be tested, not
|
|
assumed" instruction anticipates.
|
|
fix: widened the cache from 1 entry to N_ENTRIES (parameter, sized to
|
|
the real N_SLOTS in the system, default 4) with fully-associative
|
|
address-tag lookup and simple round-robin eviction. This is SAFE
|
|
regardless of sizing accuracy (an evicted-too-early entry only costs
|
|
an extra real fetch -- a performance effect, never incorrect data,
|
|
since a cache MISS always falls back to a real, address-exact
|
|
fetch) and bounded correctly by construction: each slot has at most
|
|
one outstanding request at a time (the existing memory manager's own
|
|
design), so at most N_SLOTS "pending other halves" can exist
|
|
simultaneously in real traffic -- never more than N_ENTRIES=N_SLOTS.
|
|
verification: re-ran the full N=2/N=4 D-Stress regression -- now a
|
|
REAL improvement: N=4 44,935 cycles (-9.1% vs the 49,430 baseline),
|
|
N=2 47,399 cycles (-9.1% vs the 52,161 baseline), both 256/256
|
|
bit-exact. See EXP-0046 for the full before/after comparison.
|
|
|
|
ERR-0023
|
|
timestamp: 2026-09-06T11:26:46Z
|
|
module: hardware/v2/nms/rtl/sdram_unified_backend.v (STEP19, new module)
|
|
severity: CRITICAL (real, reproducible full-system deadlock -- 0/256
|
|
neurons completed; found immediately on the first real N=4 D-Stress
|
|
integration run, before being shipped)
|
|
found_during: STEP19 -- consolidating the two previously-separate
|
|
physical memory paths (weight-fetch SDRAM + activation/result PSRAM)
|
|
into ONE physical SDRAM chip serving all three traffic classes
|
|
through a single sdram_controller.v instance, per this step's own
|
|
explicit "single external SDRAM only" mandate.
|
|
symptom: the isolated unit regression (tb_sdram_unified_backend.v,
|
|
single requester on each of the W and AR ports, no contention)
|
|
passed 40/40. The first full N=4 D-Stress integration run deadlocked
|
|
completely: jobs_allocated stuck at 12, tiles_delivered=0,
|
|
neurons_completed=0/256, watchdog timeout at 2,000,000 cycles, AR
|
|
arbiter-side utilization pegged at 100% (permanently stuck).
|
|
root_cause: this is the SAME bug class as ERR-0019/ERR-0020 in sdram_
|
|
controller.v itself, recurring one layer up. sdram_unified_backend.v
|
|
's own top-level S_IDLE arbitration between its W (weight) and AR
|
|
(activation/result) ports only checked the LIVE w_req/ar_req signals
|
|
-- a single-cycle req pulse (this project's own established
|
|
convention) arriving on a cycle this backend happened to be busy
|
|
servicing the OTHER port was silently dropped, and the caller
|
|
(slot_mem_arbiter.v or slot_mem_arbiter_wide.v) waited forever for a
|
|
`ready` that would never come. Under real N=4 contention (weight
|
|
prefetch traffic and the very first activation-fill read racing for
|
|
the same physical port from the start), this raced and deadlocked
|
|
immediately -- the isolated single-requester test structurally could
|
|
not reach this case (no contention exists there).
|
|
fix (round 1): added req_pending latches for both W and AR ports,
|
|
mirroring sdram_controller.v's own corrected ERR-0020 fix (latch
|
|
unconditionally, every cycle, regardless of current state). This
|
|
introduced a SECOND, different real bug (see below) before the full
|
|
fix was correct.
|
|
fix (round 2, the actual bug this ID documents): the unconditional
|
|
latch fires whenever w_req is seen, INCLUDING the cycle a request is
|
|
ALSO being fully serviced via the W-port's own single-cycle cache-hit
|
|
fast path (w_cache_hit, reused from sdram_weight_backend_pack128.v's
|
|
own STEP18 caching logic) -- that fast path does not go through the
|
|
same-cycle "new transaction" branch and therefore never clears the
|
|
just-set w_req_pending latch. The result: after every cache-hit
|
|
service, w_req_pending is left spuriously set, and on the FOLLOWING
|
|
idle cycle the backend issues a bogus extra real SDRAM fetch using
|
|
the STALE latched address from the already-completed request --
|
|
which then delivers ITS OWN (stale, wrong) data as if it were the
|
|
response to the NEXT genuinely new request, shifting every
|
|
subsequent W-port response by one position. Found via the isolated
|
|
regression itself regressing from 40/40 to 26/40 after round-1's own
|
|
fix, with a clear "off by one iteration" failure signature (got ==
|
|
previous iteration's own expected value) -- traced by hand-cycle
|
|
tracing the interaction between the unconditional latch and the
|
|
case-statement's own NBA ordering (last assignment in the same
|
|
always block, same time step, wins).
|
|
fix (final): the w_cache_hit branch now explicitly clears
|
|
w_req_pending itself (`w_req_pending <= 1'b0`), overriding the
|
|
spurious set from the SAME cycle's unconditional latch (textually
|
|
later in the same always block, so it wins per standard Verilog NBA
|
|
ordering) -- cancelling the latch precisely when (and only when) the
|
|
SAME cycle's request was ALREADY fully handled via the fast path.
|
|
verification: re-ran the isolated unit regression (40/40 PASS,
|
|
restored) and the full N=4 AND N=2 D-Stress integration benchmark --
|
|
both now PASS bit-exact (256/256 neurons), with real, sustained
|
|
multi-hundred-refresh-interval operation and zero deadlock/timeout/
|
|
dropped/duplicated results. See EXP-0048 for the full integration
|
|
results.
|
|
|
|
ERR-0024 (Icarus Verilog v13.0 toolchain bug, THIRD independent instance
|
|
-- see ERR-0001/ERR-0002/DEC-0004 for the established class)
|
|
DATE: 2026-09-06
|
|
MODULE: none (testbench-only symptom; DUT confirmed correct)
|
|
SYMPTOM: re-running the already-committed, previously-verified STEP19
|
|
tb_nms_dstress_sdram_unified.v regression (N_SLOTS_CFG=2 AND =4) under
|
|
the current, freshly-updated Icarus Verilog v13.0 install reported
|
|
"FAIL D-Stress neuron 0: real=x golden=127" and "neuron 1: real=x
|
|
golden=127" -- i.e. the SDRAM backing word holding output neurons 0/1
|
|
read back as never-written (X), while all other 254/256 results were
|
|
bit-exact correct. Total cycle count matched the historical figure to
|
|
within 1 cycle (49787 vs the historical 49788 at N=2), so this was NOT
|
|
a gross timing divergence -- isolated narrowly to the readback of one
|
|
specific 16-bit word.
|
|
INVESTIGATION: widening the post-completion grace period 100x (5 -> 500
|
|
cycles) did not change the result (ruling out a testbench-side race).
|
|
Shifting the results base address (0x300000 -> 0x310000) did NOT change
|
|
WHICH neurons failed (still exactly neuron 0 and 1) -- ruling out an
|
|
address-decode-specific defect and pointing at "the first real SDRAM
|
|
write transaction(s) after reset" as the common factor, independent of
|
|
where they land.
|
|
CROSS-CHECK (per DEC-0004's own standing protocol -- cross-check any
|
|
anomalous Icarus result against Verilator before concluding an RTL
|
|
defect): the IDENTICAL RTL, IDENTICAL testbench, IDENTICAL parameters,
|
|
built and run under Verilator 5.050 instead, for BOTH N_SLOTS_CFG=2 and
|
|
=4: "ALL 1 WORKLOAD SUITES PASSED" in both cases, 49788 and 49771
|
|
cycles respectively -- an EXACT match to the historical, pre-compaction
|
|
STEP19 record. CONFIRMS this is a fourth-ish real, reproducible Icarus
|
|
Verilog v13.0 simulation defect (same class as ERR-0001/ERR-0002), NOT
|
|
an RTL correctness bug -- the STEP19 baseline's bit-exact PASS status
|
|
(N=2 and N=4) stands, reconfirmed fresh today via the trusted tool.
|
|
STATUS: NOT reported upstream (out of scope). Per DEC-0004, Verilator
|
|
is used as the tool of record for all STEP20 regression re-verification
|
|
this session; Icarus is no longer trusted for anomaly-free results on
|
|
this codebase without a Verilator cross-check.
|
|
|
|
ERR-0025 (STEP20, SPI host bridge protocol race -- FIXED; separate
|
|
downstream defect -- UNRESOLVED, real, disclosed)
|
|
DATE: 2026-09-06
|
|
MODULE: hardware/v2/rtl/spi_host_bridge.v (fixed); downstream location
|
|
NOT YET root-caused (see below).
|
|
PART A -- FIXED: spi_host_bridge.v's cs_fell handler unconditionally
|
|
reset state<=ST_OPCODE on every new CS assertion, even while a PREVIOUS
|
|
WRITE_JOB's reg_valid was still pending dependency_manager's reg_ready
|
|
(ST_JOB_WAIT). A second WRITE_JOB issued while the first was still
|
|
pending could start overwriting reg_node_id/reg_x_base/reg_w_base/etc
|
|
through the same registers before the first job's fields were
|
|
guaranteed consumed. Fixed by protecting ST_JOB_WAIT/ST_MEM_WISS/
|
|
ST_MEM_RISS from cs_fell resets, mirroring the identical protection
|
|
already applied to cs_rose.
|
|
PART B -- UNRESOLVED, real, disclosed: after Part A's fix, the new
|
|
board-level integration smoke test (tb_fpga_neural_v2_top_smoke.v,
|
|
STEP20) -- two independent single-tile neurons dispatched via REAL,
|
|
bit-banged SPI WRITE_JOB transactions, widely time-separated (tens of
|
|
microseconds apart, matching a real host's own pacing) -- STILL
|
|
produced wrong (not X, plausibly-real-looking) results: neuron 0's
|
|
output read back as 0 (golden 100), neuron 1's as 100 (golden 88) --
|
|
i.e. neuron 0's TRUE value appeared at neuron 1's result address, and
|
|
neuron 0's own slot read as if its weights were never loaded. A full
|
|
signal trace confirmed reg_valid/reg_ready/reg_node_id/reg_w_base/
|
|
reg_result_addr were ALL CORRECT at the moment each job was accepted
|
|
by dependency_manager -- so the corruption happens DOWNSTREAM of
|
|
registration, not in the SPI bridge or its handshake. The already-
|
|
verified STEP19 tb_nms_dstress_sdram_unified.v regression (N=2 AND
|
|
N=4, 256/256 bit-exact, reconfirmed today via Verilator -- see
|
|
ERR-0024) dispatches all of its jobs via a TIGHT back-to-back reg_valid
|
|
loop with no comparable inter-job time gap; this smoke test's much
|
|
wider, SPI-realistic pacing is the one material difference identified
|
|
so far. Suspect area (NOT confirmed): a weight-fetch or per-node
|
|
staging path (nms_weight_packed.v / weight_prefetch_engine_wide.v /
|
|
nms_activation_fill_ctrl_v3.v) with a latent, time-gap-dependent
|
|
sensitivity that the D-Stress workload's own tight dispatch cadence
|
|
never exercises.
|
|
STATUS: UNRESOLVED. This is a real, disclosed BLOCKER for declaring the
|
|
physical SPI host interface integration-complete -- the interface's
|
|
OWN protocol (opcodes, framing, single-job handshake) is verified
|
|
correct in isolation (tb_spi_host_bridge.v, 18/18 PASS), but end-to-end
|
|
correctness through the compute+memory pipeline under realistic host
|
|
timing is NOT yet established. Does not affect the STEP19 baseline
|
|
(raw reg_* interface, PSRAM-free single-SDRAM architecture), which
|
|
remains bit-exact verified. Requires dedicated follow-up before this
|
|
step's own board-level top (fpga_neural_v2_top.v) can be considered
|
|
hardware-release-ready.
|
|
|
|
ERR-0025 Part B -- RESOLUTION (STEP20, ERR-0025 Part B closed)
|
|
DATE: 2026-09-06
|
|
ROOT CAUSE: nms_weight_packed.v and nms_activation_replicated.v both
|
|
used a REGISTERED read port (`rd_data_reg <= mem[addr]`, gated by
|
|
rd_en -- a full clock cycle of latency from address/enable to valid
|
|
output), but nms_memory_manager_stream_wide.v's own pipelined read-
|
|
ahead consumer (`rd_pending`) is designed around a COMBINATIONAL read
|
|
(issue this cycle -> already-valid data captured next cycle). Traced
|
|
via a full internal signal walk (dependency_manager -> neural_director
|
|
-> per-slot memory_manager -> weight_prefetch_engine_wide/nms_weight_
|
|
packed -> nms_activation_fill_ctrl_v3/nms_activation_replicated):
|
|
job registration, slot dispatch, and per-job context capture (w_base_
|
|
reg/result_addr_reg) were ALL confirmed correct at every stage: the
|
|
corruption traced all the way down to the exact cycle where `buf_
|
|
weight`/`buf_input` are captured, which used the PRE-edge (stale)
|
|
value of the SRAM's own registered output -- one real cycle before
|
|
that register's own update (from the read issued the SAME cycle)
|
|
actually committed. A busy, multi-tile job (e.g. STEP19's D-Stress,
|
|
128 inputs/16 tiles per neuron) never exposes this: its own weight/
|
|
activation prefetch runs far enough ahead (PREFETCH_DISTANCE=8) that
|
|
by the time any given tile is actually consumed, that tile's data has
|
|
already been sitting stable in the SRAM for many cycles, masking the
|
|
extra latency completely. An uncontested single-tile job (STEP20's own
|
|
board-level SPI smoke test, tb_fpga_neural_v2_top_smoke.v) has ZERO
|
|
such margin: its one-and-only tile's read fires on the exact edge the
|
|
data nominally becomes "ready", landing squarely on the missing cycle
|
|
-- permanently latching stale/zero weight and activation data. This
|
|
explains BOTH observed symptoms exactly: neuron 0 (first job, first
|
|
real weight fetch of the whole session) read back 0 (all-zero SRAM
|
|
reset content); neuron 1 (second job, reusing the same physical slot)
|
|
read back 100 -- neuron 0's own TRUE value, one full cycle "behind"
|
|
where it should have been.
|
|
FIX: made both SRAMs' read ports combinational (`assign rd_data = mem
|
|
[addr]`, replacing the registered `always @(posedge clk) rd_data_reg
|
|
<= mem[addr]`), with an explicit same-cycle fill/read address-match
|
|
bypass (forwarding fill_data directly) for the one hazard a plain
|
|
combinational read alone would still miss -- a fill and a read to the
|
|
IDENTICAL address landing on the IDENTICAL edge, where mem[] itself
|
|
would not yet reflect that same-edge write. Files changed: hardware/
|
|
v2/nms/rtl/nms_weight_packed.v, hardware/v2/nms/rtl/nms_activation_
|
|
replicated.v. No change to any FSM, arbiter, SDRAM controller, or
|
|
dependency-tracking logic -- this is a pure, minimal, two-file SRAM
|
|
read-timing fix.
|
|
VERIFICATION (all via Verilator, the trusted tool per DEC-0004):
|
|
- tb_fpga_neural_v2_top_smoke.v: 11/11 PASS -- single job alone,
|
|
two jobs back-to-back, two jobs with a ~85us realistic SPI-paced
|
|
gap, and a parametric sweep of inter-job gaps (100ns/5000ns/
|
|
50000ns), all bit-exact.
|
|
- tb_nms_dstress_sdram_unified.v N=2: 49788 cycles, 256/256
|
|
bit-exact -- IDENTICAL cycle count to before this fix (zero
|
|
regression).
|
|
- tb_nms_dstress_sdram_unified.v N=4: 49771 cycles, 256/256
|
|
bit-exact -- IDENTICAL cycle count to before this fix (zero
|
|
regression).
|
|
- tb_sdram_unified_backend.v: 40/40 PASS (unaffected, unrelated file).
|
|
- tb_spi_host_bridge.v: 18/18 PASS (unaffected, confirms Part A's own
|
|
fix still holds).
|
|
STATUS: ERR-0025 (Parts A and B) fully RESOLVED. The physical SPI host
|
|
interface is now verified correct end-to-end (SPI -> dependency_
|
|
manager -> compute -> SDRAM -> result) under both tight and realistic-
|
|
gap job pacing, with zero regression to the STEP19 baseline.
|
|
|
|
ERR-0026 -- T_MRD datasheet-unit mismatch (SDRAM init sequence, 64MHz)
|
|
|
|
DATE: 2026-09-06
|
|
FOUND DURING: PRE-PCB VERIFICATION FREEZE, section-8 SDRAM datasheet-
|
|
level audit (real Alliance Memory AS4C4M16SA-6TIN datasheet, Rev 5.0,
|
|
Oct 2018, Table 17 "Electrical Characteristics/AC Operating
|
|
Conditions", -6 speed grade).
|
|
ROOT CAUSE: hardware/v2/nms/rtl/sdram_controller.v modeled tMRD (LOAD
|
|
MODE REGISTER -> any command spacing) as `ns_to_cycles(12)`, i.e. as
|
|
if it were a nanosecond-based timing spec like tRCD/tRP. The real
|
|
datasheet specifies tMRD as a FIXED CYCLE COUNT, "2 tCK", independent
|
|
of clock frequency -- the same category of spec as CAS_LATENCY, which
|
|
this same file already correctly models as a fixed value two lines
|
|
below. `ns_to_cycles(12)` rounds to exactly 2 cycles at every
|
|
frequency this design had previously been tested at (100/133/166MHz),
|
|
so the wrong unit model was silently masked by coincidence. At the
|
|
real V2 board's own 64MHz operating point it rounds to only 1 cycle --
|
|
one cycle short of the real, fixed 2-tCK minimum -- a genuine,
|
|
datasheet-violating under-provisioning of the one-time SDRAM power-up/
|
|
mode-register-set sequence. Confirmed NOT a bug in tRCD/tRP (correctly
|
|
ns-based, correctly use ns_to_cycles()), tRAS (satisfied by construction:
|
|
the fixed tRCD+CAS_LATENCY+BURST_LEN dispatch sequence is always >=6
|
|
cycles, 93.75ns >= the real 42ns minimum at 64MHz), or tWR (folded in
|
|
conservatively via T_RP+1, giving 3 cycles >= the real 2-tCK minimum).
|
|
FIX: hardcoded `localparam T_MRD = 2;` (matching how CAS_LATENCY is
|
|
already modeled), replacing `localparam T_MRD = ns_to_cycles(12);`.
|
|
Only affects the one-time SDRAM init sequence, not per-transaction
|
|
timing.
|
|
VERIFICATION (all via Verilator, the trusted tool per DEC-0004):
|
|
- tb_sdram_controller.v: full 9-config legacy sweep (100/133/166MHz
|
|
x BURST_LEN 1/4/8), 461/461 PASS each config, zero regression.
|
|
- tb_sdram_controller.v: NEW dedicated 64MHz/BURST_LEN=4 config (the
|
|
real board target, not covered by the legacy sweep) -- 461/461
|
|
PASS, confirming the fix is correct at the frequency where the
|
|
bug actually manifested.
|
|
- tb_nms_dstress_sdram_unified.v N=2: 49788 cycles, 256/256
|
|
bit-exact -- IDENTICAL to pre-fix (expected: T_MRD only affects
|
|
one-time init, not steady-state per-transaction timing).
|
|
- tb_nms_dstress_sdram_unified.v N=4: 49771 cycles, 256/256
|
|
bit-exact -- IDENTICAL to pre-fix.
|
|
- tb_fpga_neural_v2_top_smoke.v: 11/11 PASS, unaffected.
|
|
STATUS: RESOLVED. Files changed: hardware/v2/nms/rtl/sdram_
|
|
controller.v (single localparam, plus explanatory comment).
|
|
|
|
ERR-0027 -- neural_director.v: runtime-indexed demux write regressed
|
|
timing after the ADDR_WIDTH memory-upgrade widening
|
|
|
|
DATE: 2026-09-07
|
|
FOUND DURING: memory-capacity upgrade investigation (AS4C4M16SA-6TIN
|
|
8MB -> AS4C32M16SA/SB-7TIN 64MB), real nextpnr-ecp5 P&R re-verification
|
|
after ADDR_WIDTH grew from 23 to 26 bits.
|
|
ROOT CAUSE: neural_director.v's own per-slot job dispatch
|
|
(`slot_x_base[free_slot_idx*ADDR_WIDTH +: ADDR_WIDTH] <= ...`, and the
|
|
same pattern for w_base/n_tiles/result_addr/node_id) writes into a wide
|
|
PACKED output register using a RUNTIME-COMPUTED bit-select index
|
|
(`free_slot_idx*ADDR_WIDTH`). Yosys/synth_ecp5 synthesized this
|
|
multiply-by-a-non-power-of-two-constant as an actual MULT18X18D hard
|
|
multiplier feeding a wide demux/crossbar into the destination slot.
|
|
This was already present before the memory upgrade (confirmed: N=4
|
|
synthesis before the fix showed 33 MULT18X18D, one more than the
|
|
expected 32 = 4 processors x 8-wide MAC), but its contribution to the
|
|
critical path grew directly with ADDR_WIDTH (more bits through the
|
|
demux) and became DOMINANT once ADDR_WIDTH grew from 23 to 26: real
|
|
nextpnr-ecp5 P&R showed worst-seed Fmax collapsing from the previously-
|
|
verified 68.51MHz (ADDR_WIDTH=23, PRE_PCB_VERIFICATION.md section 5)
|
|
to 40.27MHz (ADDR_WIDTH=26), FAILING the 64MHz target across all 8
|
|
seeds tested.
|
|
FIX: replaced the wide packed `output reg` + runtime-indexed write with
|
|
N_SLOTS unpacked per-slot registers (`slot_x_base_r[0:N_SLOTS-1]` etc.)
|
|
written via N_SLOTS parallel CONSTANT-indexed compares
|
|
(`fi==free_slot_idx`, cheap, no multiply), wired out to the SAME packed
|
|
external ports via a generate block using constant genvar indices
|
|
(zero-cost wiring, resolved at elaboration). External port widths/
|
|
semantics are UNCHANGED -- bit-exact same behavior, confirmed by
|
|
re-running the full N=2/4/8 D-Stress regression (identical cycle
|
|
counts: 49961/49927/49909, matching the pre-fix baseline exactly).
|
|
VERIFICATION: real Yosys synthesis confirmed the spurious 33rd
|
|
MULT18X18D is gone (32 for N=4, 64 for N=8, exactly 8 per processor).
|
|
Real nextpnr-ecp5 P&R, N=4, 8 seeds: ALL PASS at 64MHz (65.02-72.01MHz,
|
|
mean ~68.8MHz) -- back to approximately the pre-upgrade baseline range.
|
|
STATUS: RESOLVED for N=4. File changed: hardware/v2/rtl/neural_director.v.
|
|
|
|
ERR-0028 -- nms_activation_fill_ctrl_v3.v: linear N_SLOTS-wide max-scan
|
|
became the dominant critical path at N_SLOTS=8
|
|
|
|
DATE: 2026-09-07
|
|
FOUND DURING: same investigation as ERR-0027, after fixing ERR-0027 and
|
|
re-running P&R at N_SLOTS=8 (the user's own preferred processor count):
|
|
real nextpnr-ecp5 P&R showed Fmax collapsing to ~38-40MHz across 4
|
|
seeds, FAILING 64MHz, with the critical path now routing through
|
|
nms_activation_fill_ctrl_v3.v's own `max_n_tiles_comb` computation --
|
|
a flat, sequential N_SLOTS-wide scan (`for (j=0;j<N_SLOTS;j=j+1) if
|
|
(n_tiles_masked[j] > max_n_tiles_comb) max_n_tiles_comb =
|
|
n_tiles_masked[j];`), already flagged as "an N_SLOTS-wide sequential
|
|
chain" by this file's OWN prior header comment -- a pre-existing,
|
|
known characteristic (not newly introduced), but one whose carry-chain
|
|
critical path scales linearly with N_SLOTS and became dominant at
|
|
N_SLOTS=8 (twice the comparison depth of N_SLOTS=4, where it was not
|
|
the bottleneck).
|
|
FIX: replaced the flat scan with an explicit, hand-written balanced
|
|
binary max-tree (log2(N_SLOTS) comparison levels instead of N_SLOTS),
|
|
using distinctly-named per-level wires (NOT a multi-dimensional
|
|
generate-indexed array -- a first attempt using a shared 2D array
|
|
triggered a real simulator UNOPTFLAT "circular combinational logic"
|
|
false-positive, since that tool's array-flattening circularity check
|
|
could not prove the (acyclic) per-level dependency safe for a shared
|
|
array). Explicit cases for N_SLOTS=1/2/4/8 (the only real
|
|
configurations this project uses), with a safe (unoptimized) fallback
|
|
for any other value. SAME single-cycle latency as the scan it replaces
|
|
(max_n_tiles_reg is still registered exactly one cycle behind
|
|
n_tiles_masked) -- no FSM timing change, purely a combinational-depth
|
|
reduction. Confirmed bit-exact via the same N=2/4/8 D-Stress regression
|
|
re-run (identical cycle counts, zero regression).
|
|
VERIFICATION: real nextpnr-ecp5 P&R, N=8, 8 seeds: 5/8 PASS at 64MHz
|
|
(65.27-70.78MHz), 3/8 FAIL narrowly (55.84/61.00/63.42MHz) -- a large
|
|
improvement over the pre-fix ~38-40MHz, but NOT yet fully reliable
|
|
across every placement seed at N_SLOTS=8 (unlike N_SLOTS=4, which
|
|
passes all 8 tested seeds). The remaining bottleneck (confirmed via a
|
|
fresh critical-path trace on a failing seed) is sdram_unified_backend.
|
|
v's own weight-cache hit-index scan -- the SAME pre-existing critical-
|
|
path class already documented in PRE_PCB_VERIFICATION.md section 5
|
|
("alternates... between dependency_manager.v's own priority-encoder
|
|
scan and sdram_unified_backend.v's own weight-cache hit-index logic"),
|
|
not a new defect, but not further optimized this session (time/scope
|
|
boundary -- a similar tree-based fix is the clear next step, not
|
|
attempted here to avoid rushing an unverified third change).
|
|
STATUS: PARTIALLY RESOLVED. N_SLOTS=4 fully reliable at 64MHz (8/8
|
|
seeds). N_SLOTS=8 significantly improved but OPEN -- 5/8 seeds close
|
|
timing at 64MHz, 3/8 do not. File changed: hardware/v2/nms/rtl/
|
|
nms_activation_fill_ctrl_v3.v.
|
|
|
|
NOTE (2026-09-07, re-measurement): a fresh, full 8-seed P&R matrix run
|
|
under ERR-0029's own investigation (below) found N_SLOTS=8 @ 64MHz
|
|
PRE-FIX pass count to actually be 4/8 (seeds 2,3,5,6 PASS; 0,1,4,7
|
|
FAIL; worst seed1 55.84MHz/WNS=-2.284ns), not the 5/8 recorded above --
|
|
the "5/8" figure came from an earlier, less exhaustive P&R pass and is
|
|
SUPERSEDED by this note and by MEMORY_UPGRADE_64MB_N8.md's own tables.
|
|
This does not change ERR-0028's fix or its STATUS, only the precise
|
|
pass-count evidence.
|
|
|
|
ERR-0029 -- sdram_unified_backend.v: serial "last-match-wins" weight-
|
|
cache hit-index scan was the N_SLOTS=8 @ 64MHz critical path
|
|
|
|
DATE: 2026-09-07
|
|
FOUND DURING: the user's own mandated exhaustive final-verification
|
|
pass (all 8 seeds, 4 configs: N=4/N=8 x 64/80MHz) plus an explicit,
|
|
user-directed root-cause investigation into the N_SLOTS=8 @ 64MHz
|
|
worst-case critical path (real nextpnr-ecp5 P&R report.json, seed1:
|
|
Fmax=55.84MHz, WNS=-2.284ns, total delay 17.909ns, 84% routing).
|
|
ROOT CAUSE: `sdram_unified_backend.v`'s W-port cache (W_ENTRIES=4,
|
|
NOT parameterized by N_SLOTS -- confirmed by inspecting its only
|
|
instantiation, in nms_neural_multiprocessor_sdram_unified.v, which
|
|
overrides neither W_ENTRIES nor ADDR_WIDTH's dependents) computed its
|
|
hit index with a sequential `for (ei=0;ei<W_ENTRIES;ei=ei+1) if
|
|
(w_cache_valid[ei] && w_cache_addr[ei]==w_addr) ... w_hit_idx_c=ei;`
|
|
loop -- "last valid+matching entry wins" by unconditional overwrite,
|
|
with each iteration's result depending on the previous one. Yosys/
|
|
nextpnr-ecp5 synthesized this as a serially-dependent cascade of
|
|
PFUMX/OFX ECP5 fast-mux primitives (confirmed via the real critical-
|
|
path segment dump: w_cache_valid[2] -> nested PFUMX/OFX chain ->
|
|
w_hit_idx_c -> further LUT cascade -> control logic gating w_rdata's
|
|
own clock enable). Because W_ENTRIES stays fixed at 4 regardless of
|
|
N_SLOTS, this logic's OWN complexity does not grow with N_SLOTS -- the
|
|
real cause of the N=8-specific slowdown is that N=8's larger overall
|
|
design (double the slot_mem_arbiter_wide/weight_prefetch_engine_wide
|
|
instances of N=4) forces nextpnr to spread placement further across
|
|
the die, stretching the SAME serially-dependent logic chain over
|
|
longer physical routes: 3 of 5 major hops measured at 2.5-2.8ns each
|
|
(typical local ECP5 routing is 0.3-0.9ns), confirming routing
|
|
distance, not logic complexity or LUT/FF utilization (25.2%/25.4% at
|
|
N=8, far from the device ceiling), as the dominant cost -- exactly as
|
|
diagnosed from the real segment data before any fix was attempted.
|
|
FIX: replaced the serial priority-scan with a flat, fully parallel
|
|
one-hot compare (`w_match_oh[ei] = w_cache_valid[ei] && w_cache_addr
|
|
[ei]==w_addr`, W_ENTRIES independent comparators, no inter-entry
|
|
dependency, via a `generate`/`genvar` block) followed by a single-
|
|
level `casez` priority encode reproducing the EXACT original "highest
|
|
index wins" semantics bit-for-bit. Guarded behind `if (W_ENTRIES==4)`
|
|
(the only real, instantiated configuration in this project) with a
|
|
fallback branch preserving the original scan for any other W_ENTRIES
|
|
value, matching this project's own established fix pattern (ERR-0028).
|
|
Purely combinational, same-cycle, zero latency change -- w_ready/
|
|
w_rdata timing behavior is externally unchanged.
|
|
VERIFICATION (bit-exact): isolated `tb_sdram_unified_backend.v`, real
|
|
Verilator 5.050, 40/40 PASS, identical to pre-fix. Full D-Stress
|
|
regression (real Verilator, golden-model comparison), both N_SLOTS=4
|
|
(49927 cycles, 256/256 neurons bit-exact) and N_SLOTS=8 (49909 cycles,
|
|
256/256 neurons bit-exact) -- PASS, zero functional regression.
|
|
VERIFICATION (timing, real nextpnr-ecp5, fresh 32-seed matrix: N=4/N=8
|
|
x 64/80MHz, 8 seeds each, POST-FIX):
|
|
N=4 @ 64MHz: 8/8 seeds PASS (unchanged pass count), but worst-case
|
|
margin DROPPED: Fmax 70.68/66.58/74.74/71.98/74.48/75.65/67.41/
|
|
68.47 MHz (worst seed1 66.58MHz, WNS=+0.605ns -- down from the
|
|
PRE-FIX worst of 74.17MHz/WNS=+2.143ns). Real cause: the FIX moved
|
|
the N=4 critical path OFF the (now-shortened) hit-index chain and
|
|
onto a DIFFERENT, previously-second-worst path inside the same
|
|
module (mostly u_arbiter_wide.m_addr -> u_sdram_backend.state/
|
|
ctrl_wdata), which this specific placement seed happens to route
|
|
less favorably than the original hit-index path did. Still a real,
|
|
honest 8/8 PASS -- not a hardware concern by the mandate's own
|
|
numeric acceptance criteria -- but a disclosed, measured trade-off,
|
|
not hidden.
|
|
N=8 @ 64MHz: PASS COUNT IMPROVED, 4/8 -> 5/8 (seeds 0,1,3,4,5 PASS;
|
|
2,6,7 FAIL). Worst-case improved: Fmax 66.45/65.28/61.21/66.96/
|
|
66.66/65.71/60.12/62.70 MHz (worst seed6: 60.12MHz, WNS=-1.009ns --
|
|
up from the PRE-FIX worst of 55.84MHz/WNS=-2.284ns). Real,
|
|
measured improvement, not yet full closure.
|
|
N=4/N=8 @ 80MHz: unchanged conclusion, still 0/8 PASS both configs
|
|
(identical achieved-Fmax values to the 64MHz-labeled netlists,
|
|
confirming the fabric's achievable ceiling, not the PLL target, is
|
|
the limit) -- NO-GO, as before.
|
|
STATUS: N_SLOTS=8 measurably improved (not fully closed: 3/8 seeds
|
|
still fail at 64MHz, now failing through a DIFFERENT critical path than
|
|
before -- see MEMORY_UPGRADE_64MB_N8.md for the updated worst-case
|
|
trace). N_SLOTS=4 remains 8/8 PASS at 64MHz but with reduced margin,
|
|
disclosed above -- flagged, not hidden, per this project's own real-
|
|
data-only reporting standard. File changed: hardware/v2/nms/rtl/
|
|
sdram_unified_backend.v.
|
|
|
|
ERR-0030 -- constraints/v2_unified.lpf's FREQUENCY PORT "clk" directive
|
|
was stale at 80 MHz, silently mislabeling every real-pin-constrained
|
|
nextpnr PASS/FAIL result against the wrong bar
|
|
|
|
DATE: 2026-09-15
|
|
CONTEXT: new brief to close N=8 timing, retarget to LFE5U-85F, and
|
|
sweep 4/8/16 processors x 1/2 SDRAM banks. Phase 0 baseline: re-ran
|
|
nms_neural_multiprocessor_sdram_unified (N_SLOTS=4/8) through the real
|
|
existing flow. First attempt used `--lpf-allow-unconstrained` with NO
|
|
`--lpf` file at all (no real pin locations) -- achieved Fmax came out
|
|
implausibly high (N=4 worst-seed 86.86MHz, mean ~100MHz across 8
|
|
seeds) versus the last real production baseline (DEC-0042: N=4
|
|
worst-seed 64.55MHz). Re-ran the same N=4 netlist WITH the real
|
|
`constraints/v2_unified.lpf` (real H5/B4 clk/rst balls, real SDRAM
|
|
balls) -- achieved Fmax dropped to a much more plausible 70.03MHz
|
|
(seed0), confirming free/unconstrained IO placement was the source of
|
|
the gap, not an RTL difference.
|
|
ROOT CAUSE (real, found while investigating): `v2_unified.lpf` line 35
|
|
still reads `FREQUENCY PORT "clk" 80 MHZ` from its original STEP19
|
|
freeze -- never updated when the project's achievable/target bar moved
|
|
to 64MHz (DEC-0038 onward). Passing `--freq 64` on the nextpnr-ecp5
|
|
command line does NOT override this: nextpnr uses the LPF's own
|
|
explicit per-port FREQUENCY constraint for any port that has one, and
|
|
`--freq` only supplies a default for ports left unconstrained. Real
|
|
effect verified: with the LPF loaded, nextpnr-ecp5 reported "70.03 MHz
|
|
(FAIL at 80.00 MHz)" even with `--freq 64` passed on the command line --
|
|
the achieved-MHz number itself was real, but the PASS/FAIL label next
|
|
to it was being judged against the wrong, stale 80MHz bar the whole
|
|
time this LPF has been used for any 64MHz-era measurement.
|
|
FIX: updated `constraints/v2_unified.lpf` line 35 to `FREQUENCY PORT
|
|
"clk" 64 MHZ`, with a comment explaining why. This does not touch any
|
|
physical pin/ball assignment (SDRAM/clk/rst LOCATE COMP lines are
|
|
unchanged) and has zero effect on RTL/datapath semantics -- it only
|
|
corrects which frequency nextpnr-ecp5 optimizes/labels against.
|
|
OPEN QUESTION (disclosed, not resolved here): it is not established
|
|
whether earlier 64MHz-labeled results in decisions.log (e.g. DEC-0042)
|
|
were themselves generated WITH this same stale-80MHz LPF (in which
|
|
case their own PASS/FAIL labels would have the same defect, though
|
|
their raw achieved-MHz numbers would remain valid and comparable) or
|
|
via a different invocation (e.g. no LPF, matching this session's first,
|
|
discarded attempt). Re-verify against a fresh N=4 8-seed sweep with the
|
|
corrected LPF before treating DEC-0042's specific pass-count (8/8) as
|
|
still authoritative under real pin constraints.
|
|
|
|
ERR-0031 -- Phase 0 baseline was being measured against the wrong
|
|
top-level module (nms_neural_multiprocessor_sdram_unified.v, an
|
|
obsolete testbench-only wrapper), giving falsely-optimistic Fmax
|
|
|
|
DATE: 2026-09-15
|
|
CONTEXT: same brief as ERR-0030. After fixing the stale 80MHz LPF
|
|
constraint, re-ran the N=4 8-seed sweep against
|
|
`nms_neural_multiprocessor_sdram_unified.v` + `constraints/
|
|
v2_unified.lpf`: 8/8 PASS at 64MHz, worst-seed 84.91MHz, mean 92.65MHz
|
|
-- still implausibly high, HIGHER than DEC-0042's own best-case seed
|
|
(77.21MHz), not just its worst.
|
|
ROOT CAUSE (found by re-reading decisions.log DEC-0038 entry, which
|
|
explicitly documents this): `nms_neural_multiprocessor_sdram_unified.v`
|
|
is NOT instantiated by the real board-level top. Per decisions.log's
|
|
own prior finding (search "NOT instantiated by the real board-level
|
|
top"), `fpga_neural_v2_top.v` instantiates `nms_dataflow_core_sdram`
|
|
and `sdram_unified_backend` directly, bypassing this wrapper entirely.
|
|
The wrapper is only still used by one testbench
|
|
(`tb_nms_dstress_sdram_unified.v`) as a bit-exact regression oracle --
|
|
it was never the real synthesis target. `constraints/v2_unified.lpf`
|
|
itself says as much in its own superseding file's header
|
|
(`v2_board_top.lpf`: "This supersedes ... v2_unified.lpf"), which this
|
|
session missed on the first two attempts.
|
|
The real board top adds an EHXPLLL PLL (16MHz osc -> 64MHz internal),
|
|
a reset_sync POR synchronizer, a real spi_host_bridge (replacing the
|
|
wrapper's raw 110-pin reg_* testbench bus), and a SECOND arbitration
|
|
level (host SPI raw-memory port vs. compute AR stream) -- all real
|
|
logic and real routing congestion the wrapper never has, which is
|
|
exactly why the wrapper measured faster.
|
|
FIX: re-target synthesis+P&R to `hardware/v2/nms/rtl/
|
|
fpga_neural_v2_top.v` (chparam N_SLOTS) with `hardware/v2/constraints/
|
|
v2_board_top.lpf` (all 17 real ports ball-assigned; PLL's own
|
|
FREQUENCY_PIN_CLKI=16/FREQUENCY_PIN_CLKOP=64 attributes read
|
|
automatically by nextpnr-ecp5 for the internal clock domain, no manual
|
|
--freq needed). Sanity check, single seed: 72.23MHz achieved, PASS at
|
|
64MHz -- squarely inside DEC-0042/ERR-0029's real historical range
|
|
(60-77MHz), confirming this is now the right target.
|
|
LESSON: ERR-0030's fix (to v2_unified.lpf) was not wrong on its own
|
|
terms (the 80MHz value really was stale) but was applied to a file
|
|
that should not be used for this brief's baseline going forward --
|
|
`v2_unified.lpf` remains valid only for whoever still exercises the
|
|
`nms_neural_multiprocessor_sdram_unified.v` testbench wrapper, not for
|
|
any real hardware Fmax claim.
|
|
STATUS: Phase 0 N=4 baseline being re-measured now against the correct
|
|
target (fresh 8-seed sweep in progress). N=8 baseline deferred by user
|
|
request until N=4 is fully understood.
|