Implements M4: memory_manager.v (arbitration/buffering/forwarding/ latency hiding/double buffering, §12) + prefetch_engine.v (double-buffered tile fetch, §13), sitting on the REAL, UNMODIFIED V1 PSRAM backend chain (int8_memory_access.v -> memory_interface.v -> psram_controller.v, per §15's explicit mandate not to touch the controller). Verified fully end-to-end with Verilator: real neural_processor (M1) fed entirely by memory_manager, computing against PSRAM-resident X/W tiles (double-buffered prefetch across up to 5 tiles) and writing its result back to PSRAM -- checked via an independent PSRAM read-back, with poison bytes around the operand regions to catch addressing errors. 3/3 jobs pass (1/3/5-tile configurations). Three real RTL bugs found and fixed during integration (full diagnostic trail in errors.log ERR-0006): prefetch_engine had no single-in-flight-request discipline, letting a queued request corrupt the bank bookkeeping of a fetch already running; the fix's own !pf_busy guard had a one-cycle blind spot (pf_busy lags pf_start by a clock) that needed an explicit !pf_start term; and a state-based mux for the shared backend port was off by one cycle, silently dropping the PSRAM result write entirely. Real synthesis: 0 CHECK problems, 851 LUT4/789 FF/108 CCU2C/0 DSP (expected, no multiplication in this module). Real place&route (via a synthesis-only timing harness, needed for the same TRELLIS_IO pin- budget reason as M2's array): Fmax 165.86 MHz, PASS at 80MHz. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013xXuuRUWZScuo1DeYJxs3v
179 lines
10 KiB
Plaintext
179 lines
10 KiB
Plaintext
# V2 errors log -- solo append, mai troncato/sovrascritto (vedi README.md)
|
|
# Nessuna entry ancora -- popolato incrementalmente man mano che avanza lo sviluppo V2.
|
|
|
|
ERR-0001 (Icarus Verilog v13.0 toolchain bug, TASK/SCOPE-ENTRY DESYNC)
|
|
DATE: 2026-09-05
|
|
MODULE: hardware/v2/sim/tb_neural_processor.v (M1 testbench development)
|
|
SYMPTOM: a task (or any named `begin:label` block) whose FIRST executable
|
|
statement is a blocking assignment to a signal read by another module's
|
|
`always @(posedge clk)`, when the task/block is entered immediately after
|
|
a time-consuming statement in the caller with no intervening
|
|
`@(posedge clk)`, can make that FIRST assignment invisible to the DUT at
|
|
the very next clock edge (the DUT's own always block behaves as if the
|
|
signal never changed). Confirmed at V1's own frozen `neuron_parallel.v`
|
|
(unmodified, already certified) via a minimal 3-statement task
|
|
(`v1_start=1; @(posedge clk); v1_start=0;`) -- `busy` never asserted.
|
|
REPRODUCTION: /tmp/mt6.v-style repro (not committed, transient scratch
|
|
file) -- see conversation record for the exact minimal case.
|
|
DIAGNOSIS METHOD: bisected from the full dual-DUT testbench down to a
|
|
standalone ~15-line repro, ruling out RTL, port connections, and
|
|
operator precedence one at a time.
|
|
WORKAROUND: always begin such a task with an explicit `@(posedge clk);`
|
|
before its first assignment (matches the pre-existing convention in
|
|
hardware/v1/sim's own tasks, e.g. neuron_parallel_tb.v's run_neuron,
|
|
which is presumably why V1's own test suite was never affected).
|
|
STATUS: WORKAROUND APPLIED in hardware/v2/sim/tb_neural_processor.v's
|
|
run_case. NOT reported upstream (out of scope for this session). See
|
|
ERR-0004 for the broader consequence of this finding.
|
|
|
|
ERR-0002 (Icarus Verilog v13.0 toolchain bug, SPURIOUS CONDITION EVALUATION)
|
|
DATE: 2026-09-05
|
|
MODULE: hardware/v2/rtl/neural_processor.v (protocol-violation guard,
|
|
removed -- see decisions.log DEC-0003)
|
|
SYMPTOM: `if (operand_valid && !operand_ready && (state-is-one-of-four))`
|
|
inside `always @(posedge clk)` evaluated TRUE at an edge where
|
|
`operand_valid` was independently confirmed (via $display in the same
|
|
timestep, and via the testbench's own port-connected signal) to be 0.
|
|
Bisected term-by-term: even `if (operand_valid && !operand_ready)`
|
|
alone, and even `if (operand_valid)` alone with explicit `== 1'b1`
|
|
comparisons, still fired spuriously. Confirmed NOT a precedence issue
|
|
(parens are unambiguous) and NOT specific to this exact expression
|
|
shape (multiple simplified variants all reproduced it).
|
|
CROSS-CHECK: root-caused further via a minimal 2-state FSM
|
|
(`if (go) st<=B;`) with NO relation to the removed guard -- Icarus
|
|
failed to transition on an ODD-numbered testbench clock edge
|
|
(`repeat(3)` before the pulse) but succeeded on an EVEN-numbered one
|
|
(`repeat(4)`), reproduced identically with both `always #5 clk=~clk`
|
|
and `initial ... forever #5 clk=~clk` clock generators. VERILATOR
|
|
5.050 gives the CORRECT result for the same repro in both cases.
|
|
This suggests ERR-0001 and ERR-0002 are two symptoms of the same
|
|
underlying VVP scheduling defect (edge-count/thread-parity dependent),
|
|
not two unrelated bugs.
|
|
STATUS: the offending RTL block (protocol-violation detection) was
|
|
REMOVED rather than chased further -- see DEC-0003. Root cause not
|
|
fully isolated (documented honestly, not overclaimed).
|
|
|
|
ERR-0003 (real RTL bug in hardware/v2/rtl/neural_processor.v, FOUND AND FIXED)
|
|
DATE: 2026-09-05
|
|
MODULE: hardware/v2/rtl/neural_processor.v, stage 0 (input
|
|
alignment/register)
|
|
SYMPTOM: back-to-back single-tile jobs (and some multi-tile jobs)
|
|
produced result_data=0 instead of the correct value, while the
|
|
internal `y7` register (one stage upstream of the FSM's capture)
|
|
showed the CORRECT value one cycle later than `valid7` first asserted.
|
|
ROOT CAUSE: `last0 <= tile_last;` was unconditional, while
|
|
`valid0 <= operand_valid && operand_ready;` was correctly gated. A
|
|
master asserting `tile_last` before `operand_ready` rises (legal
|
|
valid-before-ready behavior) let a "last" tag propagate through the
|
|
pipeline (last1, last_tree[], last5, last6) with NO corresponding
|
|
valid tile behind it, arriving at stage 7 one cycle ahead of the
|
|
real valid/data pair and causing the FSM to capture a stale/wrong
|
|
`y7`.
|
|
EVIDENCE: isolated to a single-DUT, no-task, no-V1 repro
|
|
(hardware/v2/sim/tb_neural_processor.v run under Verilator, with a
|
|
cycle-by-cycle dump of valid5/last5/valid6/last6/valid7/y7) --
|
|
`last5=1` while `valid5=0` on the same cycle, confirmed the
|
|
desync's exact origin at stage 0.
|
|
FIX: `last0 <= (operand_valid && operand_ready) ? tile_last : 1'b0;`
|
|
-- last0 is now gated identically to valid0.
|
|
VERIFICATION: full 7-test bit-exact-vs-V1 regression
|
|
(hardware/v2/sim/tb_neural_processor.v under Verilator) -- 7/7 PASS
|
|
after the fix, including the back-to-back and single-tile-after-
|
|
multi-tile cases that exposed it.
|
|
STATUS: FIXED, verified.
|
|
|
|
ERR-0004 (methodology consequence of ERR-0001/ERR-0002)
|
|
DATE: 2026-09-05
|
|
NOTE: this session's V1 certification campaign (docs/validation/,
|
|
hardware/v1/docs/validation/) was verified exclusively with Icarus
|
|
Verilog v13.0, the ONLY simulator available on this machine at the
|
|
time. ERR-0001/ERR-0002 show that v13.0 has at least one real,
|
|
reproducible scheduling defect around clock-edge/task-entry timing.
|
|
V1's own testbenches were NOT observed to trigger it in this session
|
|
(V1's `run_neuron`-style tasks already begin with `@(posedge clk)`,
|
|
which incidentally avoids ERR-0001's trigger condition), and V1
|
|
remains frozen/untouched regardless. This is flagged here for
|
|
honesty, not to imply V1's certification is wrong -- re-verifying
|
|
the full V1 suite under Verilator was explicitly OUT OF SCOPE for
|
|
this V2-kickoff session (V1 is frozen, not to be touched) and was
|
|
not performed. See decisions.log DEC-0004.
|
|
STATUS: OPEN CAVEAT, not actioned in this session by design.
|
|
|
|
ERR-0005 (synthesis measurement artifact, WORKED AROUND, not an RTL bug)
|
|
DATE: 2026-09-05
|
|
MODULE: hardware/v2/rtl/neural_processor_array.v
|
|
SYMPTOM: synthesizing neural_processor_array as a bare top-level
|
|
module (every per-processor job/operand/result field exposed as a
|
|
real TRELLIS_IO pin) works at N_PROCESSORS=1 but fails place&route
|
|
at N_PROCESSORS=2 with "Unable to place cell ...$tr_io, no BELs
|
|
remaining to implement cell type 'TRELLIS_IO'".
|
|
ROOT CAUSE: not a logic/timing limit -- the LFE5U-45F-8BG381 package
|
|
has 245 TRELLIS_IO pins total; the array's wide per-processor buses
|
|
(input_data/weight_data alone are DATA_WIDTH*P_IN*N_PROCESSORS bits)
|
|
exceed that budget once N_PROCESSORS>=2, purely because these ports
|
|
have no on-chip consumer yet (the Memory Manager/M4 and Neural
|
|
Director/M5 that will drive them in the real system don't exist
|
|
yet).
|
|
WORKAROUND: hardware/v2/synthesis/harness_neural_processor_array.v --
|
|
a synthesis-only wrapper (NOT part of rtl/, not a functional
|
|
deliverable) that drives all wide buses from an internal free-
|
|
running LFSR and reduces outputs to a small checksum, keeping only
|
|
clk/rst/seed/checksum as real top-level pins. See its own header
|
|
comment and experiments.log EXP-0003 for the resulting real
|
|
resource/Fmax numbers.
|
|
STATUS: WORKED AROUND. Will become moot once M4/M5 exist and the array
|
|
is synthesized as part of a larger design with on-chip ports instead
|
|
of a bare top-level module.
|
|
|
|
ERR-0006 (real RTL bugs in hardware/v2/rtl/memory_manager.v, FOUND AND FIXED)
|
|
DATE: 2026-09-05
|
|
MODULE: hardware/v2/rtl/memory_manager.v
|
|
SYMPTOM: end-to-end M4 testbench (real V1 PSRAM backend + real M1
|
|
neural_processor) hung permanently partway through the first
|
|
multi-tile job -- bank_ready for the "current" bank never became 1,
|
|
even though the byte-level backend clearly kept completing read
|
|
transactions (observed via cycle-by-cycle hierarchical tracing of
|
|
memory_manager/prefetch_engine internal state).
|
|
ROOT CAUSES (three, found together during the same debugging session):
|
|
1. The tile-N+2 prefetch request, queued on tile-N's handoff, could
|
|
be issued (pf_start asserted) while prefetch_engine was STILL
|
|
mid-fetch for tile-N+1 -- there was no single-in-flight-request
|
|
discipline at all in the first draft. Fixed by adding a
|
|
single-entry pf_pending register: requests are queued, not
|
|
issued directly, and a dedicated rule launches the queued
|
|
request only once the engine reports free.
|
|
2. Even with that queue, pf_busy does not read 1 until the cycle
|
|
AFTER pf_start was first observed by prefetch_engine (its own
|
|
fetch_busy<=1 lags its own fetch_start sampling by one clock) --
|
|
checking only `!pf_busy` left a genuine one-cycle window where a
|
|
second queued request would fire on top of the one just
|
|
launched, silently overwriting pf_target_bank (and pf_x_addr/
|
|
pf_w_addr) for the fetch already in flight. The address corruption
|
|
was harmless (prefetch_engine had already latched the correct
|
|
address into its own state that same edge), but pf_target_bank
|
|
corruption meant the eventually-completed fetch's real data got
|
|
filed into the WRONG bank's bank_ready/bank_x/bank_w, permanently
|
|
starving the bank actually needed next. Fixed by gating the issue
|
|
rule on `!pf_busy && !pf_start` (the extra term closes exactly
|
|
this one-cycle window).
|
|
3. A combinational mux selecting between prefetch_engine's own
|
|
backend wires and the result-write-back FSM's wires was gated on
|
|
`state == MM_WRITE_RESULT`, but wr_mem_req (asserted while
|
|
state==MM_WRITE_RESULT) only becomes valid the FOLLOWING cycle,
|
|
i.e. while state==MM_DONE -- the mux therefore selected the wrong
|
|
source for the one cycle the write request pulse was actually
|
|
high, silently dropping the PSRAM write entirely. Fixed by
|
|
widening the mux's select condition to cover both states.
|
|
DIAGNOSIS METHOD: cycle-by-cycle hierarchical signal dumps (mm.state,
|
|
tile_idx, bank_ready, pf_busy, pf_pending, pf_target_bank,
|
|
prefetch_engine's own state) under Verilator, printed only on
|
|
signal-change to keep the trace readable, comparing against the
|
|
hand-derived expected sequence of events for a 3-tile job.
|
|
VERIFICATION: hardware/v2/sim/tb_memory_manager.v -- 3/3 tests PASS
|
|
after all three fixes, including a 5-tile job (steady-state
|
|
double-buffer swap across more than 2 tiles) and independent
|
|
PSRAM read-back of the written-back result byte (not just internal
|
|
signal inspection).
|
|
STATUS: FIXED, verified end-to-end with the real (unmodified) V1
|
|
PSRAM backend chain and a real M1 neural_processor.
|