From 932aec2490916fd85a23ae5fe204f521a0f02291 Mon Sep 17 00:00:00 2001 From: manvalan Date: Sun, 20 Sep 2026 22:54:52 +0200 Subject: [PATCH] feat: real first step of 4x4 hybrid systolic architecture (EXP-0089) Adds packed_pe.v (packed_slot.v's compute+activation-fetch+writeback subsystem, reusing ddr_prefetch_mgr.v/neural_processor_packed.v/ result_writeback.v completely unmodified, with its own private weight-fetch removed) and systolic_group.v (one real layer_prefetch_ ctrl.v+layer_weight_buffer.v+weight_tile_gather.v shared by 4x packed_pe.v via a real, barrier-synchronized broadcast bus). Real design choice confirmed with the user before writing any RTL (AskUserQuestion, concrete topology preview): shared-weight broadcast, not a literal PE-to-PE systolic shift register -- achieves the real, quantified rationale (4x reduction in redundant weight-fetch DDR3 traffic per group of 4 PEs) with much lower real risk than genuine inter-PE pipeline fill/drain. The real new design is the barrier: each PE's own tcnt is the join key against the group's broadcast tcnt, self-synchronizing regardless of which PE is momentarily ahead/behind (e.g. a real DDR3 row-switch stall on one PE's own activation fetch). Found and fixed a real bug during verification (not by inspection): the first full test run reported every result as undefined despite every control-flow signal tracing correctly -- root-caused via real signal tracing down to a 5-way test arbiter bus mis-sliced at the wrong slot offset (single-bit handshake buses happened to use a correct range and masked it from the control-flow trace; only the wide, byte-offset buses were wrong). Verified in isolation (tb_systolic_group.v, real Icarus xsim, real sdram_arbiter_n.v generalized to NUM_REQ=5 with zero changes): 8/8 PASS across 2 consecutive group jobs (exercising the barrier's own per-job reset path, not just cold start). Deliberately scoped to the isolated mechanism only, per this project's "one variable at a time" discipline -- Director/SPI job dispatch for group jobs, a real N=16 top-level, and real P&R are real, disclosed next steps, not done here. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01MUG92aM9m68TRc4rG55BcC --- docs/ARCHITECTURE_ANALYSIS.md | 61 +++--- hardware/v2/logs/experiments.log | 131 +++++++++++ hardware/v3/rtl/packed_pe.v | 322 +++++++++++++++++++++++++++ hardware/v3/rtl/systolic_group.v | 323 ++++++++++++++++++++++++++++ hardware/v3/sim/tb_systolic_group.v | 292 +++++++++++++++++++++++++ 5 files changed, 1104 insertions(+), 25 deletions(-) create mode 100644 hardware/v3/rtl/packed_pe.v create mode 100644 hardware/v3/rtl/systolic_group.v create mode 100644 hardware/v3/sim/tb_systolic_group.v diff --git a/docs/ARCHITECTURE_ANALYSIS.md b/docs/ARCHITECTURE_ANALYSIS.md index c60d366..384b002 100644 --- a/docs/ARCHITECTURE_ANALYSIS.md +++ b/docs/ARCHITECTURE_ANALYSIS.md @@ -659,12 +659,18 @@ specifically to document where/how it breaks rather than to succeed): --- -### 5.6 [EXPLORATORY — captured, not decided, not built] Hybrid systolic scaling: 4 groups × 4-PE weight-stationary chains +### 5.6 [First step DONE, EXP-0089] Hybrid systolic scaling: 4 groups × 4-PE weight-stationary chains -Captured from a 2026-09-20 brainstorming session (same continuation), before -any N=4/8/16 scaling work starts, so the direction isn't lost. **Nothing in -this subsection is implemented or committed to — it's a working hypothesis -for a future architecture, explicitly not yet an RTL task.** +Captured from a 2026-09-20 brainstorming session as a purely exploratory +idea; the same day, per the user's own explicit reprioritization, the real +open design question below (shared-weight broadcast vs. a literal PE-to-PE +systolic shift register) was resolved with the user directly (not guessed), +and the **first real, isolated step is now built and verified**: +`systolic_group.v` + `packed_pe.v` (EXP-0089) — one real group of 4 PEs +sharing a single broadcast weight fetch, real barrier-synchronized, real +xsim-verified (8/8 PASS across 2 consecutive group jobs). **Not yet done**: +Director/SPI-level job dispatch for group jobs, a real N=16 (4-group) +top-level, and any real P&R for this — see EXP-0089's own `next_action`. **The problem it targets**: plain N=16 independent cores (§5.5's own "documentary, expected to break" framing) means 16 independent DDR3 @@ -701,27 +707,32 @@ flagged explicitly as a projection): naive flat-N=16 baseline — a real, meaningful, but NOT a full solve by itself; still needs real measurement once anything is built. -**Real open questions, not resolved yet** (deliberately not designed -further until the current in-flight work — §5.4's channel widening, N=2/4/8 -real testing — lands first, per this project's own "one variable at a time" -discipline): -- Intra-chain dataflow RTL (result propagation between adjacent PEs, - pipeline drain/fill at chain boundaries) is a real, new design, not a - trivial extension — needs its own isolated verification before wiring - into anything real, same as every other module in this project. -- Result collection actually gets SIMPLER under this model versus flat - N=16 (4 chain-output events instead of 16 independent ones) — relevant - to §5.3's result-writeback engine, worth designing writeback with this - in mind rather than for flat N=16 if this direction is pursued. -- Arbitration simplifies too: 4 group-level requesters instead of 16, - though `sdram_arbiter_n.v`'s own NUM_REQ parameter already generalizes to - either case without changes. +**Real open question, RESOLVED (EXP-0089)**: intra-chain dataflow — the +user was asked directly (`AskUserQuestion`, concrete side-by-side preview +of both real topologies) rather than guessed, given how different their +real risk/complexity profiles are. Chosen: **shared-weight broadcast** (one +real weight fetch per group of 4 PEs, no inter-PE result/data propagation, +each PE computes its own independent activation positions in parallel) — +**not** a literal PE-to-PE systolic shift register. This achieves the real, +quantified rationale above (4× reduction in redundant weight-fetch DDR3 +traffic per group) without the real complexity/risk of genuine pipeline +fill/drain at chain boundaries, which the literal-systolic alternative +would have required. Real result-writeback stays exactly as §5.3 already +built it — `packed_pe.v` reuses `result_writeback.v` completely unmodified, +one real DDR3 write per PE, not a new "4 chain-output events" model (that +framing, from the original brainstorm, doesn't apply to the broadcast +design actually built). Arbitration: confirmed real — `sdram_arbiter_n.v`'s +own `NUM_REQ` parameter generalized to a 5-way arbiter (1 group weight-fetch ++ 4 PE activation/writeback) in `tb_systolic_group.v` with zero changes to +that module. -**Decision**: not decided. Revisit after §5.4 (32-bit widening) and the -real N=2/4/8 flat-core scaling tests produce real numbers — those numbers -will tell us whether flat scaling is "good enough" up to some N, making -this restructuring unnecessary, or whether the real congestion at N=8/16 -justifies it. +**Decision**: first real step DONE (EXP-0089) — build and verify the +isolated group mechanism before any Director/SPI/top-level integration, +per this project's own "one variable at a time" discipline. Full N=16 +integration (Director-level group job dispatch, SPI protocol extension, +real top-level, real P&R) is real, disclosed, deliberately deferred work, +not yet done — see EXP-0089's own `next_action` in +`hardware/v2/logs/experiments.log`. #### 5.6.1 [EXPLORATORY] Opportunistic BRAM cache for activation tiles diff --git a/hardware/v2/logs/experiments.log b/hardware/v2/logs/experiments.log index 4c642a4..11d01db 100644 --- a/hardware/v2/logs/experiments.log +++ b/hardware/v2/logs/experiments.log @@ -5948,3 +5948,134 @@ design confirmed by the user -- NOT a literal PE-to-PE systolic shift register -- verified in isolation before any Director/SPI-protocol integration, matching this project's own "one variable at a time" discipline). + +EXP-0089 -- real, isolated first step of the 4x4 hybrid systolic +architecture: shared-weight-broadcast group, built and verified +(2026-09-20, user's own explicit reprioritization: "Fai la parte +realmente mancante prima, il RESULT-WRITEBACK e poi implementa la 4x4 +sistolica"; real A/B design decision confirmed by the user before any +RTL was written: shared-weight BROADCAST, not a literal PE-to-PE +systolic shift register) + +CONTEXT: docs/ARCHITECTURE_ANALYSIS.md S5.6 captured this direction as +purely exploratory ("nothing in this subsection is implemented"), +explicitly flagging "intra-chain dataflow RTL... is a real, new design, +not a trivial extension" as an open question. Before writing any RTL, +the user was asked (AskUserQuestion, with a concrete side-by-side +preview of both real candidate topologies) to resolve that open +question directly: (A) one shared weight fetch per group of 4 PEs, +broadcast to all 4, each PE computing its own independent activation +positions in parallel -- achieves the doc's own quantified rationale +(4x reduction in redundant weight-fetch DDR3 traffic per group) with +much lower real risk; or (B) a literal PE-to-PE systolic shift +register (weight physically translating PE by PE, real pipeline +fill/drain at chain boundaries). User confirmed (A), explicitly noting +(B) can be revisited later if real data justifies it. + +DESIGN: two new modules. + - `packed_pe.v`: packed_slot.v's own compute+activation-fetch+ + writeback subsystem (ddr_prefetch_mgr.v, neural_processor_packed.v, + result_writeback.v -- all THREE reused completely unmodified), + with packed_slot.v's own private layer_prefetch_ctrl.v/layer_ + weight_buffer.v/weight_tile_gather.v REMOVED -- weight tile data + arrives via a real, group-broadcast interface instead + (group_tcnt/group_tile_data/group_tile_valid). + - `systolic_group.v`: ONE real layer_prefetch_ctrl.v + layer_weight_ + buffer.v + weight_tile_gather.v (group-level, shared by reference, + not duplicated -- identical instances to what packed_slot.v already + owned per-slot), driving 4x packed_pe.v. + +REAL SYNCHRONIZATION (the actual new design, not a trivial extension -- +confirmed by building and testing it, not asserted): each packed_pe.v's +own `tcnt` IS the join key -- a PE in S_TILEWAIT waits for +`group_tile_valid && (group_tcnt == tcnt)` before consuming, a real, +self-synchronizing comparison immune to which PE happens to be +momentarily ahead or behind (e.g. one PE's own activation fetch hit a +real DDR3 row switch the others didn't). systolic_group.v's own barrier +extends this project's established 2-source "S_TILEWAIT join" +discipline to 4 independent sources: it only advances to the next +tile's weight fetch once ALL 4 PEs have ack'd the current one +(`pe_acked`, individually latched, using a `pe_acked_next` combinational +fold-in to detect same-cycle acks without an extra latency cycle). A +similar accumulator (`pe_done_latch`) tracks all 4 PEs' own job_done +pulses, deliberately made UNCONDITIONAL (not state-gated) to correctly +catch a real race: a fast PE's own job_done can pulse the SAME cycle +the group's own barrier clears the LAST tile (i.e. the same cycle the +group transitions toward waiting for completion) -- a state-gated +accumulator would have silently dropped that pulse. + +REAL BUGS FOUND AND FIXED DURING DESIGN (before compiling, via re- +deriving against already-proven code, not guessed) AND DURING +VERIFICATION (via real signal tracing, not by inspection): + 1. First draft of packed_pe.v omitted the neural_processor_packed.v + job_valid/job_ready handshake's own real gating (packed_slot.v's + own proven S_JOBSTART state) -- would have raced operand_valid + against job acceptance. Caught by re-deriving against packed_ + slot.v's own real sequencing before ever compiling, not by + simulation. + 2. systolic_group.v's first draft indexed an expression directly + (`(tcnt + 16'd1)[BUFADDRW-1:0]`) -- not valid plain Verilog syntax + (only nets/regs, not arbitrary expressions, can be part-selected). + Fixed with an intermediate `tcnt_next` wire. + 3. A duplicate `pe_acked_pulse` wire declaration (real copy-paste + leftover from restructuring the generate block). + 4. THE real bug, found via real hierarchical signal tracing after the + first full test run reported EVERY result as `x` (undefined) even + though `job_done` fired and every control-flow signal traced + correctly (barrier, tcnt, state transitions all real and correct) + -- proving the bug was in DATA, not control flow. Traced down + through ddr_prefetch_mgr.v's own real, already-proven internals + (confirmed innocent) to the actual root cause: the ISOLATED + testbench's own 5-way arbiter wiring (1 group weight-fetch slot + + 4 PE slots) sliced the flattened `pe_ctrl_addr/wdata/wmask/rdata` + buses starting at the WRONG offset (`4*WIDTH`, i.e. slot 4, instead + of `1*WIDTH`, slot 1 -- slot 0 is the group's own weight-fetch). + The single-bit handshake buses (req/grant/ready/busy) happened to + use a correct `[4:1]` bit range and so masked the bug from the + control-flow trace entirely -- only the wide, byte-offset-computed + buses were wrong, which is exactly why data silently corrupted + while every handshake still looked correct. A real, honest example + of why "the control flow works" is not suffient evidence that "the + data path works" -- this project's own standing discipline (root- + cause via real signal tracing, verify don't assume) is what caught + it, not luck. + +VERIFICATION: new `tb_systolic_group.v`, real Icarus xsim, real +`burst_mem_model32.v` backend shared via a real `sdram_arbiter_n.v` +(NUM_REQ=5, confirmed its own NUM_REQ parameter already generalizes to +this shape with zero changes, per this project's own prior note). +Preloads 2 real layers' worth of weights + 16 real activation +positions, runs TWO consecutive group jobs (deliberately, to exercise +the barrier's own per-job reset path, not just a single cold-start). +**8/8 PASS (2 jobs x 4 PEs), 0 errors**, real, independently-reproduced +golden model (same formula convention as every other v3 testbench). +This verifies the shared-weight-broadcast mechanism itself is +functionally correct -- it does NOT re-verify result_writeback.v's own +real DDR3 read-after-write correctness, since packed_pe.v reuses that +module completely unmodified and it already has its own real +read-after-write verification (EXP-0088, tb_packed_slot.v). + +DECISION: real, verified first step. This is deliberately SCOPED to the +isolated group mechanism only, per this project's own "one variable at +a time" discipline (same precedent as act_tile_fetch.v EXP-0079 or +ddr_prefetch_mgr.v EXP-0083, both built and verified standalone well +before system-level integration). NOT yet integrated: Director-level +job dispatch (neural_director_packed.v only knows how to dispatch +2-position jobs to flat slots, not 4-position jobs to a group), +SPI/WRITE_JOB protocol support for group-level job submission, a real +N=16 (4 groups x 4 PEs) top-level module, and real P&R signoff for any +of this. These are real, disclosed, deliberately deferred next steps, +not overlooked. + +next_action: (1) real P&R for systolic_group.v in isolation (out-of- +context first, matching this project's own precedent for brand-new +modules before real system integration) to get a first real sense of +its area/timing cost. (2) Design the real Director/SPI protocol +extension needed for group-level job dispatch (a real, disclosed, +non-trivial addition -- WRITE_JOB's current payload only carries one +job's worth of x_base/w_base/result_addr/node_id, not 4). (3) Build a +real N=16 top-level (4x systolic_group.v) once (1) and (2) are real and +verified, with its own real P&R signoff. (4) Revisit the flat N=2/4/8/16 +core-count scaling tests (deferred by the user's own explicit +reprioritization this session) once there's a real basis for comparing +flat vs. grouped scaling with real numbers from both. diff --git a/hardware/v3/rtl/packed_pe.v b/hardware/v3/rtl/packed_pe.v new file mode 100644 index 0000000..9bcbc20 --- /dev/null +++ b/hardware/v3/rtl/packed_pe.v @@ -0,0 +1,322 @@ +`timescale 1ns/1ps + +// ============================================================ +// V3 -- packed_pe.v: real weight-sharing PE for a systolic_group.v +// (docs/ARCHITECTURE_ANALYSIS.md S5.6, EXP-0089), derived directly from +// packed_slot.v -- SAME activation-fetch (ddr_prefetch_mgr.v), compute +// (neural_processor_packed.v), and result-writeback (result_writeback.v) +// subsystems, unmodified. The ONLY real difference: this module does +// NOT own a layer_prefetch_ctrl.v/layer_weight_buffer.v/weight_tile_ +// gather.v of its own -- weight tile data is received via a real, +// broadcast, GROUP-shared interface instead (group_tcnt/group_tile_ +// data/group_tile_valid), since the whole point of grouping 4 PEs is +// ONE real weight fetch shared by all 4 (docs' own quantified +// rationale: DDR3 requester count reduction), not 4 independent ones. +// +// REAL SYNCHRONIZATION (the part that needed real design, not a +// trivial extension -- see docs S5.6's own "not a trivial extension" +// disclosure): this PE's own `tcnt` IS the join key. It waits in +// S_TILEWAIT for `group_tile_valid && (group_tcnt == tcnt)` before +// consuming -- a real, self-synchronizing comparison, not a bare +// pulse/level race. A PE that's briefly slower than its groupmates +// (e.g. its own activation fetch hit a real DDR3 row switch the others +// didn't) simply keeps waiting; when it finally reaches S_TILEWAIT for +// its own `tcnt`, the comparison is either already true (if the group +// had to wait for THIS PE, i.e. this PE IS the slow one) or becomes +// true the moment the group's own barrier (systolic_group.v, all 4 +// pe_tile_ack seen) lets it advance -- correct regardless of which PE +// is momentarily ahead or behind, no risk of double-consuming or +// skipping a tile. +// +// group_n_tiles (broadcast, held stable for the whole job -- all 4 PEs +// in a group process the SAME layer, same real weight-stationary +// premise this module's own name comes from) replaces packed_slot.v's +// own per-instance n_tiles input. +// ============================================================ +module packed_pe #( + parameter DATA_WIDTH = 8, + parameter P_IN = 8, + parameter ACC_WIDTH = 32, + parameter BURST_LEN = 8, + parameter ADDR_WIDTH = 26 +)( + input wire clk, + input wire rst, + + // ---- own per-PE job trigger: x_base_a/b (this PE's own two + // activation positions), result_addr_a/b, node_id_a/b -- w_base and + // n_tiles are NOT here, they're group-broadcast (see group_n_tiles + // below; w_base never reaches this module at all, only the group + // controller needs it). ---- + input wire job_start, + input wire [ADDR_WIDTH-1:0] x_base_a, + input wire [ADDR_WIDTH-1:0] x_base_b, + input wire [ADDR_WIDTH-1:0] result_addr_a, + input wire [ADDR_WIDTH-1:0] result_addr_b, + input wire [15:0] node_id_a, + input wire [15:0] node_id_b, + output reg job_done, // one-cycle pulse + + output reg signed [DATA_WIDTH-1:0] result_data_a, + output reg signed [DATA_WIDTH-1:0] result_data_b, + output reg [15:0] result_node_id_a, + output reg [15:0] result_node_id_b, + output reg [ADDR_WIDTH-1:0] result_addr_a_out, + output reg [ADDR_WIDTH-1:0] result_addr_b_out, + + // ---- group-broadcast weight interface (real, level-held-until- + // consumed, same discipline ddr_prefetch_mgr.v's own tile_valid + // already established) ---- + input wire [15:0] group_n_tiles, + input wire [15:0] group_tcnt, + input wire [DATA_WIDTH*P_IN-1:0] group_tile_data, + input wire group_tile_valid, + output reg pe_tile_ack, // one-shot pulse + + // ---- own activation-fetch + result-writeback arbiter port (still + // one per PE -- activation data is NOT shared across PEs, each PE + // computes different positions) ---- + output wire mem_active, + input wire mem_grant, + + output wire ctrl_req, + output wire ctrl_wr, + output wire [ADDR_WIDTH-2:0] ctrl_addr, + output wire [32*BURST_LEN-1:0] ctrl_wdata, + output wire [4*BURST_LEN-1:0] ctrl_wmask, + input wire [32*BURST_LEN-1:0] ctrl_rdata, + input wire ctrl_ready, + input wire ctrl_busy +); + localparam S_IDLE = 3'd0, + S_JOBSTART = 3'd1, + S_TILEWAIT = 3'd2, + S_OPERAND = 3'd3, + S_RESULT = 3'd4, + S_WRITEBACK = 3'd5; + + reg [2:0] state; + reg [ADDR_WIDTH-1:0] x_base_a_lat, x_base_b_lat; + reg [15:0] n_tiles_lat; + reg [ADDR_WIDTH-1:0] result_addr_a_lat, result_addr_b_lat; + reg [15:0] node_id_a_lat, node_id_b_lat; + reg [15:0] tcnt; + + // ---- ddr_prefetch_mgr.v: own activation-tile look-ahead, exactly + // as packed_slot.v already uses it ---- + reg ddrpf_job_start; + wire ddrpf_tile_valid; + reg ddrpf_tile_consume; + wire signed [DATA_WIDTH*P_IN-1:0] act_data_a_w, act_data_b_w; + wire act_mem_active; + + wire act_ctrl_req, act_ctrl_wr; + wire [ADDR_WIDTH-2:0] act_ctrl_addr; + wire [32*BURST_LEN-1:0] act_ctrl_wdata; + wire [4*BURST_LEN-1:0] act_ctrl_wmask; + + ddr_prefetch_mgr #( + .DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .BURST_LEN(BURST_LEN), .ADDR_WIDTH(ADDR_WIDTH-1) + ) u_ddrpf ( + .clk(clk), .rst(rst), + .job_start(ddrpf_job_start), + .base_a(x_base_a_lat[ADDR_WIDTH-2:0]), .base_b(x_base_b_lat[ADDR_WIDTH-2:0]), + .n_tiles(n_tiles_lat), + .tile_valid(ddrpf_tile_valid), .data_a(act_data_a_w), .data_b(act_data_b_w), + .tile_consume(ddrpf_tile_consume), + .mem_active(act_mem_active), .mem_grant(mem_grant), + .ctrl_req(act_ctrl_req), .ctrl_wr(act_ctrl_wr), .ctrl_addr(act_ctrl_addr), + .ctrl_wdata(act_ctrl_wdata), .ctrl_wmask(act_ctrl_wmask), + .ctrl_rdata(ctrl_rdata), .ctrl_ready(ctrl_ready), .ctrl_busy(ctrl_busy) + ); + + // ---- result_writeback.v: own result, own DDR3 write, exactly as + // packed_slot.v already uses it ---- + reg wb_start; + wire wb_busy, wb_done; + wire wb_mem_active; + + wire wb_ctrl_req, wb_ctrl_wr; + wire [ADDR_WIDTH-2:0] wb_ctrl_addr; + wire [32*BURST_LEN-1:0] wb_ctrl_wdata; + wire [4*BURST_LEN-1:0] wb_ctrl_wmask; + + result_writeback #( + .BURST_LEN(BURST_LEN), .DATA_WIDTH(DATA_WIDTH), + .JOB_ADDR_WIDTH(ADDR_WIDTH), .ADDR_WIDTH(ADDR_WIDTH-1) + ) u_wb ( + .clk(clk), .rst(rst), + .start(wb_start), + .result_addr_a(result_addr_a_lat), .result_addr_b(result_addr_b_lat), + .result_data_a(result_data_a), .result_data_b(result_data_b), + .result_node_id_a(result_node_id_a), .result_node_id_b(result_node_id_b), + .busy(wb_busy), .done(wb_done), + .mem_active(wb_mem_active), .mem_grant(mem_grant), + .ctrl_req(wb_ctrl_req), .ctrl_wr(wb_ctrl_wr), .ctrl_addr(wb_ctrl_addr), + .ctrl_wdata(wb_ctrl_wdata), .ctrl_wmask(wb_ctrl_wmask), + .ctrl_rdata(ctrl_rdata), .ctrl_ready(ctrl_ready), .ctrl_busy(ctrl_busy) + ); + + // mutually exclusive by FSM construction (activation fetch always + // finishes each tile's own consume before writeback ever starts, + // and writeback only starts once the whole tile loop is done) -- + // 2-way mux, one fewer branch than packed_slot.v's own 3-way (no + // local weight-fetch mux here, weight is group-broadcast). + assign ctrl_req = act_mem_active ? act_ctrl_req : wb_ctrl_req; + assign ctrl_wr = act_mem_active ? act_ctrl_wr : wb_ctrl_wr; + assign ctrl_addr = act_mem_active ? act_ctrl_addr : wb_ctrl_addr; + assign ctrl_wdata = act_mem_active ? act_ctrl_wdata : wb_ctrl_wdata; + assign ctrl_wmask = act_mem_active ? act_ctrl_wmask : wb_ctrl_wmask; + + assign mem_active = act_mem_active || wb_mem_active; + + // ---- neural_processor_packed.v (unmodified, same as packed_slot.v) ---- + reg job_valid_np; + wire job_ready_np; + reg [1:0] job_activation; + reg signed [DATA_WIDTH-1:0] job_bias; + + reg operand_valid; + wire operand_ready; + reg signed [DATA_WIDTH*P_IN-1:0] input_data_a_r, input_data_b_r; + reg [DATA_WIDTH*P_IN-1:0] weight_data_r; + reg tile_last; + + wire result_valid_np; + reg result_ready; + wire signed [DATA_WIDTH-1:0] result_data_a_np, result_data_b_np; + wire [15:0] result_node_id_a_np, result_node_id_b_np; + wire [3:0] np_state; + wire np_error; + + neural_processor_packed #( + .DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .ACC_WIDTH(ACC_WIDTH) + ) u_np ( + .clk(clk), .rst(rst), + .job_valid(job_valid_np), .job_ready(job_ready_np), + .job_node_id_a(node_id_a_lat), .job_node_id_b(node_id_b_lat), + .job_bias(job_bias), .job_activation(job_activation), + .operand_valid(operand_valid), .operand_ready(operand_ready), + .input_data_a(input_data_a_r), .input_data_b(input_data_b_r), + .weight_data(weight_data_r), .tile_last(tile_last), + .result_valid(result_valid_np), .result_ready(result_ready), + .result_data_a(result_data_a_np), .result_data_b(result_data_b_np), + .result_node_id_a(result_node_id_a_np), .result_node_id_b(result_node_id_b_np), + .np_state(np_state), .np_error(np_error) + ); + + localparam ACT_RELU = 2'd1; + + always @(posedge clk) begin + if (rst) begin + state <= S_IDLE; + job_done <= 1'b0; + ddrpf_job_start <= 1'b0; + ddrpf_tile_consume <= 1'b0; + job_valid_np <= 1'b0; + operand_valid<= 1'b0; + tile_last <= 1'b0; + result_ready <= 1'b0; + wb_start <= 1'b0; + pe_tile_ack <= 1'b0; + job_bias <= {DATA_WIDTH{1'b0}}; + job_activation <= ACT_RELU; + tcnt <= 16'd0; + end else begin + job_done <= 1'b0; + ddrpf_job_start <= 1'b0; + ddrpf_tile_consume <= 1'b0; + wb_start <= 1'b0; + pe_tile_ack <= 1'b0; + + case (state) + S_IDLE: begin + if (job_start) begin + x_base_a_lat <= x_base_a; + x_base_b_lat <= x_base_b; + n_tiles_lat <= group_n_tiles; + result_addr_a_lat <= result_addr_a; + result_addr_b_lat <= result_addr_b; + node_id_a_lat <= node_id_a; + node_id_b_lat <= node_id_b; + job_bias <= {DATA_WIDTH{1'b0}}; + job_activation <= ACT_RELU; + tcnt <= 16'd0; + job_valid_np <= 1'b1; + state <= S_JOBSTART; + end + end + + // real, proven sequencing (matches packed_slot.v's own + // S_JOBSTART exactly): only kick off the activation + // look-ahead loop and enter the tile-consumption join + // AFTER neural_processor_packed.v has actually accepted + // the job -- issuing operand_valid before job_ready_np + // would race its own internal job-acceptance state. + S_JOBSTART: begin + if (job_valid_np && job_ready_np) begin + job_valid_np <= 1'b0; + ddrpf_job_start <= 1'b1; // one-shot: kicks off this PE's + // own activation look-ahead loop + state <= S_TILEWAIT; + end + end + + // real join: group's broadcast weight tile (matched by + // tcnt, see header) AND this PE's own activation fetch + // -- same 2-source join shape packed_slot.v's own + // S_TILEWAIT already uses, just with group_tile_valid+ + // tcnt-match replacing the local tile_valid. + S_TILEWAIT: begin + if (group_tile_valid && (group_tcnt == tcnt) && ddrpf_tile_valid) begin + weight_data_r <= group_tile_data; + input_data_a_r <= act_data_a_w; + input_data_b_r <= act_data_b_w; + pe_tile_ack <= 1'b1; + ddrpf_tile_consume <= 1'b1; + tile_last <= (tcnt == n_tiles_lat - 16'd1); + operand_valid <= 1'b1; + state <= S_OPERAND; + end + end + + S_OPERAND: begin + if (operand_valid && operand_ready) begin + operand_valid <= 1'b0; + tile_last <= 1'b0; + if (tcnt == n_tiles_lat - 16'd1) begin + result_ready <= 1'b1; + state <= S_RESULT; + end else begin + tcnt <= tcnt + 16'd1; + state <= S_TILEWAIT; + end + end + end + + S_RESULT: begin + if (result_valid_np) begin + result_data_a <= result_data_a_np; + result_data_b <= result_data_b_np; + result_node_id_a <= result_node_id_a_np; + result_node_id_b <= result_node_id_b_np; + result_addr_a_out <= result_addr_a_lat; + result_addr_b_out <= result_addr_b_lat; + result_ready <= 1'b0; + wb_start <= 1'b1; + state <= S_WRITEBACK; + end + end + + S_WRITEBACK: begin + if (wb_done) begin + job_done <= 1'b1; + state <= S_IDLE; + end + end + + default: state <= S_IDLE; + endcase + end + end +endmodule diff --git a/hardware/v3/rtl/systolic_group.v b/hardware/v3/rtl/systolic_group.v new file mode 100644 index 0000000..54ef796 --- /dev/null +++ b/hardware/v3/rtl/systolic_group.v @@ -0,0 +1,323 @@ +`timescale 1ns/1ps + +// ============================================================ +// V3 -- systolic_group.v: real group controller for the "4 groups x +// 4-PE weight-stationary chains" hybrid scaling direction +// (docs/ARCHITECTURE_ANALYSIS.md S5.6, EXP-0089). User-confirmed real +// design choice (2026-09-20, explicit A/B decision): SHARED-WEIGHT +// BROADCAST, not a literal PE-to-PE systolic shift register -- one +// real weight fetch per group of 4 PEs (instead of 4 independent +// fetches), each PE then computes its own, independent activation +// positions in parallel. This achieves the doc's own real, quantified +// rationale (reduce redundant weight-fetch DDR3 traffic 4x per group) +// with far less real risk/complexity than a literal systolic pipeline +// -- no inter-PE result propagation, no pipeline fill/drain at chain +// boundaries, both of which the doc's own S5.6 explicitly flagged as +// "a real, new design, not a trivial extension". +// +// STRUCTURE: ONE real layer_prefetch_ctrl.v + layer_weight_buffer.v + +// weight_tile_gather.v (unmodified, identical instances to what +// packed_slot.v already owns per-slot -- just now group-level, shared +// by reference not duplicated), driving 4x packed_pe.v (packed_slot.v +// minus its own weight-fetch, see packed_pe.v's own header) via a +// real, level-held, tcnt-tagged broadcast bus. +// +// REAL BARRIER SYNCHRONIZATION (the actual new design, not asserted +// correct without real verification -- see tb_systolic_group.v): the +// group only advances to tile N+1's weight fetch once ALL 4 PEs have +// ack'd tile N (`pe_tile_ack[i]`, individually latched since PEs may +// consume at different real cycles -- e.g. one PE's own activation +// fetch hit a real DDR3 row switch the others didn't). This is a real, +// necessary extension of the "S_TILEWAIT join" discipline every other +// module in this project already uses for 2-source joins (weight + +// activation) -- here it's a 4-way join (one group weight source, 4 +// independent PE acks), same underlying principle: never advance past +// a shared resource until every real consumer has confirmed it read +// what it needed. +// ============================================================ +module systolic_group #( + parameter DATA_WIDTH = 8, + parameter P_IN = 8, + parameter ACC_WIDTH = 32, + parameter BURST_LEN = 8, + parameter ADDR_WIDTH = 26, + parameter LAYER_BYTES = 128, + parameter BUFADDRW = $clog2(LAYER_BYTES) +)( + input wire clk, + input wire rst, + + // ---- group-level job dispatch: ONE shared w_base/n_tiles (all 4 + // PEs process the SAME layer, weight-stationary), 4x independent + // per-PE x_base_a/b/result_addr_a/b/node_id_a/b ---- + input wire job_start, + input wire [ADDR_WIDTH-1:0] w_base, + input wire [15:0] n_tiles, + input wire [4*ADDR_WIDTH-1:0] pe_x_base_a, pe_x_base_b, + input wire [4*ADDR_WIDTH-1:0] pe_result_addr_a, pe_result_addr_b, + input wire [4*16-1:0] pe_node_id_a, pe_node_id_b, + output reg job_done, // one-cycle pulse, ALL 4 PEs done + + output wire [4*DATA_WIDTH-1:0] pe_result_data_a, pe_result_data_b, + output wire [4*16-1:0] pe_result_node_id_a, pe_result_node_id_b, + output wire [4*ADDR_WIDTH-1:0] pe_result_addr_a_out, pe_result_addr_b_out, + + // ---- group's own single arbiter port, for the shared weight + // fetch only -- each of the 4 PEs still owns ITS OWN separate + // arbiter port for activation-fetch+writeback (see top-level + // integration; not this module's own concern) ---- + output wire mem_active, + input wire mem_grant, + output wire ctrl_req, + output wire ctrl_wr, + output wire [ADDR_WIDTH-2:0] ctrl_addr, + output wire [32*BURST_LEN-1:0] ctrl_wdata, + output wire [4*BURST_LEN-1:0] ctrl_wmask, + input wire [32*BURST_LEN-1:0] ctrl_rdata, + input wire ctrl_ready, + input wire ctrl_busy, + + // ---- 4 independent per-PE arbiter ports (activation-fetch + + // result-writeback, NOT shared -- flattened NUM_PE*width buses) ---- + output wire [3:0] pe_mem_active, + input wire [3:0] pe_mem_grant, + output wire [3:0] pe_ctrl_req, + output wire [3:0] pe_ctrl_wr, + output wire [4*(ADDR_WIDTH-1)-1:0] pe_ctrl_addr, + output wire [4*32*BURST_LEN-1:0] pe_ctrl_wdata, + output wire [4*4*BURST_LEN-1:0] pe_ctrl_wmask, + input wire [4*32*BURST_LEN-1:0] pe_ctrl_rdata, + input wire [3:0] pe_ctrl_ready, + input wire [3:0] pe_ctrl_busy +); + localparam S_IDLE = 4'd0, + S_MEMWAIT = 4'd1, + S_PREFETCH = 4'd2, + S_SWAP = 4'd3, + S_PEJOBSTART = 4'd4, + S_TILELOOP = 4'd5, + S_WAITDONE = 4'd6; + + reg [3:0] state; + reg [ADDR_WIDTH-1:0] w_base_lat; + reg [15:0] n_tiles_lat; + reg [15:0] tcnt; + + // ---- shared weight fetch: layer_prefetch_ctrl.v -> layer_weight_ + // buffer.v -> weight_tile_gather.v, IDENTICAL instances to what + // packed_slot.v already owns per-slot, just group-level now ---- + reg pf_start; + wire pf_busy, pf_done; + wire pf_fill_we; + wire [BUFADDRW-1:0] pf_fill_addr; + wire [DATA_WIDTH-1:0] pf_fill_data; + reg consume_done; + + layer_prefetch_ctrl #( + .DATA_WIDTH(DATA_WIDTH), .LAYER_BYTES(LAYER_BYTES), .BURST_LEN(BURST_LEN), .ADDR_WIDTH(ADDR_WIDTH-1) + ) u_pf ( + .clk(clk), .rst(rst), + .start(pf_start), .layer_base(w_base_lat[ADDR_WIDTH-2:0]), .busy(pf_busy), .done(pf_done), + .fill_we(pf_fill_we), .fill_addr(pf_fill_addr), .fill_data(pf_fill_data), + .ctrl_req(ctrl_req), .ctrl_wr(ctrl_wr), .ctrl_addr(ctrl_addr), + .ctrl_wdata(ctrl_wdata), .ctrl_wmask(ctrl_wmask), + .ctrl_rdata(ctrl_rdata), .ctrl_ready(ctrl_ready), .ctrl_busy(ctrl_busy) + ); + + assign mem_active = (state == S_MEMWAIT) || (state == S_PREFETCH); + + wire [BUFADDRW-1:0] lwb_rd_addr; + wire [DATA_WIDTH-1:0] lwb_rd_data; + + layer_weight_buffer #(.DATA_WIDTH(DATA_WIDTH), .LAYER_DEPTH(LAYER_BYTES)) u_lwb ( + .clk(clk), .rst(rst), + .fill_we(pf_fill_we), .fill_addr(pf_fill_addr), .fill_data(pf_fill_data), .fill_done(pf_done), + .rd_addr(lwb_rd_addr), .rd_data(lwb_rd_data), .consume_done(consume_done), + .active_sel(), .swapped() + ); + + reg tile_req; + reg [BUFADDRW-1:0] tile_base; + wire tile_valid; + wire [DATA_WIDTH*P_IN-1:0] tile_data; + wire [15:0] tcnt_next = tcnt + 16'd1; + + weight_tile_gather #( + .DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .BUFADDRW(BUFADDRW) + ) u_gather ( + .clk(clk), .rst(rst), + .tile_req(tile_req), .tile_base(tile_base), + .tile_valid(tile_valid), .tile_data(tile_data), + .rd_addr(lwb_rd_addr), .rd_data(lwb_rd_data) + ); + + // ---- real broadcast bus to all 4 PEs (level-held, tcnt-tagged -- + // see packed_pe.v's own header for the real join discipline) ---- + reg group_tile_valid; + reg [DATA_WIDTH*P_IN-1:0] group_tile_data_r; + reg [3:0] pe_acked; // per-PE ack latch, cleared each tile + + wire [3:0] pe_job_start_w = {4{(state == S_PEJOBSTART)}}; + wire [3:0] pe_job_done_w; + wire [3:0] pe_acked_pulse; + + genvar gi; + generate + for (gi = 0; gi < 4; gi = gi + 1) begin : GEN_PE + packed_pe #( + .DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .ACC_WIDTH(ACC_WIDTH), + .BURST_LEN(BURST_LEN), .ADDR_WIDTH(ADDR_WIDTH) + ) u_pe ( + .clk(clk), .rst(rst), + .job_start(pe_job_start_w[gi]), + .x_base_a(pe_x_base_a[gi*ADDR_WIDTH +: ADDR_WIDTH]), + .x_base_b(pe_x_base_b[gi*ADDR_WIDTH +: ADDR_WIDTH]), + .result_addr_a(pe_result_addr_a[gi*ADDR_WIDTH +: ADDR_WIDTH]), + .result_addr_b(pe_result_addr_b[gi*ADDR_WIDTH +: ADDR_WIDTH]), + .node_id_a(pe_node_id_a[gi*16 +: 16]), .node_id_b(pe_node_id_b[gi*16 +: 16]), + .job_done(pe_job_done_w[gi]), + .result_data_a(pe_result_data_a[gi*DATA_WIDTH +: DATA_WIDTH]), + .result_data_b(pe_result_data_b[gi*DATA_WIDTH +: DATA_WIDTH]), + .result_node_id_a(pe_result_node_id_a[gi*16 +: 16]), + .result_node_id_b(pe_result_node_id_b[gi*16 +: 16]), + .result_addr_a_out(pe_result_addr_a_out[gi*ADDR_WIDTH +: ADDR_WIDTH]), + .result_addr_b_out(pe_result_addr_b_out[gi*ADDR_WIDTH +: ADDR_WIDTH]), + .group_n_tiles(n_tiles_lat), .group_tcnt(tcnt), + .group_tile_data(group_tile_data_r), .group_tile_valid(group_tile_valid), + .pe_tile_ack(pe_acked_pulse[gi]), + .mem_active(pe_mem_active[gi]), .mem_grant(pe_mem_grant[gi]), + .ctrl_req(pe_ctrl_req[gi]), .ctrl_wr(pe_ctrl_wr[gi]), + .ctrl_addr(pe_ctrl_addr[gi*(ADDR_WIDTH-1) +: (ADDR_WIDTH-1)]), + .ctrl_wdata(pe_ctrl_wdata[gi*32*BURST_LEN +: 32*BURST_LEN]), + .ctrl_wmask(pe_ctrl_wmask[gi*4*BURST_LEN +: 4*BURST_LEN]), + .ctrl_rdata(pe_ctrl_rdata[gi*32*BURST_LEN +: 32*BURST_LEN]), + .ctrl_ready(pe_ctrl_ready[gi]), .ctrl_busy(pe_ctrl_busy[gi]) + ); + end + endgenerate + + reg [3:0] pe_done_latch; + // combinational "what pe_acked/pe_done_latch would be if we also + // fold in THIS cycle's own pulses" -- used both to DECIDE the + // barrier this cycle (no extra latency) and, explicitly, as what + // gets written back when the barrier hasn't cleared yet. Kept as + // named wires (not relying on nonblocking-assignment-order + // last-write-wins semantics) so the real intent is unambiguous to + // a future reader, not just technically correct. + wire [3:0] pe_acked_next = pe_acked | pe_acked_pulse; + wire [3:0] pe_done_latch_next = pe_done_latch | pe_job_done_w; + + always @(posedge clk) begin + if (rst) begin + state <= S_IDLE; + job_done <= 1'b0; + pf_start <= 1'b0; + consume_done <= 1'b0; + tile_req <= 1'b0; + group_tile_valid <= 1'b0; + pe_acked <= 4'b0; + pe_done_latch <= 4'b0; + tcnt <= 16'd0; + end else begin + job_done <= 1'b0; + pf_start <= 1'b0; + consume_done <= 1'b0; + tile_req <= 1'b0; + + // real race, deliberately handled: a PE's own job_done can + // pulse the SAME cycle S_TILELOOP's barrier clears for the + // LAST tile (i.e. the same cycle the group transitions to + // S_WAITDONE) -- its own downstream writeback can complete + // before the group has even formally entered S_WAITDONE. + // Accumulate unconditionally, every cycle, so no early + // pe_job_done_w pulse is ever missed; S_WAITDONE's own + // success branch below explicitly overrides this back to 0 + // (Verilog's own last-nonblocking-write-wins rule within + // one always block -- intentional here, unlike pe_acked's + // window which is fully contained inside S_TILELOOP and so + // uses the more explicit if/else form instead). + pe_done_latch <= pe_done_latch_next; + + case (state) + S_IDLE: begin + if (job_start) begin + w_base_lat <= w_base; + n_tiles_lat <= n_tiles; + tcnt <= 16'd0; + state <= S_MEMWAIT; + end + end + + S_MEMWAIT: begin + if (mem_grant) begin + pf_start <= 1'b1; + state <= S_PREFETCH; + end + end + + S_PREFETCH: begin + if (pf_done) begin + consume_done <= 1'b1; + state <= S_SWAP; + end + end + + S_SWAP: begin + // one settle cycle, same real reason packed_slot.v's + // own S_SWAP exists (layer_weight_buffer.v's do_swap). + state <= S_PEJOBSTART; + end + + S_PEJOBSTART: begin + // pe_job_start_w is combinational on (state == + // S_PEJOBSTART), so all 4 PEs see job_start the + // SAME cycle -- real, established one-shot-pulse + // discipline (EXP-0066), now applied 4-way. + tile_req <= 1'b1; + tile_base <= tcnt[BUFADDRW-1:0]*P_IN[BUFADDRW-1:0]; + state <= S_TILELOOP; + end + + // real barrier: hold this tile's data broadcast until + // ALL 4 PEs have ack'd it (pe_acked_next all-ones), THEN + // advance tcnt and issue the next weight_tile_gather + // request -- see this module's own header for why a + // bare pulse/level race would be unsafe here. pe_acked + // is explicitly written on EVERY path through this + // state (either reset to 0 when the barrier clears, or + // folded forward via pe_acked_next otherwise) -- never + // relies on assignment order elsewhere in the block. + S_TILELOOP: begin + if (tile_valid && !group_tile_valid) begin + group_tile_data_r <= tile_data; + group_tile_valid <= 1'b1; + end + if (group_tile_valid && (&pe_acked_next)) begin + group_tile_valid <= 1'b0; + pe_acked <= 4'b0; + if (tcnt == n_tiles_lat - 16'd1) begin + state <= S_WAITDONE; + end else begin + tcnt <= tcnt_next; + tile_req <= 1'b1; + tile_base <= tcnt_next[BUFADDRW-1:0]*P_IN[BUFADDRW-1:0]; + state <= S_TILELOOP; + end + end else begin + pe_acked <= pe_acked_next; + end + end + + S_WAITDONE: begin + if (&pe_done_latch_next) begin + pe_done_latch <= 4'b0; + job_done <= 1'b1; + state <= S_IDLE; + end + end + + default: state <= S_IDLE; + endcase + end + end +endmodule diff --git a/hardware/v3/sim/tb_systolic_group.v b/hardware/v3/sim/tb_systolic_group.v new file mode 100644 index 0000000..268790f --- /dev/null +++ b/hardware/v3/sim/tb_systolic_group.v @@ -0,0 +1,292 @@ +`timescale 1ns/1ps + +// ============================================================ +// EXP-0089 -- isolated correctness test for systolic_group.v: one +// group of 4 packed_pe.v instances sharing ONE real weight fetch +// (broadcast, not a literal PE-to-PE systolic shift register -- see +// systolic_group.v's own header for the real, user-confirmed design +// choice). Same real discipline as every other new module in this +// project (act_tile_fetch.v EXP-0079, ddr_prefetch_mgr.v EXP-0083, +// result_writeback.v EXP-0088): verify in isolation, with a real +// memory backend and a real, independently-reproduced golden model, +// BEFORE any Director/SPI-protocol integration. +// +// Real backend: burst_mem_model32.v (same EXP-0084 model every other +// v3 isolated testbench uses) + sdram_arbiter_n.v with NUM_REQ=5 (1 +// group-level weight-fetch requester + 4 independent per-PE +// activation-fetch/writeback requesters) -- sdram_arbiter_n.v's own +// NUM_REQ already generalizes to this without any change, confirmed +// by direct reuse here, not by inspection. +// +// Runs TWO consecutive group jobs (different positions/weights each +// time) specifically to catch any "forgot to clear a per-job latch" +// bug in the group's own barrier state (pe_acked/pe_done_latch) -- +// a single-job test would not exercise that reset path at all. +// ============================================================ +module tb; + localparam BURST_LEN = 8; + localparam SDRAM_ADDR_WIDTH = 25; + localparam CLK_FREQ_MHZ = 64; + localparam CLK_PERIOD_NS = 1000.0/CLK_FREQ_MHZ; + + localparam DATA_WIDTH = 8; + localparam P_IN = 8; + localparam ACC_WIDTH = 32; + localparam ADDR_WIDTH = 26; + localparam N_INPUTS = 128; + localparam N_TILES = N_INPUTS/P_IN; + localparam LAYER_BYTES = N_INPUTS; + localparam WORDS_PER_LAYER = LAYER_BYTES/2; + localparam NUM_REQ = 5; // 1 group weight-fetch + 4 PE activation/writeback + + reg clk = 0; + always #(CLK_PERIOD_NS/2.0) clk = ~clk; + reg rst; + integer cyc; + always @(posedge clk) if (!rst) cyc <= cyc + 1; + + // ---- real burst-memory backend, shared via a real 5-way arbiter ---- + wire ctrl_req, ctrl_wr; + wire [SDRAM_ADDR_WIDTH-1:0] ctrl_addr; + wire [32*BURST_LEN-1:0] ctrl_wdata; + wire [4*BURST_LEN-1:0] ctrl_wmask; + wire [32*BURST_LEN-1:0] ctrl_rdata; + wire ctrl_ready, ctrl_busy; + + reg wpre_req, wpre_wr; + reg [SDRAM_ADDR_WIDTH-1:0] wpre_addr; + reg [32*BURST_LEN-1:0] wpre_wdata; + reg pre_active; + + wire [NUM_REQ-1:0] arb_active, arb_grant, arb_req, arb_wr; + wire [NUM_REQ*SDRAM_ADDR_WIDTH-1:0] arb_addr; + wire [NUM_REQ*32*BURST_LEN-1:0] arb_wdata, arb_rdata; + wire [NUM_REQ*4*BURST_LEN-1:0] arb_wmask; + wire [NUM_REQ-1:0] arb_ready, arb_busy; + + wire real_ctrl_req, real_ctrl_wr; + wire [SDRAM_ADDR_WIDTH-1:0] real_ctrl_addr; + wire [32*BURST_LEN-1:0] real_ctrl_wdata; + wire [4*BURST_LEN-1:0] real_ctrl_wmask; + + sdram_arbiter_n #(.NUM_REQ(NUM_REQ), .ADDR_WIDTH(SDRAM_ADDR_WIDTH), .BURST_LEN(BURST_LEN)) u_arb ( + .clk(clk), .rst(rst), + .req_active(arb_active), .req_grant(arb_grant), + .req_req(arb_req), .req_wr(arb_wr), .req_addr(arb_addr), + .req_wdata(arb_wdata), .req_wmask(arb_wmask), + .req_rdata(arb_rdata), .req_ready(arb_ready), .req_busy(arb_busy), + .ctrl_req(real_ctrl_req), .ctrl_wr(real_ctrl_wr), .ctrl_addr(real_ctrl_addr), + .ctrl_wdata(real_ctrl_wdata), .ctrl_wmask(real_ctrl_wmask), + .ctrl_rdata(ctrl_rdata), .ctrl_ready(ctrl_ready), .ctrl_busy(ctrl_busy) + ); + + assign ctrl_req = pre_active ? wpre_req : real_ctrl_req; + assign ctrl_wr = pre_active ? wpre_wr : real_ctrl_wr; + assign ctrl_addr = pre_active ? wpre_addr : real_ctrl_addr; + assign ctrl_wdata = pre_active ? wpre_wdata : real_ctrl_wdata; + assign ctrl_wmask = pre_active ? {(4*BURST_LEN){1'b0}} : real_ctrl_wmask; + + burst_mem_model32 #( + .BURST_LEN(BURST_LEN), .ADDR_WIDTH(SDRAM_ADDR_WIDTH) + ) u_mem ( + .clk(clk), .rst(rst), + .req(ctrl_req), .wr(ctrl_wr), .addr(ctrl_addr), .wdata(ctrl_wdata), .wmask(ctrl_wmask), + .rdata(ctrl_rdata), .ready(ctrl_ready), .busy(ctrl_busy) + ); + + function automatic signed [7:0] weight_byte(input integer li, input integer t); + reg [7:0] tmp; + begin + tmp = (li*17 + t*29 + 13) & 8'hFF; + weight_byte = $signed(tmp); + end + endfunction + function automatic signed [7:0] input_byte(input integer li, input integer pos, input integer t); + reg [7:0] tmp; + begin + tmp = (li*11 + pos*41 + t*7 + 3) & 8'hFF; + input_byte = $signed(tmp); + end + endfunction + + task automatic sdram_write_burst(input [SDRAM_ADDR_WIDTH-1:0] word_addr, input [32*BURST_LEN-1:0] data); + begin + @(posedge clk); while (ctrl_busy) @(posedge clk); + wpre_req = 1'b1; wpre_wr = 1'b1; wpre_addr = word_addr; wpre_wdata = data; + @(posedge clk); wpre_req = 1'b0; + while (!ctrl_ready) @(posedge clk); + end + endtask + + task automatic preload_sdram_layer(input integer li); + integer bi, wb, tt; + reg [32*BURST_LEN-1:0] burst_data; + begin + for (bi = 0; bi < (LAYER_BYTES/(4*BURST_LEN)); bi = bi + 1) begin + for (wb = 0; wb < BURST_LEN; wb = wb + 1) begin + tt = bi*(4*BURST_LEN) + wb*4; + burst_data[wb*32 +: 32] = {weight_byte(li, tt+3), weight_byte(li, tt+2), + weight_byte(li, tt+1), weight_byte(li, tt)}; + end + sdram_write_burst((li*WORDS_PER_LAYER + bi*BURST_LEN), burst_data); + end + end + endtask + + localparam [ADDR_WIDTH-1:0] ACT_MEM_BASE = 26'h10000; + function automatic [ADDR_WIDTH-1:0] act_x_base(input integer li, input integer pos); + act_x_base = ACT_MEM_BASE + (li*16 + pos) * ((N_TILES/4)*BURST_LEN); + endfunction + + task automatic preload_sdram_activation(input integer li, input integer pos); + integer tq, qi; + reg [32*BURST_LEN-1:0] burst_data; + reg [ADDR_WIDTH-1:0] base; + begin + base = act_x_base(li, pos); + for (tq = 0; tq < N_TILES/4; tq = tq + 1) begin + burst_data = {(32*BURST_LEN){1'b0}}; + for (qi = 0; qi < 4; qi = qi + 1) + burst_data[qi*64 +: 64] = {input_byte(li, pos, (4*tq+qi)*P_IN + 7), input_byte(li, pos, (4*tq+qi)*P_IN + 6), + input_byte(li, pos, (4*tq+qi)*P_IN + 5), input_byte(li, pos, (4*tq+qi)*P_IN + 4), + input_byte(li, pos, (4*tq+qi)*P_IN + 3), input_byte(li, pos, (4*tq+qi)*P_IN + 2), + input_byte(li, pos, (4*tq+qi)*P_IN + 1), input_byte(li, pos, (4*tq+qi)*P_IN + 0)}; + sdram_write_burst(base[SDRAM_ADDR_WIDTH-1:0] + tq*BURST_LEN, burst_data); + end + end + endtask + + // ---- systolic_group.v (DUT) ---- + reg job_start; + reg [ADDR_WIDTH-1:0] w_base; + reg [15:0] n_tiles_in; + reg [4*ADDR_WIDTH-1:0] pe_x_base_a, pe_x_base_b, pe_result_addr_a, pe_result_addr_b; + reg [4*16-1:0] pe_node_id_a, pe_node_id_b; + wire job_done; + wire [4*DATA_WIDTH-1:0] pe_result_data_a, pe_result_data_b; + wire [4*16-1:0] pe_result_node_id_a, pe_result_node_id_b; + wire [4*ADDR_WIDTH-1:0] pe_result_addr_a_out, pe_result_addr_b_out; + + wire grp_mem_active; + wire [3:0] pe_mem_active; + + systolic_group #( + .DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .ACC_WIDTH(ACC_WIDTH), + .BURST_LEN(BURST_LEN), .ADDR_WIDTH(ADDR_WIDTH), .LAYER_BYTES(LAYER_BYTES) + ) dut ( + .clk(clk), .rst(rst), + .job_start(job_start), .w_base(w_base), .n_tiles(n_tiles_in), + .pe_x_base_a(pe_x_base_a), .pe_x_base_b(pe_x_base_b), + .pe_result_addr_a(pe_result_addr_a), .pe_result_addr_b(pe_result_addr_b), + .pe_node_id_a(pe_node_id_a), .pe_node_id_b(pe_node_id_b), + .job_done(job_done), + .pe_result_data_a(pe_result_data_a), .pe_result_data_b(pe_result_data_b), + .pe_result_node_id_a(pe_result_node_id_a), .pe_result_node_id_b(pe_result_node_id_b), + .pe_result_addr_a_out(pe_result_addr_a_out), .pe_result_addr_b_out(pe_result_addr_b_out), + .mem_active(grp_mem_active), .mem_grant(arb_grant[0]), + .ctrl_req(arb_req[0]), .ctrl_wr(arb_wr[0]), .ctrl_addr(arb_addr[0*SDRAM_ADDR_WIDTH +: SDRAM_ADDR_WIDTH]), + .ctrl_wdata(arb_wdata[0*32*BURST_LEN +: 32*BURST_LEN]), .ctrl_wmask(arb_wmask[0*4*BURST_LEN +: 4*BURST_LEN]), + .ctrl_rdata(arb_rdata[0*32*BURST_LEN +: 32*BURST_LEN]), .ctrl_ready(arb_ready[0]), .ctrl_busy(arb_busy[0]), + .pe_mem_active(pe_mem_active), .pe_mem_grant(arb_grant[4:1]), + .pe_ctrl_req(arb_req[4:1]), .pe_ctrl_wr(arb_wr[4:1]), + .pe_ctrl_addr(arb_addr[1*SDRAM_ADDR_WIDTH +: 4*SDRAM_ADDR_WIDTH]), + .pe_ctrl_wdata(arb_wdata[1*32*BURST_LEN +: 4*32*BURST_LEN]), + .pe_ctrl_wmask(arb_wmask[1*4*BURST_LEN +: 4*4*BURST_LEN]), + .pe_ctrl_rdata(arb_rdata[1*32*BURST_LEN +: 4*32*BURST_LEN]), + .pe_ctrl_ready(arb_ready[4:1]), .pe_ctrl_busy(arb_busy[4:1]) + ); + + assign arb_active[0] = grp_mem_active; + assign arb_active[4:1] = pe_mem_active; + + integer errors, tests; + integer li_i, gi, wd; + integer acc, s; + reg signed [DATA_WIDTH-1:0] expected [0:7]; // 4 PEs x 2 lanes + + task automatic run_group_job(input integer li, input integer pos_base); + integer pe, lane, pos, tt; + reg [DATA_WIDTH-1:0] got_a, got_b; + begin + tests = tests + 1; + @(posedge clk); + job_start = 1'b1; + w_base = li*WORDS_PER_LAYER; + n_tiles_in = N_TILES[15:0]; + for (pe = 0; pe < 4; pe = pe + 1) begin + pe_x_base_a[pe*ADDR_WIDTH +: ADDR_WIDTH] = act_x_base(li, pos_base + pe*2); + pe_x_base_b[pe*ADDR_WIDTH +: ADDR_WIDTH] = act_x_base(li, pos_base + pe*2 + 1); + pe_result_addr_a[pe*ADDR_WIDTH +: ADDR_WIDTH] = 26'h9000 + pe*2; + pe_result_addr_b[pe*ADDR_WIDTH +: ADDR_WIDTH] = 26'h9000 + pe*2 + 1; + pe_node_id_a[pe*16 +: 16] = li*100 + pos_base + pe*2; + pe_node_id_b[pe*16 +: 16] = li*100 + pos_base + pe*2 + 1; + end + @(posedge clk); + job_start = 1'b0; + + // real, independently-reproduced golden model (same formula + // as tb_packed_slot.v's own, applied per PE/lane) + for (pe = 0; pe < 4; pe = pe + 1) begin + for (lane = 0; lane < 2; lane = lane + 1) begin + pos = pos_base + pe*2 + lane; + acc = 0; + for (tt = 0; tt < N_INPUTS; tt = tt + 1) + acc = acc + (input_byte(li, pos, tt) * weight_byte(li, tt)); + s = acc; + if (s <= 0) expected[pe*2+lane] = 0; + else if (s > 127) expected[pe*2+lane] = 8'sd127; + else expected[pe*2+lane] = s[DATA_WIDTH-1:0]; + end + end + + wd = 0; + while (!job_done && wd < 4000) begin @(posedge clk); wd = wd + 1; end + if (!job_done) begin + $display("FAIL li=%0d pos_base=%0d: TIMEOUT waiting for group job_done", li, pos_base); + errors = errors + 1; + end else begin + for (pe = 0; pe < 4; pe = pe + 1) begin + got_a = pe_result_data_a[pe*DATA_WIDTH +: DATA_WIDTH]; + got_b = pe_result_data_b[pe*DATA_WIDTH +: DATA_WIDTH]; + if (got_a !== expected[pe*2] || got_b !== expected[pe*2+1]) begin + $display("FAIL li=%0d pos_base=%0d PE%0d: got_a=%0d got_b=%0d expected_a=%0d expected_b=%0d", + li, pos_base, pe, $signed(got_a), $signed(got_b), + $signed(expected[pe*2]), $signed(expected[pe*2+1])); + errors = errors + 1; + end else begin + $display("PASS li=%0d pos_base=%0d PE%0d: a=%0d b=%0d (systolic_group.v)", + li, pos_base, pe, $signed(got_a), $signed(got_b)); + end + end + end + end + endtask + + initial begin + errors = 0; tests = 0; cyc = 0; + rst = 1; pre_active = 1'b1; + wpre_req = 0; wpre_wr = 0; wpre_addr = 0; wpre_wdata = 0; + job_start = 0; w_base = 0; n_tiles_in = 0; + pe_x_base_a = 0; pe_x_base_b = 0; pe_result_addr_a = 0; pe_result_addr_b = 0; + pe_node_id_a = 0; pe_node_id_b = 0; + repeat(5) @(posedge clk); + rst = 0; + @(posedge clk); while (ctrl_busy) @(posedge clk); + + $display("=== preload SDRAM: 2 layers' weights + 16 activation positions ==="); + for (li_i = 0; li_i < 2; li_i = li_i + 1) begin + preload_sdram_layer(li_i); + for (gi = 0; gi < 8; gi = gi + 1) preload_sdram_activation(li_i, gi); + end + @(posedge clk); + pre_active = 1'b0; + + $display("=== systolic_group.v: 2 group jobs (4 PEs x 2 lanes each) ==="); + run_group_job(0, 0); + run_group_job(1, 0); + + $display("=== %0d/%0d tests, %0d errors ===", tests-errors, tests, errors); + if (errors == 0) $display("ALL TESTS PASSED (tb_systolic_group)"); + $finish; + end +endmodule