michele/FPGA-Neural
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

exp: N=16 timing closure fixed (EXP-0056), weight-reuse gives real 7.16x memory speedup without DDR3 (EXP-0057)

Browse Source 
EXP-0056: N_SLOTS=16 failed timing on LFE5U-85F (23-24MHz vs 64MHz
target). First hypothesis (dependency_manager.v's serial ready-scan)
was wrong but real -- built and verified priority_encoder_lsb.v (a
generic recursive tree encoder) and dependency_manager_fast.v, bit-
exact equivalent to the original, but integrated it made no real
difference (24.26MHz). The real cause, found from nextpnr's own
critical-path report: nms_activation_fill_ctrl_v3.v's balanced max-
tree was only ever extended to N_SLOTS in {1,2,4,8}, silently falling
back to the original slow scan for 16. Added the missing case
(nms_activation_fill_ctrl_v3_n16.v), verified isolated (10017/10017)
and functionally (D-Stress N=16 still 256/256 bit-exact). Real result:
71.01MHz, PASS at 64MHz (single seed so far).

EXP-0057: built layer_weight_buffer.v, a double-buffered per-layer
weight scratchpad (fill one buffer in the background from SDRAM while
compute reads many times from the other -- weight-stationary reuse,
as opposed to D-Stress's own deliberately zero-reuse pattern). Wired
to the real sdram_controller_openrow.v + sdram_model.v, no new
hardware. For the same 32768 bytes of useful data: zero-reuse costs
27048 real cycles, reuse costs 3777 -- 7.16x real measured speedup on
the SAME SDR SDRAM, no DDR3, no clock change. This is the answer to
whether DDR3 is necessary for a workload class that actually has
reuse (e.g. conv-style face recognition, unlike D-Stress) -- it isn't,
at least not for this reason.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MUG92aM9m68TRc4rG55BcC
This commit is contained in:
michele
2026-09-16 12:01:11 +02:00
co-authored by Claude Sonnet 5
parent fce8ff2d66
commit 1ce78dff6e
25 changed files with 25848 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hardware/v2/nms/rtl/fpga_neural_v2_top_openrow_fast.v
+265
View File
@@ -0,0 +1,265 @@
`timescale 1ns/1ps
// ================================================================
// FPGA-Neural V2 -- BOARD-LEVEL TOP (STEP20, real physical interface)
//
// Wraps the STEP19 frozen compute+memory design (the same submodules
// nms_neural_multiprocessor_sdram_unified.v instantiates -- that file
// itself is NOT instantiated here, since its own reg_*/N_SLOTS+1-port
// AR arbitration needs a second arbitration LEVEL added for the new
// host-raw-SDRAM-access port; this module reproduces that same
// internal wiring plus the extra level, rather than modifying the
// frozen file) with the three things a real physical board needs that
// a testbench does not:
//
// 1. A real SPI host interface (spi_host_bridge.v) in place of the
// 110-pin reg_* testbench bus -- reg_valid/reg_ready/reg_node_id/
// etc are now DRIVEN BY THE BRIDGE, not exposed as top ports.
// 2. A real ECP5 PLL (ecp5_pll_sys_clk.v, EHXPLLL) generating the
// system clock from the board's 16MHz oscillator, instead of
// assuming an already-correct-frequency clock input.
// 3. A real reset/POR synchronizer (reset_sync.v).
//
// nms_dataflow_core_sdram.v, dependency_manager.v, neural_processor.v,
// neural_director.v, slot_mem_arbiter.v, slot_mem_arbiter_wide.v,
// sdram_unified_backend.v, sdram_controller.v are ALL byte-for-byte
// unchanged (STEP19/STEP20 standing constraint) -- this file only
// ADDS one more, already-proven, generically-parameterized
// slot_mem_arbiter instance (N_PORTS=2) to arbitrate the SPI bridge's
// raw host memory port against the existing compute-side AR stream,
// both funneling into the SAME single sdram_unified_backend/
// sdram_controller/AS4C4M16SA-6TIN physical chain STEP19 already
// validated. No V1 RTL is instantiated (STEP19's "zero V1 files in
// the V2 compile list" property is preserved).
// ================================================================
module fpga_neural_v2_top_openrow_fast #(
parameter DATA_WIDTH = 8,
parameter P_IN = 8,
parameter ACC_WIDTH = 32,
parameter ADDR_WIDTH = 26,
parameter N_SLOTS = 4,
parameter N_NODES = 16,
parameter MAX_DEPS = 4,
parameter QUEUE_DEPTH = 8,
parameter MAX_TILES = 16,
parameter PREFETCH_DISTANCE = 8,
parameter CLK_FREQ_MHZ = 64
)(
input wire osc_clk, // 16 MHz board oscillator
input wire ext_rst_n, // external POR/supervisor, active-low
// ---- physical SPI host interface ----
input wire spi_sclk,
input wire spi_mosi,
output wire spi_miso,
input wire spi_cs_n,
// ---- single physical SDRAM (weights + activations + results) ----
// sdram_clk: the real SDRAM chip's own CLK pin -- an external
// chip, it needs this driven from a real output ball, NOT just
// internal routing. Found missing entirely during this session's
// schematic review (clk_sys was purely internal, never reached a
// pad) -- added here, real free clock-capable ball (bank 6).
output wire sdram_clk,
output wire sdram_cke,
output wire sdram_cs_n,
output wire sdram_ras_n,
output wire sdram_cas_n,
output wire sdram_we_n,
output wire [1:0] sdram_ba,
output wire [12:0] sdram_a,
inout wire [15:0] sdram_dq,
output wire [1:0] sdram_dqm,
// FPGA_DATA_READY: high once the whole registered graph has
// finished (system-idle sticky flag, self-clearing on new work) --
// see nms_dataflow_core_sdram.v for the full design comment.
output wire data_ready,
output wire pll_locked
);
// ============================================================
// CLOCK / RESET
// ============================================================
wire clk_sys;
ecp5_pll_sys_clk u_pll (
.clk_16mhz(osc_clk), .clk_sys(clk_sys), .locked(pll_locked)
);
assign sdram_clk = clk_sys;
wire clk = clk_sys;
wire rst;
reset_sync u_reset_sync (
.clk_sys(clk_sys), .ext_rst_n(ext_rst_n), .pll_locked(pll_locked), .rst(rst)
);
wire soft_rst_pulse;
wire core_rst = rst | soft_rst_pulse;
// ============================================================
// SPI HOST BRIDGE (replaces the 110-pin reg_* testbench bus)
// ============================================================
wire reg_valid, reg_ready;
wire [$clog2(N_NODES)-1:0] reg_node_id;
wire [$clog2(MAX_DEPS+1)-1:0] reg_required;
wire [MAX_DEPS*$clog2(N_NODES)-1:0] reg_producer_ids;
wire [ADDR_WIDTH-1:0] reg_x_base, reg_w_base, reg_result_addr;
wire [15:0] reg_n_tiles;
wire host_mem_req, host_mem_wr, host_mem_lb_n, host_mem_ub_n;
wire [ADDR_WIDTH-1:0] host_mem_addr;
wire [15:0] host_mem_wdata, host_mem_rdata;
wire host_mem_ready;
spi_host_bridge #(
.ADDR_WIDTH(ADDR_WIDTH), .N_NODES(N_NODES), .MAX_DEPS(MAX_DEPS)
) u_spi_bridge (
.clk(clk), .rst(rst),
.sclk(spi_sclk), .mosi(spi_mosi), .miso(spi_miso), .cs_n(spi_cs_n),
.reg_valid(reg_valid), .reg_ready(reg_ready), .reg_node_id(reg_node_id),
.reg_required(reg_required), .reg_producer_ids(reg_producer_ids),
.reg_x_base(reg_x_base), .reg_w_base(reg_w_base),
.reg_n_tiles(reg_n_tiles), .reg_result_addr(reg_result_addr),
.mem_req(host_mem_req), .mem_wr(host_mem_wr), .mem_addr(host_mem_addr),
.mem_wdata(host_mem_wdata), .mem_lb_n(host_mem_lb_n), .mem_ub_n(host_mem_ub_n),
.mem_rdata(host_mem_rdata), .mem_ready(host_mem_ready),
.soft_rst_pulse(soft_rst_pulse)
);
// ============================================================
// COMPUTE + MEMORY (same wiring as nms_neural_multiprocessor_
// sdram_unified.v, plus the new host-arb level)
// ============================================================
wire [N_SLOTS:0] slot_mem_req, slot_mem_wr;
wire [ADDR_WIDTH*(N_SLOTS+1)-1:0] slot_mem_addr;
wire [16*(N_SLOTS+1)-1:0] slot_mem_wdata, slot_mem_rdata;
wire [N_SLOTS:0] slot_mem_lb_n, slot_mem_ub_n;
wire [N_SLOTS:0] slot_mem_ready;
wire [N_SLOTS-1:0] wide_slot_mem_req;
wire [ADDR_WIDTH*N_SLOTS-1:0] wide_slot_mem_addr;
wire [64*N_SLOTS-1:0] wide_slot_mem_rdata;
wire [N_SLOTS-1:0] wide_slot_mem_ready;
nms_dataflow_core_sdram_fast #(
.DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .ACC_WIDTH(ACC_WIDTH), .ADDR_WIDTH(ADDR_WIDTH),
.N_SLOTS(N_SLOTS), .N_NODES(N_NODES), .MAX_DEPS(MAX_DEPS), .QUEUE_DEPTH(QUEUE_DEPTH),
.MAX_TILES(MAX_TILES), .PREFETCH_DISTANCE(PREFETCH_DISTANCE)
) u_dataflow_core (
.clk(clk), .rst(core_rst),
.reg_valid(reg_valid), .reg_ready(reg_ready), .reg_node_id(reg_node_id),
.reg_required(reg_required), .reg_producer_ids(reg_producer_ids),
.reg_x_base(reg_x_base), .reg_w_base(reg_w_base), .reg_n_tiles(reg_n_tiles),
.reg_result_addr(reg_result_addr),
.data_ready(data_ready),
.slot_mem_req(slot_mem_req), .slot_mem_wr(slot_mem_wr), .slot_mem_addr(slot_mem_addr),
.slot_mem_wdata(slot_mem_wdata), .slot_mem_lb_n(slot_mem_lb_n), .slot_mem_ub_n(slot_mem_ub_n),
.slot_mem_rdata(slot_mem_rdata), .slot_mem_ready(slot_mem_ready),
.wide_slot_mem_req(wide_slot_mem_req), .wide_slot_mem_addr(wide_slot_mem_addr),
.wide_slot_mem_rdata(wide_slot_mem_rdata), .wide_slot_mem_ready(wide_slot_mem_ready)
);
// ---- AR level 1 (unchanged): activation-fill + per-slot result
// writeback, exactly as nms_neural_multiprocessor_sdram_unified.v ----
wire arb_m_req, arb_m_wr;
wire [ADDR_WIDTH-1:0] arb_m_addr;
wire [15:0] arb_m_wdata;
wire arb_m_lb_n, arb_m_ub_n;
wire [15:0] arb_m_rdata;
wire arb_m_ready;
slot_mem_arbiter #(
.ADDR_WIDTH(ADDR_WIDTH), .N_PORTS(N_SLOTS+1)
) u_arbiter (
.clk(clk), .rst(core_rst),
.s_req(slot_mem_req), .s_wr(slot_mem_wr), .s_addr(slot_mem_addr),
.s_wdata(slot_mem_wdata), .s_lb_n(slot_mem_lb_n), .s_ub_n(slot_mem_ub_n),
.s_rdata(slot_mem_rdata), .s_ready(slot_mem_ready),
.m_req(arb_m_req), .m_wr(arb_m_wr), .m_addr(arb_m_addr), .m_wdata(arb_m_wdata),
.m_lb_n(arb_m_lb_n), .m_ub_n(arb_m_ub_n),
.m_rdata(arb_m_rdata), .m_ready(arb_m_ready)
);
// ---- AR level 2 (NEW, STEP20): compute-side AR stream (port0)
// vs. SPI host raw memory port (port1) -- reuses slot_mem_arbiter
// completely unchanged, just at N_PORTS=2, its own already-proven
// pending-latch discipline applying equally to a 2-port instance ----
wire [1:0] host_arb_s_req, host_arb_s_wr, host_arb_s_lb_n, host_arb_s_ub_n, host_arb_s_ready;
wire [ADDR_WIDTH*2-1:0] host_arb_s_addr;
wire [16*2-1:0] host_arb_s_wdata, host_arb_s_rdata;
assign host_arb_s_req = {host_mem_req, arb_m_req};
assign host_arb_s_wr = {host_mem_wr, arb_m_wr};
assign host_arb_s_lb_n = {host_mem_lb_n, arb_m_lb_n};
assign host_arb_s_ub_n = {host_mem_ub_n, arb_m_ub_n};
assign host_arb_s_addr = {host_mem_addr, arb_m_addr};
assign host_arb_s_wdata = {host_mem_wdata, arb_m_wdata};
assign arb_m_ready = host_arb_s_ready[0];
assign arb_m_rdata = host_arb_s_rdata[15:0];
assign host_mem_ready = host_arb_s_ready[1];
assign host_mem_rdata = host_arb_s_rdata[31:16];
wire final_ar_req, final_ar_wr;
wire [ADDR_WIDTH-1:0] final_ar_addr;
wire [15:0] final_ar_wdata;
wire final_ar_lb_n, final_ar_ub_n;
wire [15:0] final_ar_rdata;
wire final_ar_ready;
slot_mem_arbiter #(
.ADDR_WIDTH(ADDR_WIDTH), .N_PORTS(2)
) u_host_arb (
.clk(clk), .rst(core_rst),
.s_req(host_arb_s_req), .s_wr(host_arb_s_wr), .s_addr(host_arb_s_addr),
.s_wdata(host_arb_s_wdata), .s_lb_n(host_arb_s_lb_n), .s_ub_n(host_arb_s_ub_n),
.s_rdata(host_arb_s_rdata), .s_ready(host_arb_s_ready),
.m_req(final_ar_req), .m_wr(final_ar_wr), .m_addr(final_ar_addr), .m_wdata(final_ar_wdata),
.m_lb_n(final_ar_lb_n), .m_ub_n(final_ar_ub_n),
.m_rdata(final_ar_rdata), .m_ready(final_ar_ready)
);
// ---- W: weight fetch (unchanged) ----
wire [N_SLOTS-1:0] wide_s_wr = {N_SLOTS{1'b0}};
wire [64*N_SLOTS-1:0] wide_s_wdata = {(64*N_SLOTS){1'b0}};
wire [N_SLOTS-1:0] wide_s_lb_n = {N_SLOTS{1'b0}};
wire [N_SLOTS-1:0] wide_s_ub_n = {N_SLOTS{1'b0}};
wire wide_arb_m_req, wide_arb_m_wr;
wire [ADDR_WIDTH-1:0] wide_arb_m_addr;
wire [63:0] wide_arb_m_wdata;
wire wide_arb_m_lb_n, wide_arb_m_ub_n;
wire [63:0] wide_arb_m_rdata;
wire wide_arb_m_ready;
slot_mem_arbiter_wide #(
.ADDR_WIDTH(ADDR_WIDTH), .N_PORTS(N_SLOTS), .DATA_WIDTH(64)
) u_arbiter_wide (
.clk(clk), .rst(core_rst),
.s_req(wide_slot_mem_req), .s_wr(wide_s_wr), .s_addr(wide_slot_mem_addr),
.s_wdata(wide_s_wdata), .s_lb_n(wide_s_lb_n), .s_ub_n(wide_s_ub_n),
.s_rdata(wide_slot_mem_rdata), .s_ready(wide_slot_mem_ready),
.m_req(wide_arb_m_req), .m_wr(wide_arb_m_wr), .m_addr(wide_arb_m_addr), .m_wdata(wide_arb_m_wdata),
.m_lb_n(wide_arb_m_lb_n), .m_ub_n(wide_arb_m_ub_n),
.m_rdata(wide_arb_m_rdata), .m_ready(wide_arb_m_ready)
);
// ---- ONE physical SDRAM backend, both W and (now 2-source-
// arbitrated) AR ports ----
sdram_unified_backend_openrow #(
.ADDR_WIDTH(ADDR_WIDTH), .CLK_FREQ_MHZ(CLK_FREQ_MHZ)
) u_sdram_backend (
.clk(clk), .rst(core_rst),
.w_req(wide_arb_m_req), .w_addr(wide_arb_m_addr),
.w_rdata(wide_arb_m_rdata), .w_ready(wide_arb_m_ready),
.ar_req(final_ar_req), .ar_wr(final_ar_wr), .ar_addr(final_ar_addr), .ar_wdata(final_ar_wdata),
.ar_lb_n(final_ar_lb_n), .ar_ub_n(final_ar_ub_n),
.ar_rdata(final_ar_rdata), .ar_ready(final_ar_ready),
.sdram_cke(sdram_cke), .sdram_cs_n(sdram_cs_n), .sdram_ras_n(sdram_ras_n),
.sdram_cas_n(sdram_cas_n), .sdram_we_n(sdram_we_n),
.sdram_ba(sdram_ba), .sdram_a(sdram_a), .sdram_dq(sdram_dq), .sdram_dqm(sdram_dqm)
);
endmodule
hardware/v2/nms/rtl/nms_activation_fill_ctrl_v3_n16.v
+310
View File
@@ -0,0 +1,310 @@
// ============================================================
// Neural Memory System (NMS) -- shared Activation fill controller.
//
// One instance per neural_memory_system (shared across all N_SLOTS),
// backing nms_activation_replicated.v's single broadcast-write fill
// port. Owns ONE prefetch_engine.v instance (real word-level PSRAM
// fetch, DEC-0015 convention, reused verbatim -- it is generic
// P_IN-byte-tile fetch logic, not weight-specific despite its name).
//
// Single-tag design (same honest limitation as the superseded
// hardware/v2/rtl/activation_cache.v, DEC-0016): tracks ONE resident
// x_base at a time. Refills (resident_count resets to 0, restarts
// fetching from tile 0) whenever the lowest-indexed currently-active
// slot's own x_base differs from what is resident -- correct always,
// but can thrash under interleaved, genuinely-different-x_base
// concurrent traffic; not exercised by this project's own realistic
// dense-layer workloads (shared-producer dispatch, many slots given
// the SAME x_base together).
//
// resident_count extends to the MAX n_tiles needed by any currently
// active slot that shares resident_tag (not just the reference slot
// that triggered the refill), so a later-joining slot with a deeper
// need is served without a second refill.
// ============================================================
module nms_activation_fill_ctrl_v3_n16 #(
parameter DATA_WIDTH = 8,
parameter P_IN = 8,
parameter N_SLOTS = 4,
parameter ADDR_WIDTH = 26,
parameter MAX_TILES = 16,
// TIW indexes the SRAM fill address (0..MAX_TILES-1); CNTW is for
// resident_count, which must represent the VALUE MAX_TILES itself
// (e.g. a fully-resident 16-tile vector with MAX_TILES=16) -- one
// bit wider than TIW, same distinction/bug as
// nms_memory_manager.v's own tile_idx/wgt_fetched (see that file's
// header for the real deadlock this caused before the fix).
parameter TIW = (MAX_TILES <= 1) ? 1 : $clog2(MAX_TILES),
parameter CNTW = $clog2(MAX_TILES+1)
)(
input wire clk,
input wire rst,
// ---- per-slot job status (levels, held while that slot's job is active) ----
input wire [N_SLOTS-1:0] job_active,
input wire [N_SLOTS*ADDR_WIDTH-1:0] x_base_flat,
input wire [N_SLOTS*16-1:0] n_tiles_flat,
// ---- broadcast status (every slot compares this against its own x_base) ----
output reg [ADDR_WIDTH-1:0] resident_tag,
output reg [CNTW-1:0] resident_count,
// ---- fill port into nms_activation_replicated.v ----
output wire fill_we,
output wire [TIW-1:0] fill_addr,
output wire [DATA_WIDTH*P_IN-1:0] fill_data,
// ---- real word-level PSRAM backend (arbitrated externally) ----
output wire mem_req,
output wire mem_wr,
output wire [ADDR_WIDTH-1:0] mem_addr,
output wire [15:0] mem_wdata,
output wire mem_lb_n,
output wire mem_ub_n,
input wire [15:0] mem_rdata,
input wire mem_ready
);
integer i;
// ---- desired x_base: lowest-indexed currently-active slot (fixed
// priority -- simple, not fairness-critical here since this only
// decides which TAG to chase, not who gets bandwidth) ----
reg desired_valid;
reg [ADDR_WIDTH-1:0] desired_x_base;
always @* begin
desired_valid = 1'b0;
desired_x_base = {ADDR_WIDTH{1'b0}};
for (i = N_SLOTS-1; i >= 0; i = i - 1) begin
if (job_active[i]) begin
desired_valid = 1'b1;
desired_x_base = x_base_flat[i*ADDR_WIDTH +: ADDR_WIDTH];
end
end
end
// ---- STEP14/EXP-0029->EXP-0030: max_n_tiles computation split
// into TWO pipeline stages, since registering ONLY its final use
// (v2, DEC-0026) left the computation ITSELF as the new critical
// path (EXP-0030, N=4 Fmax=72.78MHz, still FAIL@80MHz): the
// original single-cycle logic mixed, PER SLOT, a 23-bit tag
// equality check (x_base_flat[i]==resident_tag) together with an
// N_SLOTS-wide SEQUENTIALLY-CHAINED 16-bit running-max fold (each
// iteration's update depends on the previous one) -- both
// combinational, both in the same cycle as the register that
// captures the result.
//
// Stage 1 (independent per-slot work, no chain dependency between
// slots): register a per-slot "counts toward this refill" mask
// (job_active[i] && tag-match) and, gated by that mask, each
// slot's own n_tiles value (0 if it doesn't count) -- N_SLOTS
// independent 23-bit equality checks, no data dependency between
// slots, so their combined depth does not grow with N_SLOTS the
// way a sequential fold does.
// Stage 2 (the actual reduction): fold the REGISTERED, already-
// masked per-slot values into max_n_tiles_reg -- still an
// N_SLOTS-wide sequential chain (same fold as before), but now
// operating alone, without the equality check sharing the same
// cycle.
reg [15:0] n_tiles_masked [0:N_SLOTS-1];
genvar gsi;
generate
for (gsi = 0; gsi < N_SLOTS; gsi = gsi + 1) begin : GEN_MASK
wire slot_counts = job_active[gsi] &&
(x_base_flat[gsi*ADDR_WIDTH +: ADDR_WIDTH] == resident_tag);
always @(posedge clk) begin
if (rst) n_tiles_masked[gsi] <= 16'h0;
else n_tiles_masked[gsi] <= slot_counts ? n_tiles_flat[gsi*16 +: 16] : 16'h0;
end
end
endgenerate
// Balanced binary max-tree (log2(N_SLOTS) comparison levels)
// instead of the flat N_SLOTS-wide sequential scan this file's own
// header comment above already flagged as "an N_SLOTS-wide
// sequential chain". Found and fixed this session: that chain's
// own carry-chain critical path became the DOMINANT critical path
// at N_SLOTS=8 (real nextpnr-ecp5 P&R: Fmax collapsed to ~40MHz,
// failing the 64MHz target across every measured seed). A tree
// has the SAME single-cycle combinational timing as the scan it
// replaces (max_n_tiles_reg is still registered exactly one cycle
// behind n_tiles_masked -- no FSM/latency change, purely a
// combinational-depth reduction: log2(N_SLOTS) levels instead of
// N_SLOTS).
//
// Written as explicit, uniquely-named per-level wires (NOT a
// multi-dimensional generate-indexed array) -- a first attempt
// using a shared 2D `wire max_tree[level][idx]` array triggered a
// real simulator UNOPTFLAT "circular combinational logic" warning.
// The actual dependency graph IS acyclic (level L+1 only ever
// reads level L), but that tool's array-flattening circularity
// check could not prove that for a shared 2D array; distinctly-
// named per-level wires sidestep the ambiguity entirely for both
// simulation and synthesis. N_SLOTS is a power of two for every
// real configuration this project uses (1/2/4/8); anything else
// falls back, explicitly, to the original flat scan (correct but not
// optimized) rather than silently doing the wrong thing.
reg [15:0] max_n_tiles_reg;
generate
if (N_SLOTS == 1) begin : GEN_MAXTREE_N1
always @(posedge clk) begin
if (rst) max_n_tiles_reg <= 16'h0;
else max_n_tiles_reg <= n_tiles_masked[0];
end
end else if (N_SLOTS == 2) begin : GEN_MAXTREE_N2
wire [15:0] max_final = (n_tiles_masked[0] > n_tiles_masked[1]) ? n_tiles_masked[0] : n_tiles_masked[1];
always @(posedge clk) begin
if (rst) max_n_tiles_reg <= 16'h0;
else max_n_tiles_reg <= max_final;
end
end else if (N_SLOTS == 4) begin : GEN_MAXTREE_N4
wire [15:0] m0 = (n_tiles_masked[0] > n_tiles_masked[1]) ? n_tiles_masked[0] : n_tiles_masked[1];
wire [15:0] m1 = (n_tiles_masked[2] > n_tiles_masked[3]) ? n_tiles_masked[2] : n_tiles_masked[3];
wire [15:0] max_final = (m0 > m1) ? m0 : m1;
always @(posedge clk) begin
if (rst) max_n_tiles_reg <= 16'h0;
else max_n_tiles_reg <= max_final;
end
end else if (N_SLOTS == 8) begin : GEN_MAXTREE_N8
wire [15:0] m0 = (n_tiles_masked[0] > n_tiles_masked[1]) ? n_tiles_masked[0] : n_tiles_masked[1];
wire [15:0] m1 = (n_tiles_masked[2] > n_tiles_masked[3]) ? n_tiles_masked[2] : n_tiles_masked[3];
wire [15:0] m2 = (n_tiles_masked[4] > n_tiles_masked[5]) ? n_tiles_masked[4] : n_tiles_masked[5];
wire [15:0] m3 = (n_tiles_masked[6] > n_tiles_masked[7]) ? n_tiles_masked[6] : n_tiles_masked[7];
wire [15:0] m01 = (m0 > m1) ? m0 : m1;
wire [15:0] m23 = (m2 > m3) ? m2 : m3;
wire [15:0] max_final = (m01 > m23) ? m01 : m23;
always @(posedge clk) begin
if (rst) max_n_tiles_reg <= 16'h0;
else max_n_tiles_reg <= max_final;
end
end else if (N_SLOTS == 16) begin : GEN_MAXTREE_N16
// EXP-0056: same balanced-tree pattern as N_SLOTS==8 above,
// one more level. This is the exact case that used to fall
// through to GEN_MAXTREE_FALLBACK's own flat sequential
// scan -- the real critical path measured blocking
// N_SLOTS=16 timing closure on the LFE5U-85F (worst
// 23.52-24.64MHz vs 64MHz target across two independent
// seeds, both pre-fix; see experiments.log EXP-0056).
wire [15:0] m0 = (n_tiles_masked[0] > n_tiles_masked[1]) ? n_tiles_masked[0] : n_tiles_masked[1];
wire [15:0] m1 = (n_tiles_masked[2] > n_tiles_masked[3]) ? n_tiles_masked[2] : n_tiles_masked[3];
wire [15:0] m2 = (n_tiles_masked[4] > n_tiles_masked[5]) ? n_tiles_masked[4] : n_tiles_masked[5];
wire [15:0] m3 = (n_tiles_masked[6] > n_tiles_masked[7]) ? n_tiles_masked[6] : n_tiles_masked[7];
wire [15:0] m4 = (n_tiles_masked[8] > n_tiles_masked[9]) ? n_tiles_masked[8] : n_tiles_masked[9];
wire [15:0] m5 = (n_tiles_masked[10] > n_tiles_masked[11]) ? n_tiles_masked[10] : n_tiles_masked[11];
wire [15:0] m6 = (n_tiles_masked[12] > n_tiles_masked[13]) ? n_tiles_masked[12] : n_tiles_masked[13];
wire [15:0] m7 = (n_tiles_masked[14] > n_tiles_masked[15]) ? n_tiles_masked[14] : n_tiles_masked[15];
wire [15:0] m01 = (m0 > m1) ? m0 : m1;
wire [15:0] m23 = (m2 > m3) ? m2 : m3;
wire [15:0] m45 = (m4 > m5) ? m4 : m5;
wire [15:0] m67 = (m6 > m7) ? m6 : m7;
wire [15:0] m0123 = (m01 > m23) ? m01 : m23;
wire [15:0] m4567 = (m45 > m67) ? m45 : m67;
wire [15:0] max_final = (m0123 > m4567) ? m0123 : m4567;
always @(posedge clk) begin
if (rst) max_n_tiles_reg <= 16'h0;
else max_n_tiles_reg <= max_final;
end
end else begin : GEN_MAXTREE_FALLBACK
reg [15:0] max_n_tiles_comb_fallback;
integer j;
always @* begin
max_n_tiles_comb_fallback = 16'h0;
for (j = 0; j < N_SLOTS; j = j + 1)
if (n_tiles_masked[j] > max_n_tiles_comb_fallback)
max_n_tiles_comb_fallback = n_tiles_masked[j];
end
always @(posedge clk) begin
if (rst) max_n_tiles_reg <= 16'h0;
else max_n_tiles_reg <= max_n_tiles_comb_fallback;
end
end
endgenerate
localparam ST_IDLE = 1'd0;
localparam ST_FETCH = 1'd1;
reg state;
reg pf_start;
reg [ADDR_WIDTH-1:0] pf_addr;
wire pf_busy, pf_done;
wire signed [DATA_WIDTH*P_IN-1:0] pf_tile;
prefetch_engine #(
.DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .ADDR_WIDTH(ADDR_WIDTH)
) u_pf (
.clk(clk), .rst(rst),
.fetch_start(pf_start), .w_addr(pf_addr),
.fetch_busy(pf_busy), .fetch_done(pf_done), .tile_w(pf_tile),
.mem_req(mem_req), .mem_wr(mem_wr), .mem_addr(mem_addr), .mem_wdata(mem_wdata),
.mem_lb_n(mem_lb_n), .mem_ub_n(mem_ub_n),
.mem_rdata(mem_rdata), .mem_ready(mem_ready)
);
reg fill_we_reg;
reg [TIW-1:0] fill_addr_reg;
reg [DATA_WIDTH*P_IN-1:0] fill_data_reg;
assign fill_we = fill_we_reg;
assign fill_addr = fill_addr_reg;
assign fill_data = fill_data_reg;
always @(posedge clk) begin
if (rst) begin
resident_tag <= {ADDR_WIDTH{1'b1}}; // sentinel: matches no real x_base at reset
resident_count <= {CNTW{1'b0}};
state <= ST_IDLE;
pf_start <= 1'b0;
fill_we_reg <= 1'b0;
end else begin
pf_start <= 1'b0;
fill_we_reg <= 1'b0;
// latch a completed fetch into the replicated activation
// memory's broadcast fill port
if (pf_done) begin
fill_we_reg <= 1'b1;
fill_addr_reg <= resident_count[TIW-1:0]; // valid: gated < max_n_tiles <= MAX_TILES
fill_data_reg <= pf_tile;
resident_count <= resident_count + 1'b1;
end
// refill trigger: the reference slot wants a DIFFERENT tag,
// and the fetch engine is genuinely idle (never interrupt an
// in-flight fetch -- same discipline as memory_manager.v's
// own pf_pending guard, ERR-0006). Stays in ST_IDLE (not
// ST_FETCH): only updates resident_tag/resident_count here;
// the ST_IDLE case below is what actually issues pf_start,
// reading the NEW resident_tag starting next cycle -- this
// path must NOT itself jump to ST_FETCH without a matching
// pf_start, or the engine would sit in ST_FETCH forever
// waiting for a pf_done that was never triggered.
if (desired_valid && (desired_x_base != resident_tag) && !pf_busy && (state == ST_IDLE)) begin
resident_tag <= desired_x_base;
resident_count <= {CNTW{1'b0}};
end
case (state)
ST_IDLE: begin
// stay idle: fetching further tiles for the CURRENT
// tag (if any active slot still needs more) is
// handled below, symmetric to the refill case.
if (!pf_busy && !pf_start && (resident_count < max_n_tiles_reg) &&
desired_valid && (desired_x_base == resident_tag)) begin
pf_start <= 1'b1;
pf_addr <= resident_tag + (resident_count * P_IN[ADDR_WIDTH-1:0]);
state <= ST_FETCH;
end
end
ST_FETCH: begin
if (pf_done) begin
// resident_count already bumped above this cycle;
// decide whether more remain once back in IDLE.
state <= ST_IDLE;
end
end
default: state <= ST_IDLE;
endcase
end
end
endmodule
hardware/v2/nms/rtl/nms_dataflow_core_sdram_fast.v
+325
View File
@@ -0,0 +1,325 @@
`timescale 1ns/1ps
// ================================================================
// Neural Memory System (NMS) -- STEP8 full integration, mirrors
// hardware/v2/rtl/dataflow_core.v's own scope exactly (M6 Dependency
// Manager -> M5 Neural Director -> N_SLOTS x (memory manager + neural
// processor)), but replaces the M4 memory_manager.v +
// activation_cache.v cluster with the NMS's own decided pieces
// (DEC-0019/DEC-0020):
// - nms_activation_replicated.v: N_SLOTS private full-vector
// activation copies, broadcast-filled by...
// - nms_activation_fill_ctrl.v: the shared dedup/fetch controller
// (single logical tag, same honest thrash-under-interleaved-
// different-x_base limitation as the superseded activation_cache.v)
// - nms_weight_packed.v: N_SLOTS private, per-MAC-lane packed weight
// copies (never shared, no arbitration needed)
// - nms_memory_manager.v: per-slot job FSM, reads directly from the
// two SRAMs above instead of double-buffering 2 banks (the whole
// vector is resident, not just 2 tiles worth)
//
// hardware/v2/rtl/dependency_manager.v and neural_director.v are
// REUSED VERBATIM, unmodified -- the node-registration and slot-
// dispatch protocol did not change at all; only what happens between
// "job dispatched to a slot" and "job_done" changed.
//
// Memory Backend Interface: exposed N_SLOTS+1 wide exactly like
// dataflow_core.v (indices [0,N_SLOTS) = per-slot memory managers'
// own weight-fetch+result-write port, index [N_SLOTS] = the shared
// activation fill controller's own port) -- arbitrated one level up,
// reusing hardware/v2/rtl/slot_mem_arbiter.v unchanged.
//
// STEP16 Phase 5: forked from nms_dataflow_core_dual32.v (STEP15's
// own dual-chip-32-bit variant) with ONLY the wide weight-fetch port
// widened from 32 to 64 bits (MEM_DATA_WIDTH=64 in the per-slot
// nms_memory_manager_stream_wide instance below) -- the natural
// P_IN*DATA_WIDTH/16=4-word SDRAM burst identified in Phase 1 means
// ONE mem_req now fetches exactly one whole tile (WORDS_PER_TILE=1),
// same as the dual32 case fetched one whole tile per 32-bit request.
// Everything else (activation path, dependency manager, neural
// director, per-slot neural_processor) is BYTE-FOR-BYTE UNCHANGED --
// per the governing spec's own "do not create an artificial
// benchmark" instruction, only the piece under test (weight-fetch
// physical memory) differs from the validated dual32 baseline.
// ================================================================
module nms_dataflow_core_sdram_fast #(
parameter DATA_WIDTH = 8,
parameter P_IN = 8,
parameter ACC_WIDTH = 32,
parameter ADDR_WIDTH = 26,
parameter N_SLOTS = 4,
parameter N_NODES = 16,
parameter MAX_DEPS = 4,
parameter QUEUE_DEPTH = 8,
parameter MAX_TILES = 16,
parameter PREFETCH_DISTANCE = 8,
parameter TIW = (MAX_TILES <= 1) ? 1 : $clog2(MAX_TILES),
// Must match nms_memory_manager.v's/nms_activation_fill_ctrl.v's
// own CNTW exactly -- this top-level wire connecting the two was
// left at the narrower TIW after those modules were widened,
// silently truncating resident_count's real value (16) back down
// to 0 right when it should have reached MAX_TILES, deadlocking
// the very last tile of any n_tiles==MAX_TILES job forever (found
// via simulation: D-Stress's real 16-tile neurons hung 1 tile
// short, act_resident_count visibly reset to 0 the exact cycle it
// should have become 16).
parameter CNTW = $clog2(MAX_TILES+1)
)(
input wire clk,
input wire rst,
input wire reg_valid,
output wire reg_ready,
input wire [$clog2(N_NODES)-1:0] reg_node_id,
input wire [$clog2(MAX_DEPS+1)-1:0] reg_required,
input wire [MAX_DEPS*$clog2(N_NODES)-1:0] reg_producer_ids,
input wire [ADDR_WIDTH-1:0] reg_x_base,
input wire [ADDR_WIDTH-1:0] reg_w_base,
input wire [15:0] reg_n_tiles,
input wire [ADDR_WIDTH-1:0] reg_result_addr,
// FPGA_DATA_READY: see the assignment site (below u_director) for
// the full design comment.
output wire data_ready,
output wire [N_SLOTS:0] slot_mem_req,
output wire [N_SLOTS:0] slot_mem_wr,
output wire [ADDR_WIDTH*(N_SLOTS+1)-1:0] slot_mem_addr,
output wire [16*(N_SLOTS+1)-1:0] slot_mem_wdata,
output wire [N_SLOTS:0] slot_mem_lb_n,
output wire [N_SLOTS:0] slot_mem_ub_n,
input wire [16*(N_SLOTS+1)-1:0] slot_mem_rdata,
input wire [N_SLOTS:0] slot_mem_ready,
// ---- STEP16 Phase 5: wide (64-bit logical) weight-fetch backend
// interface, per slot -- N_SLOTS wide (NOT N_SLOTS+1: the shared
// activation-fill controller stays on the ORIGINAL 16-bit port
// above, unchanged). Arbitrated one level up (slot_mem_arbiter_
// wide.v, reused unchanged at DATA_WIDTH=64) down to the real
// SDRAM physical interface (sdram_weight_backend.v). ----
output wire [N_SLOTS-1:0] wide_slot_mem_req,
output wire [ADDR_WIDTH*N_SLOTS-1:0] wide_slot_mem_addr,
input wire [64*N_SLOTS-1:0] wide_slot_mem_rdata,
input wire [N_SLOTS-1:0] wide_slot_mem_ready
);
localparam NODE_IDW = $clog2(N_NODES);
wire dm_ready_valid;
wire dm_ready_ready;
wire [NODE_IDW-1:0] dm_ready_node_id;
wire [ADDR_WIDTH-1:0] dm_ready_x_base, dm_ready_w_base, dm_ready_result_addr;
wire [15:0] dm_ready_n_tiles;
wire dm_producer_done_valid;
wire [NODE_IDW-1:0] dm_producer_done_node_id;
wire dm_any_pending;
dependency_manager_fast #(
.N_NODES(N_NODES), .MAX_DEPS(MAX_DEPS), .ADDR_WIDTH(ADDR_WIDTH)
) u_dep_mgr (
.clk(clk), .rst(rst),
.reg_valid(reg_valid), .reg_ready(reg_ready), .reg_node_id(reg_node_id),
.reg_required(reg_required), .reg_producer_ids(reg_producer_ids),
.reg_x_base(reg_x_base), .reg_w_base(reg_w_base), .reg_n_tiles(reg_n_tiles),
.reg_result_addr(reg_result_addr),
.producer_done_valid(dm_producer_done_valid), .producer_done_node_id(dm_producer_done_node_id),
.ready_valid(dm_ready_valid), .ready_ready(dm_ready_ready), .ready_node_id(dm_ready_node_id),
.ready_x_base(dm_ready_x_base), .ready_w_base(dm_ready_w_base),
.ready_n_tiles(dm_ready_n_tiles), .ready_result_addr(dm_ready_result_addr),
.any_pending(dm_any_pending)
);
wire [15:0] dm_ready_node_id_ext = {{(16-NODE_IDW){1'b0}}, dm_ready_node_id};
wire [N_SLOTS-1:0] dir_slot_job_start;
wire [ADDR_WIDTH*N_SLOTS-1:0] dir_slot_x_base, dir_slot_w_base, dir_slot_result_addr;
wire [16*N_SLOTS-1:0] dir_slot_n_tiles, dir_slot_node_id;
wire [N_SLOTS-1:0] dir_slot_job_done;
wire dir_job_out_done;
wire [$clog2(N_SLOTS)-1:0] dir_job_out_slot;
wire [3:0] dir_state;
wire dir_error;
wire dir_queue_empty;
neural_director #(
.ADDR_WIDTH(ADDR_WIDTH), .N_SLOTS(N_SLOTS), .QUEUE_DEPTH(QUEUE_DEPTH)
) u_director (
.clk(clk), .rst(rst),
.job_in_valid(dm_ready_valid), .job_in_ready(dm_ready_ready),
.job_in_x_base(dm_ready_x_base), .job_in_w_base(dm_ready_w_base),
.job_in_n_tiles(dm_ready_n_tiles), .job_in_result_addr(dm_ready_result_addr),
.job_in_node_id(dm_ready_node_id_ext),
.slot_job_start(dir_slot_job_start), .slot_x_base(dir_slot_x_base), .slot_w_base(dir_slot_w_base),
.slot_n_tiles(dir_slot_n_tiles), .slot_result_addr(dir_slot_result_addr),
.slot_node_id(dir_slot_node_id), .slot_job_done(dir_slot_job_done),
.job_out_done(dir_job_out_done), .job_out_slot(dir_job_out_slot),
.dir_state(dir_state), .dir_error(dir_error), .queue_empty(dir_queue_empty)
);
// ---- FPGA_DATA_READY: system-idle detection (see decisions.log
// for the full design rationale) ----
// sys_busy: true while ANY of {a slot is active, the director's
// dispatch queue is non-empty, dependency_manager has a node not
// yet dispatched} holds. data_ready is a sticky level that goes
// HIGH on the busy->idle falling edge (a graph just finished) and
// LOW again the instant any new work starts (registration or
// dispatch) -- self-clearing, no explicit host ACK needed. Correct
// ONLY if the host finishes registering every node of a graph
// before the first one completes (documented assumption, see
// decisions.log) -- registration (microseconds over SPI) is far
// faster than per-neuron compute (~195 real measured cycles) for
// every workload this project has characterized.
wire sys_busy = (|job_active) || (!dir_queue_empty) || dm_any_pending;
reg sys_busy_prev;
reg data_ready_reg;
always @(posedge clk) begin
if (rst) begin
sys_busy_prev <= 1'b0;
data_ready_reg <= 1'b0;
end else begin
sys_busy_prev <= sys_busy;
if (sys_busy) data_ready_reg <= 1'b0;
else if (sys_busy_prev) data_ready_reg <= 1'b1;
end
end
assign data_ready = data_ready_reg;
wire [15:0] completed_node_id_16 = dir_slot_node_id[dir_job_out_slot*16 +: 16];
assign dm_producer_done_valid = dir_job_out_done;
assign dm_producer_done_node_id = completed_node_id_16[NODE_IDW-1:0];
// ---- NMS memory: shared Activation SRAM (replicated) + private
// Weight SRAM (packed), per DEC-0019/DEC-0020 ----
wire fill_we;
wire [TIW-1:0] fill_addr;
wire signed [DATA_WIDTH*P_IN-1:0] fill_data;
wire [ADDR_WIDTH-1:0] act_resident_tag;
wire [CNTW-1:0] act_resident_count;
wire [N_SLOTS-1:0] act_rd_en;
wire [N_SLOTS*TIW-1:0] act_rd_addr_flat;
wire signed [DATA_WIDTH*P_IN*N_SLOTS-1:0] act_rd_data_flat;
nms_activation_replicated #(
.DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .N_SLOTS(N_SLOTS), .MAX_TILES(MAX_TILES)
) u_act_mem (
.clk(clk), .rst(rst),
.fill_we(fill_we), .fill_addr(fill_addr), .fill_data(fill_data),
.rd_en(act_rd_en), .rd_addr_flat(act_rd_addr_flat), .rd_data_flat(act_rd_data_flat)
);
wire [N_SLOTS-1:0] job_active;
wire [ADDR_WIDTH*N_SLOTS-1:0] job_x_base_flat;
wire [16*N_SLOTS-1:0] job_n_tiles_flat;
nms_activation_fill_ctrl_v3_n16 #(
.DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .N_SLOTS(N_SLOTS), .ADDR_WIDTH(ADDR_WIDTH), .MAX_TILES(MAX_TILES)
) u_act_fill (
.clk(clk), .rst(rst),
.job_active(job_active), .x_base_flat(job_x_base_flat), .n_tiles_flat(job_n_tiles_flat),
.resident_tag(act_resident_tag), .resident_count(act_resident_count),
.fill_we(fill_we), .fill_addr(fill_addr), .fill_data(fill_data),
.mem_req(slot_mem_req[N_SLOTS]), .mem_wr(slot_mem_wr[N_SLOTS]),
.mem_addr(slot_mem_addr[N_SLOTS*ADDR_WIDTH +: ADDR_WIDTH]),
.mem_wdata(slot_mem_wdata[N_SLOTS*16 +: 16]),
.mem_lb_n(slot_mem_lb_n[N_SLOTS]), .mem_ub_n(slot_mem_ub_n[N_SLOTS]),
.mem_rdata(slot_mem_rdata[N_SLOTS*16 +: 16]), .mem_ready(slot_mem_ready[N_SLOTS])
);
wire [N_SLOTS-1:0] wgt_fill_we;
wire [N_SLOTS*TIW-1:0] wgt_fill_addr_flat;
wire signed [DATA_WIDTH*P_IN*N_SLOTS-1:0] wgt_fill_data_flat;
wire [N_SLOTS-1:0] wgt_rd_en;
wire [N_SLOTS*TIW-1:0] wgt_rd_addr_flat;
wire signed [DATA_WIDTH*P_IN*N_SLOTS-1:0] wgt_rd_data_flat;
nms_weight_packed #(
.DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .N_SLOTS(N_SLOTS), .MAX_TILES(MAX_TILES)
) u_wgt_mem (
.clk(clk), .rst(rst),
.fill_we(wgt_fill_we), .fill_addr_flat(wgt_fill_addr_flat), .fill_data_flat(wgt_fill_data_flat),
.rd_en(wgt_rd_en), .rd_addr_flat(wgt_rd_addr_flat), .rd_data_flat(wgt_rd_data_flat)
);
genvar g;
generate
for (g = 0; g < N_SLOTS; g = g + 1) begin : GEN_SLOT
wire mm_operand_valid, mm_operand_ready;
wire signed [DATA_WIDTH*P_IN-1:0] mm_input_data, mm_weight_data;
wire mm_tile_last;
wire mm_result_valid, mm_result_ready;
wire signed [DATA_WIDTH-1:0] mm_result_data;
nms_memory_manager_stream_wide #(
.DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .ADDR_WIDTH(ADDR_WIDTH), .MAX_TILES(MAX_TILES),
.PREFETCH_DISTANCE(PREFETCH_DISTANCE), .MEM_DATA_WIDTH(64)
) u_mm (
.clk(clk), .rst(rst),
.job_start(dir_slot_job_start[g]),
.x_base(dir_slot_x_base[g*ADDR_WIDTH +: ADDR_WIDTH]),
.w_base(dir_slot_w_base[g*ADDR_WIDTH +: ADDR_WIDTH]),
.n_tiles(dir_slot_n_tiles[g*16 +: 16]),
.result_addr(dir_slot_result_addr[g*ADDR_WIDTH +: ADDR_WIDTH]),
.job_done(dir_slot_job_done[g]),
.operand_valid(mm_operand_valid), .operand_ready(mm_operand_ready),
.input_data(mm_input_data), .weight_data(mm_weight_data), .tile_last(mm_tile_last),
.result_valid(mm_result_valid), .result_ready(mm_result_ready), .result_data(mm_result_data),
.job_active(job_active[g]),
.job_x_base(job_x_base_flat[g*ADDR_WIDTH +: ADDR_WIDTH]),
.job_n_tiles(job_n_tiles_flat[g*16 +: 16]),
.act_resident_tag(act_resident_tag), .act_resident_count(act_resident_count),
.act_rd_en(act_rd_en[g]),
.act_rd_addr(act_rd_addr_flat[g*TIW +: TIW]),
.act_rd_data(act_rd_data_flat[g*DATA_WIDTH*P_IN +: DATA_WIDTH*P_IN]),
.wgt_fill_we(wgt_fill_we[g]),
.wgt_fill_addr(wgt_fill_addr_flat[g*TIW +: TIW]),
.wgt_fill_data(wgt_fill_data_flat[g*DATA_WIDTH*P_IN +: DATA_WIDTH*P_IN]),
.wgt_rd_en(wgt_rd_en[g]),
.wgt_rd_addr(wgt_rd_addr_flat[g*TIW +: TIW]),
.wgt_rd_data(wgt_rd_data_flat[g*DATA_WIDTH*P_IN +: DATA_WIDTH*P_IN]),
.mem_req(slot_mem_req[g]), .mem_wr(slot_mem_wr[g]),
.mem_addr(slot_mem_addr[g*ADDR_WIDTH +: ADDR_WIDTH]),
.mem_wdata(slot_mem_wdata[g*16 +: 16]),
.mem_lb_n(slot_mem_lb_n[g]), .mem_ub_n(slot_mem_ub_n[g]),
.mem_rdata(slot_mem_rdata[g*16 +: 16]), .mem_ready(slot_mem_ready[g]),
.wide_mem_req(wide_slot_mem_req[g]),
.wide_mem_addr(wide_slot_mem_addr[g*ADDR_WIDTH +: ADDR_WIDTH]),
.wide_mem_rdata(wide_slot_mem_rdata[g*64 +: 64]),
.wide_mem_ready(wide_slot_mem_ready[g])
);
reg job_valid_np;
wire job_ready_np;
wire result_valid_np;
wire signed [DATA_WIDTH-1:0] result_data_np;
wire [3:0] np_state;
wire np_error;
neural_processor #(
.DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .ACC_WIDTH(ACC_WIDTH)
) u_np (
.clk(clk), .rst(rst),
.job_valid(job_valid_np), .job_ready(job_ready_np),
.job_node_id(16'h0), .job_bias(8'sd0), .job_activation(2'd1),
.operand_valid(mm_operand_valid), .operand_ready(mm_operand_ready),
.input_data(mm_input_data), .weight_data(mm_weight_data), .tile_last(mm_tile_last),
.result_valid(result_valid_np), .result_ready(mm_result_ready),
.result_data(result_data_np), .result_node_id(),
.np_state(np_state), .np_error(np_error)
);
assign mm_result_valid = result_valid_np;
assign mm_result_data = result_data_np;
always @(posedge clk) begin
if (rst) job_valid_np <= 1'b0;
else if (dir_slot_job_start[g]) job_valid_np <= 1'b1;
else if (job_valid_np && job_ready_np) job_valid_np <= 1'b0;
end
end
endgenerate
endmodule
hardware/v2/nms/rtl/nms_neural_multiprocessor_sdram_openrow_fast.v
+171
View File
@@ -0,0 +1,171 @@
`timescale 1ns/1ps
// ================================================================
// Neural Memory System (NMS) -- STEP19 real hardware-facing top level.
//
// SINGLE EXTERNAL SDRAM ONLY. Forked from nms_neural_multiprocessor_
// sdram_pack128.v (STEP18) with the ONE change this step's own
// governing spec mandates: the real hardware/v1/rtl/psram_controller.v
// + memory_interface.v pairing (activation-fill + result-writeback,
// 16-bit) is REMOVED from the V2 physical path entirely and replaced
// by sdram_unified_backend.v's own AR port, sharing the SAME single
// physical AS4C4M16SA-6TIN SDRAM chip and the SAME single sdram_
// controller.v instance the weight-fetch path (W port) already uses.
//
// slot_mem_arbiter.v (16-bit, activation+result) and slot_mem_
// arbiter_wide.v (64-bit, weight) are BOTH reused completely
// UNCHANGED -- their own downstream ports now both terminate at
// sdram_unified_backend.v instead of two separate physical chains.
// nms_dataflow_core_sdram.v, nms_activation_fill_ctrl_v3.v, nms_
// memory_manager_stream_wide.v, weight_prefetch_engine_wide.v, and
// neural_processor.v are ALL byte-for-byte unchanged -- this is a
// pure memory-side substitution, per the governing spec's own
// explicit instruction.
//
// V1 (hardware/v1/**) is untouched -- psram_controller.v and memory_
// interface.v simply are no longer INSTANTIATED by this top-level;
// neither file was modified, and V1's own golden-reference status is
// unaffected.
//
// Real pin count (weight+activation+result, ALL through ONE chip):
// 2(BA)+12(A)+1(CKE)+1(CS#)+1(RAS#)+1(CAS#)+1(WE#)+2(DQM)+16(DQ) = 37
// pins total -- the SAME 37 pins the weight-only path already used in
// STEP16-18 (no NEW physical SDRAM pins are needed to add activation/
// result traffic, since it shares the identical physical bus).
// ================================================================
module nms_neural_multiprocessor_sdram_openrow_fast #(
parameter DATA_WIDTH = 8,
parameter P_IN = 8,
parameter ACC_WIDTH = 32,
parameter ADDR_WIDTH = 26,
parameter N_SLOTS = 2,
parameter N_NODES = 16,
parameter MAX_DEPS = 4,
parameter QUEUE_DEPTH = 8,
parameter MAX_TILES = 16,
parameter PREFETCH_DISTANCE = 8,
parameter CLK_FREQ_MHZ = 80
)(
input wire clk,
input wire rst,
input wire reg_valid,
output wire reg_ready,
input wire [$clog2(N_NODES)-1:0] reg_node_id,
input wire [$clog2(MAX_DEPS+1)-1:0] reg_required,
input wire [MAX_DEPS*$clog2(N_NODES)-1:0] reg_producer_ids,
input wire [ADDR_WIDTH-1:0] reg_x_base,
input wire [ADDR_WIDTH-1:0] reg_w_base,
input wire [15:0] reg_n_tiles,
input wire [ADDR_WIDTH-1:0] reg_result_addr,
// FPGA_DATA_READY: system-idle sticky flag, see nms_dataflow_core_sdram.v
output wire data_ready,
// ---- STEP19: ONE physical SDRAM interface, ALL traffic
// (weights + activations + results) ----
output wire sdram_cke,
output wire sdram_cs_n,
output wire sdram_ras_n,
output wire sdram_cas_n,
output wire sdram_we_n,
output wire [1:0] sdram_ba,
output wire [12:0] sdram_a,
inout wire [15:0] sdram_dq,
output wire [1:0] sdram_dqm
);
wire [N_SLOTS:0] slot_mem_req, slot_mem_wr;
wire [ADDR_WIDTH*(N_SLOTS+1)-1:0] slot_mem_addr;
wire [16*(N_SLOTS+1)-1:0] slot_mem_wdata, slot_mem_rdata;
wire [N_SLOTS:0] slot_mem_lb_n, slot_mem_ub_n;
wire [N_SLOTS:0] slot_mem_ready;
wire [N_SLOTS-1:0] wide_slot_mem_req;
wire [ADDR_WIDTH*N_SLOTS-1:0] wide_slot_mem_addr;
wire [64*N_SLOTS-1:0] wide_slot_mem_rdata;
wire [N_SLOTS-1:0] wide_slot_mem_ready;
nms_dataflow_core_sdram_fast #(
.DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .ACC_WIDTH(ACC_WIDTH), .ADDR_WIDTH(ADDR_WIDTH),
.N_SLOTS(N_SLOTS), .N_NODES(N_NODES), .MAX_DEPS(MAX_DEPS), .QUEUE_DEPTH(QUEUE_DEPTH),
.MAX_TILES(MAX_TILES), .PREFETCH_DISTANCE(PREFETCH_DISTANCE)
) u_dataflow_core (
.clk(clk), .rst(rst),
.reg_valid(reg_valid), .reg_ready(reg_ready), .reg_node_id(reg_node_id),
.reg_required(reg_required), .reg_producer_ids(reg_producer_ids),
.reg_x_base(reg_x_base), .reg_w_base(reg_w_base), .reg_n_tiles(reg_n_tiles),
.reg_result_addr(reg_result_addr),
.data_ready(data_ready),
.slot_mem_req(slot_mem_req), .slot_mem_wr(slot_mem_wr), .slot_mem_addr(slot_mem_addr),
.slot_mem_wdata(slot_mem_wdata), .slot_mem_lb_n(slot_mem_lb_n), .slot_mem_ub_n(slot_mem_ub_n),
.slot_mem_rdata(slot_mem_rdata), .slot_mem_ready(slot_mem_ready),
.wide_slot_mem_req(wide_slot_mem_req), .wide_slot_mem_addr(wide_slot_mem_addr),
.wide_slot_mem_rdata(wide_slot_mem_rdata), .wide_slot_mem_ready(wide_slot_mem_ready)
);
// ---- AR: activation-fill (shared, 1 port) + per-slot result
// writeback (N_SLOTS ports), arbitrated exactly as before ----
wire arb_m_req, arb_m_wr;
wire [ADDR_WIDTH-1:0] arb_m_addr;
wire [15:0] arb_m_wdata;
wire arb_m_lb_n, arb_m_ub_n;
wire [15:0] arb_m_rdata;
wire arb_m_ready;
slot_mem_arbiter #(
.ADDR_WIDTH(ADDR_WIDTH), .N_PORTS(N_SLOTS+1)
) u_arbiter (
.clk(clk), .rst(rst),
.s_req(slot_mem_req), .s_wr(slot_mem_wr), .s_addr(slot_mem_addr),
.s_wdata(slot_mem_wdata), .s_lb_n(slot_mem_lb_n), .s_ub_n(slot_mem_ub_n),
.s_rdata(slot_mem_rdata), .s_ready(slot_mem_ready),
.m_req(arb_m_req), .m_wr(arb_m_wr), .m_addr(arb_m_addr), .m_wdata(arb_m_wdata),
.m_lb_n(arb_m_lb_n), .m_ub_n(arb_m_ub_n),
.m_rdata(arb_m_rdata), .m_ready(arb_m_ready)
);
// ---- W: weight fetch (N_SLOTS ports), arbitrated exactly as
// before -- weight fetch never writes, same tie-off convention
// as STEP16-18 ----
wire [N_SLOTS-1:0] wide_s_wr = {N_SLOTS{1'b0}};
wire [64*N_SLOTS-1:0] wide_s_wdata = {(64*N_SLOTS){1'b0}};
wire [N_SLOTS-1:0] wide_s_lb_n = {N_SLOTS{1'b0}};
wire [N_SLOTS-1:0] wide_s_ub_n = {N_SLOTS{1'b0}};
wire wide_arb_m_req, wide_arb_m_wr;
wire [ADDR_WIDTH-1:0] wide_arb_m_addr;
wire [63:0] wide_arb_m_wdata;
wire wide_arb_m_lb_n, wide_arb_m_ub_n;
wire [63:0] wide_arb_m_rdata;
wire wide_arb_m_ready;
slot_mem_arbiter_wide #(
.ADDR_WIDTH(ADDR_WIDTH), .N_PORTS(N_SLOTS), .DATA_WIDTH(64)
) u_arbiter_wide (
.clk(clk), .rst(rst),
.s_req(wide_slot_mem_req), .s_wr(wide_s_wr), .s_addr(wide_slot_mem_addr),
.s_wdata(wide_s_wdata), .s_lb_n(wide_s_lb_n), .s_ub_n(wide_s_ub_n),
.s_rdata(wide_slot_mem_rdata), .s_ready(wide_slot_mem_ready),
.m_req(wide_arb_m_req), .m_wr(wide_arb_m_wr), .m_addr(wide_arb_m_addr), .m_wdata(wide_arb_m_wdata),
.m_lb_n(wide_arb_m_lb_n), .m_ub_n(wide_arb_m_ub_n),
.m_rdata(wide_arb_m_rdata), .m_ready(wide_arb_m_ready)
);
// ---- STEP19: ONE physical SDRAM backend, both W and AR ports ----
sdram_unified_backend_openrow #(
.ADDR_WIDTH(ADDR_WIDTH), .CLK_FREQ_MHZ(CLK_FREQ_MHZ)
) u_sdram_backend (
.clk(clk), .rst(rst),
.w_req(wide_arb_m_req), .w_addr(wide_arb_m_addr),
.w_rdata(wide_arb_m_rdata), .w_ready(wide_arb_m_ready),
.ar_req(arb_m_req), .ar_wr(arb_m_wr), .ar_addr(arb_m_addr), .ar_wdata(arb_m_wdata),
.ar_lb_n(arb_m_lb_n), .ar_ub_n(arb_m_ub_n),
.ar_rdata(arb_m_rdata), .ar_ready(arb_m_ready),
.sdram_cke(sdram_cke), .sdram_cs_n(sdram_cs_n), .sdram_ras_n(sdram_ras_n),
.sdram_cas_n(sdram_cas_n), .sdram_we_n(sdram_we_n),
.sdram_ba(sdram_ba), .sdram_a(sdram_a), .sdram_dq(sdram_dq), .sdram_dqm(sdram_dqm)
);
endmodule
hardware/v2/nms/sim/tb_fpga_neural_v2_top_openrow_fast_smoke.v
+281
View File
@@ -0,0 +1,281 @@
`timescale 1ns/1ps
// ================================================================
// FPGA-Neural V2 -- board-level top INTEGRATION SMOKE TEST (STEP20)
//
// Proves the NEW STEP20 wiring end-to-end: real SPI transactions (bit-
// banged, mode 0) drive job registration THROUGH spi_host_bridge.v,
// through the real compute+memory pipeline (byte-for-byte identical
// to the already-verified STEP19 nms_neural_multiprocessor_sdram_
// unified.v internals) via the NEW 2-level host-arb AR arbitration,
// down to the SAME single sdram_unified_backend/sdram_controller/
// AS4C4M16SA-6TIN chain -- checked against a real, backdoor-peeked
// SDRAM result. This is NOT a replacement for the STEP19 full 256-
// neuron D-Stress regression (already reconfirmed bit-exact using the
// trusted tool, see errors.log ERR-0024) -- it exists purely to
// validate the NEW pieces this step adds (SPI bridge, PLL-bypass
// clocking, reset_sync, the extra host-arb arbiter level) that
// D-Stress's own tight, back-to-back dispatch loop never exercises:
// realistic, WIDELY TIME-SEPARATED job pacing, as a real host would
// actually issue over SPI.
//
// STATUS (STEP20, ERR-0025 Part B): FIXED. Root cause: nms_weight_
// packed.v / nms_activation_replicated.v used a REGISTERED read (one
// full extra clock of latency) while nms_memory_manager_stream_wide.v's
// own read-ahead pipeline (`rd_pending`) assumes a COMBINATIONAL read
// (issue this cycle, data valid to capture next cycle). A busy multi-
// tile job's own prefetch lead time always absorbs the extra cycle
// invisibly; an uncontested single-tile job's first (only) tile has
// zero such margin and captured stale/zero data permanently. Fixed by
// making both SRAMs' reads combinational (with an explicit same-cycle
// fill/read bypass for the one hazard a combinational read alone would
// still miss). Verified: this test now passes, AND the STEP19 D-Stress
// regression (N=2 49788 cycles, N=4 49771 cycles, both 256/256
// bit-exact) is UNCHANGED -- cycle-for-cycle identical to before the
// fix, since D-Stress's own prefetch margin never depended on the
// extra (buggy) register cycle in the first place.
//
// Six scenarios below, using disjoint SDRAM regions so none interfere:
// A) two jobs, realistic wide SPI pacing (the original failing case)
// B) a single job dispatched alone (twice: neuron0 alone, neuron1 alone)
// C) two jobs back-to-back (minimal CS gap)
// D) two jobs with a large gap (same as A, kept as its own named case)
// G) parametric sweep across several distinct inter-job gaps, proving
// the fix does not depend on any particular cycle count
//
// Weights/activations are preloaded via the same backdoor poke
// convention already used by tb_nms_dstress_sdram_unified.v (direct
// writes into u_sdram.mem[]) -- only JOB REGISTRATION goes through the
// real, physical SPI path, since that is the actual integration
// surface under test. `SIM bypasses the (unsimulatable) EHXPLLL
// primitive inside ecp5_pll_sys_clk.v with a direct pass-through, per
// that module's own documented, declared limitation.
// ================================================================
`define SIM
module tb_fpga_neural_v2_top_smoke;
localparam ADDR_WIDTH = 26; // AS4C32M16SA memory upgrade
localparam N_SLOTS = 2;
localparam N_NODES = 16;
localparam MAX_DEPS = 4;
reg osc_clk = 0;
// Driven at the REAL 64MHz clk_sys rate (not the board's own 16MHz
// osc_clk) -- under the `SIM PLL bypass (clk_sys = osc_clk
// directly, see ecp5_pll_sys_clk.v), this reproduces the real
// board's actual system-clock rate for this test, matching
// CLK_FREQ_MHZ(64) above (a previous draft left both this and the
// controller's own CLK_FREQ_MHZ at a stale, pre-freeze value).
always #7.8125 osc_clk = ~osc_clk; // 64MHz
reg ext_rst_n = 0;
reg spi_sclk = 0, spi_mosi = 0, spi_cs_n = 1;
wire spi_miso;
wire sdram_cke, sdram_cs_n, sdram_ras_n, sdram_cas_n, sdram_we_n;
wire [1:0] sdram_ba;
wire [12:0] sdram_a;
wire [15:0] sdram_dq;
wire [1:0] sdram_dqm;
wire pll_locked;
fpga_neural_v2_top_openrow_fast #(
.ADDR_WIDTH(ADDR_WIDTH), .N_SLOTS(N_SLOTS), .N_NODES(N_NODES), .MAX_DEPS(MAX_DEPS),
.CLK_FREQ_MHZ(64)
) dut (
.osc_clk(osc_clk), .ext_rst_n(ext_rst_n),
.spi_sclk(spi_sclk), .spi_mosi(spi_mosi), .spi_miso(spi_miso), .spi_cs_n(spi_cs_n),
.sdram_cke(sdram_cke), .sdram_cs_n(sdram_cs_n), .sdram_ras_n(sdram_ras_n),
.sdram_cas_n(sdram_cas_n), .sdram_we_n(sdram_we_n),
.sdram_ba(sdram_ba), .sdram_a(sdram_a), .sdram_dq(sdram_dq), .sdram_dqm(sdram_dqm),
.pll_locked(pll_locked)
);
sdram_model #(.CLK_FREQ_MHZ(64)) u_sdram (
.clk(dut.clk_sys), .cke(sdram_cke), .cs_n(sdram_cs_n), .ras_n(sdram_ras_n),
.cas_n(sdram_cas_n), .we_n(sdram_we_n), .ba(sdram_ba), .a(sdram_a),
.dq(sdram_dq), .dqm(sdram_dqm)
);
function automatic signed [7:0] relu_sat(input signed [31:0] acc);
begin
if (acc < 0) relu_sat = 8'sd0;
else if (acc > 127) relu_sat = 8'sd127;
else relu_sat = acc[7:0];
end
endfunction
task poke_byte(input [ADDR_WIDTH-1:0] byte_addr, input signed [7:0] val);
reg [24:0] word_addr;
begin
word_addr = byte_addr[ADDR_WIDTH-1:1];
if (byte_addr[0] == 1'b0) u_sdram.mem[word_addr][7:0] = val;
else u_sdram.mem[word_addr][15:8] = val;
end
endtask
function automatic signed [7:0] peek_byte(input [ADDR_WIDTH-1:0] byte_addr);
reg [24:0] word_addr;
begin
word_addr = byte_addr[ADDR_WIDTH-1:1];
peek_byte = (byte_addr[0] == 1'b0) ? u_sdram.mem[word_addr][7:0] : u_sdram.mem[word_addr][15:8];
end
endfunction
// ---- SPI master BFM (matches spi_host_bridge.v's own protocol,
// same realistic 500ns-bit-period convention as tb_spi_host_
// bridge.v -- see that module's header on the CDC margin reason) ----
task spi_byte(input [7:0] tx, output [7:0] rx);
integer i;
begin
rx = 8'h00;
for (i = 7; i >= 0; i = i - 1) begin
spi_mosi = tx[i];
#200; spi_sclk = 1; #50; rx = {rx[6:0], spi_miso}; #50; spi_sclk = 0; #200;
end
end
endtask
task write_job(input [3:0] node_id, input [2:0] required, input [15:0] producer_ids,
input [ADDR_WIDTH-1:0] x_base, input [ADDR_WIDTH-1:0] w_base, input [15:0] n_tiles,
input [ADDR_WIDTH-1:0] result_addr);
reg [7:0] rxb;
begin
spi_cs_n = 0; #20;
spi_byte(8'h10, rxb);
spi_byte({4'b0, node_id}, rxb);
spi_byte({5'b0, required}, rxb);
spi_byte(producer_ids[15:8], rxb);
spi_byte(producer_ids[7:0], rxb);
spi_byte({6'b0, x_base[25:24]}, rxb);
spi_byte(x_base[23:16], rxb);
spi_byte(x_base[15:8], rxb);
spi_byte(x_base[7:0], rxb);
spi_byte({6'b0, w_base[25:24]}, rxb);
spi_byte(w_base[23:16], rxb);
spi_byte(w_base[15:8], rxb);
spi_byte(w_base[7:0], rxb);
spi_byte(n_tiles[15:8], rxb);
spi_byte(n_tiles[7:0], rxb);
spi_byte({6'b0, result_addr[25:24]}, rxb);
spi_byte(result_addr[23:16], rxb);
spi_byte(result_addr[15:8], rxb);
spi_byte(result_addr[7:0], rxb);
// hold CS through the reg_valid/reg_ready handshake (may
// need a few extra idle clocks if the target slot is busy)
#2000;
spi_cs_n = 1; #200;
end
endtask
integer errors, tests;
integer node_ctr; // fresh node_id per sub-test (dependency_manager never reclaims a dispatched id)
task check_neuron(input [22:0] x_base, input [22:0] w_base, input [22:0] res_addr,
input [255:0] label);
integer k;
reg signed [31:0] acc;
reg signed [7:0] golden, real_y;
begin
acc = 0;
for (k = 0; k < 8; k = k + 1)
acc = acc + peek_byte(x_base + k) * peek_byte(w_base + k);
golden = relu_sat(acc);
real_y = peek_byte(res_addr);
tests = tests + 1;
if (real_y !== golden) begin
errors = errors + 1;
$display("FAIL %0s: real=%0d golden=%0d", label, real_y, golden);
end else begin
$display("PASS %0s: real=%0d golden=%0d", label, real_y, golden);
end
end
endtask
// One independent, disjoint scratch region per pair-test invocation,
// so scenarios never interfere with each other's SDRAM content:
// x_base=region, w0=region+0x100, w1=region+0x110, res=region+0x200/0x201
task run_pair(input [22:0] region, input integer gap_ns, input [255:0] label);
reg [22:0] x_base, w0, w1, res0, res1;
integer k, n;
begin
x_base = region;
w0 = region + 26'h100;
w1 = region + 26'h110;
res0 = region + 26'h200;
res1 = region + 26'h201;
for (k = 0; k < 8; k = k + 1) poke_byte(x_base + k, k[7:0] + 1);
for (n = 0; n < 2; n = n + 1)
for (k = 0; k < 8; k = k + 1)
poke_byte((n == 0 ? w0 : w1) + k, ((n + k) % 4) + 1);
poke_byte(res0, 8'sd0);
poke_byte(res1, 8'sd0);
write_job(node_ctr[3:0], 3'd0, 16'h0000, x_base, w0, 16'd1, res0);
node_ctr = node_ctr + 1;
if (gap_ns > 0) #gap_ns;
write_job(node_ctr[3:0], 3'd0, 16'h0000, x_base, w1, 16'd1, res1);
node_ctr = node_ctr + 1;
repeat (3000) @(posedge dut.clk_sys);
check_neuron(x_base, w0, res0, {label, "-A"});
check_neuron(x_base, w1, res1, {label, "-B"});
end
endtask
// Single, standalone job (scenario B) -- no second job at all.
task run_single(input [22:0] region, input [255:0] label);
reg [22:0] x_base, w0, res0;
integer k;
begin
x_base = region;
w0 = region + 26'h100;
res0 = region + 26'h200;
for (k = 0; k < 8; k = k + 1) poke_byte(x_base + k, k[7:0] + 3);
for (k = 0; k < 8; k = k + 1) poke_byte(w0 + k, ((k) % 3) + 1);
poke_byte(res0, 8'sd0);
write_job(node_ctr[3:0], 3'd0, 16'h0000, x_base, w0, 16'd1, res0);
node_ctr = node_ctr + 1;
repeat (3000) @(posedge dut.clk_sys);
check_neuron(x_base, w0, res0, label);
end
endtask
initial begin
errors = 0; tests = 0; node_ctr = 0;
ext_rst_n = 0;
repeat (20) @(posedge osc_clk);
ext_rst_n = 1;
repeat (10) @(posedge osc_clk);
wait (dut.u_sdram_backend.u_sdram_ctrl.state == dut.u_sdram_backend.u_sdram_ctrl.S_IDLE);
@(posedge dut.clk_sys);
// B) single job, alone
run_single(26'h001000, "B-single-neuron0");
// A/D) two jobs, realistic wide SPI pacing (~85us worth of SPI
// framing plus an explicit extra gap -- the original failing case)
run_pair(26'h004000, 20000, "A-wide-gap");
// C) two jobs back-to-back (minimal CS-high gap between them)
run_pair(26'h007000, 0, "C-back-to-back");
// G) parametric sweep across several distinct inter-job gaps
run_pair(26'h00A000, 100, "G-gap100ns");
run_pair(26'h00D000, 5000, "G-gap5000ns");
run_pair(26'h010000, 50000, "G-gap50000ns");
$display("=== tb_fpga_neural_v2_top_smoke: %0d/%0d PASS ===", tests-errors, tests);
if (errors != 0) $display("*** %0d FAILURES ***", errors);
$finish;
end
endmodule
hardware/v2/nms/sim/tb_nms_dstress_sdram_openrow_fast.v
+880
View File
@@ -0,0 +1,880 @@
`timescale 1ns/1ps
// ================================================================
// FPGA-Neural V2 -- Final Benchmark Campaign (post-M10, real
// end-to-end characterization, docs/v2-description.md §22/§30/§32)
//
// One testbench, compiled once per N_SLOTS configuration (N_SLOTS_CFG
// parameter, overridden at Verilator invocation via -GN_SLOTS_CFG=N),
// running SIX representative workloads back-to-back through the REAL
// neural_multiprocessor.v (M8: dataflow_core + slot_mem_arbiter + the
// real, unmodified V1 PSRAM chain), with:
// - a software "golden" model replicating neural_processor.v's exact
// integer math (sum(x*w) over all tiles, ReLU + INT8 saturate --
// dataflow_core.v hardcodes bias=0/ACT_RELU for every job, so the
// golden model only needs to replicate that one path)
// - bit-exact verification of EVERY neuron's real result against
// that golden model (peek_byte from the real psram_model backing
// array -- an oracle independent of the RTL under test)
// - real cycle-accounting instrumentation (testbench-only, no RTL
// touched): per-slot busy/idle cycles, shared PSRAM port busy/idle
// cycles, REAL tiles delivered per slot (operand_valid&&
// operand_ready pulses -- one pulse = one whole P_IN-wide tile
// consumed by neural_processor, NOT one byte), director/dependency
// bookkeeping (jobs allocated/completed, ready-queue occupancy,
// WAITING/READY/DISPATCHED node counts, producer-done wakeups)
//
// Workloads (node_id ranges are disjoint across all six so the WHOLE
// campaign runs in ONE continuous simulation -- only ONE real PSRAM
// power-up wait, no reset between phases, closer to real sustained
// operation than resetting between every workload):
// A) Small -- 16 independent neurons, 8 inputs each
// B) Medium -- 64 independent neurons, 32 inputs each
// C) Large -- 128 independent neurons, 128 inputs each
// D) Stress -- 256 independent neurons, 128 inputs each
// E) Multilayer -- 8 layer-1 neurons (RANDOM data, logged seed) feed
// a shared 8-byte hidden vector; 2 layer-2 neurons
// consume that vector (real cross-node data
// forwarding through real PSRAM, real dependency
// wake-up, "shared producer/multiple consumers")
// F) DAG -- 6-node diamond+fan-in graph (A,B independent; C
// dep on A; D dep on B; E dep on BOTH C and D
// [2-hop transitive wake-up]; F dep on A,B,C [mixed
// direct+1-hop, 3 producers])
//
// All workloads A-D use a REALISTIC dense-layer shape: one shared
// input activation vector, N independent weight vectors (one per
// neuron) -- exactly how a real fully-connected layer's neurons share
// their layer's input. This is not an isolated synthetic microbench.
//
// Verified with Verilator (decisions.log DEC-0004).
// ================================================================
// ================================================================
// STEP11 variant: identical D-Stress workload/golden-model/correctness
// criteria as tb_nms_dstress.v (STEP9's own official benchmark), but
// instantiating nms_neural_multiprocessor_pf (REAL weight prefetch
// engine, weight_prefetch_engine.v) instead of the baseline
// nms_neural_multiprocessor.v, with an added PFD_CFG (PREFETCH_DISTANCE)
// parameter, plus NEW instrumentation (testbench-only, no RTL touched)
// for the two STEP11-mandated metrics that cannot be derived from the
// STEP9 instrumentation alone:
// weight_stall_cycles = cycles a slot is otherwise ready to
// present a tile (activation resident,
// in bounds) but blocked purely because
// tile_idx >= wgt_ready_count
// prefetch_effectiveness = tiles consumed with ZERO such
// weight-blocking cycles beforehand
// (i.e. the weight was ALREADY resident
// the moment the tile became eligible)
// / total tiles consumed
// per STEP11's own explicit metric definitions.
// ================================================================
module tb #(
parameter N_SLOTS_CFG = 2,
parameter PFD_CFG = 8
);
localparam ADDR_WIDTH = 26; // AS4C32M16SA: 25-bit word address + 1 byte-select bit
localparam DATA_WIDTH = 8;
localparam P_IN = 8;
localparam ACC_WIDTH = 32;
// N_NODES must exceed the HIGHEST node_id used by ANY workload
// (node_base + count - 1) -- workload D's own range alone
// (node_base=400, 256 neurons) reaches id 655. An earlier draft
// used N_NODES=512: D's ids silently wrapped (9-bit truncation)
// past id 511, colliding with workload A's already-DISPATCHED
// node 0 (dependency_manager never reclaims dispatched node slots,
// DEC-0008) and deadlocking register_node's reg_ready wait
// forever. A real consequence of DEC-0008's design choice, not an
// RTL bug -- fixed here by sizing N_NODES generously above the
// real id range used below (see decisions.log DEC-0008 and the
// final benchmark report's Limitations section).
localparam N_NODES = 1024;
localparam MAX_DEPS = 8;
localparam QUEUE_DEPTH = 8;
localparam NODE_IDW = $clog2(N_NODES);
localparam CLK_PERIOD = 12.5; // 80 MHz, matches psram_controller's CLK_FREQ_MHZ
reg clk, rst;
initial begin clk = 1'b0; forever #(CLK_PERIOD/2.0) clk = ~clk; end
reg reg_valid;
wire reg_ready;
reg [NODE_IDW-1:0] reg_node_id;
reg [$clog2(MAX_DEPS+1)-1:0] reg_required;
reg [MAX_DEPS*NODE_IDW-1:0] reg_producer_ids;
reg [ADDR_WIDTH-1:0] reg_x_base, reg_w_base, reg_result_addr;
reg [15:0] reg_n_tiles;
// STEP19: ONE physical SDRAM interface. weights, activations, and
// results ALL share this single bus/chip now -- no PSRAM anywhere.
wire sdram_cke, sdram_cs_n, sdram_ras_n, sdram_cas_n, sdram_we_n;
wire [1:0] sdram_ba;
wire [12:0] sdram_a;
wire [15:0] sdram_dq;
wire [1:0] sdram_dqm;
nms_neural_multiprocessor_sdram_openrow_fast #(
.DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .ACC_WIDTH(ACC_WIDTH), .ADDR_WIDTH(ADDR_WIDTH),
.N_SLOTS(N_SLOTS_CFG), .N_NODES(N_NODES), .MAX_DEPS(MAX_DEPS), .QUEUE_DEPTH(QUEUE_DEPTH),
.MAX_TILES(16), .PREFETCH_DISTANCE(PFD_CFG), .CLK_FREQ_MHZ(80)
) u_nmp (
.clk(clk), .rst(rst),
.reg_valid(reg_valid), .reg_ready(reg_ready), .reg_node_id(reg_node_id),
.reg_required(reg_required), .reg_producer_ids(reg_producer_ids),
.reg_x_base(reg_x_base), .reg_w_base(reg_w_base), .reg_n_tiles(reg_n_tiles),
.reg_result_addr(reg_result_addr),
.sdram_cke(sdram_cke), .sdram_cs_n(sdram_cs_n), .sdram_ras_n(sdram_ras_n),
.sdram_cas_n(sdram_cas_n), .sdram_we_n(sdram_we_n),
.sdram_ba(sdram_ba), .sdram_a(sdram_a), .sdram_dq(sdram_dq), .sdram_dqm(sdram_dqm)
);
sdram_model #(.CLK_FREQ_MHZ(80)) u_sdram (
.clk(clk), .cke(sdram_cke), .cs_n(sdram_cs_n), .ras_n(sdram_ras_n),
.cas_n(sdram_cas_n), .we_n(sdram_we_n), .ba(sdram_ba), .a(sdram_a),
.dq(sdram_dq), .dqm(sdram_dqm)
);
// ============================================================
// STEP19: byte-level backdoor access (test setup/verification
// only) -- weights, activations, AND results now ALL live on the
// single real SDRAM physical interface (u_sdram); there is no
// PSRAM anywhere in this system anymore. poke_byte/peek_byte (used
// by activation+result call sites) and poke_byte_weight/peek_byte
// _weight (used by weight call sites) are now identical in
// implementation -- kept as two names rather than merged, to avoid
// touching every one of their many existing call sites for a
// cosmetic rename; both correctly target the same u_sdram.mem
// backing array via the same byte_addr>>1 / byte_addr[0] pattern.
// ============================================================
task automatic poke_byte(input [ADDR_WIDTH-1:0] byte_addr, input signed [7:0] val);
reg [24:0] word_addr;
begin
word_addr = byte_addr[ADDR_WIDTH-1:1];
if (byte_addr[0] == 1'b0) u_sdram.mem[word_addr][7:0] = val;
else u_sdram.mem[word_addr][15:8] = val;
end
endtask
function automatic signed [7:0] peek_byte(input [ADDR_WIDTH-1:0] byte_addr);
reg [24:0] word_addr;
begin
word_addr = byte_addr[ADDR_WIDTH-1:1];
peek_byte = (byte_addr[0] == 1'b0) ? u_sdram.mem[word_addr][7:0] : u_sdram.mem[word_addr][15:8];
end
endfunction
// sdram_model.v's own `mem` array is flat-indexed by the 25-bit
// word address directly (bank*ROWS*COLS + row*COLS + col, which,
// given ROWS=8192/COLS=1024 are both powers of 2, is numerically
// IDENTICAL to treating the address as one flat 25-bit integer --
// confirmed against sdram_model.v's own BANKS/ROWS/COLS localparams
// before writing this, not assumed) -- so this is the exact same
// byte_addr>>1 / byte_addr[0] pattern as the original single-chip
// poke_byte/peek_byte above, just against u_sdram.mem instead of
// u_psram.mem.
task automatic poke_byte_weight(input [ADDR_WIDTH-1:0] byte_addr, input signed [7:0] val);
reg [24:0] word_addr;
begin
word_addr = byte_addr[ADDR_WIDTH-1:1];
if (byte_addr[0] == 1'b0) u_sdram.mem[word_addr][7:0] = val;
else u_sdram.mem[word_addr][15:8] = val;
end
endtask
function automatic signed [7:0] peek_byte_weight(input [ADDR_WIDTH-1:0] byte_addr);
reg [24:0] word_addr;
begin
word_addr = byte_addr[ADDR_WIDTH-1:1];
peek_byte_weight = (byte_addr[0] == 1'b0) ? u_sdram.mem[word_addr][7:0] : u_sdram.mem[word_addr][15:8];
end
endfunction
// Golden model: exactly replicates neural_processor.v's real path
// through dataflow_core (bias=0, ACT_RELU always -- see
// dataflow_core.v's own hardcoded job_bias/job_activation).
function automatic signed [7:0] relu_sat(input integer acc);
begin
if (acc <= 0) relu_sat = 8'sd0;
else if (acc > 127) relu_sat = 8'sd127;
else relu_sat = acc[7:0];
end
endfunction
// ============================================================
// Node registration (generalized to MAX_DEPS=8 producers, passed
// as a packed array; n_producers of them are meaningful, the rest
// ignored since reg_required gates how many entries the RTL
// actually reads).
// ============================================================
task automatic register_node(
input [NODE_IDW-1:0] nid,
input [$clog2(MAX_DEPS+1)-1:0] required,
input [MAX_DEPS*NODE_IDW-1:0] producer_ids_packed,
input [ADDR_WIDTH-1:0] xb, input [ADDR_WIDTH-1:0] wb,
input [15:0] nt, input [ADDR_WIDTH-1:0] resaddr
);
begin
@(posedge clk);
reg_node_id = nid;
reg_required = required;
reg_producer_ids = producer_ids_packed;
reg_x_base = xb; reg_w_base = wb; reg_n_tiles = nt; reg_result_addr = resaddr;
reg_valid = 1'b1;
while (!reg_ready) @(posedge clk);
@(posedge clk);
reg_valid = 1'b0;
end
endtask
// ============================================================
// M10+ real cycle-accounting instrumentation (testbench-only, no
// RTL touched -- same idiom as EXP-0013).
// ============================================================
reg measure_en;
integer total_cycles;
integer psram_busy_cycles;
integer ni; // moved up from its original later declaration point
// (STEP20 tooling-compatibility fix, zero behavior
// change -- see nms_memory_manager_stream_wide.v's own
// header note on icarus 13.0's stricter declared-
// before-use rule for procedural blocks)
genvar gi;
reg [N_SLOTS_CFG-1:0] slot_busy_bit; // memory_manager.state != MM_IDLE, this cycle
reg [N_SLOTS_CFG-1:0] slot_tile_bit; // operand_valid && operand_ready, this cycle
integer slot_busy_cycles [0:N_SLOTS_CFG-1];
integer slot_tiles_delivered [0:N_SLOTS_CFG-1];
generate
for (gi = 0; gi < N_SLOTS_CFG; gi = gi + 1) begin : GEN_SLOT_MON
always @(*) begin
slot_busy_bit[gi] = (u_nmp.u_dataflow_core.GEN_SLOT[gi].u_mm.state != 3'd0);
slot_tile_bit[gi] = u_nmp.u_dataflow_core.GEN_SLOT[gi].mm_operand_valid &&
u_nmp.u_dataflow_core.GEN_SLOT[gi].mm_operand_ready;
end
end
endgenerate
// ============================================================
// STEP17 Part B/C: cycle-decomposition + SDRAM effectiveness
// instrumentation (testbench-only, no RTL touched).
// ============================================================
integer active_count; // popcount(slot_busy_bit) this cycle
integer active_hist [0:4]; // cycles with exactly k active slots, k=0..4
integer useful_mac_cycles; // sum over cycles of (#slots with slot_tile_bit this cycle)
integer first_tile_cyc; // total_cycles value at the first tile ever delivered (startup boundary)
integer last_tile_cyc; // total_cycles value at the most recent tile delivered (drain boundary)
integer any_tile_bit;
// SDRAM controller-port instrumentation (real signals on the
// actual sdram_controller.v instance servicing all weight fetch)
integer sdram_req_count, sdram_ready_count, sdram_wr_count;
integer sdram_busy_cycles, sdram_refresh_count;
integer sdram_req_start_cyc, sdram_lat_sum, sdram_lat_min, sdram_lat_max, sdram_lat_n;
reg sdram_prev_state_is_refwait;
initial begin
active_hist[0]=0; active_hist[1]=0; active_hist[2]=0; active_hist[3]=0; active_hist[4]=0;
useful_mac_cycles = 0; first_tile_cyc = -1; last_tile_cyc = -1;
sdram_req_count=0; sdram_ready_count=0; sdram_wr_count=0;
sdram_busy_cycles=0; sdram_refresh_count=0;
sdram_req_start_cyc=0; sdram_lat_sum=0; sdram_lat_min=999999; sdram_lat_max=0; sdram_lat_n=0;
sdram_prev_state_is_refwait=1'b0;
end
always @(posedge clk) begin
if (measure_en) begin
active_count = slot_busy_bit[0];
for (ni = 1; ni < N_SLOTS_CFG; ni = ni + 1) active_count = active_count + slot_busy_bit[ni];
active_hist[active_count] <= active_hist[active_count] + 1;
any_tile_bit = slot_tile_bit[0];
for (ni = 1; ni < N_SLOTS_CFG; ni = ni + 1) any_tile_bit = any_tile_bit | slot_tile_bit[ni];
for (ni = 0; ni < N_SLOTS_CFG; ni = ni + 1)
if (slot_tile_bit[ni]) useful_mac_cycles <= useful_mac_cycles + 1;
if (any_tile_bit) begin
if (first_tile_cyc < 0) first_tile_cyc <= total_cycles;
last_tile_cyc <= total_cycles;
end
// ---- real SDRAM controller port (single physical chip,
// all weight-fetch traffic funnels through this one
// instance) ----
if (u_nmp.u_sdram_backend.u_sdram_ctrl.req) begin
sdram_req_count <= sdram_req_count + 1;
sdram_req_start_cyc <= total_cycles;
if (u_nmp.u_sdram_backend.u_sdram_ctrl.wr) sdram_wr_count <= sdram_wr_count + 1;
end
if (u_nmp.u_sdram_backend.u_sdram_ctrl.ready) begin
sdram_ready_count <= sdram_ready_count + 1;
sdram_lat_sum <= sdram_lat_sum + (total_cycles - sdram_req_start_cyc);
sdram_lat_n <= sdram_lat_n + 1;
if ((total_cycles - sdram_req_start_cyc) < sdram_lat_min) sdram_lat_min <= (total_cycles - sdram_req_start_cyc);
if ((total_cycles - sdram_req_start_cyc) > sdram_lat_max) sdram_lat_max <= (total_cycles - sdram_req_start_cyc);
end
if (u_nmp.u_sdram_backend.u_sdram_ctrl.busy) sdram_busy_cycles <= sdram_busy_cycles + 1;
sdram_prev_state_is_refwait <= (u_nmp.u_sdram_backend.u_sdram_ctrl.state == 5'd9);
if (u_nmp.u_sdram_backend.u_sdram_ctrl.state == 5'd9 && !sdram_prev_state_is_refwait)
sdram_refresh_count <= sdram_refresh_count + 1;
end
end
task automatic report_step17_instrumentation;
real active_pct [0:4];
real util_pct, startup_cycles, drain_cycles;
real sdram_avg_lat, sdram_busy_pct, sdram_bytes_per_cycle;
integer kk, total_tiles_all;
begin
total_tiles_all = 0;
for (kk = 0; kk < N_SLOTS_CFG; kk = kk + 1) total_tiles_all = total_tiles_all + slot_tiles_delivered[kk];
$display(" ---- STEP17 Part B: cycle decomposition ----");
for (kk = 0; kk <= N_SLOTS_CFG; kk = kk + 1) begin
active_pct[kk] = (total_cycles > 0) ? (100.0*active_hist[kk]/total_cycles) : 0.0;
$display(" active_slots=%0d: %0d cycles (%0.2f%%)", kk, active_hist[kk], active_pct[kk]);
end
util_pct = (total_cycles > 0) ? (100.0*useful_mac_cycles/(total_cycles*1.0*N_SLOTS_CFG)) : 0.0;
$display(" useful_mac_cycles (slot-tile-delivery events, summed)=%0d (%0.2f%% of total_cycles*N_SLOTS)", useful_mac_cycles, util_pct);
startup_cycles = (first_tile_cyc >= 0) ? (1.0*first_tile_cyc) : 0.0;
drain_cycles = (last_tile_cyc >= 0) ? (1.0*(total_cycles - last_tile_cyc)) : 0.0;
$display(" startup (cycles before first tile delivered anywhere)=%0.0f", startup_cycles);
$display(" drain (cycles after last tile delivered, until job completion)=%0.0f", drain_cycles);
$display(" ---- STEP17 Part C: SDRAM effectiveness ----");
sdram_avg_lat = (sdram_lat_n > 0) ? (1.0*sdram_lat_sum/sdram_lat_n) : 0.0;
sdram_busy_pct = (total_cycles > 0) ? (100.0*sdram_busy_cycles/total_cycles) : 0.0;
sdram_bytes_per_cycle = (total_cycles > 0) ? (8.0*sdram_ready_count/total_cycles) : 0.0;
$display(" sdram_req_count=%0d sdram_ready_count=%0d sdram_wr_count=%0d (real reads vs writes)",
sdram_req_count, sdram_ready_count, sdram_wr_count);
$display(" sdram_busy_cycles=%0d/%0d (%0.2f%%)", sdram_busy_cycles, total_cycles, sdram_busy_pct);
$display(" sdram_refresh_count=%0d (real AUTO REFRESH commands issued)", sdram_refresh_count);
$display(" sdram_request_latency: min=%0d max=%0d avg=%0.2f cycles (req-to-ready, single controller port)",
sdram_lat_min, sdram_lat_max, sdram_avg_lat);
$display(" sdram_avg_bytes_per_cycle (8 bytes/transaction * ready_count / total_cycles)=%0.4f", sdram_bytes_per_cycle);
end
endtask
// ---- STEP11: weight-stall / prefetch-effectiveness instrumentation ----
// slot_could_present_act: this slot's tile_idx is in-bounds and the
// activation operand for it is already resident -- i.e. everything
// EXCEPT the weight is ready. slot_weight_blocking: on top of that,
// the weight specifically is NOT yet ready (tile_idx>=wgt_ready_count)
// and the FSM is genuinely stalled on it (not mid-read-pipeline, not
// already holding a valid operand).
reg [N_SLOTS_CFG-1:0] slot_could_present_act;
reg [N_SLOTS_CFG-1:0] slot_weight_blocking;
reg [N_SLOTS_CFG-1:0] slot_stalled_this_tile; // sticky per current tile_idx
reg [31:0] prev_tile_idx [0:N_SLOTS_CFG-1];
integer weight_stall_cycles [0:N_SLOTS_CFG-1];
integer tiles_prefetched_clean [0:N_SLOTS_CFG-1]; // consumed w/ zero weight-blocking cycles
integer tiles_consumed_total [0:N_SLOTS_CFG-1];
// plain (non-hierarchical) mirrors of each slot's tile_idx, populated
// combinationally inside the genvar-indexed generate block below --
// a generate-block instance array (GEN_SLOT[.]) can only be indexed
// by a constant genvar, not a runtime `for` variable, so the
// sequential accumulation loop reads these plain arrays instead of
// reaching back into the hierarchy with a runtime index.
wire [31:0] slot_tile_idx_w [0:N_SLOTS_CFG-1];
generate
for (gi = 0; gi < N_SLOTS_CFG; gi = gi + 1) begin : GEN_SLOT_PF_MON
assign slot_tile_idx_w[gi] = {16'b0, u_nmp.u_dataflow_core.GEN_SLOT[gi].u_mm.tile_idx};
always @(*) begin
slot_could_present_act[gi] =
({{16{1'b0}}, u_nmp.u_dataflow_core.GEN_SLOT[gi].u_mm.tile_idx} <
{16'b0, u_nmp.u_dataflow_core.GEN_SLOT[gi].u_mm.n_tiles_reg}) &&
({{16{1'b0}}, u_nmp.u_dataflow_core.GEN_SLOT[gi].u_mm.tile_idx} <
{16'b0, u_nmp.u_dataflow_core.GEN_SLOT[gi].u_mm.usable_act});
// nms_memory_manager_stream.v has no read_issued/
// read_ready states (replaced by the rd_ptr/rd_pending
// read-ahead pipeline) -- the equivalent "blocked
// purely on weight readiness, nothing buffered yet"
// condition is simply: consumption pointer in bounds,
// activation ready, weight NOT ready, and no operand
// currently held in the skid buffer awaiting NP.
slot_weight_blocking[gi] =
slot_could_present_act[gi] &&
!(u_nmp.u_dataflow_core.GEN_SLOT[gi].u_mm.tile_idx <
u_nmp.u_dataflow_core.GEN_SLOT[gi].u_mm.wgt_ready_count) &&
!u_nmp.u_dataflow_core.GEN_SLOT[gi].u_mm.operand_valid;
end
end
endgenerate
always @(posedge clk) begin
if (measure_en) begin
for (ni = 0; ni < N_SLOTS_CFG; ni = ni + 1) begin
if (prev_tile_idx[ni] != slot_tile_idx_w[ni]) begin
// moved on to a new tile: clear the sticky flag for it
slot_stalled_this_tile[ni] <= 1'b0;
prev_tile_idx[ni] <= slot_tile_idx_w[ni];
end else if (slot_weight_blocking[ni]) begin
slot_stalled_this_tile[ni] <= 1'b1;
weight_stall_cycles[ni] <= weight_stall_cycles[ni] + 1;
end
if (slot_tile_bit[ni]) begin
tiles_consumed_total[ni] <= tiles_consumed_total[ni] + 1;
if (!slot_stalled_this_tile[ni])
tiles_prefetched_clean[ni] <= tiles_prefetched_clean[ni] + 1;
end
end
end
end
// Director/dependency bookkeeping
integer jobs_allocated, jobs_completed, wakeups;
integer waiting_sum, ready_sum, dispatched_sum, sample_count;
// Occupancy sampling is EXPENSIVE (a full N_NODES=512 scan) and is
// only needed for the small/structural workloads (A/B/E/F), not
// for the large neuron counts (C/D) where it would dominate
// simulation wall-time for no real benefit (per-slot/PSRAM/tile
// counters below are cheap and always collected). Gated by
// sample_occupancy, set per-workload.
reg sample_occupancy;
integer scan_i;
integer waiting_now, ready_now, dispatched_now;
always @(posedge clk) begin
if (measure_en) begin
total_cycles <= total_cycles + 1;
if (u_nmp.u_arbiter.owner != 0) psram_busy_cycles <= psram_busy_cycles + 1;
for (ni = 0; ni < N_SLOTS_CFG; ni = ni + 1) begin
if (slot_busy_bit[ni]) slot_busy_cycles[ni] <= slot_busy_cycles[ni] + 1;
if (slot_tile_bit[ni]) slot_tiles_delivered[ni] <= slot_tiles_delivered[ni] + 1;
end
if (u_nmp.u_dataflow_core.dm_ready_valid && u_nmp.u_dataflow_core.dm_ready_ready)
jobs_allocated <= jobs_allocated + 1;
if (u_nmp.u_dataflow_core.dir_job_out_done)
jobs_completed <= jobs_completed + 1;
if (u_nmp.u_dataflow_core.dm_producer_done_valid)
wakeups <= wakeups + 1;
if (sample_occupancy) begin
waiting_now = 0; ready_now = 0; dispatched_now = 0;
for (scan_i = 0; scan_i < N_NODES; scan_i = scan_i + 1) begin
case (u_nmp.u_dataflow_core.u_dep_mgr.node_state[scan_i])
2'd1: waiting_now = waiting_now + 1;
2'd2: ready_now = ready_now + 1;
2'd3: dispatched_now = dispatched_now + 1;
default: ;
endcase
end
waiting_sum <= waiting_sum + waiting_now;
ready_sum <= ready_sum + ready_now;
dispatched_sum <= dispatched_sum + dispatched_now;
sample_count <= sample_count + 1;
end
end
end
task automatic reset_instrumentation(input do_sample_occupancy);
integer k;
begin
active_hist[0]=0; active_hist[1]=0; active_hist[2]=0; active_hist[3]=0; active_hist[4]=0;
useful_mac_cycles = 0; first_tile_cyc = -1; last_tile_cyc = -1;
sdram_req_count=0; sdram_ready_count=0; sdram_wr_count=0;
sdram_busy_cycles=0; sdram_refresh_count=0;
sdram_req_start_cyc=0; sdram_lat_sum=0; sdram_lat_min=999999; sdram_lat_max=0; sdram_lat_n=0;
total_cycles = 0; psram_busy_cycles = 0;
jobs_allocated = 0; jobs_completed = 0; wakeups = 0;
waiting_sum = 0; ready_sum = 0; dispatched_sum = 0; sample_count = 0;
sample_occupancy = do_sample_occupancy;
for (k = 0; k < N_SLOTS_CFG; k = k + 1) begin
slot_busy_cycles[k] = 0;
slot_tiles_delivered[k] = 0;
weight_stall_cycles[k] = 0;
tiles_prefetched_clean[k] = 0;
tiles_consumed_total[k] = 0;
slot_stalled_this_tile[k] = 1'b0;
prev_tile_idx[k] = 32'hFFFFFFFF;
end
end
endtask
task automatic report_instrumentation(input [255:0] label, input integer n_neurons_completed);
integer k, total_tiles;
integer total_weight_stall_cycles, total_tiles_consumed_all, total_tiles_prefetched_clean;
real avg_waiting, avg_ready, avg_dispatched;
real psram_util, sustained_mac_per_cycle, wallclock_us;
real processor_utilization, weight_stall_pct, prefetch_effectiveness_pct;
begin
total_tiles = 0;
for (k = 0; k < N_SLOTS_CFG; k = k + 1) total_tiles = total_tiles + slot_tiles_delivered[k];
avg_waiting = (sample_count > 0) ? (1.0*waiting_sum/sample_count) : 0.0;
avg_ready = (sample_count > 0) ? (1.0*ready_sum/sample_count) : 0.0;
avg_dispatched = (sample_count > 0) ? (1.0*dispatched_sum/sample_count) : 0.0;
psram_util = (total_cycles > 0) ? (100.0*psram_busy_cycles/total_cycles) : 0.0;
sustained_mac_per_cycle = (total_cycles > 0) ? (1.0*total_tiles*P_IN/total_cycles) : 0.0;
wallclock_us = total_cycles * CLK_PERIOD / 1000.0;
$display("---- BENCHMARK REPORT: %0s ----", label);
$display(" total_cycles=%0d wallclock_us=%0.3f", total_cycles, wallclock_us);
$display(" neurons_completed=%0d tiles_delivered(real)=%0d", n_neurons_completed, total_tiles);
$display(" jobs_allocated=%0d jobs_completed=%0d dependency_wakeups=%0d", jobs_allocated, jobs_completed, wakeups);
$display(" shared AR (activation+result) arbiter-side utilization: %0.1f%% (%0d/%0d busy cycles)", psram_util, psram_busy_cycles, total_cycles);
for (k = 0; k < N_SLOTS_CFG; k = k + 1)
$display(" slot %0d: busy=%0d/%0d (%0.1f%%) tiles=%0d", k, slot_busy_cycles[k], total_cycles,
(total_cycles>0)?(100.0*slot_busy_cycles[k]/total_cycles):0.0, slot_tiles_delivered[k]);
if (sample_count > 0)
$display(" dependency_manager avg occupancy (sampled every measured cycle): waiting=%0.2f ready=%0.2f dispatched=%0.2f", avg_waiting, avg_ready, avg_dispatched);
else
$display(" dependency_manager occupancy: NOT SAMPLED for this workload (N_NODES scan skipped for large neuron counts to keep simulation time reasonable)");
$display(" DERIVED: sustained end-to-end MAC/cycle = %0.4f (real tiles*%0d / real total_cycles)", sustained_mac_per_cycle, P_IN);
if (n_neurons_completed > 0)
$display(" DERIVED: cycles/neuron = %0.2f", 1.0*total_cycles/n_neurons_completed);
if (total_tiles > 0)
$display(" DERIVED: cycles/tile = %0.2f", 1.0*total_cycles/total_tiles);
// ---- STEP11 metrics ----
total_weight_stall_cycles = 0; total_tiles_consumed_all = 0; total_tiles_prefetched_clean = 0;
for (k = 0; k < N_SLOTS_CFG; k = k + 1) begin
total_weight_stall_cycles = total_weight_stall_cycles + weight_stall_cycles[k];
total_tiles_consumed_all = total_tiles_consumed_all + tiles_consumed_total[k];
total_tiles_prefetched_clean = total_tiles_prefetched_clean + tiles_prefetched_clean[k];
end
processor_utilization = (total_cycles > 0) ? (100.0*total_tiles/(total_cycles*1.0)) : 0.0;
weight_stall_pct = (total_cycles > 0) ? (100.0*total_weight_stall_cycles/(total_cycles*N_SLOTS_CFG*1.0)) : 0.0;
prefetch_effectiveness_pct = (total_tiles_consumed_all > 0) ?
(100.0*total_tiles_prefetched_clean/(total_tiles_consumed_all*1.0)) : 0.0;
$display(" [STEP11] PFD=%0d weight_stall_cycles(sum,all slots)=%0d (%0.2f%% of total_cycles*N_SLOTS)",
PFD_CFG, total_weight_stall_cycles, weight_stall_pct);
$display(" [STEP11] tiles_consumed=%0d tiles_prefetched_clean(zero weight-block before consumption)=%0d",
total_tiles_consumed_all, total_tiles_prefetched_clean);
$display(" [STEP11] DERIVED: prefetch_effectiveness = %0.2f%%", prefetch_effectiveness_pct);
$display(" [STEP11] DERIVED: processor_utilization (tiles*P_IN-equivalent proxy, see sustained MAC/cycle) reference sustained_mac_per_cycle=%0.4f", sustained_mac_per_cycle);
end
endtask
// ============================================================
// Workload generators
// ============================================================
integer errors, tests;
integer wd;
// A/B/C/D: shared-input dense layer. Generates the shared X
// vector, then N independent (neuron, weight-vector) jobs, each
// verified bit-exact against the golden model.
task automatic run_dense_layer(
input [255:0] label,
input integer n_neurons,
input integer n_tiles_count,
input [NODE_IDW-1:0] node_base,
input [ADDR_WIDTH-1:0] x_base,
input [ADDR_WIDTH-1:0] w_base,
input [ADDR_WIDTH-1:0] res_base,
input sample_occ
);
integer n, t, k, len, acc;
reg signed [7:0] xv, wv, golden, real_y;
reg [MAX_DEPS*NODE_IDW-1:0] no_deps;
integer completed, wd2;
begin
len = n_tiles_count * P_IN;
no_deps = {(MAX_DEPS*NODE_IDW){1'b0}};
// shared input vector
for (k = 0; k < len; k = k + 1)
poke_byte(x_base + k, ((k % 8) + 1));
reset_instrumentation(sample_occ);
measure_en = 1'b1;
for (n = 0; n < n_neurons; n = n + 1) begin
acc = 0;
for (t = 0; t < n_tiles_count; t = t + 1) begin
for (k = 0; k < P_IN; k = k + 1) begin
xv = peek_byte(x_base + t*P_IN + k);
wv = (((n + t*P_IN + k) % 8) + 1);
poke_byte_weight(w_base + n*len + t*P_IN + k, wv);
acc = acc + xv*wv;
end
end
golden = relu_sat(acc);
poke_byte(res_base + n, 8'sd0); // poison, must NOT still be 0 after completion (unless golden IS 0 -- checked separately)
register_node(node_base + n[NODE_IDW-1:0], 0, no_deps,
x_base, w_base + n*len, n_tiles_count[15:0], res_base + n);
if ((n % 32) == 0) begin
$display(" [%0s] registered %0d/%0d", label, n+1, n_neurons);
$fflush;
end
end
$display(" [%0s] all %0d neurons registered, waiting for completion...", label, n_neurons);
$fflush;
// wait for all n_neurons completions
completed = 0; wd2 = 0;
while (completed < n_neurons && wd2 < 2000000) begin
@(posedge clk);
wd2 = wd2 + 1;
completed = jobs_completed;
if ((wd2 % 20000) == 0) begin
$display(" [%0s] watchdog %0d: completed=%0d/%0d total_cycles=%0d", label, wd2, completed, n_neurons, total_cycles);
`ifdef STEP16_DEBUG_TRACE
$display(" slot0: mm.state=%0d tile_idx=%0d n_tiles_reg=%0d wgt_ready_count=%0d usable_act=%0d op_valid=%0d op_ready=%0d",
u_nmp.u_dataflow_core.GEN_SLOT[0].u_mm.state,
u_nmp.u_dataflow_core.GEN_SLOT[0].u_mm.tile_idx,
u_nmp.u_dataflow_core.GEN_SLOT[0].u_mm.n_tiles_reg,
u_nmp.u_dataflow_core.GEN_SLOT[0].u_mm.wgt_ready_count,
u_nmp.u_dataflow_core.GEN_SLOT[0].u_mm.usable_act,
u_nmp.u_dataflow_core.GEN_SLOT[0].mm_operand_valid,
u_nmp.u_dataflow_core.GEN_SLOT[0].mm_operand_ready);
$display(" slot1: mm.state=%0d tile_idx=%0d n_tiles_reg=%0d wgt_ready_count=%0d usable_act=%0d op_valid=%0d op_ready=%0d",
u_nmp.u_dataflow_core.GEN_SLOT[1].u_mm.state,
u_nmp.u_dataflow_core.GEN_SLOT[1].u_mm.tile_idx,
u_nmp.u_dataflow_core.GEN_SLOT[1].u_mm.n_tiles_reg,
u_nmp.u_dataflow_core.GEN_SLOT[1].u_mm.wgt_ready_count,
u_nmp.u_dataflow_core.GEN_SLOT[1].u_mm.usable_act,
u_nmp.u_dataflow_core.GEN_SLOT[1].mm_operand_valid,
u_nmp.u_dataflow_core.GEN_SLOT[1].mm_operand_ready);
$display(" sdram: req=%0d busy=%0d ready=%0d req_pending=%0d state=%0d | arb: owner=%0d pending=%0b wide_req=%0b wide_ready=%0b",
u_nmp.u_sdram_backend.u_sdram_ctrl.req,
u_nmp.u_sdram_backend.u_sdram_ctrl.busy,
u_nmp.u_sdram_backend.u_sdram_ctrl.ready,
u_nmp.u_sdram_backend.u_sdram_ctrl.req_pending,
u_nmp.u_sdram_backend.u_sdram_ctrl.state,
u_nmp.u_arbiter_wide.owner,
u_nmp.u_arbiter_wide.pending,
u_nmp.wide_slot_mem_req,
u_nmp.wide_slot_mem_ready);
`endif
$fflush;
end
end
repeat(5) @(posedge clk);
measure_en = 1'b0;
tests = tests + 1;
if (completed < n_neurons) begin
$display("FAIL %0s: only %0d/%0d neurons completed within watchdog", label, completed, n_neurons);
errors = errors + 1;
end else begin : check_block
integer local_errors;
local_errors = 0;
for (n = 0; n < n_neurons; n = n + 1) begin
acc = 0;
for (t = 0; t < n_tiles_count; t = t + 1)
for (k = 0; k < P_IN; k = k + 1)
acc = acc + peek_byte(x_base + t*P_IN + k) * peek_byte_weight(w_base + n*len + t*P_IN + k);
golden = relu_sat(acc);
real_y = peek_byte(res_base + n);
if (real_y !== golden) begin
$display("FAIL %0s neuron %0d: real=%0d golden=%0d", label, n, real_y, golden);
local_errors = local_errors + 1;
end
end
if (local_errors == 0)
$display("PASS %0s: all %0d neurons bit-exact vs golden", label, n_neurons);
else
errors = errors + 1;
end
report_instrumentation(label, n_neurons);
report_step17_instrumentation;
end
endtask
// E: Multilayer (8 layer-1 random neurons -> shared hidden vector
// -> 2 layer-2 neurons consuming it, real dependency wake-up +
// real cross-node data forwarding through real PSRAM).
localparam L1_N = 8;
localparam L2_N = 2;
integer rand_seed;
task automatic run_multilayer(
input [NODE_IDW-1:0] node_base,
input [ADDR_WIDTH-1:0] l1x_base, input [ADDR_WIDTH-1:0] l1w_base,
input [ADDR_WIDTH-1:0] hidden_base,
input [ADDR_WIDTH-1:0] l2w_base, input [ADDR_WIDTH-1:0] l2res_base
);
integer n, k, acc, completed, wd2;
reg signed [7:0] xv, wv, golden_l1 [0:L1_N-1], golden_l2, real_y;
reg [MAX_DEPS*NODE_IDW-1:0] no_deps, l2_deps;
integer local_errors;
begin
no_deps = {(MAX_DEPS*NODE_IDW){1'b0}};
l2_deps = {(MAX_DEPS*NODE_IDW){1'b0}};
for (n = 0; n < L1_N; n = n + 1)
l2_deps[n*NODE_IDW +: NODE_IDW] = node_base + n[NODE_IDW-1:0];
rand_seed = 32'hC0FFEE01;
$display("RANDOM SEED (workload E, layer-1 data) = 32'h%08h", rand_seed);
reset_instrumentation(1'b1);
measure_en = 1'b1;
for (n = 0; n < L1_N; n = n + 1) begin
acc = 0;
for (k = 0; k < P_IN; k = k + 1) begin
xv = $random(rand_seed) % 9; // deterministic PRNG stream, range roughly [-8,8]
wv = $random(rand_seed) % 9;
poke_byte(l1x_base + n*P_IN + k, xv);
poke_byte(l1w_base + n*P_IN + k, wv);
acc = acc + xv*wv;
end
golden_l1[n] = relu_sat(acc);
poke_byte(hidden_base + n, 8'sd0); // poison hidden slot
register_node(node_base + n[NODE_IDW-1:0], 0, no_deps,
l1x_base + n*P_IN, l1w_base + n*P_IN, 16'd1, hidden_base + n);
end
for (n = 0; n < L2_N; n = n + 1) begin
for (k = 0; k < P_IN; k = k + 1)
poke_byte(l2w_base + n*P_IN + k, ((n + k) % 6) + 1);
register_node(node_base + L1_N[NODE_IDW-1:0] + n[NODE_IDW-1:0], L1_N[$clog2(MAX_DEPS+1)-1:0], l2_deps,
hidden_base, l2w_base + n*P_IN, 16'd1, l2res_base + n);
end
completed = 0; wd2 = 0;
while (completed < (L1_N+L2_N) && wd2 < 2000000) begin
@(posedge clk); wd2 = wd2 + 1; completed = jobs_completed;
end
repeat(5) @(posedge clk);
measure_en = 1'b0;
tests = tests + 1;
local_errors = 0;
if (completed < (L1_N+L2_N)) begin
$display("FAIL Multilayer: only %0d/%0d nodes completed", completed, L1_N+L2_N);
local_errors = local_errors + 1;
end else begin
for (n = 0; n < L1_N; n = n + 1) begin
real_y = peek_byte(hidden_base + n);
if (real_y !== golden_l1[n]) begin
$display("FAIL Multilayer L1 neuron %0d: real=%0d golden=%0d", n, real_y, golden_l1[n]);
local_errors = local_errors + 1;
end
end
for (n = 0; n < L2_N; n = n + 1) begin
acc = 0;
for (k = 0; k < P_IN; k = k + 1)
acc = acc + golden_l1[k] * peek_byte(l2w_base + n*P_IN + k);
golden_l2 = relu_sat(acc);
real_y = peek_byte(l2res_base + n);
if (real_y !== golden_l2) begin
$display("FAIL Multilayer L2 neuron %0d: real=%0d golden=%0d (using REAL L1 hidden values)", n, real_y, golden_l2);
local_errors = local_errors + 1;
end
end
end
if (local_errors == 0) $display("PASS Multilayer: 8 L1 (random) -> 2 L2 neurons, all bit-exact, real cross-node forwarding via real PSRAM");
else errors = errors + 1;
report_instrumentation("E-Multilayer", L1_N+L2_N);
end
endtask
// F: DAG diamond+fan-in (A,B indep; C dep-A; D dep-B; E dep-C&D
// [2-hop]; F dep-A,B,C [mixed, 3 producers])
task automatic run_dag(
input [NODE_IDW-1:0] node_base,
input [ADDR_WIDTH-1:0] x_base, input [ADDR_WIDTH-1:0] w_base, input [ADDR_WIDTH-1:0] res_base
);
integer n, k, acc, completed, wd2, local_errors;
reg signed [7:0] golden [0:5];
reg signed [7:0] real_y;
reg [MAX_DEPS*NODE_IDW-1:0] deps;
reg [NODE_IDW-1:0] idA, idB, idC, idD, idE, idF;
begin
idA = node_base+0; idB = node_base+1; idC = node_base+2;
idD = node_base+3; idE = node_base+4; idF = node_base+5;
// Each of the 6 nodes: its own small independent 8-input
// job (deterministic, distinct per node) -- dependencies
// here are purely about SCHEDULING/wake-up order, not
// data forwarding (workload E already covers that).
for (n = 0; n < 6; n = n + 1) begin
acc = 0;
for (k = 0; k < P_IN; k = k + 1) begin
poke_byte(x_base + n*P_IN + k, ((n+k)%4)+1);
poke_byte(w_base + n*P_IN + k, ((n+k)%5)+1);
acc = acc + peek_byte(x_base+n*P_IN+k)*peek_byte(w_base+n*P_IN+k);
end
golden[n] = relu_sat(acc);
poke_byte(res_base + n, 8'sd0);
end
reset_instrumentation(1'b1);
measure_en = 1'b1;
deps = {(MAX_DEPS*NODE_IDW){1'b0}};
register_node(idA, 0, deps, x_base+0*P_IN, w_base+0*P_IN, 16'd1, res_base+0);
register_node(idB, 0, deps, x_base+1*P_IN, w_base+1*P_IN, 16'd1, res_base+1);
deps = {(MAX_DEPS*NODE_IDW){1'b0}}; deps[0*NODE_IDW+:NODE_IDW] = idA;
register_node(idC, 1, deps, x_base+2*P_IN, w_base+2*P_IN, 16'd1, res_base+2);
deps = {(MAX_DEPS*NODE_IDW){1'b0}}; deps[0*NODE_IDW+:NODE_IDW] = idB;
register_node(idD, 1, deps, x_base+3*P_IN, w_base+3*P_IN, 16'd1, res_base+3);
deps = {(MAX_DEPS*NODE_IDW){1'b0}}; deps[0*NODE_IDW+:NODE_IDW] = idC; deps[1*NODE_IDW+:NODE_IDW] = idD;
register_node(idE, 2, deps, x_base+4*P_IN, w_base+4*P_IN, 16'd1, res_base+4);
deps = {(MAX_DEPS*NODE_IDW){1'b0}}; deps[0*NODE_IDW+:NODE_IDW] = idA; deps[1*NODE_IDW+:NODE_IDW] = idB; deps[2*NODE_IDW+:NODE_IDW] = idC;
register_node(idF, 3, deps, x_base+5*P_IN, w_base+5*P_IN, 16'd1, res_base+5);
completed = 0; wd2 = 0;
while (completed < 6 && wd2 < 2000000) begin @(posedge clk); wd2=wd2+1; completed = jobs_completed; end
repeat(5) @(posedge clk);
measure_en = 1'b0;
tests = tests + 1;
local_errors = 0;
if (completed < 6) begin
$display("FAIL DAG: only %0d/6 nodes completed", completed);
local_errors = local_errors + 1;
end else begin
for (n = 0; n < 6; n = n + 1) begin
real_y = peek_byte(res_base+n);
if (real_y !== golden[n]) begin
$display("FAIL DAG node %0d: real=%0d golden=%0d", n, real_y, golden[n]);
local_errors = local_errors + 1;
end
end
end
if (local_errors == 0) $display("PASS DAG: 6-node diamond+fan-in (2-hop transitive wake-up, 3-producer mixed-depth dependency), all bit-exact");
else errors = errors + 1;
report_instrumentation("F-DAG", 6);
end
endtask
initial begin
errors = 0; tests = 0;
rst = 1; reg_valid = 0; reg_node_id = 0; reg_required = 0; reg_producer_ids = 0;
reg_x_base = 0; reg_w_base = 0; reg_n_tiles = 0; reg_result_addr = 0;
measure_en = 0;
repeat(5) @(posedge clk);
rst = 0;
$display("========================================");
$display("NMS D-Stress benchmark (STEP19, SINGLE SDRAM (AS4C4M16SA-6TIN) for weights+activations+results, no PSRAM anywhere) -- N_SLOTS_CFG=%0d PFD_CFG=%0d", N_SLOTS_CFG, PFD_CFG);
$display("========================================");
wait (u_nmp.u_sdram_backend.u_sdram_ctrl.state == u_nmp.u_sdram_backend.u_sdram_ctrl.S_IDLE);
@(posedge clk);
// Official V2 memory map (datasheet ch.5): weights @ 0x010000,
// activations @ 0x200000, results @ 0x300000 -- non-overlapping
// 1MB-aligned regions in the single SDRAM.
run_dense_layer("D-Stress", 256, 16, 16'd400, 26'h200000, 26'h010000, 26'h300000, 1'b0);
// FPGA_DATA_READY check: the whole graph (256 nodes) just
// finished and no new work has been registered -- data_ready
// must be asserted (system-idle sticky flag, see
// nms_dataflow_core_sdram.v). A few idle cycles for the
// busy->idle edge to settle before sampling.
repeat (4) @(posedge clk);
if (u_nmp.data_ready !== 1'b1) begin
$display("FAIL data_ready: expected 1 after graph completion, got %b", u_nmp.data_ready);
errors = errors + 1;
end else begin
$display("PASS data_ready: correctly asserted after graph completion");
end
$display("========================================");
if (errors == 0)
$display("ALL %0d WORKLOAD SUITES PASSED (N_SLOTS_CFG=%0d, PFD_CFG=%0d, SINGLE SDRAM for weights+activations+results, no PSRAM)", tests, N_SLOTS_CFG, PFD_CFG);
else
$display("FAILED: %0d/%0d workload suite(s) had errors -- see messages above", errors, tests);
$display("========================================");
$finish;
end
endmodule