`timescale 1ns/1ps // ============================================================ // V3 -- packed_slot.v: real synthesizable per-slot sequencer, the // piece that promotes EXP-0062's own PROCEDURAL testbench sequence // (prefetch -> swap -> job dispatch -> tile-by-tile operand feed -> // result capture) into real RTL, exactly the same class of promotion // weight_tile_gather.v already did for the byte-gather step // (EXP-0061). // // Wraps: layer_prefetch_ctrl.v -> layer_weight_buffer.v -> // weight_tile_gather.v -> neural_processor_packed.v, driven by a new // sequencing FSM, presenting the external contract neural_director_ // packed.v already expects (job_start/x_base_a/b/w_base/n_tiles/ // node_id_a/b -> job_done/result_data_a/b/result_node_id_a/b). // // ACTIVATION FETCH (EXP-0079, real, closes the gap this header used to // disclose as deferred; EXP-0083 upgrades it to a look-ahead prefetch): // ddr_prefetch_mgr.v wraps act_tile_fetch.v with a depth-2 ping-pong // buffer, issuing tile N+1's fetch the instant the fetch engine is // free rather than waiting for this slot to finish CONSUMING tile N -- // overlapping "fetch next tile" with "consume current tile" (see // ddr_prefetch_mgr.v's own header for the real, honest, measured scope // of the benefit -- it does not raise the physical DDR3 ceiling, only // removes small real per-tile re-request overhead). It shares THIS // slot's own single ctrl_req/addr/etc port with layer_prefetch_ctrl.v // (u_pf): the two are mutually exclusive in time by FSM construction // (weight prefetch always fully completes, including its own // consume_done, before the tile loop that needs activation data ever // starts), muxed below on act_mem_active. The outer arbiter's grant // (mem_active/mem_grant, this module's own top-level ports) is now // also needed during activation fetch, not just weight prefetch -- // held PER TILE (one 2-burst fetch, lane A then lane B), released // between tiles, matching this project's own established "lock the // grant for one whole logical fetch, not longer" discipline (avoids // starving the other slot for the whole tile loop's duration). // // MEMORY LAYOUT this requires of activation data in DDR3 (EXP-0081, // v2 convention): two consecutive tiles share one full BURST_LEN=8- // word burst (even index low 64 bits, odd index high 64 bits) -- see // act_tile_fetch.v's own header and docs/PHYSICAL_REALIZATION.md S4. // // RESULT WRITEBACK (EXP-0088, real, closes the gap this header used to // disclose as deferred): result_writeback.v (u_wb) writes each // completed job's result INTO DDR3 at result_addr_a/b, sharing this // slot's own ctrl port the same way u_pf/u_ddrpf already do (mutually // exclusive in time -- writeback only ever starts in S_RESULT, after // the whole tile loop has finished). This was the last real blocker // for scaling past a handful of slots: result_data_a/b/etc. remain // real output ports here for debug/testbench visibility, but the // TOP-LEVEL n2_system_ddr3_top.v no longer exposes them as literal, // N-scaled package pins -- see result_writeback.v's own header. // // EVERY job re-fetches its layer from SDRAM (no resident-weight-skip // optimization) -- correctness first; EXP-0057's own measured // prefetch/reuse PERFORMANCE benefit is a property of the buffer // being read MANY times per fetch (many reuse positions per Director- // dispatched pair's own tile loop is NOT what's being reused here -- // see note in the FSM below), not of skipping fetches across // DIFFERENT Director dispatches; adding that optimization is future // work, not a correctness requirement. // ============================================================ module packed_slot #( parameter DATA_WIDTH = 8, parameter P_IN = 8, parameter ACC_WIDTH = 32, parameter BURST_LEN = 8, parameter ADDR_WIDTH = 26, parameter LAYER_BYTES = 128, parameter BUFADDRW = $clog2(LAYER_BYTES) )( input wire clk, input wire rst, // ---- Director interface (matches neural_director_packed.v's own // per-slot output ports exactly) ---- input wire job_start, input wire [ADDR_WIDTH-1:0] x_base_a, input wire [ADDR_WIDTH-1:0] x_base_b, input wire [ADDR_WIDTH-1:0] w_base, input wire [15:0] n_tiles, input wire [ADDR_WIDTH-1:0] result_addr_a, input wire [ADDR_WIDTH-1:0] result_addr_b, input wire [15:0] node_id_a, input wire [15:0] node_id_b, output reg job_done, // one-cycle pulse output reg signed [DATA_WIDTH-1:0] result_data_a, output reg signed [DATA_WIDTH-1:0] result_data_b, output reg [15:0] result_node_id_a, output reg [15:0] result_node_id_b, output reg [ADDR_WIDTH-1:0] result_addr_a_out, output reg [ADDR_WIDTH-1:0] result_addr_b_out, // high exactly while this slot needs exclusive access to the // shared SDRAM controller (its own weight-fetch OR activation- // fetch phase) -- a shared-controller arbiter uses this to lock a // grant for the whole multi-burst fetch, not just one transaction. output wire mem_active, // grant from a shared-controller arbiter (see mem_active's own // comment): must be asserted before this slot may pulse its own // layer_prefetch_ctrl.v start, since that module's ctrl_req is a // one-shot pulse with no retry -- issuing it before the arbiter // has actually granted this slot the bus loses it permanently // (found empirically integrating N=2 slots behind sdram_slot_ // arbiter2.v: a slot could hang forever in S_WAIT with ctrl_req // already dropped and ctrl_ready never coming). Tie high for a // single-slot (N=1, no arbiter) system. input wire mem_grant, // ---- SDRAM controller port (connects directly, or through a // shared arbiter for N>1 slots) ---- output wire ctrl_req, output wire ctrl_wr, output wire [ADDR_WIDTH-2:0] ctrl_addr, output wire [32*BURST_LEN-1:0] ctrl_wdata, output wire [4*BURST_LEN-1:0] ctrl_wmask, input wire [32*BURST_LEN-1:0] ctrl_rdata, input wire ctrl_ready, input wire ctrl_busy ); localparam S_IDLE = 4'd0, S_MEMWAIT = 4'd1, S_PREFETCH = 4'd2, S_SWAP = 4'd3, S_JOBSTART = 4'd4, S_TILEREQ = 4'd5, S_TILEWAIT = 4'd6, S_OPERAND = 4'd7, S_RESULT = 4'd8, S_DONE = 4'd9, S_WRITEBACK = 4'd10; reg [3:0] state; reg [ADDR_WIDTH-1:0] w_base_lat, x_base_a_lat, x_base_b_lat; reg [15:0] n_tiles_lat; reg [ADDR_WIDTH-1:0] result_addr_a_lat, result_addr_b_lat; reg [15:0] node_id_a_lat, node_id_b_lat; reg [15:0] tcnt; // ---- layer_prefetch_ctrl.v ---- reg pf_start; wire pf_busy, pf_done; wire pf_fill_we; wire [BUFADDRW-1:0] pf_fill_addr; wire [DATA_WIDTH-1:0] pf_fill_data; wire pf_ctrl_req, pf_ctrl_wr; wire [ADDR_WIDTH-2:0] pf_ctrl_addr; wire [32*BURST_LEN-1:0] pf_ctrl_wdata; wire [4*BURST_LEN-1:0] pf_ctrl_wmask; layer_prefetch_ctrl #( .DATA_WIDTH(DATA_WIDTH), .LAYER_BYTES(LAYER_BYTES), .BURST_LEN(BURST_LEN), .ADDR_WIDTH(ADDR_WIDTH-1) ) u_pf ( .clk(clk), .rst(rst), .start(pf_start), .layer_base(w_base_lat[ADDR_WIDTH-2:0]), .busy(pf_busy), .done(pf_done), .fill_we(pf_fill_we), .fill_addr(pf_fill_addr), .fill_data(pf_fill_data), .ctrl_req(pf_ctrl_req), .ctrl_wr(pf_ctrl_wr), .ctrl_addr(pf_ctrl_addr), .ctrl_wdata(pf_ctrl_wdata), .ctrl_wmask(pf_ctrl_wmask), .ctrl_rdata(ctrl_rdata), .ctrl_ready(ctrl_ready), .ctrl_busy(ctrl_busy) ); // ---- ddr_prefetch_mgr.v (EXP-0083): look-ahead activation fetch, // shares this slot's own ctrl port with u_pf above (mutually // exclusive in time -- see header). Job-level start (once per job, // not once per tile -- the whole tile loop's lookahead sequencing // happens inside this module). reg ddrpf_job_start; wire ddrpf_tile_valid; reg ddrpf_tile_consume; wire signed [DATA_WIDTH*P_IN-1:0] act_data_a_w, act_data_b_w; wire act_mem_active; wire act_ctrl_req, act_ctrl_wr; wire [ADDR_WIDTH-2:0] act_ctrl_addr; wire [32*BURST_LEN-1:0] act_ctrl_wdata; wire [4*BURST_LEN-1:0] act_ctrl_wmask; ddr_prefetch_mgr #( .DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .BURST_LEN(BURST_LEN), .ADDR_WIDTH(ADDR_WIDTH-1) ) u_ddrpf ( .clk(clk), .rst(rst), .job_start(ddrpf_job_start), .base_a(x_base_a_lat[ADDR_WIDTH-2:0]), .base_b(x_base_b_lat[ADDR_WIDTH-2:0]), .n_tiles(n_tiles_lat), .tile_valid(ddrpf_tile_valid), .data_a(act_data_a_w), .data_b(act_data_b_w), .tile_consume(ddrpf_tile_consume), .mem_active(act_mem_active), .mem_grant(mem_grant), .ctrl_req(act_ctrl_req), .ctrl_wr(act_ctrl_wr), .ctrl_addr(act_ctrl_addr), .ctrl_wdata(act_ctrl_wdata), .ctrl_wmask(act_ctrl_wmask), .ctrl_rdata(ctrl_rdata), .ctrl_ready(ctrl_ready), .ctrl_busy(ctrl_busy) ); // ---- result_writeback.v (EXP-0088): real result-writeback engine, // shares this slot's own ctrl port with u_pf/u_ddrpf above // (mutually exclusive in time by FSM construction -- writeback only // ever starts in S_RESULT, strictly after the whole tile loop that // drives u_ddrpf has already finished). Writes both lanes' results // into DDR3 at their own job-supplied result_addr_a/b -- see // result_writeback.v's own header for the real addressing/format. reg wb_start; wire wb_busy, wb_done; wire wb_mem_active; wire wb_ctrl_req, wb_ctrl_wr; wire [ADDR_WIDTH-2:0] wb_ctrl_addr; wire [32*BURST_LEN-1:0] wb_ctrl_wdata; wire [4*BURST_LEN-1:0] wb_ctrl_wmask; result_writeback #( .BURST_LEN(BURST_LEN), .DATA_WIDTH(DATA_WIDTH), .JOB_ADDR_WIDTH(ADDR_WIDTH), .ADDR_WIDTH(ADDR_WIDTH-1) ) u_wb ( .clk(clk), .rst(rst), .start(wb_start), .result_addr_a(result_addr_a_lat), .result_addr_b(result_addr_b_lat), .result_data_a(result_data_a), .result_data_b(result_data_b), .result_node_id_a(result_node_id_a), .result_node_id_b(result_node_id_b), .busy(wb_busy), .done(wb_done), .mem_active(wb_mem_active), .mem_grant(mem_grant), .ctrl_req(wb_ctrl_req), .ctrl_wr(wb_ctrl_wr), .ctrl_addr(wb_ctrl_addr), .ctrl_wdata(wb_ctrl_wdata), .ctrl_wmask(wb_ctrl_wmask), .ctrl_rdata(ctrl_rdata), .ctrl_ready(ctrl_ready), .ctrl_busy(ctrl_busy) ); // mutually exclusive by FSM construction (weight prefetch always // fully completes, incl. consume_done, before the tile loop that // triggers ddrpf_job_start ever fires; writeback only ever starts // in S_RESULT, strictly after that same tile loop has already // finished) -- safe to select on act_mem_active/wb_mem_active alone. assign ctrl_req = act_mem_active ? act_ctrl_req : (wb_mem_active ? wb_ctrl_req : pf_ctrl_req); assign ctrl_wr = act_mem_active ? act_ctrl_wr : (wb_mem_active ? wb_ctrl_wr : pf_ctrl_wr); assign ctrl_addr = act_mem_active ? act_ctrl_addr : (wb_mem_active ? wb_ctrl_addr : pf_ctrl_addr); assign ctrl_wdata = act_mem_active ? act_ctrl_wdata : (wb_mem_active ? wb_ctrl_wdata : pf_ctrl_wdata); assign ctrl_wmask = act_mem_active ? act_ctrl_wmask : (wb_mem_active ? wb_ctrl_wmask : pf_ctrl_wmask); assign mem_active = (state == S_MEMWAIT) || (state == S_PREFETCH) || act_mem_active || wb_mem_active; // ---- layer_weight_buffer.v ---- wire [BUFADDRW-1:0] lwb_rd_addr; wire [DATA_WIDTH-1:0] lwb_rd_data; reg consume_done; layer_weight_buffer #(.DATA_WIDTH(DATA_WIDTH), .LAYER_DEPTH(LAYER_BYTES)) u_lwb ( .clk(clk), .rst(rst), .fill_we(pf_fill_we), .fill_addr(pf_fill_addr), .fill_data(pf_fill_data), .fill_done(pf_done), .rd_addr(lwb_rd_addr), .rd_data(lwb_rd_data), .consume_done(consume_done), .active_sel(), .swapped() ); // ---- weight_tile_gather.v ---- reg tile_req; reg [BUFADDRW-1:0] tile_base; reg tile_seen; // S_TILEWAIT join latch (weight side only -- see header; // the activation side, ddrpf_tile_valid, is level-held by // ddr_prefetch_mgr.v so it needs no separate latch) wire tile_valid; wire [DATA_WIDTH*P_IN-1:0] tile_data; weight_tile_gather #( .DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .BUFADDRW(BUFADDRW) ) u_gather ( .clk(clk), .rst(rst), .tile_req(tile_req), .tile_base(tile_base), .tile_valid(tile_valid), .tile_data(tile_data), .rd_addr(lwb_rd_addr), .rd_data(lwb_rd_data) ); // ---- neural_processor_packed.v ---- reg job_valid_np; wire job_ready_np; reg [1:0] job_activation; reg signed [DATA_WIDTH-1:0] job_bias; reg operand_valid; wire operand_ready; reg signed [DATA_WIDTH*P_IN-1:0] input_data_a_r, input_data_b_r; reg [DATA_WIDTH*P_IN-1:0] weight_data_r; reg tile_last; wire result_valid_np; reg result_ready; wire signed [DATA_WIDTH-1:0] result_data_a_np, result_data_b_np; wire [15:0] result_node_id_a_np, result_node_id_b_np; wire [3:0] np_state; wire np_error; neural_processor_packed #( .DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .ACC_WIDTH(ACC_WIDTH) ) u_np ( .clk(clk), .rst(rst), .job_valid(job_valid_np), .job_ready(job_ready_np), .job_node_id_a(node_id_a_lat), .job_node_id_b(node_id_b_lat), .job_bias(job_bias), .job_activation(job_activation), .operand_valid(operand_valid), .operand_ready(operand_ready), .input_data_a(input_data_a_r), .input_data_b(input_data_b_r), .weight_data(weight_data_r), .tile_last(tile_last), .result_valid(result_valid_np), .result_ready(result_ready), .result_data_a(result_data_a_np), .result_data_b(result_data_b_np), .result_node_id_a(result_node_id_a_np), .result_node_id_b(result_node_id_b_np), .np_state(np_state), .np_error(np_error) ); localparam ACT_RELU = 2'd1; always @(posedge clk) begin if (rst) begin state <= S_IDLE; job_done <= 1'b0; pf_start <= 1'b0; consume_done <= 1'b0; tile_req <= 1'b0; ddrpf_job_start <= 1'b0; ddrpf_tile_consume <= 1'b0; tile_seen <= 1'b0; job_valid_np <= 1'b0; operand_valid<= 1'b0; tile_last <= 1'b0; result_ready <= 1'b0; wb_start <= 1'b0; job_bias <= {DATA_WIDTH{1'b0}}; job_activation <= ACT_RELU; tcnt <= 16'd0; end else begin job_done <= 1'b0; pf_start <= 1'b0; consume_done <= 1'b0; tile_req <= 1'b0; ddrpf_job_start <= 1'b0; ddrpf_tile_consume <= 1'b0; wb_start <= 1'b0; case (state) S_IDLE: begin if (job_start) begin w_base_lat <= w_base; x_base_a_lat <= x_base_a; x_base_b_lat <= x_base_b; n_tiles_lat <= n_tiles; result_addr_a_lat <= result_addr_a; result_addr_b_lat <= result_addr_b; node_id_a_lat <= node_id_a; node_id_b_lat <= node_id_b; job_bias <= {DATA_WIDTH{1'b0}}; job_activation <= ACT_RELU; state <= S_MEMWAIT; end end S_MEMWAIT: begin if (mem_grant) begin pf_start <= 1'b1; state <= S_PREFETCH; end end S_PREFETCH: begin if (pf_done) begin consume_done <= 1'b1; state <= S_SWAP; end end S_SWAP: begin // one settle cycle for layer_weight_buffer.v's own // do_swap (fill_done_latched already set from // pf_done above; consume_done pulsed this cycle) -- // matches EXP-0058/0062's own tested sequencing. job_valid_np <= 1'b1; state <= S_JOBSTART; end S_JOBSTART: begin if (job_valid_np && job_ready_np) begin job_valid_np <= 1'b0; tcnt <= 16'd0; ddrpf_job_start <= 1'b1; // one-shot: kicks off the whole job's // look-ahead tile loop inside u_ddrpf state <= S_TILEREQ; end end S_TILEREQ: begin tile_req <= 1'b1; tile_base <= tcnt[BUFADDRW-1:0]*P_IN[BUFADDRW-1:0]; tile_seen <= 1'b0; state <= S_TILEWAIT; end // Real join: weight_tile_gather.v's tile_valid (fast, // on-chip, one-cycle pulse -- latched via tile_seen) // and u_ddrpf's ddrpf_tile_valid (real DDR3 latency, // but LEVEL-held by the prefetch manager's own ping- // pong buffer, possibly already true this cycle if the // look-ahead fetch completed early) do NOT arrive on // the same cycle in general -- proceed once BOTH are // available. ddrpf_tile_valid needs no separate latch // since it stays high until this slot pulses // ddrpf_tile_consume itself. S_TILEWAIT: begin if (tile_valid) begin weight_data_r <= tile_data; tile_seen <= 1'b1; end if ((tile_valid || tile_seen) && ddrpf_tile_valid) begin input_data_a_r <= act_data_a_w; input_data_b_r <= act_data_b_w; ddrpf_tile_consume <= 1'b1; tile_last <= (tcnt == n_tiles_lat - 16'd1); operand_valid <= 1'b1; state <= S_OPERAND; end end S_OPERAND: begin if (operand_valid && operand_ready) begin operand_valid <= 1'b0; tile_last <= 1'b0; if (tcnt == n_tiles_lat - 16'd1) begin result_ready <= 1'b1; state <= S_RESULT; end else begin tcnt <= tcnt + 16'd1; state <= S_TILEREQ; end end end // EXP-0088: result_data_a/b/result_node_id_a/b/ // result_addr_a_out/b_out remain real output ports // (unchanged, still updated exactly as before -- kept // for debug/testbench visibility), but job_done is now // held back until the real DDR3 writeback itself // completes (S_WRITEBACK), not just the moment // result_valid_np fires -- the whole point of this // module's own EXP-0088 upgrade is that job_done means // "the result is durably in DDR3", not merely "captured // in a register only this slot's own top-level pins // could see". S_RESULT: begin if (result_valid_np) begin result_data_a <= result_data_a_np; result_data_b <= result_data_b_np; result_node_id_a <= result_node_id_a_np; result_node_id_b <= result_node_id_b_np; result_addr_a_out <= result_addr_a_lat; result_addr_b_out <= result_addr_b_lat; result_ready <= 1'b0; wb_start <= 1'b1; state <= S_WRITEBACK; end end S_WRITEBACK: begin if (wb_done) begin job_done <= 1'b1; state <= S_IDLE; end end default: state <= S_IDLE; endcase end end endmodule