michele/FPGA-Neural
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

perf(v2): shared activation cache - further 1.66-2.00x real speedup (DEC-0016)

Browse Source 
Implements optimization #2 from the final benchmark campaign's own
recommendation, on top of DEC-0015's word-level burst rewrite: a new
shared activation_cache.v module fetches a given activation (X)
vector from PSRAM once instead of once per neuron sharing it - the
exact redundant traffic pattern the dense-layer workloads in this
project's benchmark suite exhibit.

Each memory_manager's own prefetch_engine now fetches WEIGHTS only;
the activation half is requested from the shared cache instead
(single-tag, tile-granular, N_SLOTS request ports, its own real
word-level PSRAM backend via a new dedicated arbiter port).
dataflow_core.v/slot_mem_arbiter.v/neural_multiprocessor.v widened to
N_SLOTS+1 ports to arbitrate the cache's traffic alongside each
slot's weight traffic.

Two real bugs found and fixed during implementation (ERR-0010): a
target-bank/pending-bank race in memory_manager.v's activation-cache
wiring (the same bug class ERR-0006 already fixed once for
pf_target_bank - a later handoff's queued request can overwrite which
bank an earlier, still-in-flight request's ack applies to), and a
repeat of ERR-0009's N_SLOTS=1 zero-width replication bug in
activation_cache.v itself.

Real, measured results: the full final-benchmark campaign (24/24
workload/config combinations) re-verified bit-exact. D-Stress cycles
fall a further 1.66-2.00x on top of DEC-0015 (~4x combined vs the
original byte-level baseline). But the cache's real Fmax cost is much
steeper than DEC-0015's own: N_SLOTS=2 (the recommended default,
DEC-0014) drops from 133.58 to 87.72 MHz (-34%, margin over 80MHz
shrinks from +67% to +9.7%), and N_SLOTS=4 drops to 65.01 MHz - now
FAILING the 80MHz target it previously passed. Combined real
wall-clock speedup vs the original baseline: N=1 3.86x, N=2 2.45x
(both real net wins); N=4 is a real regression once its own now-failing
Fmax is honestly used, though N=4 was never the recommended
configuration.

N_SLOTS=2 remains the recommended default (DEC-0014 unaffected) with
a thinner but still real Fmax margin. Cache hit-detection pipelining
is flagged as concrete follow-up work if N_SLOTS>2 is ever needed with
the cache active - not attempted this round.

Logged: simulation/synthesis/timing/benchmark/decisions (DEC-0016)/
experiments (EXP-0016)/errors (ERR-0010)/development.log, ROADMAP.md
updated.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013xXuuRUWZScuo1DeYJxs3v
This commit is contained in:
michele
2026-09-05 20:59:14 +02:00
co-authored by Claude Sonnet 5
parent e4a5540b6e
commit 63cac6a7e5
16 changed files with 931 additions and 211 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hardware/v2/rtl/activation_cache.v
+224
View File
@@ -0,0 +1,224 @@
`timescale 1ns/1ps
// ================================================================
// FPGA-Neural V2 -- Shared Activation Cache (post-M10, docs/v2-
// description.md §14; decisions.log DEC-0016)
//
// User-requested optimization #2, following the final benchmark
// campaign's own recommendation: in the realistic dense-layer
// workloads this project benchmarks (hardware/v2/docs/benchmarks/
// final-benchmark.md), many neurons in the same layer share the
// EXACT SAME activation (X) input vector -- each of dataflow_core's
// N_SLOTS memory_manager instances re-fetching that identical vector
// from PSRAM independently was real, measured, redundant traffic on
// the one shared PSRAM port. This module fetches a given X vector
// from PSRAM ONCE (tile by tile, on first use) and serves every
// subsequent request for the SAME x_base/tile directly from an
// on-chip buffer -- no PSRAM access at all on a hit.
//
// Single-tag design: one active cached x_base at a time, filled
// tile-by-tile up to `filled_up_to` (tiles [0, filled_up_to) are
// valid). A request for a DIFFERENT x_base invalidates the cache and
// restarts filling from tile 0 for the new tag. This is correct
// (never serves stale/wrong data -- a tag switch always resets
// filled_up_to, so a later request against the OLD tag is treated as
// a fresh miss, refetched from scratch) but can THRASH under
// interleaved concurrent requests for genuinely different x_base
// values (falls back to no worse than the pre-cache behavior, never
// incorrect -- see decisions.log DEC-0016 for the full analysis).
// Fine for this project's own realistic workload shape (a "layer" of
// neurons dispatched together, sharing one x_base for the whole
// phase); a multi-way cache would avoid thrashing for interleaved
// multi-layer traffic, deferred until measured to matter.
//
// Request protocol: each of N_SLOTS ports issues a ONE-CYCLE req
// pulse (x_base + tile_idx); the cache LATCHES it into a per-slot
// pending register regardless of hit/miss/fetch-in-progress state --
// the same single-entry "queue, don't drop the request" idiom already
// used by memory_manager's own pf_pending register (ERR-0006) and
// slot_mem_arbiter's own pending latch (ERR-0008) -- so a request
// arriving while the cache is busy filling a miss for another slot is
// never lost. ack pulses exactly once per request, the cycle its
// data becomes available (immediately, if already a hit at latch
// time; after the real PSRAM fetch completes, on a miss). Multiple
// slots pending on tiles that become valid the SAME cycle a fetch
// completes are all acked that same cycle (broadcast hit).
//
// Backend: word-level (16-bit + lb_n/ub_n), same convention as
// prefetch_engine.v post-DEC-0015 -- talks to memory_interface.v's
// own 16-bit word interface via the shared slot_mem_arbiter.v (one
// more arbiter port, dedicated to this cache).
// ================================================================
module activation_cache #(
parameter DATA_WIDTH = 8,
parameter P_IN = 8,
parameter ADDR_WIDTH = 23,
parameter N_SLOTS = 4,
parameter MAX_TILES = 16 // max cacheable vector length, in tiles
)(
input wire clk,
input wire rst,
// ---- per-slot request port (one per memory_manager) ----
input wire [N_SLOTS-1:0] req,
input wire [ADDR_WIDTH*N_SLOTS-1:0] req_x_base,
input wire [16*N_SLOTS-1:0] req_tile_idx,
output reg [N_SLOTS-1:0] ack,
output reg signed [DATA_WIDTH*P_IN*N_SLOTS-1:0] tile_x_out,
// ---- shared backend port (word-level, -> slot_mem_arbiter.v) ----
output reg mem_req,
output reg mem_wr,
output reg [ADDR_WIDTH-1:0] mem_addr, // WORD address
output reg [15:0] mem_wdata, // unused (read-only), tied 0
output reg mem_lb_n,
output reg mem_ub_n,
input wire [15:0] mem_rdata,
input wire mem_ready
);
localparam WORDS_PER_TILE = P_IN/2;
localparam WIW = $clog2(WORDS_PER_TILE+1);
localparam TIW = $clog2(MAX_TILES+1);
localparam ST_IDLE = 1'd0;
localparam ST_FETCH = 1'd1;
reg state;
reg [ADDR_WIDTH-1:0] tag;
reg tag_valid;
reg [TIW-1:0] filled_up_to;
reg signed [DATA_WIDTH*P_IN-1:0] tile_store [0:MAX_TILES-1];
reg [TIW-1:0] fetch_tile_idx;
reg [WIW-1:0] word_idx;
// ---- per-slot pending-request latch (see file header) ----
reg [N_SLOTS-1:0] pending;
reg [ADDR_WIDTH*N_SLOTS-1:0] pending_x_base;
reg [16*N_SLOTS-1:0] pending_tile_idx;
integer pi;
wire [N_SLOTS-1:0] hit;
wire [N_SLOTS-1:0] miss;
genvar gi;
generate
for (gi = 0; gi < N_SLOTS; gi = gi + 1) begin : GEN_HITCHK
assign hit[gi] = pending[gi] && tag_valid &&
(pending_x_base[gi*ADDR_WIDTH +: ADDR_WIDTH] == tag) &&
(pending_tile_idx[gi*16 +: 16] < {{(16-TIW){1'b0}}, filled_up_to});
assign miss[gi] = pending[gi] && !hit[gi];
end
endgenerate
// Fixed lowest-index-wins priority scan over MISS requests (same
// convention as neural_director/dependency_manager/slot_mem_arbiter).
reg [$clog2(N_SLOTS)-1:0] miss_idx;
reg any_miss;
integer mi;
always @(*) begin
miss_idx = '0; // '0 self-sizes for any width incl. 0 (N_SLOTS=1) -- see errors.log ERR-0009
any_miss = 1'b0;
for (mi = N_SLOTS-1; mi >= 0; mi = mi - 1) begin
if (miss[mi]) begin
miss_idx = mi[$clog2(N_SLOTS)-1:0];
any_miss = 1'b1;
end
end
end
wire [ADDR_WIDTH-1:0] miss_x_base = pending_x_base[miss_idx*ADDR_WIDTH +: ADDR_WIDTH];
wire miss_is_new_tag = !tag_valid || (miss_x_base != tag);
wire [TIW-1:0] next_fetch_tile = miss_is_new_tag ? {TIW{1'b0}} : filled_up_to;
wire [ADDR_WIDTH-1:0] next_word_base = miss_x_base[ADDR_WIDTH-1:1] +
(next_fetch_tile * WORDS_PER_TILE[TIW-1:0]);
always @(posedge clk) begin
if (rst) begin
state <= ST_IDLE;
tag <= {ADDR_WIDTH{1'b0}};
tag_valid <= 1'b0;
filled_up_to <= {TIW{1'b0}};
fetch_tile_idx <= {TIW{1'b0}};
word_idx <= {WIW{1'b0}};
pending <= {N_SLOTS{1'b0}};
pending_x_base <= {(ADDR_WIDTH*N_SLOTS){1'b0}};
pending_tile_idx <= {(16*N_SLOTS){1'b0}};
ack <= {N_SLOTS{1'b0}};
tile_x_out <= {(DATA_WIDTH*P_IN*N_SLOTS){1'b0}};
mem_req <= 1'b0;
mem_wr <= 1'b0;
mem_addr <= {ADDR_WIDTH{1'b0}};
mem_wdata <= 16'h0000;
mem_lb_n <= 1'b1;
mem_ub_n <= 1'b1;
end else begin
mem_req <= 1'b0;
ack <= {N_SLOTS{1'b0}};
// Latch every incoming request pulse (never dropped, see
// file header).
for (pi = 0; pi < N_SLOTS; pi = pi + 1) begin
if (req[pi]) begin
pending[pi] <= 1'b1;
pending_x_base[pi*ADDR_WIDTH +: ADDR_WIDTH] <= req_x_base[pi*ADDR_WIDTH +: ADDR_WIDTH];
pending_tile_idx[pi*16 +: 16] <= req_tile_idx[pi*16 +: 16];
end
end
// Serve every currently-pending HIT this same cycle
// (broadcast -- see file header). Safe against colliding
// with the latch loop above: a slot only ever hits while
// its OWN pending bit was already set on an EARLIER cycle
// (this cycle's freshly-latched requests read `filled_up_to`/
// `tag` at their OWN NEXT evaluation, not this one).
for (pi = 0; pi < N_SLOTS; pi = pi + 1) begin
if (hit[pi]) begin
ack[pi] <= 1'b1;
tile_x_out[pi*DATA_WIDTH*P_IN +: DATA_WIDTH*P_IN] <= tile_store[pending_tile_idx[pi*16 +: 16]];
pending[pi] <= 1'b0;
end
end
case (state)
ST_IDLE: begin
if (any_miss) begin
tag <= miss_x_base;
tag_valid <= 1'b1;
filled_up_to <= miss_is_new_tag ? {TIW{1'b0}} : filled_up_to;
fetch_tile_idx <= next_fetch_tile;
word_idx <= {WIW{1'b0}};
mem_req <= 1'b1;
mem_wr <= 1'b0;
mem_lb_n <= 1'b0;
mem_ub_n <= 1'b0;
mem_addr <= next_word_base;
state <= ST_FETCH;
end
end
ST_FETCH: begin
if (mem_ready) begin
tile_store[fetch_tile_idx][word_idx*16 +: 16] <= mem_rdata;
if (word_idx == WORDS_PER_TILE[WIW-1:0] - 1'b1) begin
filled_up_to <= fetch_tile_idx + 1'b1;
state <= ST_IDLE;
end else begin
word_idx <= word_idx + 1'b1;
mem_req <= 1'b1;
mem_wr <= 1'b0;
mem_lb_n <= 1'b0;
mem_ub_n <= 1'b0;
mem_addr <= tag[ADDR_WIDTH-1:1] + fetch_tile_idx*WORDS_PER_TILE[TIW-1:0] + word_idx + 1'b1;
end
end
end
default: state <= ST_IDLE;
endcase
end
end
endmodule
hardware/v2/rtl/dataflow_core.v
+70 -26
View File
@@ -34,19 +34,32 @@
// component gluing the two together.
//
// Scope (see hardware/v2/logs/decisions.log DEC-0009):
// - activation_buffer.v/weight_buffer.v/result_buffer.v (M3) are NOT
// instantiated inside dataflow_core yet -- they belong on the OTHER
// side of the Memory Backend Interface (§15's own diagram: Memory
// Manager -> Memory Backend Interface -> PSRAM Controller), and
// each memory_manager instance already owns its own prefetch double
// buffer (M4) for the fast path. Wiring the M3 buffers in as a
// shared on-chip cache in front of PSRAM is real future work, not
// done here (no measured need for it yet, §22/§30).
// - each slot's byte-level Memory Backend Interface port is exposed
// SEPARATELY (N_SLOTS independent ports) rather than arbitrated
// down to one shared PSRAM master -- real PSRAM integration
// (including whatever arbitration N_SLOTS>1 requires) is explicitly
// M8's job, not this one's.
// - activation_buffer.v/weight_buffer.v/result_buffer.v (M3, BRAM-
// backed FIFOs) are NOT instantiated here -- superseded by a
// different, measurement-driven shared cache (activation_cache.v,
// post-M10 DEC-0016, see below), not the original M3 modules
// themselves.
// - each slot's Memory Backend Interface port is exposed SEPARATELY
// (N_SLOTS independent ports) rather than arbitrated down to one
// shared PSRAM master -- real PSRAM integration (including whatever
// arbitration N_SLOTS>1 requires) is done one level up, in
// neural_multiprocessor.v (M8).
//
// Post-M10 (decisions.log DEC-0016): a single shared activation_cache
// instance sits alongside the N_SLOTS memory_managers, serving the
// ACTIVATION (X) half of each tile fetch -- in the realistic dense-
// layer workloads this project benchmarks, many neurons share the
// exact same X vector, and fetching it from PSRAM once instead of
// once per memory_manager instance is real, measured, redundant-
// traffic elimination (see hardware/v2/docs/benchmarks/
// final-benchmark.md's own recommendation #2). Each memory_manager's
// own prefetch_engine now fetches WEIGHTS only. The exposed
// slot_mem_* arrays are sized N_SLOTS+1: indices [0, N_SLOTS) are the
// per-slot memory_managers' own weight+write-back backend ports
// (unchanged in spirit from before), index [N_SLOTS] is the shared
// activation_cache's own backend port -- all N_SLOTS+1 arbitrated
// together by neural_multiprocessor.v's slot_mem_arbiter.v (N_PORTS
// widened to N_SLOTS+1 there to match).
// ================================================================
module dataflow_core #(
@@ -73,19 +86,20 @@ module dataflow_core #(
input wire [15:0] reg_n_tiles,
input wire [ADDR_WIDTH-1:0] reg_result_addr,
// ---- per-slot Memory Backend Interface (arrayed, one per slot --
// see file header on why arbitration to one shared PSRAM port is
// NOT done here). WORD-level (16-bit) post-M10 (decisions.log
// DEC-0015) -- see memory_manager.v/prefetch_engine.v's own
// headers for why. ----
output wire [N_SLOTS-1:0] slot_mem_req,
output wire [N_SLOTS-1:0] slot_mem_wr,
output wire [ADDR_WIDTH*N_SLOTS-1:0] slot_mem_addr, // WORD address
output wire [16*N_SLOTS-1:0] slot_mem_wdata,
output wire [N_SLOTS-1:0] slot_mem_lb_n,
output wire [N_SLOTS-1:0] slot_mem_ub_n,
input wire [16*N_SLOTS-1:0] slot_mem_rdata,
input wire [N_SLOTS-1:0] slot_mem_ready
// ---- Memory Backend Interface, arrayed N_SLOTS+1 wide (see file
// header: indices [0,N_SLOTS) are the per-slot memory_managers'
// own weight+write-back ports, index [N_SLOTS] is the shared
// activation_cache's own port). WORD-level (16-bit) post-M10
// (decisions.log DEC-0015) -- see memory_manager.v/
// prefetch_engine.v's own headers for why. ----
output wire [N_SLOTS:0] slot_mem_req,
output wire [N_SLOTS:0] slot_mem_wr,
output wire [ADDR_WIDTH*(N_SLOTS+1)-1:0] slot_mem_addr, // WORD address
output wire [16*(N_SLOTS+1)-1:0] slot_mem_wdata,
output wire [N_SLOTS:0] slot_mem_lb_n,
output wire [N_SLOTS:0] slot_mem_ub_n,
input wire [16*(N_SLOTS+1)-1:0] slot_mem_rdata,
input wire [N_SLOTS:0] slot_mem_ready
);
localparam NODE_IDW = $clog2(N_NODES);
@@ -153,6 +167,14 @@ module dataflow_core #(
assign dm_producer_done_valid = dir_job_out_done;
assign dm_producer_done_node_id = completed_node_id_16[NODE_IDW-1:0];
// ---- shared activation_cache request bus (one port per slot,
// collected here for the cache instance below) ----
wire [N_SLOTS-1:0] xc_req;
wire [ADDR_WIDTH*N_SLOTS-1:0] xc_x_base;
wire [16*N_SLOTS-1:0] xc_tile_idx;
wire [N_SLOTS-1:0] xc_ack;
wire signed [DATA_WIDTH*P_IN*N_SLOTS-1:0] xc_tile_x;
// ---- N_SLOTS x (Memory Manager (M4) + Neural Processor (M1)) ----
genvar g;
generate
@@ -177,6 +199,11 @@ module dataflow_core #(
.operand_valid(mm_operand_valid), .operand_ready(mm_operand_ready),
.input_data(mm_input_data), .weight_data(mm_weight_data), .tile_last(mm_tile_last),
.result_valid(mm_result_valid), .result_ready(mm_result_ready), .result_data(mm_result_data),
.xc_req(xc_req[g]),
.xc_x_base(xc_x_base[g*ADDR_WIDTH +: ADDR_WIDTH]),
.xc_tile_idx(xc_tile_idx[g*16 +: 16]),
.xc_ack(xc_ack[g]),
.xc_tile_x(xc_tile_x[g*DATA_WIDTH*P_IN +: DATA_WIDTH*P_IN]),
.mem_req(slot_mem_req[g]), .mem_wr(slot_mem_wr[g]),
.mem_addr(slot_mem_addr[g*ADDR_WIDTH +: ADDR_WIDTH]),
.mem_wdata(slot_mem_wdata[g*16 +: 16]),
@@ -215,4 +242,21 @@ module dataflow_core #(
end
endgenerate
// ---- shared activation_cache (M10+, DEC-0016) -- serves the
// ACTIVATION half of every slot's tile fetch, using arbiter port
// index N_SLOTS (the last one) for its own PSRAM traffic on a
// cache miss. ----
activation_cache #(
.DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .ADDR_WIDTH(ADDR_WIDTH), .N_SLOTS(N_SLOTS)
) u_activation_cache (
.clk(clk), .rst(rst),
.req(xc_req), .req_x_base(xc_x_base), .req_tile_idx(xc_tile_idx),
.ack(xc_ack), .tile_x_out(xc_tile_x),
.mem_req(slot_mem_req[N_SLOTS]), .mem_wr(slot_mem_wr[N_SLOTS]),
.mem_addr(slot_mem_addr[N_SLOTS*ADDR_WIDTH +: ADDR_WIDTH]),
.mem_wdata(slot_mem_wdata[N_SLOTS*16 +: 16]),
.mem_lb_n(slot_mem_lb_n[N_SLOTS]), .mem_ub_n(slot_mem_ub_n[N_SLOTS]),
.mem_rdata(slot_mem_rdata[N_SLOTS*16 +: 16]), .mem_ready(slot_mem_ready[N_SLOTS])
);
endmodule
hardware/v2/rtl/memory_manager.v
+150 -87
View File
@@ -10,42 +10,41 @@
// The processor sees only "data available" (operand_valid/ready,
// tile_last) -- never PSRAM request/wait cycles directly (§12).
//
// Post-M10 (decisions.log DEC-0015): this port talks directly to
// memory_interface.v's own 16-bit word interface instead of routing
// through int8_memory_access.v's byte-splitting layer -- every real
// transaction now moves a full PSRAM word (2 bytes) instead of
// discarding half of one, halving the real transaction count for
// prefetch_engine's own reads. int8_memory_access.v itself is
// Post-M10 (decisions.log DEC-0015): the WEIGHT backend port talks
// directly to memory_interface.v's own 16-bit word interface instead
// of routing through int8_memory_access.v's byte-splitting layer --
// every real transaction now moves a full PSRAM word (2 bytes)
// instead of discarding half of one. int8_memory_access.v itself is
// untouched (still frozen V1); V2 simply no longer instantiates it in
// this datapath, reusing the lower (word-level) layer directly
// instead, the same "reuse what fits" precedent slot_mem_arbiter.v
// already set for hardware/v1/rtl/mem_arbiter.v.
//
// Post-M10 (decisions.log DEC-0016): the ACTIVATION (X) side is no
// longer fetched from PSRAM by this module's own prefetch_engine at
// all -- it is requested from a shared activation_cache.v instance
// (one per dataflow_core, not one per slot), which fetches a given
// X vector from PSRAM once and serves every memory_manager sharing
// that same x_base directly on-chip. Each bank therefore becomes
// ready only once BOTH its activation half (cache ack) AND its
// weight half (prefetch_engine's own pf_done, now W-only) have
// arrived -- bank_ready[b] = bank_x_ready[b] && bank_w_ready[b].
//
// Double-buffered prefetch (§13): while the processor consumes tile
// N from bank "current", this module retargets the single
// prefetch_engine instance (M4) at bank "next" to fetch tile N+1
// N from bank "current", this module retargets its single
// prefetch_engine instance (M4, W-only) and issues a fresh
// activation_cache request at bank "next" to fetch tile N+1
// concurrently. On tile handoff, banks swap; if a bank isn't ready in
// time (prefetch slower than compute for this run), operand_valid
// simply stays low until it is -- a real stall, not hidden, so its
// frequency is genuinely measurable (§22, deferred to M9). NOTE
// (measured characteristic, not yet optimized -- see
// hardware/v2/logs/decisions.log DEC-0006): the bank-swap-and-check
// control path itself costs a minimum 1 idle cycle per tile handoff
// even when the next bank was already prefetched in time, unlike
// neural_processor.v's own zero-gap tile acceptance -- a real,
// deliberately-not-hidden overhead of this first Memory Manager
// implementation, left for M10 (Optimization) to revisit with real
// stall-percentage data (§22) rather than optimized blindly now.
// time, operand_valid simply stays low until it is -- a real stall,
// not hidden (§22). NOTE (measured characteristic, not yet optimized
// -- see decisions.log DEC-0006): the bank-swap-and-check control path
// itself costs a minimum 1 idle cycle per tile handoff even when the
// next bank was already prefetched in time, unlike neural_processor.v's
// own zero-gap tile acceptance.
//
// One job = one neuron's worth of tiles (n_tiles), read from x_base/
// w_base (PSRAM byte addresses), followed by writing the single
// INT8 result back to result_addr. The result write only happens
// after the last tile has been handed off and prefetch_engine is
// idle (temporally disjoint from prefetching by construction), so no
// separate backend arbiter is needed at this milestone -- see
// decisions.log DEC-0006 for why, and what changes once multiple
// concurrent jobs/processors need to share one backend port
// (deferred, not yet needed).
// w_base (PSRAM byte addresses), followed by writing the single INT8
// result back to result_addr.
// ================================================================
module memory_manager #(
@@ -56,8 +55,7 @@ module memory_manager #(
input wire clk,
input wire rst,
// ---- job control (from a future Neural Director, M5; driven
// directly by a testbench at M4) ----
// ---- job control (from Neural Director, M5) ----
input wire job_start,
input wire [ADDR_WIDTH-1:0] x_base,
input wire [ADDR_WIDTH-1:0] w_base,
@@ -78,12 +76,19 @@ module memory_manager #(
output reg result_ready,
input wire signed [DATA_WIDTH-1:0] result_data,
// ---- Memory Backend Interface (word-level, matches
// ---- shared activation_cache.v request port (post-M10 DEC-0016
// -- one per memory_manager instance, cache is shared/instantiated
// once per dataflow_core) ----
output reg xc_req, // one-cycle pulse
output reg [ADDR_WIDTH-1:0] xc_x_base,
output reg [15:0] xc_tile_idx,
input wire xc_ack, // one-cycle pulse
input wire signed [DATA_WIDTH*P_IN-1:0] xc_tile_x,
// ---- WEIGHT Memory Backend Interface (word-level, matches
// hardware/v1/rtl/memory_interface.v's contract exactly -- see
// prefetch_engine.v's own header and decisions.log DEC-0015 for
// why this is now word- rather than byte-level: int8_memory_access.v
// is no longer in the datapath, each transaction moves a full
// 16-bit PSRAM word instead of discarding half of it) ----
// prefetch_engine.v's own header and decisions.log DEC-0015/
// DEC-0016 for why this is word-level and weight-only) ----
output wire mem_req,
output wire mem_wr,
output wire [ADDR_WIDTH-1:0] mem_addr, // WORD address
@@ -108,17 +113,55 @@ module memory_manager #(
reg [15:0] tile_idx; // tile currently presented (bank `current`)
reg current_bank; // 0 or 1
reg [1:0] bank_ready; // bank_ready[b] = bank b holds valid, unconsumed prefetched data
// ---- double-buffer storage: X half filled by the shared cache,
// W half filled by this module's own prefetch_engine -- a bank is
// usable once BOTH halves have arrived. ----
reg [1:0] bank_x_ready, bank_w_ready;
wire [1:0] bank_ready = bank_x_ready & bank_w_ready;
// ---- double-buffer storage (owned here, filled by prefetch_engine) ----
reg signed [DATA_WIDTH*P_IN-1:0] bank_x [0:1];
reg signed [DATA_WIDTH*P_IN-1:0] bank_w [0:1];
// ---- single prefetch_engine instance, retargeted per bank ----
// ---- activation_cache request bookkeeping: single-entry pending
// (same idiom as pf_pending below -- only one outstanding cache
// request at a time, one instance to serve, one bank as its target). ----
reg xc_pending;
reg [ADDR_WIDTH-1:0] xc_pending_x_base;
reg [15:0] xc_pending_tile_idx;
reg xc_pending_bank;
// xc_target_bank is the bank the CURRENTLY-outstanding (already
// issued) cache request will fill -- set ONLY by the issue rule
// below, from xc_pending_bank, at the exact moment xc_req fires.
// Queueing logic (MM_IDLE/MM_PREFETCH_FIRST/MM_STREAM) writes
// xc_pending_bank, NEVER xc_target_bank directly -- writing
// xc_target_bank directly from queueing was a real bug (found via
// simulation): a later handoff can queue a NEW request (targeting
// a DIFFERENT bank) in the same cycle an EARLIER request is being
// issued, and program-order NBA "last write wins" would silently
// overwrite which bank the EARLIER (already in-flight) request's
// eventual ack gets applied to -- the exact same class of bug
// ERR-0006 already found and fixed once for pf_target_bank/
// pf_pending_bank (which already used this two-register pattern
// correctly; this module's activation-cache side did not, until
// now).
reg xc_target_bank;
// Tracks whether THIS instance's own cache request is still
// awaiting its ack (real PSRAM miss latency can easily exceed one
// neural_processor tile's own compute time, so a later handoff's
// "queue the next request" can genuinely race an earlier request
// still in flight -- the same class of race ERR-0006 already found
// and fixed once for pf_pending/pf_busy; fixed here the same way,
// with an explicit outstanding flag this module controls directly
// rather than inferring busy-ness from a signal with its own
// latency quirk).
reg xc_outstanding;
// ---- single prefetch_engine instance (WEIGHT-only post-DEC-0016),
// retargeted per bank ----
reg pf_start;
reg [ADDR_WIDTH-1:0] pf_x_addr, pf_w_addr;
reg [ADDR_WIDTH-1:0] pf_w_addr;
wire pf_busy, pf_done;
wire signed [DATA_WIDTH*P_IN-1:0] pf_tile_x, pf_tile_w;
wire signed [DATA_WIDTH*P_IN-1:0] pf_tile_w;
reg pf_target_bank; // which bank the CURRENTLY-running (or just-launched) prefetch fills
@@ -132,15 +175,17 @@ module memory_manager #(
// descriptor instead of touching pf_start directly; a single
// always-active rule issues pf_start once the engine is free.
reg pf_pending;
reg [ADDR_WIDTH-1:0] pf_pending_x, pf_pending_w;
reg [ADDR_WIDTH-1:0] pf_pending_w;
reg pf_pending_bank;
// prefetch_engine drives its OWN internal backend wires; the
// result-write FSM below drives its own. A combinational mux
// (never both at once, by construction -- see file header)
// selects which one actually reaches the real output port,
// avoiding a two-driver conflict on mem_req/mem_wr/mem_addr/
// mem_wdata/mem_lb_n/mem_ub_n.
// prefetch_engine drives its OWN internal weight-backend wires;
// the result-write FSM below drives its own. A combinational mux
// (never both at once, by construction -- MM_WRITE_RESULT/MM_DONE
// only run after every tile for this job has already been fetched,
// so prefetch_engine is guaranteed idle) selects which one actually
// reaches the real output port, same pattern as the pre-DEC-0015
// design, just weight-only now (the activation side moved to the
// shared activation_cache.v, DEC-0016).
wire pf_mem_req, pf_mem_wr;
wire [ADDR_WIDTH-1:0] pf_mem_addr;
wire [15:0] pf_mem_wdata;
@@ -150,9 +195,9 @@ module memory_manager #(
.DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .ADDR_WIDTH(ADDR_WIDTH)
) u_prefetch (
.clk(clk), .rst(rst),
.fetch_start(pf_start), .x_addr(pf_x_addr), .w_addr(pf_w_addr),
.fetch_start(pf_start), .w_addr(pf_w_addr),
.fetch_busy(pf_busy), .fetch_done(pf_done),
.tile_x(pf_tile_x), .tile_w(pf_tile_w),
.tile_w(pf_tile_w),
.mem_req(pf_mem_req), .mem_wr(pf_mem_wr), .mem_addr(pf_mem_addr), .mem_wdata(pf_mem_wdata),
.mem_lb_n(pf_mem_lb_n), .mem_ub_n(pf_mem_ub_n),
.mem_rdata(mem_rdata), .mem_ready(mem_ready)
@@ -163,12 +208,6 @@ module memory_manager #(
reg [15:0] wr_mem_wdata;
reg wr_mem_lb_n, wr_mem_ub_n;
// wr_mem_req is SET while state==MM_WRITE_RESULT but only becomes
// valid (via NBA) the FOLLOWING cycle, i.e. while state==MM_DONE --
// the mux must select the write-back source across BOTH states,
// not just the one that issues it (an off-by-one here silently
// dropped the write request entirely -- found and fixed here, see
// hardware/v2/logs/errors.log ERR-0006).
wire wr_active = (state == MM_WRITE_RESULT) || (state == MM_DONE);
assign mem_req = wr_active ? wr_mem_req : pf_mem_req;
assign mem_wr = wr_active ? 1'b1 : pf_mem_wr;
@@ -188,45 +227,63 @@ module memory_manager #(
result_ready <= 1'b0;
pf_start <= 1'b0;
current_bank <= 1'b0;
bank_ready <= 2'b00;
bank_x_ready <= 2'b00;
bank_w_ready <= 2'b00;
tile_idx <= 16'h0;
pf_pending <= 1'b0;
xc_req <= 1'b0;
xc_pending <= 1'b0;
xc_outstanding <= 1'b0;
wr_mem_req <= 1'b0;
wr_mem_addr <= {ADDR_WIDTH{1'b0}};
wr_mem_wdata <= 16'h0000;
wr_mem_lb_n <= 1'b1;
wr_mem_ub_n <= 1'b1;
pf_pending <= 1'b0;
end else begin
job_done <= 1'b0;
pf_start <= 1'b0;
xc_req <= 1'b0;
result_ready <= 1'b0;
// Latch a completed prefetch into its target bank.
// Latch a completed weight prefetch into its target bank.
if (pf_done) begin
bank_x[pf_target_bank] <= pf_tile_x;
bank_w[pf_target_bank] <= pf_tile_w;
bank_ready[pf_target_bank] <= 1'b1;
bank_w_ready[pf_target_bank] <= 1'b1;
end
// Issue a pending fetch request as soon as the (single)
// Latch a completed activation-cache fetch into its target
// bank and clear the outstanding flag (see its own
// declaration comment above).
if (xc_ack) begin
bank_x[xc_target_bank] <= xc_tile_x;
bank_x_ready[xc_target_bank] <= 1'b1;
xc_outstanding <= 1'b0;
end
// Issue a pending weight fetch as soon as the (single)
// prefetch engine is genuinely free. The `!pf_start` guard
// is required, not cosmetic: pf_busy does not read 1 until
// the cycle AFTER pf_start was first observed (prefetch_
// engine's own fetch_busy<=1 is one clock behind its own
// fetch_start sampling), so checking !pf_busy alone leaves
// a genuine one-cycle window where a second pending
// request would fire on top of the one just launched,
// silently corrupting pf_target_bank for the fetch already
// in flight (found and fixed here -- see
// hardware/v2/logs/errors.log ERR-0006).
// is required, not cosmetic -- see hardware/v2/logs/
// errors.log ERR-0006.
if (pf_pending && !pf_busy && !pf_start) begin
pf_start <= 1'b1;
pf_x_addr <= pf_pending_x;
pf_w_addr <= pf_pending_w;
pf_target_bank <= pf_pending_bank;
pf_pending <= 1'b0;
end
// Issue a pending activation-cache request only once this
// instance's own PREVIOUS request has been genuinely acked
// (xc_outstanding low) -- see that flag's own declaration
// comment for why checking xc_req alone is not enough.
if (xc_pending && !xc_outstanding) begin
xc_req <= 1'b1;
xc_outstanding <= 1'b1;
xc_x_base <= xc_pending_x_base;
xc_tile_idx <= xc_pending_tile_idx;
xc_target_bank <= xc_pending_bank;
xc_pending <= 1'b0;
end
case (state)
MM_IDLE: begin
@@ -237,30 +294,37 @@ module memory_manager #(
result_addr_reg <= result_addr;
tile_idx <= 16'h0;
current_bank <= 1'b0;
bank_ready <= 2'b00;
bank_x_ready <= 2'b00;
bank_w_ready <= 2'b00;
operand_valid <= 1'b0;
// kick off the very first fetch (tile 0 into bank 0)
pf_pending <= 1'b1;
pf_pending_x <= x_base;
pf_pending_w <= w_base;
pf_pending_bank <= 1'b0;
xc_pending <= 1'b1;
xc_pending_x_base <= x_base;
xc_pending_tile_idx <= 16'h0;
xc_pending_bank <= 1'b0;
state <= MM_PREFETCH_FIRST;
end
end
MM_PREFETCH_FIRST: begin
if (bank_ready[0] || (pf_done && pf_target_bank == 1'b0)) begin
if (bank_ready[0]) begin
// Present tile 0; concurrently start prefetching
// tile 1 into bank 1, if there is one.
operand_valid <= 1'b1;
input_data <= pf_done ? pf_tile_x : bank_x[0];
weight_data <= pf_done ? pf_tile_w : bank_w[0];
input_data <= bank_x[0];
weight_data <= bank_w[0];
tile_last <= (n_tiles_reg == 16'h1);
if (n_tiles_reg > 16'h1) begin
pf_pending <= 1'b1;
pf_pending_x <= x_base_reg + P_IN[ADDR_WIDTH-1:0];
pf_pending_w <= w_base_reg + P_IN[ADDR_WIDTH-1:0];
pf_pending_bank <= 1'b1;
xc_pending <= 1'b1;
xc_pending_x_base <= x_base_reg;
xc_pending_tile_idx <= 16'h1;
xc_pending_bank <= 1'b1;
end
state <= MM_STREAM;
end
@@ -269,7 +333,8 @@ module memory_manager #(
MM_STREAM: begin
if (operand_valid && operand_ready) begin
// This tile consumed; free its bank, swap.
bank_ready[current_bank] <= 1'b0;
bank_x_ready[current_bank] <= 1'b0;
bank_w_ready[current_bank] <= 1'b0;
current_bank <= ~current_bank;
tile_idx <= tile_idx + 16'h1;
operand_valid <= 1'b0; // re-asserted below once the new bank is ready
@@ -279,17 +344,14 @@ module memory_manager #(
state <= MM_WAIT_RESULT;
end else if (tile_idx + 16'h2 < n_tiles_reg) begin
// Queue a prefetch for the tile AFTER next into
// the bank we just freed (current_bank, pre-swap)
// -- it will actually launch once the (single)
// prefetch engine is free (see the pf_pending
// issue rule above); it is very likely still
// busy with the tile-N+1 fetch kicked off on the
// PREVIOUS handoff, so this almost always queues
// rather than launching immediately.
// the bank we just freed (current_bank, pre-swap).
pf_pending <= 1'b1;
pf_pending_x <= x_base_reg + (tile_idx + 16'h2) * P_IN[ADDR_WIDTH-1:0];
pf_pending_w <= w_base_reg + (tile_idx + 16'h2) * P_IN[ADDR_WIDTH-1:0];
pf_pending_bank <= current_bank; // the one just freed
xc_pending <= 1'b1;
xc_pending_x_base <= x_base_reg;
xc_pending_tile_idx <= tile_idx + 16'h2;
xc_pending_bank <= current_bank;
end
end else if (!operand_valid) begin
// Waiting for the new current bank to become ready
@@ -322,8 +384,9 @@ module memory_manager #(
MM_WRITE_RESULT: begin
// prefetch_engine is guaranteed idle here (no more
// tiles to fetch for this job), so driving the shared
// backend port directly is safe -- see file header.
// tiles to fetch for this job), so driving the
// shared weight backend port directly is safe --
// see file header/wr_active above.
wr_mem_req <= 1'b1;
wr_mem_addr <= result_addr_reg[ADDR_WIDTH-1:1]; // byte -> word
state <= MM_DONE;
hardware/v2/rtl/neural_multiprocessor.v
+12 -8
View File
@@ -72,12 +72,14 @@ module neural_multiprocessor #(
);
// ---- dataflow_core (M7, control logic unmodified; per-slot
// backend port widened to 16-bit + lb_n/ub_n per DEC-0015) ----
wire [N_SLOTS-1:0] slot_mem_req, slot_mem_wr;
wire [ADDR_WIDTH*N_SLOTS-1:0] slot_mem_addr;
wire [16*N_SLOTS-1:0] slot_mem_wdata, slot_mem_rdata;
wire [N_SLOTS-1:0] slot_mem_lb_n, slot_mem_ub_n;
wire [N_SLOTS-1:0] slot_mem_ready;
// backend port widened to 16-bit + lb_n/ub_n per DEC-0015, and to
// N_SLOTS+1 ports per DEC-0016 -- the extra port is the shared
// activation_cache's own backend traffic) ----
wire [N_SLOTS:0] slot_mem_req, slot_mem_wr;
wire [ADDR_WIDTH*(N_SLOTS+1)-1:0] slot_mem_addr;
wire [16*(N_SLOTS+1)-1:0] slot_mem_wdata, slot_mem_rdata;
wire [N_SLOTS:0] slot_mem_lb_n, slot_mem_ub_n;
wire [N_SLOTS:0] slot_mem_ready;
dataflow_core #(
.DATA_WIDTH(DATA_WIDTH), .P_IN(P_IN), .ACC_WIDTH(ACC_WIDTH), .ADDR_WIDTH(ADDR_WIDTH),
@@ -93,7 +95,9 @@ module neural_multiprocessor #(
.slot_mem_rdata(slot_mem_rdata), .slot_mem_ready(slot_mem_ready)
);
// ---- N_SLOTS -> 1 arbiter (M8, word-level per DEC-0015) ----
// ---- (N_SLOTS+1) -> 1 arbiter (M8, word-level per DEC-0015;
// widened to N_SLOTS+1 ports per DEC-0016 to also arbitrate the
// shared activation_cache's own backend traffic) ----
wire arb_m_req, arb_m_wr;
wire [ADDR_WIDTH-1:0] arb_m_addr;
wire [15:0] arb_m_wdata;
@@ -102,7 +106,7 @@ module neural_multiprocessor #(
wire arb_m_ready;
slot_mem_arbiter #(
.ADDR_WIDTH(ADDR_WIDTH), .N_PORTS(N_SLOTS)
.ADDR_WIDTH(ADDR_WIDTH), .N_PORTS(N_SLOTS+1)
) u_arbiter (
.clk(clk), .rst(rst),
.s_req(slot_mem_req), .s_wr(slot_mem_wr), .s_addr(slot_mem_addr),
hardware/v2/rtl/prefetch_engine.v
+36 -61
View File
@@ -1,43 +1,44 @@
`timescale 1ns/1ps
// ================================================================
// FPGA-Neural V2 -- Prefetch Engine (M4, docs/v2-description.md §13;
// word-level burst rewrite post-M10 -- see hardware/v2/logs/
// decisions.log DEC-0015)
// FPGA-Neural V2 -- Weight Prefetch Engine (M4, docs/v2-description.md
// §13; word-level burst rewrite post-M10 DEC-0015; X-fetch moved out
// to a shared activation_cache.v post-M10 DEC-0016)
//
// Fetches ONE tile (P_IN activation bytes + P_IN weight bytes) from
// the WORD-level Memory Backend Interface, P_IN/2 sixteen-bit
// transactions per array instead of P_IN single-byte ones.
// Fetches ONE tile's P_IN WEIGHT bytes from the WORD-level Memory
// Backend Interface, P_IN/2 sixteen-bit transactions instead of P_IN
// single-byte ones (DEC-0015 -- see this rationale in full below).
//
// WHY: hardware/v1/rtl/int8_memory_access.v (the byte-level backend
// this engine originally sat on) converts every 8-bit logical request
// into a FULL 16-bit PSRAM word access internally (mem_addr <= addr
// >> 1, one byte lane selected via lb_n/ub_n) -- so a byte-at-a-time
// fetch was ALREADY paying for two bytes of real PSRAM bandwidth per
// transaction while only using one. This engine now talks directly to
// hardware/v1/rtl/memory_interface.v's own 16-bit word interface
// (skipping int8_memory_access.v entirely -- both are frozen V1 files,
// unmodified either way, §1/§34; V2 is simply choosing to reuse the
// lower layer instead of the byte-splitting one on top of it, the
// same "reuse what fits" precedent already set by slot_mem_arbiter.v
// not reusing hardware/v1/rtl/mem_arbiter.v verbatim). psram_controller.v's
// own real page-mode support (already implemented, unmodified) then
// serves consecutive same-page word reads faster than a cold access --
// this engine's job is simply to stop discarding half of every word it
// already paid for, and to halve the number of real backend
// round-trips needed per tile.
// Historical note: this module used to ALSO fetch the P_IN
// ACTIVATION (X) bytes for the same tile. DEC-0016 moved that
// responsibility to a new shared activation_cache.v instead: in the
// realistic dense-layer workloads this project actually benchmarks
// (hardware/v2/docs/benchmarks/final-benchmark.md), many neurons
// share the exact same X vector, and each of memory_manager.v's own
// N_SLOTS instances re-fetching that identical vector from PSRAM
// independently was real, measured, redundant traffic on the one
// shared PSRAM port -- exactly the kind of real recommendation the
// benchmark campaign was built to surface. Weights (W) are NOT shared
// across neurons (each neuron has its own trained weight vector), so
// there is no equivalent caching opportunity on the W side -- this
// engine keeps fetching W directly from PSRAM, unchanged in spirit
// from DEC-0015, just no longer also fetching X.
//
// CONSTRAINT: P_IN must be even, and x_addr/w_addr must be word-
// aligned (even BYTE addresses) -- each 16-bit transaction covers
// BYTE addresses {addr, addr+1} as {low byte, high byte} (matches
// int8_memory_access.v's own addr[0] convention exactly, replicated
// here since that module is no longer in the datapath). A host/loader
// placing X/W tile arrays at even byte offsets (already true of every
// address used in this project's own testbenches) satisfies this
// with no special handling.
// WHY word-level (DEC-0015, unchanged rationale): int8_memory_access.v
// (the byte-level backend this engine originally sat on) converts
// every 8-bit logical request into a FULL 16-bit PSRAM word access
// internally (mem_addr <= addr >> 1, one byte lane selected via
// lb_n/ub_n) -- so a byte-at-a-time fetch was ALREADY paying for two
// bytes of real PSRAM bandwidth per transaction while only using one.
// This engine talks directly to hardware/v1/rtl/memory_interface.v's
// own 16-bit word interface (skipping int8_memory_access.v entirely --
// both are frozen V1 files, unmodified either way, §1/§34).
//
// The double-buffering strategy itself (§13) remains memory_manager.v's
// responsibility -- unchanged by this rewrite.
// CONSTRAINT: P_IN must be even, and w_addr must be word-aligned (even
// BYTE address) -- each 16-bit transaction covers BYTE addresses
// {addr, addr+1} as {low byte, high byte} (matches int8_memory_access.v's
// own addr[0] convention exactly, replicated here since that module is
// no longer in the datapath).
// ================================================================
module prefetch_engine #(
@@ -49,11 +50,9 @@ module prefetch_engine #(
input wire rst,
input wire fetch_start,
input wire [ADDR_WIDTH-1:0] x_addr, // BYTE address, word-aligned
input wire [ADDR_WIDTH-1:0] w_addr, // BYTE address, word-aligned
output reg fetch_busy,
output reg fetch_done, // one-cycle pulse
output reg signed [DATA_WIDTH*P_IN-1:0] tile_x,
output reg signed [DATA_WIDTH*P_IN-1:0] tile_w,
// ---- word-level Memory Backend Interface (matches
@@ -69,7 +68,6 @@ module prefetch_engine #(
);
localparam ST_IDLE = 2'd0;
localparam ST_READ_X = 2'd1;
localparam ST_READ_W = 2'd2;
localparam ST_DONE = 2'd3;
@@ -79,7 +77,6 @@ module prefetch_engine #(
reg [1:0] state;
reg [WIW-1:0] word_idx;
wire [ADDR_WIDTH-1:0] x_word_base = x_addr[ADDR_WIDTH-1:1];
wire [ADDR_WIDTH-1:0] w_word_base = w_addr[ADDR_WIDTH-1:1];
always @(posedge clk) begin
@@ -106,32 +103,10 @@ module prefetch_engine #(
word_idx <= 0;
mem_req <= 1'b1;
mem_wr <= 1'b0;
mem_addr <= x_word_base;
mem_addr <= w_word_base;
mem_lb_n <= 1'b0; // both byte lanes -- fetch the whole word
mem_ub_n <= 1'b0;
state <= ST_READ_X;
end
end
ST_READ_X: begin
if (mem_ready) begin
tile_x[word_idx*16 +: 16] <= mem_rdata;
if (word_idx == WORDS_PER_TILE[WIW-1:0] - 1'b1) begin
word_idx <= 0;
mem_req <= 1'b1;
mem_wr <= 1'b0;
mem_addr <= w_word_base;
mem_lb_n <= 1'b0;
mem_ub_n <= 1'b0;
state <= ST_READ_W;
end else begin
word_idx <= word_idx + 1'b1;
mem_req <= 1'b1;
mem_wr <= 1'b0;
mem_addr <= x_word_base + word_idx + 1'b1;
mem_lb_n <= 1'b0;
mem_ub_n <= 1'b0;
end
state <= ST_READ_W;
end
end