FPGA-Neural/docs/PHYSICAL_REALIZATION.md

# FPGA-Neural V3 — Physical Realization Reference

Real, verified data for the custom PCB (bare XC7A100T-CSG324-2 + DDR3, no dev
board). Every pin/part/setting below comes from a real Vivado-generated
constraint file, a real datasheet, or a real place-and-route run — none of it
is guessed. See `hardware/v2/logs/experiments.log` (EXP-0059 onward) for the
full derivation history.

## 1. Core components (real, verified availability)

| Component | Part | Notes |
|---|---|---|
| FPGA | **XC7A100T-CSG324-2** | Speed grade **-2** (corrected from an initial -1 assumption, EXP-0074) — same die/package/footprint as -1, strictly better timing margin. |
| DDR3 SDRAM | **Micron MT41J128M16JT-125:K** | 2Gb, x16, DDR3-1600-rated (run at 310.078MHz here due to -2 timing closure, see §3). Verified in-stock on LCSC. |
| Config flash | **Winbond W25Q32JVSSIQ** | 32Mbit/4MB, SOIC-8. Comfortably fits the ~30.5Mbit full XC7A100T bitstream. Verified in-stock on LCSC. Wired **exclusively** to the FPGA (see §5). |

## 2. FPGA pin assignments (real, from the routed design)

### 2.1 DDR3 (fixed by the FPGA's own PHY hardware — not a free choice)

**EXP-0084 UPDATE: real 32-bit channel (two MT41J128M16JT-125:K chips
ganged in parallel)** — dq/dqs/dm pin counts DOUBLED versus the original
16-bit design (dq 16→32, dqs 2→4 pairs, dm 2→4). Address/command/control
lines are unchanged and fanned out identically to both chips. Real,
board-accurate pin placement below (confirmed from the actual routed
design, `mig_7series_0.xdc` + this project's own top-level XDC) —
**functionally verified**, but see §3 for the real, honest timing-closure
status before treating this as a final signoff.

Generated by the Vivado MIG wizard (`mig_7series_0.xdc`), all `SSTL15` /
`DIFF_SSTL15` (1.5V), banks 34/35:

| Signal | Pin | Signal | Pin | Signal | Pin |
|---|---|---|---|---|---|
| ddr3_dq[0] | C5 | ddr3_dq[16] | F4 | ddr3_addr[0] | U3 |
| ddr3_dq[1] | B7 | ddr3_dq[17] | F3 | ddr3_addr[1] | U4 |
| ddr3_dq[2] | B6 | ddr3_dq[18] | E2 | ddr3_addr[2] | V1 |
| ddr3_dq[3] | D8 | ddr3_dq[19] | D2 | ddr3_addr[3] | U1 |
| ddr3_dq[4] | C7 | ddr3_dq[20] | C1 | ddr3_addr[4] | L5 |
| ddr3_dq[5] | E6 | ddr3_dq[21] | H1 | ddr3_addr[5] | L6 |
| ddr3_dq[6] | E5 | ddr3_dq[22] | G1 | ddr3_addr[6] | L4 |
| ddr3_dq[7] | E7 | ddr3_dq[23] | F1 | ddr3_addr[7] | K5 |
| ddr3_dq[8] | B4 | ddr3_dq[24] | F6 | ddr3_addr[8] | M2 |
| ddr3_dq[9] | A4 | ddr3_dq[25] | G4 | ddr3_addr[9] | M3 |
| ddr3_dq[10] | A3 | ddr3_dq[26] | G3 | ddr3_addr[10] | L3 |
| ddr3_dq[11] | B3 | ddr3_dq[27] | J3 | ddr3_addr[11] | K3 |
| ddr3_dq[12] | B2 | ddr3_dq[28] | J2 | ddr3_addr[12] | M1 |
| ddr3_dq[13] | D5 | ddr3_dq[29] | K2 | ddr3_addr[13] | L1 |
| ddr3_dq[14] | D4 | ddr3_dq[30] | K1 | ddr3_ba[0] | V5 |
| ddr3_dq[15] | E3 | ddr3_dq[31] | H6 | ddr3_ba[1] | V2 |
| ddr3_dm[0] | C6 | ddr3_dm[2] | C2 | ddr3_ba[2] | U2 |
| ddr3_dm[1] | C4 | ddr3_dm[3] | G6 | | |
| ddr3_dqs_p[0] | A6 (DIFF) | ddr3_dqs_n[0] | A5 (DIFF) | ddr3_cas_n | R3 |
| ddr3_dqs_p[1] | B1 (DIFF) | ddr3_dqs_n[1] | A1 (DIFF) | ddr3_cke[0] | P2 |
| ddr3_dqs_p[2] | H2 (DIFF) | ddr3_dqs_n[2] | G2 (DIFF) | ddr3_cs_n[0] | T5 |
| ddr3_dqs_p[3] | J4 (DIFF) | ddr3_dqs_n[3] | H4 (DIFF) | ddr3_odt[0] | R2 |
| ddr3_ck_p[0] | N2 (DIFF) | ddr3_ck_n[0] | N1 (DIFF) | ddr3_ras_n | V4 |
| ddr3_reset_n | D7 (LVCMOS15) | ddr3_we_n | T3 | | |
| **sys_clk_p** | **N5 (DIFF_SSTL15, bank 34)** | **sys_clk_n** | **P5 (DIFF_SSTL15, bank 34)** | | |
| **clk_ref_p** | **T14 (LVDS_25, bank 14)** | **clk_ref_n** | **T15 (LVDS_25, bank 14)** | | |

**Real board implication (EXP-0084)**: both `sys_clk` and `clk_ref` are now
genuine **differential pairs** — the board needs a differential oscillator
(or a differential buffer stage), not a single-ended one. `clk_ref` lives in
**bank 14** (not 34/35) because the MIG wizard's own UG586 placement rules
restrict that specific net to bank 14 for this part/package — not a free
choice. This directly displaced the config-flash SPI bus, which used to
share bank 14 (see §2.3 — moved to bank 16 in EXP-0084 to resolve a real
VCCO conflict, LVCMOS33 vs LVDS_25 cannot coexist in one bank).

**Bank voltage requirements**: bank 34/35 → **1.5V** (DDR3 SSTL15, plus
`sys_clk_p/n` at bank 34's own 1.5V-domain `DIFF_SSTL15`), bank 14 →
**2.5V** (`clk_ref_p/n`, LVDS_25).

`INTERNAL_VREF` for banks 34/35 is set to 0.750V by the MIG constraints
(required for SSTL15 single-ended inputs) — this is a Vivado-side setting,
not a board component, but note it if you ever inspect bitstream generation
warnings about VREF.

### 2.2 Neural-processor management SPI (ESP32 ↔ FPGA, FPGA is **slave**)

Bank 15, package edge column A/B, physically adjacent (short traces), `LVCMOS33`:

| Signal | Pin | Direction (FPGA side) |
|---|---|---|
| sclk | A15 | input |
| mosi | B16 | input |
| miso | B17 | output |
| cs_n | A16 | input |
| sys_rst | G13 | input (EXP-0084, tentative — see §7) |
| data_ready_n | D14 | output (EXP-0085, active-low sticky IRQ — see §6, tentative — see §7) |

**Bank 15 VCCO**: assumed **3.3V** — change the XDC's IOSTANDARD if your
board power plan uses a different rail for this bank.

### 2.3 Config flash SPI (FPGA ↔ flash, FPGA is **master**)

**EXP-0084 REAL PIN CHANGE**: originally these were the FPGA's own dedicated
Master-SPI configuration pins (K17/K18/L13, bank 14), reclaimed as ordinary
fabric I/O post-configuration. EXP-0084's real 32-bit DDR3 widening moved
the differential reference clock (`clk_ref_p/n`, §2.1) into bank 14 too —
a real VCCO conflict (the flash needs LVCMOS33/3.3V, `clk_ref` needs
LVDS_25/2.5V, one bank can only have one VCCO), confirmed by a real
`place_design` failure, not hypothetical. **Fixed by moving the flash bus
to bank 16** (completely unconstrained before this, no VCCO commitment):

| Signal | Pin | Direction (FPGA side) | Notes |
|---|---|---|---|
| flash_mosi | D9 | output | bank 16 (was K17/D00_MOSI, bank 14) |
| flash_miso | D10 | input | bank 16 (was K18/D01_DIN, bank 14) |
| flash_cs_n | C9 | output | bank 16 (was L13/FCS_B, bank 14) |
| (CCLK) | E9 | output | **Not a top-level port** — driven internally via the `STARTUPE2` primitive. Wire the flash's own CLK pin to package pin **E9**. Unaffected by the bank-14→16 move (CCLK was never a plain port). |

Still requires `BITSTREAM.CONFIG.PERSIST = NO` (the Vivado default — set
explicitly in the project XDC; **EXP-0084 note**: the XDC previously had
this as `FALSE`, an invalid enum value in this Vivado version — silently
not applied at all until fixed to the real `NO`/`YES` enum).

**Bank 16 VCCO**: **3.3V** (matches the flash's own VCC, typically
1.8–3.6V for the W25Q32JV) — chosen freely since bank 16 had no prior VCCO
commitment.

**Real board implication**: since the flash bridge no longer uses the
FPGA's own dedicated D00_MOSI/D01_DIN/FCS_B configuration pins at all
(K17/K18/L13 are now unused by this design), the flash chip's SPI lines on
the PCB connect to D9/D10/C9 instead — a real routing change versus any
earlier board layout draft based on the pre-EXP-0084 pinout.

**Reserved, do not use** (bank 14, unused by this design, kept clear for
any future Quad-SPI/BPI expansion — no longer near the flash bus, now just
general config-mode-reserved pins): `L16` (EMCCLK), `R16` (RDWR_B), `V15`
(CSI_B). The project's own XDC `PROHIBIT`s the underlying **sites** for
these (fixed in EXP-0084 — `PROHIBIT` is not a valid property directly on
`package_pin` objects in this Vivado version, only on their `site`; this
constraint had silently never been enforced before).

### 2.4 FPGA configuration control (dedicated, bank 0, not negotiable)

| Signal | Pin | Purpose |
|---|---|---|
| PROGRAM_B | P9 | pulse low to force a full reconfiguration from flash |
| INIT_B | P7 | goes low during config; can indicate a config error if it re-asserts |
| DONE | P10 | goes high once configuration succeeds — wire to a status LED if desired |
| M0 | P12 | mode select |
| M1 | P13 | mode select |
| M2 | P11 | mode select |
| CFGBVS | P8 | tie to match bank 0's VCCO logic level (see UG470) |

**Mode pin setting for Master SPI boot** (the flash-based autonomous boot
path, see §5): `M[2:0] = 001` (per UG470's mode pin table) — tie via pull-up/
pull-down resistors on the board, not driven dynamically.

### 2.5 JTAG (always available, independent of flash content)

| Signal | Pin |
|---|---|
| TCK | E10 |
| TDI | E11 |
| TMS | E12 |
| TDO | E13 |

Used for: (a) first-ever/factory programming when the flash is blank (see
§5), (b) recovery, (c) development/debug. This project's own plan drives
these from an ESP32 doing real JTAG bit-banging (TAP state machine, IR/DR
shifting) rather than a bench programmer — that firmware is separate,
software-side work, not covered here.

## 3. Real timing signoff (EXP-0083, the current, trustworthy number)

Real in-context Vivado place-and-route (not out-of-context, not estimated):

| Metric | Value |
|---|---|
| DDR3 PHY clock (sys_clk_i) | **310.078 MHz** (3.225ns period) |
| Compute domain clock (ui_clk, PLL-derived 2:1 from sys_clk_i) | **155.039 MHz** |
| WNS (setup slack) | **+0.073 ns** — real, still thin but improved vs every prior real run. Re-verify with a fresh P&R after ANY further logic addition. |
| WHS (hold slack) | +0.036 ns |
| Failing endpoints | 0 (setup), 0 (hold) |
| LUTs used | 5644 / 63400 (8.90%) |
| DSP48E1 used | 16 / 240 (6.67%) — 8 per compute core × 2 cores, unchanged since EXP-0059 |
| Block RAM used | 0 |
| STARTUPE2 used | 1 / 1 (100%) — the config-flash bridge |

Signoff history (every real change, same target, in-context P&R):

| EXP | WNS (ns) | LUTs | DSP48E1 |
|---|---|---|---|
| 0074 (first real DDR3 + pins) | +0.040 | 5140 | 16 |
| 0076 (+regfile, +pins, +SPI fix) | +0.056 | 5173 | 16 |
| 0078 (+flash bridge, STARTUPE2) | +0.013 | 5213 | 16 |
| 0079 (+real activation engine) | +0.030 | 5379 | 16 |
| 0082 (+denser activation packing) | +0.068 | 5437 | 16 |
| 0083 (+DDRManager phase 1, `ddr_prefetch_mgr.v`) | **+0.073** | 5644 | 16 |
| 0084 (32-bit DDR3 widening, 2900ps/172.414MHz) | **-0.618 (FAILS)** | 6418 | 16 |

### 3.1 EXP-0084: real 32-bit DDR3 widening — functionally verified, timing NOT yet closed

**Honest status, not glossed over**: the real 32-bit channel (two
MT41J128M16JT-125:K chips) is **functionally complete and real-verified**
(real xsim against the real 2-chip DDR3 model: `tb_mig_native_adapter.v`
12/12 PASS, `tb_n2_system_ddr3.v` 8/8 PASS) — but its own real P&R does
**not** close timing at the paired clock speedup (2900ps sys_clk /
172.414MHz ui_clk): **WNS = -0.618ns, 213 failing endpoints**. Route
completed 100% (a real, physically valid placement/routing), only timing
itself fails.

**Real root cause** (traced to the actual worst path): the violation is
inside `neural_processor_packed.v`'s own packed-MAC accumulation tree (a
DSP48E1 output through a 4-deep CARRY4 chain), **unchanged since EXP-0059**
— it had real positive margin at the old 155.039MHz `ui_clk` (EXP-0083:
+0.073ns) but not at the new 172.414MHz (an 11.2% real frequency increase).
This is **not caused by the 32-bit width change itself** — every module
touched for the widening (adapter, arbiter, activation fetch, etc.) is
real-verified functionally correct; the failure is in unrelated, pre-
existing compute logic that simply wasn't budgeted for a faster clock.

**The real, honest decoupling this reveals**: bandwidth = width × clock
rate. Widening to 32 bits **alone**, even at the OLD, already-closing
3225ps/155.039MHz, already delivers the FULL intended 2× bandwidth gain
(1.24 → ~2.48 GB/s physical ceiling) — the clock speedup to 2900ps was a
separate, additional optimization stacked on top in the same wizard
session, and it is specifically *that* stacking, not the width, that
breaks timing.

**Current real, trustworthy signoff therefore remains EXP-0083** (16-bit,
155.039MHz, +0.073ns) until a real P&R closes for the 32-bit configuration
— expected once the Input Clock Period is reverted toward 3225ps (keeping
Data Width=32), a real, user-gated MIG wizard step, not yet done. See
`hardware/v2/logs/experiments.log` EXP-0084 for the complete story,
including 5 real P&R/XDC bugs found and fixed along the way (stale
differential-clock ports, a real VCCO bank conflict, a stale imported XDC,
missing IOSTANDARDs, and two cosmetic but previously-silently-broken XDC
property bugs).

## 4. Real DDR3 memory layout convention

Both weight data and activation data share the same DDR3 address space,
word-addressed. **EXP-0084**: the native word width is now **32 bits**
(`BURST_LEN=8` per transaction = 256 bits/burst, up from 128 bits at the
old 16-bit width) — this is the real, current RTL (`hardware/v3/rtl/`),
functionally verified; see §3.1 for its pending timing-closure status.

- **Weights**: one layer's weight set starts at word address `layer_index *
  WORDS_PER_LAYER` (`WORDS_PER_LAYER = LAYER_BYTES/2`). Densely packed —
  `layer_prefetch_ctrl.v` reads full bursts sequentially into the on-chip
  weight buffer once per job. `BYTES_PER_BURST` is now `4*BURST_LEN` (32
  bytes/burst, up from 16) — the byte-drain logic itself is word-width-
  agnostic and needed no change beyond this.
- **Activations** (real engine, `act_tile_fetch.v`; current layout is the
  **v3 convention, EXP-0084**): **FOUR consecutive tiles (P_IN=8 INT8
  values each, 64 bits each) share ONE full `BURST_LEN=8`-word (256-bit)
  burst** — tile parity 0/1/2/3 (`tcnt[1:0]`) selects bits
  `[63:0]`/`[127:64]`/`[191:128]`/`[255:192]` of the burst response. Tile
  `t`'s burst address is `base + (t>>2)*BURST_LEN` (integer division —
  four tiles per burst), always burst-aligned by construction. This is
  **not** a further bytes-per-MAC reduction beyond EXP-0081's already-
  optimal 1 byte/MAC — it's what's required to keep that same 100% packing
  utilization at the new, larger burst size instead of leaving half of it
  newly wasted.
  - **Why this is timing-safe despite selecting a sub-burst quarter at
    read time**: the tile index's own low 2 bits (which quarter of the
    burst a given tile lives in) are known at *request* time, not at
    response time. They're latched into a register (`sel_lat`) the same
    cycle the request is accepted — many `ui_clk` cycles before the real
    DDR3 round-trip completes and `ctrl_rdata` becomes valid. The eventual
    data-select mux is an explicit 4-way `case` on constant byte offsets
    (not a runtime-indexed part-select expression) — extending the exact
    same discipline EXP-0081 established for the 1-bit case to 2 bits.
    This mux is real and functionally verified, but is unrelated to
    §3.1's own real timing failure (that violation is inside the compute
    core's accumulation tree, not this fetch path).
  - Real measured effect: back-to-back same-row DDR3 throughput was a
    fixed 1.24 GB/s at 16-bit width (measured, EXP-0080); at 32-bit width
    the real physical ceiling doubles to ~2.48 GB/s, independent of this
    packing convention — packing controls how much of that ceiling is
    *wasted*, not the ceiling itself (see `docs/ARCHITECTURE_ANALYSIS.md`
    §3 and §5.1/§5.4).
  - `base` (a job's own `x_base_a`/`x_base_b`) is chosen freely by whoever
    submits jobs (the SPI host) — just keep each position's own activation
    array in its own non-overlapping `(N_TILES/4) * BURST_LEN`-word region.

## 5. FPGA configuration (boot) procedure

Two complementary paths, both present on this board by design:

1. **Factory-first / recovery (JTAG, ESP32-driven)**: the flash starts
   blank on a fresh board — no other path can bootstrap it (a real chicken-
   and-egg constraint: the FPGA can't relay flash-programming commands over
   SPI, §5.2, until it's already running logic that does that). The ESP32
   bit-bangs JTAG (§2.5) to load a bitstream directly, or to run Vivado's own
   "indirect SPI flash programming" sequence to write the flash for the
   first time. One-time (or rare/recovery-only) step.
2. **Normal boot (Master SPI, autonomous)**: every subsequent power-on, the
   FPGA self-configures from the flash via its own dedicated hardware (mode
   pins set to Master SPI, §2.4) — no ESP32 involvement needed.
3. **Field firmware updates (SPI-through-FPGA, `FLASH_XFER` opcode 0x40)**:
   once the FPGA is running, the ESP32 can rewrite the flash by relaying raw
   SPI-NOR bytes through the FPGA over the management SPI bus (§2.2) — the
   FPGA then re-transmits them as master on the flash bus (§2.3). This is the
   **only** electrical path from ESP32 to the flash; there is no direct
   connection (by design, per explicit requirement).
   - **Real SPI-NOR opcodes** (verified against the actual W25Q32JV
     datasheet, for whoever writes the ESP32-side flashing routine):
     `0x06` Write Enable, `0x04` Write Disable, `0x05` Read Status Register-1
     (bit0=BUSY, bit1=WEL), `0x02` Page Program, `0x03` Read Data, `0x20`
     Sector Erase (4KB), `0x52` 32KB Block Erase, `0xD8` 64KB Block Erase,
     `0xC7`/`0x60` Chip Erase.
   - **Protocol timing note**: `FLASH_XFER` relays are NOT instantaneous —
     each relayed byte's real flash response is only stable starting **two**
     host-clocked bytes later (not one), so the host must clock 2 trailing
     dummy bytes after its last real command byte to safely receive the
     final response. See `spi_host_bridge_v3.v`'s own header for the full
     real-measured reasoning (EXP-0077).
   - After writing a new bitstream to the flash, reconfigure either by
     pulsing `PROGRAM_B` externally, or (future work, not built yet) via a
     `ICAPE2`-based warm self-reconfiguration triggered over the same SPI bus.

## 6. Management SPI protocol summary (for ESP32 firmware)

One opcode byte (MSB-first) per CS-low transaction, driven by
`spi_host_bridge_v3.v`:

| Opcode | Name | Payload | Purpose |
|---|---|---|---|
| 0x00 | NOP | 0 bytes | inert |
| 0x0F | RESET | 0 bytes | pulses a soft-reset |
| 0x10 | WRITE_JOB | 16 bytes | submit one inference job (node_id, x_base, w_base, n_tiles, result_addr) |
| 0x20 | STATUS | 0 bytes → 1 byte out | job_busy / mem_busy / last_job_accepted bits |
| 0x01 | WRITE_MEM | 4+2N bytes | raw DDR3 word write (N words) |
| 0x02 | READ_MEM | 6 bytes → 2N bytes out | raw DDR3 word read (N words) |
| 0x30 | REG_WRITE | 5 bytes | write a control register |
| 0x31 | REG_READ | 1 byte → 4 bytes out | read a status/ID register (0x00 DEVICE_ID, 0x01 CONTROL, 0x02 STATUS incl. DDR3-ready + Director-error, 0x03 N_SLOTS) |
| 0x40 | FLASH_XFER | N bytes → N bytes out (+2 margin) | raw passthrough to the config flash, see §5.3 |

Full byte-level field layouts are documented in `spi_host_bridge_v3.v`'s own
header comment — treat that file as the authoritative protocol spec, this
table is a summary/index.

**Real out-of-band notification (EXP-0085)**: `data_ready_n`, a real,
**active-low, sticky** top-level pin (D14, bank 15, 3.3V — see §2.2) —
lets the ESP32 be interrupt-driven instead of polling STATUS in a loop.
Asserted (driven low) when a job/pair completes (`job_out_done`) or while
a real Director error is active (`dir_error`); the job-completion latch
is **sticky** (stays low even after the underlying pulse ends) until the
host acknowledges by completing a real STATUS (0x20) or REG_READ(0x02)
transaction — a REG_READ of any *other* register does not acknowledge it.
The `dir_error` contribution is live/combinational, not latched — it
clears the moment `dir_error` itself does. Real firmware implication: the
ESP32 can wire this to a GPIO interrupt and only bother reading STATUS
when it actually fires, instead of polling every loop iteration.

## 7. Known-open items (honestly disclosed, not hidden)

- Scaling past N=2 compute cores (silicon budget allows up to ~30 per the
  DSP48E1 count) is not yet built or timing-verified. A result-writeback
  engine (currently `result_data_a/b` are literal top-level pins, fine at
  N=2 but a real pin-budget blocker beyond that) must land first — see
  `docs/ARCHITECTURE_ANALYSIS.md` §5.3.
- `sys_rst` has only a *tentative* real pin (G13, bank 15, EXP-0084 —
  chosen just to unblock real P&R, not a final board decision) — assign
  its real, permanent location once the rest of the board layout (reset
  circuit, status LEDs, etc.) is decided.
- **The 32-bit DDR3 channel's own real timing does NOT currently close**
  (EXP-0084: WNS -0.618ns at 2900ps/172.414MHz) — the current real,
  trustworthy signoff remains EXP-0083 (16-bit, 155.039MHz, +0.073ns) until
  the Input Clock Period is reverted toward 3225ps (keeping Data Width=32)
  in one more real, user-gated MIG wizard session. See §3.1 for the full
  honest story and real root cause (unrelated to the width change itself).
- ESP32-side JTAG bit-banging firmware (§5.1) does not exist yet — it's
  software work on the host side, not part of this FPGA RTL.
- `ddr_prefetch_mgr.v` (DDRManager phase 1, EXP-0083) is real, built, and
  timing-verified (at the OLD 16-bit/155MHz signoff), but its measured
  benefit (2.86% real reduction in total simulated time) is modest — see
  `docs/ARCHITECTURE_ANALYSIS.md` §5.2 for the honest number and why. The
  larger multi-slot DDRManager is not built. Its own real benefit should be
  re-measured once the 32-bit channel's timing actually closes.
- `data_ready_n` (EXP-0085, user-requested active-low sticky IRQ) is real,
  built, and functionally verified (49/49 including 10 new checks,
  `tb_spi_host_bridge_v3.v`) — see §6. Its own real P&R verification is
  deferred to the same next real P&R run already needed to close §3.1's
  timing gap, not run separately against a config already known to fail
  timing for unrelated reasons.