# Periscope — Agentic Schematic Validation

Periscope validates hardware schematics against component datasheets. It extracts constraints from PDFs, parses netlists and BOMs into a queryable graph, and runs an agentic validation loop to flag design violations.

> **Open-core note.** This is the open-source core. A small set of files are
> "gateway-owned seams" — pass-through stubs here (`frontend/src/proxy.ts`,
> `use-optional-auth.ts`, `clerk-theme-provider.tsx`,
> `components/billing/*`, `sidebar-auth.tsx`, `pricing-section.tsx`,
> `analytics/*`, `lib/csp-hosts.ts`) that the hosted-cloud repo replaces
> with auth/billing implementations. Keep their export signatures stable,
> and never import auth/billing SDKs anywhere else in the frontend. On the
> backend, everything reaches billing only through
> `backend/services/billing_hook.py:get_billing()` (a no-op here).

## System Overview

Three layers:

| Layer | Location | Purpose |
|-------|----------|---------|
| **Core library** | `backend/periscopex/` | Models, parsers, graph builder, agentic validator, passive resolver, taxonomy, BOM summary, derating |
| **Backend** | `backend/` | FastAPI app — async pipeline orchestration, SSE progress, project/file storage |
| **Frontend** | `frontend/` | Next.js 16 app — project dashboard, pipeline progress, report viewer, derating, admin dashboard |

Plus `skills/` — extraction prompts (pintable, patterns, specs) inlined locally for DeepSeek. Do not upload to Anthropic Console.

The pipeline stages: Parse BOM → Extract IC Pintables → Extract Simple Components → Extract Passives → DigiKey Auto-Resolve + Value Fallback → Build Graph → Direct Datasheet Review. Pipeline runs can be cancelled mid-execution via `POST /api/pipeline/{id}/cancel`.

## Example Project

`simple_project/` is the reference design for development and testing:

- **MCU**: TI MSPM0G3507SPTR (U3) — 48-pin LQFP
- **USB-UART Bridge**: CH340E (U2)
- **LDO Regulator**: SPX3819M5-L-3-3 (U1) — 5V to 3.3V
- **ESD Protection**: USBLC6-2SC6 (D1)
- **Crystal**: 8 MHz (X1) with 18pF load caps (C9, C10)

Files: `.asc` (PADS-PCB netlist; `.edn` EDIF 2.0.0 also accepted), `.csv`/`.xlsx` (BOM), `design_graph.json` (committed reference fixture used by tests).

## Architecture Principles

- **Modular extractors** — Domain-specific extraction per component type, unified constraint schema
- **Netlist as graph** — Queryable bipartite graph (components + nets) with traversal helpers
- **LLM API for PDF extraction** — Forced tool calls for structured output (pintable, passive patterns, specs). Default provider is DeepSeek.
- **Prompt caching** — Anthropic stamps `cache_control`; Gemini uses CachedContent; DeepSeek uses automatic prefix cache (cache-hit tokens in usage).
- **Local extraction skills** — `skills/*/SKILL.md` is inlined and `validate.py` runs in-process. Never call `scripts/upload_skills.py` (Anthropic Console).
- **Direct datasheet review** — The model reads the IC datasheet plus circuit neighborhood, compares to the reference application circuit, and flags issues via graph query tools (`find_connected_components`, `get_net_for_pin`, `get_pintable`). DeepSeek converts PDFs to text (and page images on the vision model).
- **Datasheet page trimming** — Large PDFs are keyword-trimmed to relevant pages before sending to Claude, reducing token cost (`pypdf`)
- **DigiKey fallback (exact MPN only)** — When pattern-based and direct extraction fail, DigiKey API fetches product parameters for auto-resolve. DigiKey matches only on exact MPN; fuzzy hits are rejected to avoid polluting the shared library with wrong-dielectric / wrong-voltage parts.
- **Value-string fallback** — When DigiKey misses an R/C/L/FB passive, a value-string resolver maps the BOM `Value` string to typed passive specs. Value-derived specs are persisted per-project only — never to the shared library.
- **Per-IC review error isolation** — Direct datasheet review runs each IC independently; one malformed payload or bad response cannot kill the whole run. Failed ICs surface as skipped components with the error.
- **Cross-IC excerpt budget (per-neighbor)** — To verify an interface finding the reviewer can pull a *connected* IC's datasheet pages (`get_datasheet_excerpt`). The budget is a global per-review page ceiling **plus a per-neighbor sub-budget**, so verifying one interface is never starved by pages already spent on other neighbors.
- **Finding normalization is downgrade-only** — A post-review per-IC normalize pass (`services/normalize_findings.py`) drops self-cancelling findings, merges same-root-cause findings, and re-grades severity — but only ever *downward*. A deterministic clamp caps each finding at the reviewer's calibrated severity (and any `Unverified:` finding at WARNING, preserving the prefix).
- **Cross-IC finding dedup** — After all per-IC reviews complete, a single pass (`services/dedupe_findings.py`) collapses one physical interface defect reported from both endpoints into a single finding. Gated by `cross_ic_dedup_enabled`; fail-soft.
- **Capacitor voltage derating** — Deterministic derating table computed from graph (ceramic/tantalum/electrolytic percentages, pass/fail per capacitor)
- **Deterministic checks over heuristics** — Exact checks where possible
- **Zero coupling between layers** — Backend calls periscopex functions with paths; frontend talks to backend via REST + SSE
- **Library deduplication** — Shared library (`library/extracted/`, `library/patterns/`, `library/models/`, `library/passives/`, `library/datasheets/`) caches extractions across projects
- **Content-addressed datasheets** — `library/datasheets/blobs/{md5}.pdf` stores unique PDFs once; `library/datasheets/refs/{safe_mpn}.json` maps MPNs to blobs (dedupe + multi-MPN sharing)
- **Taxonomy-driven extraction** — Living component taxonomy (`taxonomy/`) with per-subtype classification and specs schemas
- **Per-stage model config** — Each pipeline stage can use a different Claude model (e.g., Sonnet for review, Haiku for auto-resolve)
- **API call logging** — Every Claude API call is logged with token counts, cost, and timing per pipeline run
- **Report versioning** — Each project run is stamped with the current app version on the first `/start` transition (`ProjectMeta.periscope_version`). The version comes from `frontend/content/changelog.md`'s latest `##` heading — single source of truth — read at backend startup via `backend/_version.py`.

## Datasheet Extraction

Extracted data lives in `library/extracted/` (shared) or per-project under the storage backend. One JSON per MPN, schema in `backend/periscopex/models.py`.

Per-MPN IC extraction captures:
1. **Pintable** — Pin number + name (required), description + alt functions (optional)
2. **Package info** — Base family, package, pin count, description
3. **Component subtype** — Dotted taxonomy path (e.g., `ic.mcu`, `ic.power.ldo`)

For discrete/simple components:
4. **Specs** — Component specs (value, tolerance, package, voltage rating, etc.); parameters are filtered against taxonomy specs schemas

Extraction inlines **local skills** (`skills/*/SKILL.md` + `validate.py`) against DeepSeek. Do not use Anthropic Console Skills.

## Claude Console Skills

```
skills/
├── extract-pintable/    # Pin table + package info + taxonomy
│   ├── SKILL.md         # System prompt (YAML frontmatter + markdown)
│   ├── schema.json      # Tool output schema
│   └── validate.py      # Validation script
├── extract-pattern/     # Passive MPN pattern
└── extract-specs/       # Component specs (discrete, connectors, crystals, etc.)
```

## Taxonomy

Living component taxonomy in `taxonomy/` — one JSON file per top-level type (ic, passive, connector, crystal, discrete, fuse, switch, test_point, transformer). Each subtype entry includes `description` and `example_mpn`.

Key taxonomy features:
- **Ref prefix mapping** — `U→ic`, `R/C/L→passive`, `D/Q→discrete`, `X→crystal`, etc.
- **Dotted subtype paths** — e.g., `ic.mcu`, `passive.capacitor.ceramic`, `ic.protection.esd`
- **Dynamic growth** — `add_subtype()` adds new entries; concurrent-safe JSON writes
- **Specs schema auto-generation** — Type-level and subtype-level parameter specs schemas are auto-generated via Claude when a taxonomy entry has none; extraction discards parameters not in the schema (`extra_specs` field)

## Scripts

- `scripts/upload_skills.py` — leftover Claude Console uploader. **Do not run.** Skills are local + DeepSeek only.
- `scripts/migrate_datasheets_to_library.py` — One-time migration: copy per-project datasheets to `library/datasheets/` (dry-run by default, `--apply` to execute)
- `scripts/migrate_datasheets_to_blobs.py` — Migrate named-PDF datasheets into the content-addressed blobs/refs layout (dry-run by default, `--apply` to execute)
- `scripts/dedup_library_datasheets.py` — Remove redundant per-MPN datasheet PDFs when a passive pattern already has a `datasheet_key` (dry-run by default, `--apply` to execute)
- `scripts/gc_orphan_blobs.py` — Garbage-collect `library/datasheets/blobs/*.pdf` not referenced by any ref file
- `scripts/clear_rules_from_extractions.py` — Strip deprecated `rules`/`absolute_maximum_ratings` from existing library extractions

## Tech Stack

- **Core**: Python 3.12+, Pydantic 2.x, OpenAI SDK (DeepSeek), Anthropic SDK (optional), google-genai (optional), openpyxl, pypdf, PyMuPDF
- **Backend**: FastAPI, uvicorn, sse-starlette, pydantic-settings
- **Frontend**: Next.js 16 (App Router, Turbopack), React 19, Tailwind CSS v4, shadcn/ui (Base UI), react-pdf
- **AI**: DeepSeek Chat Completions (OpenAI-compatible) with forced tool calls for extraction and agentic review. Do not route stages to Anthropic.
- **Model**: `deepseek-flash` for extraction, review, auto-resolve, and normalize (per-stage overrides via `.env`)
- **Skills**: Local SKILL.md + validate.py on DeepSeek
- **External APIs**: DigiKey API v4 (OAuth2) — optional datasheet auto-fetch and parameter-based auto-resolve (`DIGIKEY_CLIENT_ID`, `DIGIKEY_CLIENT_SECRET`)

## Extracted Model Versioning

All `ComponentConstraints` extracted JSON files carry a `model_version` semver field:

- **Initial value** — set from `default_model_version` in `backend/skills_manifest.json` (starts at `1.0.0`)
- **Minor bump** — increment `default_model_version` in `skills_manifest.json` when extraction prompts change (do **not** run `upload_skills.py`).

**Rule**: When committing changes under `skills/`, bump `default_model_version` locally. Never call Anthropic.

## Development Guidelines

- Write tests against `simple_project/` — it's the ground truth
- Netlist parser and BOM parser are pure functions with no side effects
- All data structures use Pydantic models in `backend/periscopex/models.py`
- Frontend types in `frontend/src/lib/types.ts` must stay in sync with `backend/periscopex/models.py`
- Extraction prompts live in `skills/` (SKILL.md + schema.json + validate.py) and run locally against DeepSeek
- **Never swallow exceptions silently** — prefer logging or re-raising over bare `except: continue`. Silent failures hide real bugs.

## Running

```bash
# Backend (copy backend/.env.example to .env at repo root first)
python3 -m uvicorn backend.main:app --reload    # localhost:8000

# Frontend
cd frontend && npm run dev                       # localhost:3000
```

Local mode needs no cloud services and no auth — projects are stored in `data/` and you are `user_id="local"` with admin access.