periscope/CLAUDE.md

Periscope — Agentic Schematic Validation

Periscope validates hardware schematics against component datasheets. It extracts constraints from PDFs, parses netlists and BOMs into a queryable graph, and runs an agentic validation loop to flag design violations.

Open-core note. This is the open-source core. A small set of files are "gateway-owned seams" — pass-through stubs here (frontend/src/proxy.ts, use-optional-auth.ts, clerk-theme-provider.tsx, components/billing/*, sidebar-auth.tsx, pricing-section.tsx, analytics/*, lib/csp-hosts.ts) that the hosted-cloud repo replaces with auth/billing implementations. Keep their export signatures stable, and never import auth/billing SDKs anywhere else in the frontend. On the backend, everything reaches billing only through backend/services/billing_hook.py:get_billing() (a no-op here).

System Overview

Three layers:

Layer	Location	Purpose
Core library	backend/periscopex/	Models, parsers, graph builder, agentic validator, passive resolver, taxonomy, BOM summary, derating
Backend	backend/	FastAPI app — async pipeline orchestration, SSE progress, project/file storage
Frontend	frontend/	Next.js 16 app — project dashboard, pipeline progress, report viewer, derating, admin dashboard

Plus skills/ — extraction prompts (pintable, patterns, specs) inlined locally for DeepSeek. Do not upload to Anthropic Console.

The pipeline stages: Parse BOM → Extract IC Pintables → Extract Simple Components → Extract Passives → DigiKey Auto-Resolve + Value Fallback → Build Graph → Direct Datasheet Review. Pipeline runs can be cancelled mid-execution via POST /api/pipeline/{id}/cancel.

Example Project

simple_project/ is the reference design for development and testing:

• MCU: TI MSPM0G3507SPTR (U3) — 48-pin LQFP
• USB-UART Bridge: CH340E (U2)
• LDO Regulator: SPX3819M5-L-3-3 (U1) — 5V to 3.3V
• ESD Protection: USBLC6-2SC6 (D1)
• Crystal: 8 MHz (X1) with 18pF load caps (C9, C10)

Files: .asc (PADS-PCB netlist; .edn EDIF 2.0.0 also accepted), .csv/.xlsx (BOM), design_graph.json (committed reference fixture used by tests).

Architecture Principles

• Modular extractors — Domain-specific extraction per component type, unified constraint schema
• Netlist as graph — Queryable bipartite graph (components + nets) with traversal helpers
• LLM API for PDF extraction — Forced tool calls for structured output (pintable, passive patterns, specs). Default provider is DeepSeek.
• Prompt caching — Anthropic stamps cache_control; Gemini uses CachedContent; DeepSeek uses automatic prefix cache (cache-hit tokens in usage).
• Local extraction skills — skills/*/SKILL.md is inlined and validate.py runs in-process. Never call scripts/upload_skills.py (Anthropic Console).
• Direct datasheet review — The model reads the IC datasheet plus circuit neighborhood, compares to the reference application circuit, and flags issues via graph query tools (find_connected_components, get_net_for_pin, get_pintable). DeepSeek converts PDFs to text (and page images on the vision model).
• Datasheet page trimming — Large PDFs are keyword-trimmed to relevant pages before sending to Claude, reducing token cost (pypdf)
• DigiKey fallback (exact MPN only) — When pattern-based and direct extraction fail, DigiKey API fetches product parameters for auto-resolve. DigiKey matches only on exact MPN; fuzzy hits are rejected to avoid polluting the shared library with wrong-dielectric / wrong-voltage parts.
• Value-string fallback — When DigiKey misses an R/C/L/FB passive, a value-string resolver maps the BOM Value string to typed passive specs. Value-derived specs are persisted per-project only — never to the shared library.
• Per-IC review error isolation — Direct datasheet review runs each IC independently; one malformed payload or bad response cannot kill the whole run. Failed ICs surface as skipped components with the error.
• Cross-IC excerpt budget (per-neighbor) — To verify an interface finding the reviewer can pull a connected IC's datasheet pages (get_datasheet_excerpt). The budget is a global per-review page ceiling plus a per-neighbor sub-budget, so verifying one interface is never starved by pages already spent on other neighbors.
• Finding normalization is downgrade-only — A post-review per-IC normalize pass (services/normalize_findings.py) drops self-cancelling findings, merges same-root-cause findings, and re-grades severity — but only ever downward. A deterministic clamp caps each finding at the reviewer's calibrated severity (and any Unverified: finding at WARNING, preserving the prefix).
• Cross-IC finding dedup — After all per-IC reviews complete, a single pass (services/dedupe_findings.py) collapses one physical interface defect reported from both endpoints into a single finding. Gated by cross_ic_dedup_enabled; fail-soft.
• Capacitor voltage derating — Deterministic derating table computed from graph (ceramic/tantalum/electrolytic percentages, pass/fail per capacitor)
• Deterministic checks over heuristics — Exact checks where possible
• Zero coupling between layers — Backend calls periscopex functions with paths; frontend talks to backend via REST + SSE
• Library deduplication — Shared library (library/extracted/, library/patterns/, library/models/, library/passives/, library/datasheets/) caches extractions across projects
• Content-addressed datasheets — library/datasheets/blobs/{md5}.pdf stores unique PDFs once; library/datasheets/refs/{safe_mpn}.json maps MPNs to blobs (dedupe + multi-MPN sharing)
• Taxonomy-driven extraction — Living component taxonomy (taxonomy/) with per-subtype classification and specs schemas
• Per-stage model config — Each pipeline stage can use a different Claude model (e.g., Sonnet for review, Haiku for auto-resolve)
• API call logging — Every Claude API call is logged with token counts, cost, and timing per pipeline run
• Report versioning — Each project run is stamped with the current app version on the first /start transition (ProjectMeta.periscope_version). The version comes from frontend/content/changelog.md's latest ## heading — single source of truth — read at backend startup via backend/_version.py.

Datasheet Extraction

Extracted data lives in library/extracted/ (shared) or per-project under the storage backend. One JSON per MPN, schema in backend/periscopex/models.py.

Per-MPN IC extraction captures:

1. Pintable — Pin number + name (required), description + alt functions (optional)
2. Package info — Base family, package, pin count, description
3. Component subtype — Dotted taxonomy path (e.g., ic.mcu, ic.power.ldo)

For discrete/simple components: 4. Specs — Component specs (value, tolerance, package, voltage rating, etc.); parameters are filtered against taxonomy specs schemas

Extraction inlines local skills (skills/*/SKILL.md + validate.py) against DeepSeek. Do not use Anthropic Console Skills.

Claude Console Skills

skills/
├── extract-pintable/    # Pin table + package info + taxonomy
│   ├── SKILL.md         # System prompt (YAML frontmatter + markdown)
│   ├── schema.json      # Tool output schema
│   └── validate.py      # Validation script
├── extract-pattern/     # Passive MPN pattern
└── extract-specs/       # Component specs (discrete, connectors, crystals, etc.)

Taxonomy

Living component taxonomy in taxonomy/ — one JSON file per top-level type (ic, passive, connector, crystal, discrete, fuse, switch, test_point, transformer). Each subtype entry includes description and example_mpn.

Key taxonomy features:

• Ref prefix mapping — U→ic, R/C/L→passive, D/Q→discrete, X→crystal, etc.
• Dotted subtype paths — e.g., ic.mcu, passive.capacitor.ceramic, ic.protection.esd
• Dynamic growth — add_subtype() adds new entries; concurrent-safe JSON writes
• Specs schema auto-generation — Type-level and subtype-level parameter specs schemas are auto-generated via Claude when a taxonomy entry has none; extraction discards parameters not in the schema (extra_specs field)

Scripts

• scripts/upload_skills.py — leftover Claude Console uploader. Do not run. Skills are local + DeepSeek only.
• scripts/migrate_datasheets_to_library.py — One-time migration: copy per-project datasheets to library/datasheets/ (dry-run by default, --apply to execute)
• scripts/migrate_datasheets_to_blobs.py — Migrate named-PDF datasheets into the content-addressed blobs/refs layout (dry-run by default, --apply to execute)
• scripts/dedup_library_datasheets.py — Remove redundant per-MPN datasheet PDFs when a passive pattern already has a datasheet_key (dry-run by default, --apply to execute)
• scripts/gc_orphan_blobs.py — Garbage-collect library/datasheets/blobs/*.pdf not referenced by any ref file
• scripts/clear_rules_from_extractions.py — Strip deprecated rules/absolute_maximum_ratings from existing library extractions

Tech Stack

• Core: Python 3.12+, Pydantic 2.x, OpenAI SDK (DeepSeek), Anthropic SDK (optional), google-genai (optional), openpyxl, pypdf, PyMuPDF
• Backend: FastAPI, uvicorn, sse-starlette, pydantic-settings
• Frontend: Next.js 16 (App Router, Turbopack), React 19, Tailwind CSS v4, shadcn/ui (Base UI), react-pdf
• AI: DeepSeek Chat Completions (OpenAI-compatible) with forced tool calls for extraction and agentic review. Do not route stages to Anthropic.
• Model: deepseek-flash for extraction, review, auto-resolve, and normalize (per-stage overrides via .env)
• Skills: Local SKILL.md + validate.py on DeepSeek
• External APIs: DigiKey API v4 (OAuth2) — optional datasheet auto-fetch and parameter-based auto-resolve (DIGIKEY_CLIENT_ID, DIGIKEY_CLIENT_SECRET)

Extracted Model Versioning

All ComponentConstraints extracted JSON files carry a model_version semver field:

• Initial value — set from default_model_version in backend/skills_manifest.json (starts at 1.0.0)
• Minor bump — increment default_model_version in skills_manifest.json when extraction prompts change (do not run upload_skills.py).

Rule: When committing changes under skills/, bump default_model_version locally. Never call Anthropic.

Development Guidelines

• Write tests against simple_project/ — it's the ground truth
• Netlist parser and BOM parser are pure functions with no side effects
• All data structures use Pydantic models in backend/periscopex/models.py
• Frontend types in frontend/src/lib/types.ts must stay in sync with backend/periscopex/models.py
• Extraction prompts live in skills/ (SKILL.md + schema.json + validate.py) and run locally against DeepSeek
• Never swallow exceptions silently — prefer logging or re-raising over bare except: continue. Silent failures hide real bugs.

Running

# Backend (copy backend/.env.example to .env at repo root first)
python3 -m uvicorn backend.main:app --reload    # localhost:8000

# Frontend
cd frontend && npm run dev                       # localhost:3000

Local mode needs no cloud services and no auth — projects are stored in data/ and you are user_id="local" with admin access.