periscope/periscope/dependency/backend/CLAUDE.md

Periscope Backend

FastAPI application providing async pipeline orchestration, project storage, and SSE progress streaming. Wraps the periscopex/ core library — calls existing functions with local paths, adds no domain logic of its own.

Running

# From project root
python3 -m uvicorn backend.main:app --reload    # localhost:8000

Config reads from backend/.env (see config.py). Key settings: DEEPSEEK_API_KEY, DEEPSEEK_MODEL, per-stage DeepSeek overrides (model_pintable_deepseek, model_validation_deepseek, …), PROVIDER_DEFAULT (default deepseek), optional ANTHROPIC_API_KEY / GEMINI_API_KEY, CORS_ORIGINS, DigiKey keys.

For local mode, leave GCS_BUCKET empty — uses LocalStorageBackend (data/ directory) and no auth (user_id defaults to "local", admin access granted).

Architecture

backend/
├── main.py              # App entry, lifespan hook, CORS, auth middleware, router includes
├── config.py            # Pydantic Settings from .env
├── _version.py          # Reads app version from frontend/content/changelog.md (single source of truth)
├── Dockerfile           # Python 3.12-slim, copies taxonomy/ + changelog.md for runtime
├── skills_manifest.json # Claude Console Skill IDs (extract-pintable, extract-pattern, extract-specs)
├── periscopex/           # Core library (models, parsers, graph, validator, taxonomy, derating)
│   ├── utils.py         # Shared utilities: safe_mpn(), natural_sort_key()
│   └── resolve_passives.py  # Passive MPN pattern matching + value decoders (R/C/L)
├── middleware/
│   └── auth.py          # JWT verification via JWKS (enabled when CLERK_JWKS_URL is set; off in OSS mode)
├── routers/
│   ├── deps.py          # Shared router dependencies: get_storage(), get_user_id(), resolve_or_404()
│   ├── projects.py      # CRUD + file upload + library check + collaborators + DigiKey endpoints
│   ├── pipeline.py      # Start, cancel, estimate, resume, restart, regen, SSE, status
│   ├── reports.py       # Report, comments, graph, datasheet, API logs, BOM, derating
│   ├── admin.py         # Admin-only: components, users, usage, projects, runs, settings
│   ├── feedback.py      # User feedback tickets
│   └── contact.py       # Contact form (email relay; inert unless email is configured)
└── services/
    ├── storage.py            # StorageBackend protocol + LocalStorageBackend
    ├── storage_gcs.py        # GCSStorageBackend (optional, Google Cloud Storage)
    ├── projects.py           # Project CRUD + library ops via StorageBackend
    ├── pipeline.py           # Multi-stage orchestrator + EventBroker + PipelineWorkspace
    ├── extraction.py         # Async Claude API calls (pintable, patterns, specs, auto-resolve, value fallback) + skills + page trimming
    ├── validation.py         # Async agentic validation wrapper with per-IC error isolation
    ├── normalize_findings.py # Post-review per-IC normalize pass (downgrade-only)
    ├── dedupe_findings.py    # Cross-IC finding dedup
    ├── billing_hook.py       # Open-core billing seam (NullBilling here — pipelines run free)
    ├── digikey.py            # DigiKey API v4 — OAuth2, datasheet fetch, parameter fetch (exact MPN only)
    ├── purple_parts.py       # Optional external LCSC→MPN resolver (env-gated; fails soft when unset)
    ├── api_logs.py           # API call logging, cost calculation per pipeline run
    ├── cost_estimator.py     # Pre-flight pipeline estimate (read-only)
    ├── datasheet_store.py    # Content-addressed PDF storage: blobs/{md5}.pdf + refs/{safe_mpn}.json
    ├── admin_settings.py     # Admin-only settings (e.g., min_model_version threshold)
    ├── job_runner.py         # Optional Cloud Run job trigger (in-process asyncio locally)
    └── email.py              # Email notifications (optional, env-gated)

Storage Abstraction

All file I/O goes through StorageBackend (protocol in services/storage.py):

• LocalStorageBackend: Maps keys to data/ directory. Default.
• GCSStorageBackend: Uses google-cloud-storage SDK. Used when GCS_BUCKET is set.

Storage keys follow GCS-style paths: users/{user_id}/projects/{id}/uploads/bom.csv

The periscopex/ core library is unaware of storage — it operates on local paths. During pipeline execution, PipelineWorkspace downloads files to a temp dir, runs periscopex/ functions locally, then uploads results back.

Project Storage

# data/ directory (or GCS bucket)
users/{user_id}/projects/{id}/
├── project.json                    # ProjectMeta (name, status, timestamps, user_id, total_cost_usd)
├── uploads/
│   ├── bom.csv
│   ├── netlist.asc          # OR netlist.edn for EDIF uploads
│   └── datasheets/*.pdf
├── extracted/                      # Per-project IC extractions
├── patterns/                       # Per-project passive patterns
├── models/                         # Per-project resolved specs
├── design_graph.json
├── bom_summary.json                # BOM summary table (collated from design graph)
├── derating.json                   # Capacitor voltage derating table
├── report.json                     # Findings + comments
└── api_logs.jsonl                  # Claude API call log (token counts, cost, timing)

library/                            # Shared across projects
├── extracted/                      # Shared IC extractions
├── patterns/                       # Shared passive patterns
├── models/                         # Shared component specs (discrete, connectors, etc.)
├── passives/                       # DigiKey-resolved passive specs (exact-MPN only)
└── datasheets/
    ├── blobs/{md5}.pdf             # Content-addressed PDF blobs (deduped)
    └── refs/{safe_mpn}.json        # MPN → blob pointer ({hash, blob_key})

taxonomy/                           # Component taxonomy (repo taxonomy/ dir in local mode)

Library lookups happen first — if an MPN was already extracted, it's reused without re-calling the API.

Pipeline Stages

The pipeline runs async via asyncio.create_task(). Progress emitted as SSE events via EventBroker (async queue per subscriber). PipelineWorkspace handles download/upload. Pipelines can be cancelled mid-run via POST /api/pipeline/{id}/cancel.

1. Parse BOM — Read uploaded CSV/XLSX (uses stored column mappings from upload; XLSX converted to CSV via openpyxl)
2. Extract IC Pintables — Async Claude API calls for pintable per IC MPN (datasheets keyword-trimmed via pypdf). Cache-miss MPNs are extracted concurrently, up to IC_CONCURRENCY (default 6) in flight at once. 2.5. Extract Simple Components — Specs extraction for discrete/simple components with datasheets
3. Extract Passives — Pattern-based extraction per MPN group, then a specs fallback per MPN. 3.5. DigiKey Auto-Resolve (exact MPN) — Fallback for unresolved passives; parameters mapped to taxonomy specs via Haiku. Requires exact MPN match so the shared library/passives/ stays clean. 3.6. Value Fallback (R/C/L/FB only) — When DigiKey misses, parse the BOM Value string via Haiku into typed passive specs. Per-project only; never written to the shared library.
4. Build Graph — Call periscopex.graph.build_graph() with local temp paths
5. BOM Summary — Collate components from design graph (no AI)
6. Derating Table — Capacitor voltage derating computation (no AI)
7. Direct Datasheet Review — Per-IC (isolated): Claude reads the datasheet PDF + circuit neighborhood from the graph, compares to reference application circuit, and submits findings via graph query tools. ICs are reviewed concurrently, up to IC_CONCURRENCY in flight at once.

Concurrency knob — IC_CONCURRENCY (config.py: ic_concurrency, default 6) governs parallelism for stage 2 (IC extraction), stage 3.5 (passive specs fallback), and stage 7 (review). Set IC_CONCURRENCY=1 for fully sequential behavior.

Key Patterns

• StorageBackend protocol — all file I/O is abstracted; swap local/GCS via GCS_BUCKET env var
• PipelineWorkspace — downloads to temp dir, runs periscopex locally, uploads results
• BillingHook seam (open-core) — core code reaches billing exclusively through services/billing_hook.py:get_billing(). In this repo that's NullBilling: every pipeline runs free and no billing routes are mounted. Never import billing modules directly from core code — go through the hook.
• Auth middleware — JWT verification via a JWKS endpoint; disabled when CLERK_JWKS_URL is empty (local mode: user_id="local", is_admin() returns True)
• AsyncAnthropic for all Claude API calls — extraction and validation
• Claude Console Skills — extraction uses managed skills (skill_id + version from skills_manifest.json); no fallback, raises error if skill not configured
• Prompt caching — extraction and validation calls use cache_control={"type": "ephemeral"} on system prompts and input context
• Forced tool calls for extraction — structured output via tool_choice
• SSE via sse-starlette — EventBroker manages per-project async queues with history replay
• API call logging — ApiLogger in services/api_logs.py collects per-call metadata; CallMeta returned from extraction functions
• Traceability IDs — findings get IDs (format: {designator}-{001}) for audit trails
• Taxonomy specs schemas — auto-generated via Claude per type/subtype; extraction discards parameters not in schema (extra_specs)
• Shared router deps — routers/deps.py centralizes get_storage(), get_user_id(), resolve_or_404() across all routers
• DigiKey OAuth2 — Token caching in services/digikey.py; _find_product requires exact MPN (no silent first-match fallback)
• Version stamping — backend/_version.py reads the latest ## heading from frontend/content/changelog.md and exports PERISCOPE_VERSION; stamped onto ProjectMeta.periscope_version at /start
• Datasheet page trimming — _select_pages() in extraction.py keyword-trims large PDFs to reduce token costs
• Content-addressed datasheets — datasheet_store.py writes PDFs to library/datasheets/blobs/{md5}.pdf and maps MPNs via refs
• Passive value decoders — periscopex/resolve_passives.py decodes EIA-198, R-notation, letter-decimal, EIA3/EIA4 for R/C/L values
• Collaborator access — resolve_or_404() grants access to both owner and collaborators
• Per-IC review isolation — In services/validation.py, each IC review is wrapped so a single bad payload is captured as a skipped component rather than aborting the run

Guidelines

• Keep all Claude API interaction in services/extraction.py and services/validation.py; logging in services/api_logs.py
• Keep all storage operations in services/projects.py (uses StorageBackend)
• Routers are thin — validate input, call service, return response
• Thread user_id from request.state through to all service calls
• Don't import from backend/ in periscopex/ — dependency flows one way
• CORS is configured for localhost:3000 by default; override with CORS_ORIGINS env var