Native DeepSeek datasheet extraction in periscope/src (Fase C3).

Pipeline and LCSC resolve call datasheet_extract; inherited extraction.py
stays in dependency as fallback. No Anthropic Console skill ids.
This commit is contained in:
2026-09-20 15:26:21 +02:00
parent a96ee2e88a
commit 5c0c5184fb
11 changed files with 1351 additions and 18 deletions
@@ -906,7 +906,7 @@ async def auto_resolve(req: AutoResolveRequest, request: Request):
import asyncio
from backend.services.digikey import fetch_params
from backend.services.extraction import CatalogResolveMiss, auto_resolve_specs
from backend.services.datasheet_extract import CatalogResolveMiss, auto_resolve_specs
if not settings.use_digikey:
raise HTTPException(400, "DigiKey API not configured")
@@ -996,7 +996,7 @@ async def lcsc_resolve_passive(
from backend.services.api_logs import ApiLogger
from backend.services.billing_hook import InsufficientCredits, get_billing
from backend.services.extraction import auto_resolve_specs
from backend.services.datasheet_extract import auto_resolve_specs
storage = get_storage(request)
result = proj_svc.resolve_project_access(storage, get_user_id(request), project_id)
@@ -1,11 +1,12 @@
"""Async datasheet extraction using the configured LLM provider.
"""Inherited PinScope datasheet extraction (fallback, not the live pipeline).
Live path since 2.40.0: ``backend.services.datasheet_extract``. This file
stays in periscope/dependency/ — do not empty-delete it.
Ports the extraction steps from run_pipeline.py to async:
- extract_pintable: Pin table + package info + taxonomy assignment
- extract_pattern: Passive MPN pattern
- extract_specs: Component specs (discrete, connectors, crystals, etc.)
Skills (SKILL.md + validate.py) run locally against DeepSeek. Do not use Anthropic Console Skills.
"""
from __future__ import annotations
@@ -47,7 +47,7 @@ from backend.config import settings
from backend.services import admin_settings as settings_svc
from backend.services.billing_hook import InsufficientCredits, get_billing
from backend.services.datasheet_store import compute_md5_from_path, store_datasheet, store_datasheet_bytes
from backend.services import extraction, projects as proj_svc
from backend.services import datasheet_extract as extraction, projects as proj_svc
from backend.services.api_logs import ApiLogger, total_cost
from backend.services.cost_estimator import estimate_stage_cost_usd
from backend.services.storage import StorageBackend
@@ -2,6 +2,13 @@
What's new in Periscope.
## 2.40.0 — 2026-09-20 — Fase C3: native datasheet extraction
Pipeline pintable/pattern/specs/auto-resolve run from `periscope/src` (`datasheet_extract.py`): DeepSeek + local SKILL.md, no Anthropic Console skill ids. Inherited `extraction.py` stays in `dependency/`.
- [Changed] `pipeline.py` and project LCSC resolve import `datasheet_extract`.
- [Changed] Extract stages pin the DeepSeek provider.
## 2.39.1 — 2026-09-20 — src no longer imports PinScope review loop
After 2.39.0 live smoke (Emmaforo ERROR 2 / WARNING 12 / INFO 18), native modules are the only review path from `periscope/src`. PinScope `validate.py` / `validation_tools.py` remain on disk.
File diff suppressed because it is too large Load Diff
@@ -1,6 +1,6 @@
# Piano — indipendenza architettonica e di licenza da PinScope
**Stato:** split **2.38.0**. C2 native review loop **2.39.0** live. **2.39.1** `periscope/src` non importa più `validate.py` / `validation_tools.py` (PCB incluso). File PinScope restano in `dependency/`. Fork non staccato.
**Stato:** split **2.38.0**. C2 review **2.39.0/2.39.1** live. C3 extraction **2.40.0**. `validate.py` / `extraction.py` restano in `dependency/` (non chiamati dal live path). Fork non staccato.
**Gate Michele:** sostituire/smettere di chiamare un modulo `dependency/` solo dopo pytest + deploy smoke. Se la verifica fallisce, resta il path ereditato.
**Sequenza:** split → sostituzione incrementale (C2 loop → C3 extraction → C4 PCB off `validate.py`). **Mai** empty-delete. Parsers/graph (C5/E) e auto-place fuori scope. AGPL resta.
@@ -225,7 +225,7 @@ Qui sì si riscrive lengine ereditato. Ordine interno:
| C0 | Spec freeze `motore-finding.md` = unico IR | Già nativo; non diluire in `validate.py` |
| C1 | Adapter: `validate.py` emette solo `Finding` grezzi → `complete_finding` | Già parziale; chiudere i campi doppi |
| C2 | **REWRITE** loop per-IC native in `periscope/src` (DeepSeek, tools su `DesignGraph`) | **Shipped 2.39.0** live `review_ic_async``review_session`; PinScope files **kept**, not called from live loop |
| C3 | Extraction: `local_skill.py` + schemi JSON (KEEP schema se identici; REWRITE orchestrazione Anthropic) | |
| C3 | Extraction: `local_skill.py` + schemi JSON (KEEP schema se identici; REWRITE orchestrazione Anthropic) | **Shipped 2.40.0** live `datasheet_extract`; PinScope `extraction.py` kept |
| C4 | Spegnere import da `validate.py` nel PCB (`_parse_review` → parser finding nativo) | **Shipped 2.39.1** `pcb_validation``review_parse` / `review_session` |
| C5 | Test golden `simple_project` + Emmaforo: parity FACT/REQUIREMENT, non parity prose | |
+31
View File
@@ -0,0 +1,31 @@
"""Native datasheet_extract vs inherited extraction — prove before switching pipeline."""
from __future__ import annotations
import ast
from pathlib import Path
from backend.services.datasheet_extract import _coerce_abs_max
from backend.services.extraction import _coerce_abs_max as inherited_coerce
def test_coerce_abs_max_matches_inherited():
raw = [
{"parameter": "Vin", "min": 0, "max": 6, "unit": "V", "source_page": 4},
"not-a-dict",
]
assert _coerce_abs_max(raw) == inherited_coerce(raw)
assert _coerce_abs_max(None) == inherited_coerce(None)
def test_native_extract_has_no_anthropic_import():
path = Path(__import__("backend.services.datasheet_extract", fromlist=["x"]).__file__)
tree = ast.parse(path.read_text())
imported = []
for node in ast.walk(tree):
if isinstance(node, ast.ImportFrom) and node.module:
imported.append(node.module)
elif isinstance(node, ast.Import):
imported.extend(a.name for a in node.names)
assert not any("anthropic" in (m or "") for m in imported)
assert "backend.services.llm.factory" in imported
+4 -4
View File
@@ -86,12 +86,12 @@ def test_auto_resolve_skips_llm_when_catalog_parses(monkeypatch):
import asyncio
from pathlib import Path
from backend.services.extraction import auto_resolve_specs
from backend.services.datasheet_extract import auto_resolve_specs
async def boom(*_a, **_k):
raise AssertionError("LLM must not run")
monkeypatch.setattr("backend.services.extraction.call_with_fallback", boom)
monkeypatch.setattr("backend.services.datasheet_extract.call_with_fallback", boom)
tax = TAXONOMY
model = asyncio.run(
auto_resolve_specs(
@@ -110,12 +110,12 @@ def test_auto_resolve_use_llm_false_on_ferrite(monkeypatch):
import asyncio
from pathlib import Path
from backend.services.extraction import auto_resolve_specs
from backend.services.datasheet_extract import auto_resolve_specs
async def boom(*_a, **_k):
raise AssertionError("LLM must not run")
monkeypatch.setattr("backend.services.extraction.call_with_fallback", boom)
monkeypatch.setattr("backend.services.datasheet_extract.call_with_fallback", boom)
tax = TAXONOMY
model = asyncio.run(
auto_resolve_specs(
+4 -4
View File
@@ -52,12 +52,12 @@ def test_ambiguous_string_returns_none():
def test_resolve_from_value_skips_llm_when_parseable(monkeypatch):
from backend.services.extraction import resolve_from_value
from backend.services.datasheet_extract import resolve_from_value
async def boom(*_a, **_k):
raise AssertionError("LLM must not run")
monkeypatch.setattr("backend.services.extraction.call_with_fallback", boom)
monkeypatch.setattr("backend.services.datasheet_extract.call_with_fallback", boom)
tax = TAXONOMY
model = asyncio.run(
resolve_from_value(
@@ -71,12 +71,12 @@ def test_resolve_from_value_skips_llm_when_parseable(monkeypatch):
def test_resolve_from_value_placeholder_no_llm(monkeypatch):
from backend.services.extraction import resolve_from_value
from backend.services.datasheet_extract import resolve_from_value
async def boom(*_a, **_k):
raise AssertionError("LLM must not run")
monkeypatch.setattr("backend.services.extraction.call_with_fallback", boom)
monkeypatch.setattr("backend.services.datasheet_extract.call_with_fallback", boom)
tax = TAXONOMY
with pytest.raises(ValueError, match="Placeholder"):
asyncio.run(
+1 -1
View File
@@ -4,7 +4,7 @@ from __future__ import annotations
from pathlib import Path
from backend.services.extraction import _coerce_abs_max
from backend.services.datasheet_extract import _coerce_abs_max
from backend.services.llm.pdf_ingest import (
extract_pdf_text,
make_text_pdf,
+1 -1
View File
@@ -452,7 +452,7 @@ async def test_lcsc_resolve_passive_success_and_cached(tmp_path, monkeypatch):
from backend.config import settings
from backend.periscopex.models import CapacitorSpecs, ComponentModel
from backend.services import purple_parts
from backend.services import extraction as extraction_svc
from backend.services import datasheet_extract as extraction_svc
# Billing modules are absent in the open-source repo; the endpoint under
# test only charges when billing is enabled, so skip there.