Native DeepSeek datasheet extraction in periscope/src (Fase C3).
Pipeline and LCSC resolve call datasheet_extract; inherited extraction.py stays in dependency as fallback. No Anthropic Console skill ids.
This commit is contained in:
@@ -906,7 +906,7 @@ async def auto_resolve(req: AutoResolveRequest, request: Request):
|
||||
import asyncio
|
||||
|
||||
from backend.services.digikey import fetch_params
|
||||
from backend.services.extraction import CatalogResolveMiss, auto_resolve_specs
|
||||
from backend.services.datasheet_extract import CatalogResolveMiss, auto_resolve_specs
|
||||
|
||||
if not settings.use_digikey:
|
||||
raise HTTPException(400, "DigiKey API not configured")
|
||||
@@ -996,7 +996,7 @@ async def lcsc_resolve_passive(
|
||||
|
||||
from backend.services.api_logs import ApiLogger
|
||||
from backend.services.billing_hook import InsufficientCredits, get_billing
|
||||
from backend.services.extraction import auto_resolve_specs
|
||||
from backend.services.datasheet_extract import auto_resolve_specs
|
||||
|
||||
storage = get_storage(request)
|
||||
result = proj_svc.resolve_project_access(storage, get_user_id(request), project_id)
|
||||
|
||||
@@ -1,11 +1,12 @@
|
||||
"""Async datasheet extraction using the configured LLM provider.
|
||||
"""Inherited PinScope datasheet extraction (fallback, not the live pipeline).
|
||||
|
||||
Live path since 2.40.0: ``backend.services.datasheet_extract``. This file
|
||||
stays in periscope/dependency/ — do not empty-delete it.
|
||||
|
||||
Ports the extraction steps from run_pipeline.py to async:
|
||||
- extract_pintable: Pin table + package info + taxonomy assignment
|
||||
- extract_pattern: Passive MPN pattern
|
||||
- extract_specs: Component specs (discrete, connectors, crystals, etc.)
|
||||
|
||||
Skills (SKILL.md + validate.py) run locally against DeepSeek. Do not use Anthropic Console Skills.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -47,7 +47,7 @@ from backend.config import settings
|
||||
from backend.services import admin_settings as settings_svc
|
||||
from backend.services.billing_hook import InsufficientCredits, get_billing
|
||||
from backend.services.datasheet_store import compute_md5_from_path, store_datasheet, store_datasheet_bytes
|
||||
from backend.services import extraction, projects as proj_svc
|
||||
from backend.services import datasheet_extract as extraction, projects as proj_svc
|
||||
from backend.services.api_logs import ApiLogger, total_cost
|
||||
from backend.services.cost_estimator import estimate_stage_cost_usd
|
||||
from backend.services.storage import StorageBackend
|
||||
|
||||
@@ -2,6 +2,13 @@
|
||||
|
||||
What's new in Periscope.
|
||||
|
||||
## 2.40.0 — 2026-09-20 — Fase C3: native datasheet extraction
|
||||
|
||||
Pipeline pintable/pattern/specs/auto-resolve run from `periscope/src` (`datasheet_extract.py`): DeepSeek + local SKILL.md, no Anthropic Console skill ids. Inherited `extraction.py` stays in `dependency/`.
|
||||
|
||||
- [Changed] `pipeline.py` and project LCSC resolve import `datasheet_extract`.
|
||||
- [Changed] Extract stages pin the DeepSeek provider.
|
||||
|
||||
## 2.39.1 — 2026-09-20 — src no longer imports PinScope review loop
|
||||
|
||||
After 2.39.0 live smoke (Emmaforo ERROR 2 / WARNING 12 / INFO 18), native modules are the only review path from `periscope/src`. PinScope `validate.py` / `validation_tools.py` remain on disk.
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,6 +1,6 @@
|
||||
# Piano — indipendenza architettonica e di licenza da PinScope
|
||||
|
||||
**Stato:** split **2.38.0**. C2 native review loop **2.39.0** live. **2.39.1** `periscope/src` non importa più `validate.py` / `validation_tools.py` (PCB incluso). File PinScope restano in `dependency/`. Fork non staccato.
|
||||
**Stato:** split **2.38.0**. C2 review **2.39.0/2.39.1** live. C3 extraction **2.40.0**. `validate.py` / `extraction.py` restano in `dependency/` (non chiamati dal live path). Fork non staccato.
|
||||
**Gate Michele:** sostituire/smettere di chiamare un modulo `dependency/` solo dopo pytest + deploy smoke. Se la verifica fallisce, resta il path ereditato.
|
||||
**Sequenza:** split → sostituzione incrementale (C2 loop → C3 extraction → C4 PCB off `validate.py`). **Mai** empty-delete. Parsers/graph (C5/E) e auto-place fuori scope. AGPL resta.
|
||||
|
||||
@@ -225,7 +225,7 @@ Qui sì si riscrive l’engine ereditato. Ordine interno:
|
||||
| C0 | Spec freeze `motore-finding.md` = unico IR | Già nativo; non diluire in `validate.py` |
|
||||
| C1 | Adapter: `validate.py` emette solo `Finding` grezzi → `complete_finding` | Già parziale; chiudere i campi doppi |
|
||||
| C2 | **REWRITE** loop per-IC native in `periscope/src` (DeepSeek, tools su `DesignGraph`) | **Shipped 2.39.0** live `review_ic_async` → `review_session`; PinScope files **kept**, not called from live loop |
|
||||
| C3 | Extraction: `local_skill.py` + schemi JSON (KEEP schema se identici; REWRITE orchestrazione Anthropic) | |
|
||||
| C3 | Extraction: `local_skill.py` + schemi JSON (KEEP schema se identici; REWRITE orchestrazione Anthropic) | **Shipped 2.40.0** live `datasheet_extract`; PinScope `extraction.py` kept |
|
||||
| C4 | Spegnere import da `validate.py` nel PCB (`_parse_review` → parser finding nativo) | **Shipped 2.39.1** `pcb_validation` → `review_parse` / `review_session` |
|
||||
| C5 | Test golden `simple_project` + Emmaforo: parity FACT/REQUIREMENT, non parity prose | |
|
||||
|
||||
|
||||
@@ -0,0 +1,31 @@
|
||||
"""Native datasheet_extract vs inherited extraction — prove before switching pipeline."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import ast
|
||||
from pathlib import Path
|
||||
|
||||
from backend.services.datasheet_extract import _coerce_abs_max
|
||||
from backend.services.extraction import _coerce_abs_max as inherited_coerce
|
||||
|
||||
|
||||
def test_coerce_abs_max_matches_inherited():
|
||||
raw = [
|
||||
{"parameter": "Vin", "min": 0, "max": 6, "unit": "V", "source_page": 4},
|
||||
"not-a-dict",
|
||||
]
|
||||
assert _coerce_abs_max(raw) == inherited_coerce(raw)
|
||||
assert _coerce_abs_max(None) == inherited_coerce(None)
|
||||
|
||||
|
||||
def test_native_extract_has_no_anthropic_import():
|
||||
path = Path(__import__("backend.services.datasheet_extract", fromlist=["x"]).__file__)
|
||||
tree = ast.parse(path.read_text())
|
||||
imported = []
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.ImportFrom) and node.module:
|
||||
imported.append(node.module)
|
||||
elif isinstance(node, ast.Import):
|
||||
imported.extend(a.name for a in node.names)
|
||||
assert not any("anthropic" in (m or "") for m in imported)
|
||||
assert "backend.services.llm.factory" in imported
|
||||
@@ -86,12 +86,12 @@ def test_auto_resolve_skips_llm_when_catalog_parses(monkeypatch):
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
|
||||
from backend.services.extraction import auto_resolve_specs
|
||||
from backend.services.datasheet_extract import auto_resolve_specs
|
||||
|
||||
async def boom(*_a, **_k):
|
||||
raise AssertionError("LLM must not run")
|
||||
|
||||
monkeypatch.setattr("backend.services.extraction.call_with_fallback", boom)
|
||||
monkeypatch.setattr("backend.services.datasheet_extract.call_with_fallback", boom)
|
||||
tax = TAXONOMY
|
||||
model = asyncio.run(
|
||||
auto_resolve_specs(
|
||||
@@ -110,12 +110,12 @@ def test_auto_resolve_use_llm_false_on_ferrite(monkeypatch):
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
|
||||
from backend.services.extraction import auto_resolve_specs
|
||||
from backend.services.datasheet_extract import auto_resolve_specs
|
||||
|
||||
async def boom(*_a, **_k):
|
||||
raise AssertionError("LLM must not run")
|
||||
|
||||
monkeypatch.setattr("backend.services.extraction.call_with_fallback", boom)
|
||||
monkeypatch.setattr("backend.services.datasheet_extract.call_with_fallback", boom)
|
||||
tax = TAXONOMY
|
||||
model = asyncio.run(
|
||||
auto_resolve_specs(
|
||||
|
||||
@@ -52,12 +52,12 @@ def test_ambiguous_string_returns_none():
|
||||
|
||||
|
||||
def test_resolve_from_value_skips_llm_when_parseable(monkeypatch):
|
||||
from backend.services.extraction import resolve_from_value
|
||||
from backend.services.datasheet_extract import resolve_from_value
|
||||
|
||||
async def boom(*_a, **_k):
|
||||
raise AssertionError("LLM must not run")
|
||||
|
||||
monkeypatch.setattr("backend.services.extraction.call_with_fallback", boom)
|
||||
monkeypatch.setattr("backend.services.datasheet_extract.call_with_fallback", boom)
|
||||
tax = TAXONOMY
|
||||
model = asyncio.run(
|
||||
resolve_from_value(
|
||||
@@ -71,12 +71,12 @@ def test_resolve_from_value_skips_llm_when_parseable(monkeypatch):
|
||||
|
||||
|
||||
def test_resolve_from_value_placeholder_no_llm(monkeypatch):
|
||||
from backend.services.extraction import resolve_from_value
|
||||
from backend.services.datasheet_extract import resolve_from_value
|
||||
|
||||
async def boom(*_a, **_k):
|
||||
raise AssertionError("LLM must not run")
|
||||
|
||||
monkeypatch.setattr("backend.services.extraction.call_with_fallback", boom)
|
||||
monkeypatch.setattr("backend.services.datasheet_extract.call_with_fallback", boom)
|
||||
tax = TAXONOMY
|
||||
with pytest.raises(ValueError, match="Placeholder"):
|
||||
asyncio.run(
|
||||
|
||||
@@ -4,7 +4,7 @@ from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from backend.services.extraction import _coerce_abs_max
|
||||
from backend.services.datasheet_extract import _coerce_abs_max
|
||||
from backend.services.llm.pdf_ingest import (
|
||||
extract_pdf_text,
|
||||
make_text_pdf,
|
||||
|
||||
@@ -452,7 +452,7 @@ async def test_lcsc_resolve_passive_success_and_cached(tmp_path, monkeypatch):
|
||||
from backend.config import settings
|
||||
from backend.periscopex.models import CapacitorSpecs, ComponentModel
|
||||
from backend.services import purple_parts
|
||||
from backend.services import extraction as extraction_svc
|
||||
from backend.services import datasheet_extract as extraction_svc
|
||||
|
||||
# Billing modules are absent in the open-source repo; the endpoint under
|
||||
# test only charges when billing is enabled, so skip there.
|
||||
|
||||
Reference in New Issue
Block a user