michele/DigiRadio
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Fix Si4684 pitch distortion: uncalibrated crystal (CTUN/XTAL_FREQ)

Browse Source 
Root cause of FM/DAB "stonata" audio: xtalCtun=31/nominal XTAL_FREQ
defaults were never measured against this board's actual crystal
(Abracon ABM8-19.200MHZ-10-1-U-T, CL=10pF, two external 15pF load
caps). Fixed via CTUN trim by ear (31->0) plus a new live calibration
loop that reads the Si4684's own FM_RSQ FREQOFF field (broadcast
carriers are GPS-locked, so it's a free precision frequency reference)
to trim XTAL_FREQ with no lab equipment. Converged to CTUN=0,
XTAL_FREQ=19199750 Hz, residual -3/-4 ppm cross-checked on two
stations.

New tools/infrastructure (kept, not one-off):
- Si4684Driver::recalibrateXtal() + POST /api/tuner/xtal-calibrate:
  live crystal re-trim without an ESP32 reflash.
- FM_RSQ FREQOFF exposed as "freqoff_ppm" in GET /api/tuner/status.
- tools/si4684_xtal_calibration.py: automates the trim loop.
- tools/si4684_antenna_calibration.py: AN851 Appendix A ANTCAP/VARM/VARB
  sweep tool (same session, separate calibration).
- CONFIG_ESP32_I2S_TEST_TONE (off by default): isolates Si4684-specific
  audio issues from shared-downstream ones by writing a tone directly
  over the I2S bus the Si4684 also uses.
- SIGMA_WRITE_REGISTER_BLOCK/sigma_safeload_block now retry on NACK and
  verify via read-back instead of firing I2C writes blind.
- FM de-emphasis set to European 50us (was left at the US 75us default).
- DAB_ACF_ENABLE restored to its previous 0x0000 with a citation
  explaining why (tested the datasheet default of 0x0003, made things
  audibly worse).

See docs/si4684-rf-investigation-report.md's 2026-08-23 entry for the
full elimination chain and docs/TODO.md for follow-up work (persisting
calibration results to EEPROM instead of requiring a firmware edit).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
michele
2026-08-23 20:30:42 +02:00
co-authored by Claude Sonnet 5
parent a91c63fc31
commit fae2305164
24 changed files with 1309 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Software/tools/si4684_antenna_calibration.py
+341
View File
@@ -0,0 +1,341 @@
#!/usr/bin/env python3
"""si4684_antenna_calibration.py — AN851 Appendix A varactor-tuning sweep.
DigiRadio firmware — https://github.com/manvalan/DigiRadio
Copyright 2026 Michele Bigi
SPDX-License-Identifier: Apache-2.0
Runs the Si4684 antenna varactor calibration procedure described in AN851
("Si468x AM/AMHD-FM/FMHD-DAB/DAB+ antenna/matching network design
guidelines"), Appendix A, driving the device's existing HTTP tuner API
instead of a bench Test_Get_RSSI command:
1. At each of several test frequencies, sweep ANTCAP from --antcap-min
to --antcap-max (AN851: 1-128) via POST /api/tuner/tune, averaging
--samples (AN851: 5) reads per step via GET /api/tuner/status.
2. Pick the ANTCAP with the best average reading at each frequency.
3. Linear-fit best_antcap(frequency_MHz) = (m/1000)*frequency_MHz + b,
AN851's own formula, giving VARM (m, property 0x1710) and VARB
(b, property 0x1711) as signed 16-bit integers.
Scope and limits (read before running):
* This firmware's HTTP API has no endpoint to write VARM/VARB directly
(see components/net/src/SetupWebServer.cpp) -- only a single fixed
ANTCAP override can be persisted, via POST /api/tuner/calibrate-antenna.
This script only COMPUTES the fit and prints it; it does not write
anything to the device. Applying the result means hand-editing the
kFmTuneFeVarm/kFmTuneFeVarb (or the DAB kDabProps table) constants in
components/drivers/si4684/src/Si4684Driver.cpp and reflashing.
* DAB caveat: DAB_TUNE_FREQ (AN649 Command 0xB0) addresses frequencies by
an index into a chip-side table (default "European frequency list")
loaded via DAB_SET_FREQ_LIST. AN649's copy in this repo does not publish
that table's contents, and neither the firmware nor its HTTP API expose
the real MHz for a given freq_index. This script does NOT assume any
index<->MHz mapping -- for --band dab you must supply the real
frequency for each index yourself (e.g. read off a known ensemble's
published transmitter frequency). Do not guess; a wrong MHz value here
silently produces a wrong VARM/VARB fit.
* DAB metric caveat: AN851 Appendix A calls for RSSI at each step, but
this firmware's DAB status JSON does not expose RSSI (only
dab.fic_quality and dab.cnr_db -- see core/TunerStatus.hpp). This
script uses dab.cnr_db as the optimization metric for --band dab
instead. That is a deliberate substitution, not a datasheet value --
treat DAB fit results with more skepticism than FM ones.
* Known dead zone (docs/si4684-rf-investigation-report.md, 2026-08-20):
DAB freq_index 22 lost lock entirely at antcap 72 and 80 on this board.
This script does not skip any antcap value in range; a step with no
lock is logged as "NO LOCK" and simply excluded from that point's
average, not treated as a fatal error.
Usage:
python3 tools/si4684_antenna_calibration.py --host digiradio-XXXXXX.local \\
--band fm --point 88500 --point 98000 --point 107900
python3 tools/si4684_antenna_calibration.py --host 192.168.1.56 \\
--band dab --point 5:174928 --point 12:198160 --point 23:225648 \\
--raw-csv dab_sweep.csv
"""
from __future__ import annotations
import argparse
import csv
import json
import socket
import statistics
import sys
import time
import urllib.error
import urllib.request
from pathlib import Path
from typing import Optional
def parse_point(band: str, raw: str) -> dict:
"""Parse one --point argument into a tune target for the given band."""
if band == "dab":
if ":" not in raw:
raise ValueError(
"DAB points must be 'freq_index:frequency_khz', e.g. 12:198160"
)
idx_s, khz_s = raw.split(":", 1)
idx = int(idx_s)
khz = int(khz_s)
if not 0 <= idx <= 37:
raise ValueError("freq_index must be 0-37 (core/TunerJson.cpp limit)")
if khz <= 0:
raise ValueError("frequency_khz must be positive")
return {
"index": idx,
"khz": khz,
"mhz": khz / 1000.0,
"label": f"idx{idx}@{khz / 1000:.3f}MHz",
}
khz = int(raw)
if not 87500 <= khz <= 108000:
raise ValueError("FM frequency_khz should be within 87500-108000")
return {"khz": khz, "mhz": khz / 1000.0, "label": f"{khz / 1000:.3f}MHz"}
def tune_payload(band: str, point: dict, antcap: int) -> dict:
if band == "dab":
return {"band": "dab", "freq_index": point["index"], "antcap": antcap}
return {"band": "fm", "frequency_khz": point["khz"], "antcap": antcap}
def extract_metric(status: dict, band: str) -> Optional[float]:
"""AN851's optimization metric: RSSI for FM, CNR for DAB (see docstring)."""
if not status.get("locked", False):
return None
if band == "fm":
fm = status.get("fm") or {}
rssi = fm.get("rssi_dbuv")
return float(rssi) if rssi is not None else None
dab = status.get("dab") or {}
cnr = dab.get("cnr_db")
return float(cnr) if cnr is not None else None
def http_request(
host: str,
method: str,
path: str,
payload: Optional[dict],
timeout: float,
retries: int,
) -> dict:
url = f"http://{host}{path}"
# The firmware's JSON parser (components/core/src/TunerJson.cpp) does
# plain substring search for e.g. `"key":` and is not whitespace-
# tolerant -- json.dumps()'s default ": "/", " separators break every
# request with "invalid_json". Compact separators match what curl's
# -d with no spaces sends, which the firmware does accept.
data = json.dumps(payload, separators=(",", ":")).encode("utf-8") if payload is not None else None
headers = {"Content-Type": "application/json"} if data is not None else {}
last_err: Optional[BaseException] = None
for attempt in range(retries + 1):
try:
req = urllib.request.Request(url, data=data, method=method, headers=headers)
with urllib.request.urlopen(req, timeout=timeout) as resp:
body = resp.read()
return json.loads(body) if body else {}
except (urllib.error.URLError, socket.timeout, TimeoutError) as exc:
last_err = exc
if attempt < retries:
time.sleep(1.0 + attempt) # device is known to stall briefly under HTTP load
raise RuntimeError(f"{method} {path} failed after {retries + 1} attempt(s): {last_err}")
def sweep_point(
host: str,
band: str,
point: dict,
antcap_values: list,
samples: int,
settle_s: float,
sample_gap_s: float,
timeout: float,
retries: int,
raw_rows: Optional[list],
) -> list:
results = []
for antcap in antcap_values:
try:
status = http_request(
host, "POST", "/api/tuner/tune", tune_payload(band, point, antcap), timeout, retries
)
except RuntimeError as exc:
print(f" antcap={antcap:3d} tune failed: {exc}", file=sys.stderr)
results.append((antcap, None))
continue
time.sleep(settle_s)
readings = []
first = extract_metric(status, band)
if first is not None:
readings.append(first)
for _ in range(max(0, samples - 1)):
time.sleep(sample_gap_s)
try:
status = http_request(host, "GET", "/api/tuner/status", None, timeout, retries)
except RuntimeError as exc:
print(f" antcap={antcap:3d} status read failed: {exc}", file=sys.stderr)
continue
reading = extract_metric(status, band)
if reading is not None:
readings.append(reading)
if raw_rows is not None:
for i, reading in enumerate(readings):
raw_rows.append(
{"point": point["label"], "antcap": antcap, "sample": i, "metric": reading}
)
avg = statistics.mean(readings) if readings else None
state = "locked" if readings else "NO LOCK"
avg_str = f"{avg:6.2f}" if avg is not None else " - "
print(f" antcap={antcap:3d} samples={len(readings)}/{samples} avg={avg_str} [{state}]")
results.append((antcap, avg))
return results
def linear_fit(xs: list, ys: list) -> Optional[tuple]:
n = len(xs)
if n < 2:
return None
mean_x = sum(xs) / n
mean_y = sum(ys) / n
num = sum((x - mean_x) * (y - mean_y) for x, y in zip(xs, ys))
den = sum((x - mean_x) ** 2 for x in xs)
if den == 0:
return None
slope = num / den
intercept = mean_y - slope * mean_x
return slope, intercept
def main() -> int:
parser = argparse.ArgumentParser(
description="AN851 Appendix A ANTCAP sweep + VARM/VARB linear fit over the tuner HTTP API.",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=__doc__,
)
parser.add_argument("--host", required=True, help="Device IP or mDNS host, e.g. digiradio-XXXXXX.local")
parser.add_argument("--band", required=True, choices=["fm", "dab"])
parser.add_argument(
"--point",
action="append",
required=True,
metavar="FREQ_KHZ|INDEX:FREQ_KHZ",
help="FM: frequency in kHz (e.g. 98000). DAB: freq_index:frequency_khz "
"(e.g. 12:198160) -- see script docstring for why the MHz value must "
"come from you, not this script.",
)
parser.add_argument("--antcap-min", type=int, default=1)
parser.add_argument("--antcap-max", type=int, default=128)
parser.add_argument("--antcap-step", type=int, default=1)
parser.add_argument(
"--samples", type=int, default=5, help="Reads averaged per ANTCAP step (AN851 Appendix A: 5)"
)
parser.add_argument(
"--settle-ms",
type=int,
default=150,
help="Delay after tune before the first reading (engineering margin, not from AN851)",
)
parser.add_argument("--sample-gap-ms", type=int, default=80, help="Delay between repeated status reads")
parser.add_argument("--timeout", type=float, default=10.0, help="HTTP timeout per request, seconds")
parser.add_argument(
"--retries", type=int, default=2, help="Retries per request (device is known to stall briefly)"
)
parser.add_argument("--raw-csv", type=Path, default=None, help="Optional path to dump every raw sample")
args = parser.parse_args()
if not 1 <= args.antcap_min <= args.antcap_max <= 128:
print("error: antcap range must be within 1-128 (0 = auto, not part of the sweep)", file=sys.stderr)
return 1
points = []
for raw in args.point:
try:
points.append(parse_point(args.band, raw))
except ValueError as exc:
print(f"error: invalid --point {raw!r}: {exc}", file=sys.stderr)
return 1
antcap_values = list(range(args.antcap_min, args.antcap_max + 1, args.antcap_step))
raw_rows: Optional[list] = [] if args.raw_csv else None
fit_points = []
for point in points:
print(f"\n=== sweeping {args.band} @ {point['label']} ===")
results = sweep_point(
args.host,
args.band,
point,
antcap_values,
args.samples,
args.settle_ms / 1000.0,
args.sample_gap_ms / 1000.0,
args.timeout,
args.retries,
raw_rows,
)
valid = [(a, v) for a, v in results if v is not None]
if not valid:
print(f" WARNING: no locked reading anywhere -- excluding {point['label']} from the fit")
continue
best_antcap, best_val = max(valid, key=lambda t: t[1])
print(f" best: antcap={best_antcap} ({best_val:.2f})")
fit_points.append((point["mhz"], best_antcap, point["label"]))
if args.raw_csv:
with args.raw_csv.open("w", newline="") as f:
writer = csv.DictWriter(f, fieldnames=["point", "antcap", "sample", "metric"])
writer.writeheader()
writer.writerows(raw_rows)
print(f"\nraw samples written to {args.raw_csv}")
print(f"\n=== AN851 Appendix A fit (band={args.band}) ===")
if len(fit_points) < 2:
print(
"Not enough locked points to fit VARM/VARB (need >= 2 usable frequencies). "
"Try different --point values or check the antenna/signal."
)
return 1
xs = [p[0] for p in fit_points]
ys = [p[1] for p in fit_points]
fit = linear_fit(xs, ys)
if fit is None:
print("Fit failed (degenerate frequency set -- all points at the same MHz?).")
return 1
slope, intercept = fit
m = round(slope * 1000)
b = round(intercept)
m_ok = -32768 <= m <= 32767
b_ok = -32768 <= b <= 32767
print(" points used: " + ", ".join(f"{label} -> antcap {a}" for _, a, label in fit_points))
print(" varactor_value = (m/1000)*frequency_MHz + b [AN851 Appendix A]")
print(f" m (slope x1000, property 0x1710) = {m}" + ("" if m_ok else " ** OUT OF int16 RANGE **"))
print(f" b (intercept, property 0x1711) = {b}" + ("" if b_ok else " ** OUT OF int16 RANGE **"))
if m_ok and b_ok:
print(f" -> 0x1710 = 0x{m & 0xFFFF:04X} 0x1711 = 0x{b & 0xFFFF:04X}")
print()
print(" Not written to the device. To apply: edit the k{Fm,Dab}TuneFeVarm/")
print(" k{Fm,Dab}TuneFeVarb constants in components/drivers/si4684/src/Si4684Driver.cpp")
print(" (current values and AN851 Appendix B citation are next to them), reflash, then")
print(" re-run this sweep with antcap=0 (auto) at the same points to confirm auto-tune")
print(" now tracks the measured optimum.")
return 0
if __name__ == "__main__":
raise SystemExit(main())
Software/tools/si4684_xtal_calibration.py
+184
View File
@@ -0,0 +1,184 @@
#!/usr/bin/env python3
"""si4684_xtal_calibration.py — live Si4684 crystal trim via FM_RSQ FREQOFF.
DigiRadio firmware — https://github.com/manvalan/DigiRadio
Copyright 2026 Michele Bigi
SPDX-License-Identifier: Apache-2.0
Uses the Si4684's own FM_RSQ_STATUS FREQOFF field (AN649 Command 0x32,
RESP8, signed offset in units of 2 PPM) as a precision frequency reference
-- real FM broadcast transmitters are GPS/rubidium-locked, so a locked
station's carrier reads the same PPM error as the receiver's own crystal
reference error, no lab equipment required.
Procedure (matches AN649 §9.3's own recommendation to trim by measurement,
just automated over HTTP instead of by ear):
1. Tune to a strong, stable FM station.
2. Read GET /api/tuner/status -> fm.freqoff_ppm.
3. new_xtal_freq_hz = 19200000 * (1 + ppm/1e6)
4. POST /api/tuner/xtal-calibrate {"xtal_freq_hz": new_xtal_freq_hz}
(this reboots the Si4684, no ESP32 restart, no persistence yet).
5. Re-tune, re-read freqoff_ppm. Repeat until it converges near 0.
6. Optionally cross-check on a second station at the other end of the
band -- if the residual offset differs systematically between the two,
the error isn't (only) the crystal; don't chase it further with this
tool.
Nothing here is persisted to flash. Once you have a converged xtal_freq_hz,
hand it to a human/Claude to bake into the Si4684Driver::boot() default
call in main/hardware_bootstrap.cpp -- this script only calibrates the
currently-running session.
Usage:
python3 tools/si4684_xtal_calibration.py --host 192.168.1.62 \\
--frequency-khz 87600
python3 tools/si4684_xtal_calibration.py --host 192.168.1.62 \\
--frequency-khz 87600 --once # single read, no correction loop
"""
from __future__ import annotations
import argparse
import json
import sys
import time
import urllib.error
import urllib.request
from typing import Optional
def http_request(host: str, method: str, path: str, payload: Optional[dict],
timeout: float) -> dict:
url = f"http://{host}{path}"
# Compact separators: the firmware's hand-rolled JSON parser is not
# whitespace-tolerant (confirmed bug in components/core/src/TunerJson.cpp).
data = json.dumps(payload, separators=(",", ":")).encode() if payload is not None else None
headers = {"Content-Type": "application/json"} if data is not None else {}
req = urllib.request.Request(url, data=data, method=method, headers=headers)
with urllib.request.urlopen(req, timeout=timeout) as resp:
body = resp.read()
return json.loads(body) if body else {}
def tune_fm(host: str, frequency_khz: int, timeout: float) -> dict:
return http_request(host, "POST", "/api/tuner/tune",
{"band": "fm", "frequency_khz": frequency_khz}, timeout)
def read_status(host: str, timeout: float) -> dict:
return http_request(host, "GET", "/api/tuner/status", None, timeout)
def recalibrate(host: str, xtal_freq_hz: int, ibias: int, ctun: int,
timeout: float) -> dict:
return http_request(
host, "POST", "/api/tuner/xtal-calibrate",
{"xtal_freq_hz": xtal_freq_hz, "ibias": ibias, "ctun": ctun}, timeout)
def main() -> int:
parser = argparse.ArgumentParser(
description="Trim Si4684 XTAL_FREQ live using FM_RSQ_STATUS FREQOFF "
"-- no lab equipment, uses locked broadcast carriers as reference.")
parser.add_argument("--host", required=True, help="Device IP or mDNS host")
parser.add_argument("--frequency-khz", type=int, required=True,
help="A strong, stable FM station to lock onto, e.g. 87600")
parser.add_argument("--ibias", type=int, default=72, help="POWER_UP IBIAS (0-127)")
parser.add_argument("--ctun", type=int, default=0,
help="POWER_UP CTUN (0-63) -- default 0, the best value "
"found by ear on 2026-08-23; leave alone unless you have "
"a reason to also move it")
parser.add_argument("--start-xtal-freq-hz", type=int, default=19200000,
help="Starting XTAL_FREQ before the first correction")
parser.add_argument("--max-iterations", type=int, default=6)
parser.add_argument("--converge-ppm", type=float, default=1.0,
help="Stop once |freqoff_ppm| is under this many ppm")
parser.add_argument("--settle-s", type=float, default=1.0,
help="Delay after tune/recalibrate before reading status")
parser.add_argument("--samples", type=int, default=5,
help="FREQOFF reads averaged per iteration (reduces "
"reception-noise jitter in the ppm estimate)")
parser.add_argument("--sample-gap-s", type=float, default=0.3)
parser.add_argument("--damping", type=float, default=0.6,
help="Fraction of the measured ppm correction applied "
"per iteration (<1.0 avoids overshoot/oscillation "
"around the true value)")
parser.add_argument("--timeout", type=float, default=10.0)
parser.add_argument("--once", action="store_true",
help="Single read only, no correction loop")
args = parser.parse_args()
xtal_freq_hz = args.start_xtal_freq_hz
for iteration in range(1 if args.once else args.max_iterations):
try:
recalibrate(args.host, xtal_freq_hz, args.ibias, args.ctun, args.timeout)
except (urllib.error.URLError, OSError) as exc:
print(f"error: xtal-calibrate failed: {exc}", file=sys.stderr)
return 1
time.sleep(args.settle_s)
try:
status = tune_fm(args.host, args.frequency_khz, args.timeout)
except (urllib.error.URLError, OSError) as exc:
print(f"error: tune failed: {exc}", file=sys.stderr)
return 1
time.sleep(args.settle_s)
samples = []
for s in range(args.samples):
if s > 0:
time.sleep(args.sample_gap_s)
try:
status = read_status(args.host, args.timeout)
except (urllib.error.URLError, OSError) as exc:
print(f"error: status read failed: {exc}", file=sys.stderr)
return 1
fm = status.get("fm") or {}
if status.get("locked", False) and fm.get("freqoff_ppm") is not None:
samples.append((fm["freqoff_ppm"], fm.get("rssi_dbuv"), fm.get("snr_db")))
if not samples:
print(" no lock / no FREQOFF reading across all samples -- pick "
"a stronger station and retry", file=sys.stderr)
return 1
ppm_avg = sum(s[0] for s in samples) / len(samples)
rssi = samples[-1][1]
snr = samples[-1][2]
print(f"iter {iteration}: xtal_freq_hz={xtal_freq_hz} "
f"samples={len(samples)}/{args.samples} rssi={rssi} snr={snr} "
f"freqoff_ppm_avg={ppm_avg:.1f} "
f"raw={[s[0] for s in samples]}")
if args.once:
return 0
if abs(ppm_avg) <= args.converge_ppm:
print(f"\nConverged: xtal_freq_hz={xtal_freq_hz} "
f"(residual {ppm_avg:.1f} ppm avg, ibias={args.ibias}, "
f"ctun={args.ctun})")
print("Not persisted -- to make this the boot default, edit the "
"gSi4684.boot(...) call in main/hardware_bootstrap.cpp.")
return 0
# Empirically determined 2026-08-23: correcting in the "+ppm"
# direction diverges (each iteration made freqoff_ppm larger, not
# smaller). The correct sign is "-ppm" -- confirmed by a live A/B
# test, not derived from AN649 (which doesn't specify the relation
# between XTAL_FREQ and FREQOFF's sign convention). Damping avoids
# overshoot from single-reading reception noise.
xtal_freq_hz = round(xtal_freq_hz * (1.0 - args.damping * ppm_avg / 1e6))
print(f"\nDid not converge within {args.max_iterations} iterations "
f"(last xtal_freq_hz={xtal_freq_hz}). Try again, or check the "
f"second-station cross-check described in this script's docstring "
f"-- a residual that varies with frequency isn't the crystal.")
return 1
if __name__ == "__main__":
raise SystemExit(main())