From f6dc9916a48168de11ab415a4d89b2d2dda409d8 Mon Sep 17 00:00:00 2001 From: Michele Bigi Date: Fri, 21 Aug 2026 09:32:07 +0200 Subject: [PATCH] Add BT1035 background boot retry; document intermittent silence findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit BT1035 boot can fail with total UART silence even after the banner-timing fix, on otherwise-identical, correctly-powered hardware. Confirmed via multimeter (VBAT_IN, 1.8V_OUT, SYS_CTRL/RESET, TX all normal) and by observing the same physical module both succeed and fail across different boot attempts in one session, that this is intermittent, not a dead module — the crystal is sealed inside the module and not inspectable or fixable from our side. Since the fault self-clears on a later attempt, mitigate with an indefinite background retry task: if the initial boot() fails, keep retrying with no artificial delay (each attempt already takes ~25-60s) so a temporary failure becomes a bounded, self-recovering delay instead of requiring a manual power cycle. Documented the full diagnostic session, including the ruled-out theories and a possible future ESP32-S31 (native Bluetooth Classic) migration path, in the RF investigation report. Co-Authored-By: Claude Sonnet 5 --- Software/docs/TODO.md | 32 ++++- .../docs/si4684-rf-investigation-report.md | 116 ++++++++++++++++++ Software/main/hardware_bootstrap.cpp | 73 +++++++++-- 3 files changed, 209 insertions(+), 12 deletions(-) diff --git a/Software/docs/TODO.md b/Software/docs/TODO.md index 7a30580..5c82b0b 100644 --- a/Software/docs/TODO.md +++ b/Software/docs/TODO.md @@ -105,10 +105,40 @@ Short version: independently verified correct with a multimeter first — the module and PCB were never at fault. Fixed by waiting up to 25s for the banner (`kBootBannerWaitMs`); boot now succeeds on the first attempt. +- **BT1035 — a second, harder failure mode confirmed intermittent, not + hardware (2026-08-21).** Distinct from the banner-timing bug above: even + with the 25s wait already in place, boot sometimes still gets zero UART + bytes at all — no banner, no AT response, silent across all 8 probed + baud rates (9600-921600). Root-cause evidence this session: VBAT_IN + (3.3V), 1.8V_OUT (1.8V), SYS_CTRL/RESET (~3.27V, matching the firmware's + own GPIO readback), and BT1035 TX (idle-HIGH ~3.29V, no short/float) all + measured normal with a multimeter. The BT1035's 32 MHz crystal is + integrated inside the sealed Feasycom module (confirmed via the module's + own datasheet block diagram — no external crystal on our schematic), so + it can't be inspected or reworked from our side; a marginal + oscillator-startup margin inside the module is the leading suspect. + **Decisive evidence it's intermittent, not a dead unit**: the exact same + physical module booted cleanly (banner + all AT commands `OK`) on one + attempt and went totally silent on the very next attempt, no physical + changes in between. A replacement module is therefore not a guaranteed + fix — the same defect class could recur on a different unit. Mitigated + (not fixed) by an indefinite background retry task + (`hardware::bt1035RetryTask` in `main/hardware_bootstrap.cpp`): if the + initial `Bt1035Driver::boot()` fails, a background FreeRTOS task keeps + calling `boot()` again with no artificial delay between attempts (each + attempt already takes ~25-60s on its own) until it succeeds, while the + rest of the system (tuner, Wi-Fi, web UI) stays fully usable in the + meantime. Turns a permanent-until-manual-power-cycle failure into a + bounded, self-recovering delay. See + `docs/si4684-rf-investigation-report.md` (2026-08-21 entry) for the full + session narrative, including a UART TX/RX loopback test attempt that was + inconclusive (bridging the ESP32's own TX/RX pins from cold boot caused + an unrelated, reproducible, harmless early-boot hang, not yet explained). - **Still open**: intermittent multi-second HTTP unresponsiveness under load; DAB signal quality still antenna-limited; 24 KB `nvs` partition may be undersized (`saveProfile()` `store_failed` seen intermittently, - error code never captured). + error code never captured); BT1035 intermittent total-silence boot + failures (mitigated via background retry, not root-caused — see above). --- diff --git a/Software/docs/si4684-rf-investigation-report.md b/Software/docs/si4684-rf-investigation-report.md index 997cb90..59bc7ef 100644 --- a/Software/docs/si4684-rf-investigation-report.md +++ b/Software/docs/si4684-rf-investigation-report.md @@ -792,3 +792,119 @@ identified. `/api/bluetooth/status` and `/api/bluetooth/paired` correctly report `{"status":"error","reason":"at_timeout"}` while in this state; the rest of the device (tuner, web UI) stays usable per the existing non-fatal-BT1035-boot design. + +## 2026-08-21 update: BT1035 total silence confirmed intermittent (not +## hardware); background retry mitigation added + +Follow-up session dedicated entirely to the "total UART silence" BT1035 +failure mode above. Summary: **confirmed intermittent on genuinely +identical hardware, root cause narrowed to the module's internal crystal +(not our PCB, not fixable by us), and mitigated (not fixed) with an +indefinite background boot retry.** + +**Diagnostic instrumentation (temporary, added then reverted this +session)**: added `BT1035 AT TX: ` / `BT1035 UART RX RAW: >` / `BT1035 AT RESULT: OK|ERROR|TIMEOUT` logging around +`Bt1035Driver::transmitAndCollect()`, and temporarily dropped +`kBootAttempts` to 1 for single-attempt clarity. This confirmed the +failure signature precisely: `AT` is transmitted, zero bytes ever come +back (``), timeout. Reverted via `git checkout` once the manual +diagnosis was done — not kept in the codebase. + +**Multimeter checks, all normal** (scope-level checks — crystal +oscillation, power-on transient — remain out of reach without an +oscilloscope): +- VBAT_IN: 3.3V (datasheet range 3.0-4.2V) ✓ +- 1.8V_OUT (module's internal regulator): 1.8V ✓ — proves the module's + own power management *is* running, it isn't simply unpowered +- SYS_CTRL / RESET (post-boot): ~3.27V, matching the firmware's own GPIO + readback log (`post-reset: SYS_CTRL=1 RESET=1`) ✓ +- BT1035 TX pin (module side) to GND: 3.29V, idle-HIGH, no short/float/ + reversed polarity ✓ (though idle-HIGH alone doesn't prove the module's + firmware is executing — some pads default HIGH from reset state alone) +- A 10kΩ pull-down the user had added on SYS_CTRL (matching the + datasheet's own recommendation for an undriven pin) was checked and is + not the cause — the ESP32 GPIO drives push-pull and its own readback + confirms it reaches a valid HIGH regardless. + +**Crystal location determined**: the BT1035 datasheet's own block diagram +shows "32MHz Crystal" as an internal block of the QCC3056 die, and the +DigiRadio schematic netlist (`Netlist_Schematic1_2026-08-07.asc`) has no +XTAL_IN/XTAL_OUT pins wired to any external crystal for U11 — confirming +the oscillator is sealed inside the Feasycom module, not on our PCB. This +is why nothing on our side (layout, load caps, our firmware) can affect +it; if the failure really is a marginal oscillator-startup margin, it's a +property of that specific physical module unit (or the part's design +tolerance in general). + +**Decisive evidence of intermittency, not a dead unit**: across repeated +reboots in the same session (physical power-cycles and serial-port-open +resets, which also hard-reset this ESP32-S3's native USB-CDC), the +identical physical module was observed to boot **completely successfully** +at least once — spontaneous banner `+VER=FSC-BT1035,V6.1.1,20240521` + +`+DEVSTAT=1`, then `AT` and `AT+AUXCFG=3` both answered `OK` — and to fail +completely silently on other attempts, with no physical change in between. +This rules out "defective/dead module" as an explanation; ordering a +replacement module is therefore *not* a guaranteed fix, since the same +physical unit demonstrably works when it works. + +**UART loopback test attempt — inconclusive, logged for future +reference.** Tried to isolate ESP32 vs. module by bridging the ESP32-S3's +own GPIO40 (BT1035 UART TX)/GPIO41 (BT1035 UART RX) pins with a jumper +held by hand on the ESP32 module's castellated pads (no series +resistor/test point exists on this net per the schematic netlist — U8.33 +↔ U11.P$14 and U8.34 ↔ U11.P$13 directly, nothing else). Twice +reproducibly, bridging those pins from cold boot caused the ESP32 itself +to hang very early in boot (right after the bootloader's "Disabling RNG +early entropy source" line, before `app_main()` even starts) — harmless +(board recovers fully once the jumper is removed) but unexplained, and it +sidesteps the actual test rather than answering it. Not pursued further +this session given the practical difficulty of hand-holding a wire onto +castellated pads without a proper SMD test hook. If retried: attach the +jumper *after* the ESP32 has already booted past that early stage (there's +a ~25s window before the BT1035 AT command is actually sent) rather than +from a cold boot. + +**Mitigation implemented: indefinite background boot retry.** Since the +module's own internal fault (if that's what it is) isn't something we can +fix, and since it demonstrably self-clears on a later attempt rather than +needing repair, `main/hardware_bootstrap.cpp` now spawns a +`bt1035RetryTask` FreeRTOS task whenever the initial `HardwareBootstrap:: +boot()`'s call to `Bt1035Driver::boot()` fails. The task loops calling +`boot()` again with **no artificial delay** between attempts — each +attempt already blocks for ~25-60s on its own (the banner wait times +`kBootAttempts`, plus an 8-step baud-rate sweep on final failure), so no +extra backoff is needed on top — until it succeeds, at which point it runs +the same post-boot setup (device name, auto-reconnect) the normal success +path does, then exits. The rest of the system (Wi-Fi, tuner, web UI) never +blocks on this and stays fully usable throughout. Verified live: after a +forced failure (2 attempts + baud sweep, ~62s), the retry task started +immediately, the HTTP server and heartbeat came up normally in parallel, +and the retry task began a fresh attempt right away without any pause. + +**Open going forward**: root cause of the intermittent total-silence mode +is still not identified — this session's diagnosis exhausted what's +possible with a multimeter alone. Real progress would need either an +oscilloscope on SYS_CTRL/RESET/crystal across several boots to correlate +success/failure with power-on timing jitter, or a large-N automated +reboot-cycle statistic (attempted this session via a `pyserial` script, +but the ESP32-S3's native USB-CDC re-enumerating on every hardware reset +made a fully unattended multi-cycle script unreliable — a naive read loop +silently produced a false "0/5 success" result once across a reconnect +window). A future attempt at that statistic needs to detect the USB path +disappearing/reappearing and reopen the port, or use a separate +hardware UART-to-USB adapter that doesn't disconnect when the target +resets. + +**Also discussed this session (not implemented, for a future hardware +revision)**: whether a different/newer SoC could eliminate the need for +the external BT1035 module entirely. Confirmed via web search that +Espressif's new **ESP32-S31** (RISC-V, announced April 2026) has +integrated **Bluetooth 5.4 with both LE and Classic (BR/EDR)** support — +unlike the ESP32-S3 used today, which is BLE-only at the silicon level +(confirmed: no Classic BT/A2DP hardware exists on S3, this is not a +firmware limitation). An `ESP32-S31-WROOM-3` module also exists. This +would be a significant main-MCU redesign, not a drop-in swap, and its +ESP-IDF support maturity/availability wasn't independently verified this +session — worth a dedicated evaluation before committing to it for a +future hardware revision. diff --git a/Software/main/hardware_bootstrap.cpp b/Software/main/hardware_bootstrap.cpp index 87eaa0b..c2cbf71 100644 --- a/Software/main/hardware_bootstrap.cpp +++ b/Software/main/hardware_bootstrap.cpp @@ -32,6 +32,8 @@ #include "driver/spi_master.h" #include "esp_log.h" +#include "freertos/FreeRTOS.h" +#include "freertos/task.h" namespace hardware { @@ -96,6 +98,58 @@ bool gReady = false; busHandle, static_cast(board::pins::Eeprom24aaAddr)); } + +/** + * @brief applyBt1035PostBootSetup — device name + auto-reconnect, run + * once after any successful BT1035 boot (first attempt or a + * later background retry). + */ +void applyBt1035PostBootSetup() +{ + if (auto nameResult = gBt1035.setDeviceName(gDeviceIdentity.bluetoothName()); + !nameResult) { + ESP_LOGW(kTag, "BT1035 device name set failed"); + } + if (auto reconnectResult = gBt1035.setAutoReconnect(3U); !reconnectResult) { + ESP_LOGW(kTag, "BT1035 auto-reconnect set failed"); + } +} + +/** + * @brief bt1035RetryTask — keep retrying Bt1035Driver::boot() in the + * background after the initial boot() attempt fails. + * + * @dname bt1035RetryTask + * @pubstate loops gBt1035.boot() with no artificial delay between + * attempts — each call already blocks for tens of seconds + * (kBootBannerWaitMs's banner wait, times kBootAttempts), so no + * extra backoff is added on top. Exits once boot() succeeds. + * + * Why: BT1035 boot failure has been observed to be intermittent on + * identical, correctly-wired, correctly-powered hardware — the same + * physical module has booted successfully and failed silently across + * different attempts in the same session, with the crystal oscillator + * inside the (sealed, non-serviceable) module the leading suspect. Since + * the fault clears on a later attempt rather than needing repair, retrying + * indefinitely in the background turns a permanent-until-manual-reboot + * failure into a bounded, self-recovering delay. + * + * @author Michele Bigi + * @date 2026-08-21 + */ +void bt1035RetryTask(void* /*arg*/) +{ + ESP_LOGW(kTag, "BT1035 background retry started"); + while (true) { + if (auto result = gBt1035.boot(); result) { + applyBt1035PostBootSetup(); + ESP_LOGI(kTag, "BT1035 background retry succeeded"); + break; + } + ESP_LOGW(kTag, "BT1035 background retry attempt failed, trying again"); + } + vTaskDelete(nullptr); +} } // namespace std::expected HardwareBootstrap::boot() @@ -164,18 +218,15 @@ std::expected HardwareBootstrap::boot() } if (auto btResult = gBt1035.boot(); !btResult) { - ESP_LOGE(kTag, "BT1035 boot failed — continuing without Bluetooth"); + ESP_LOGE(kTag, "BT1035 boot failed — continuing without Bluetooth, " + "retrying in background"); + if (xTaskCreate(bt1035RetryTask, "bt1035_retry", 4096, nullptr, 3, + nullptr) + != pdPASS) { + ESP_LOGW(kTag, "BT1035 background retry task create failed"); + } } else { - if (auto nameResult = - gBt1035.setDeviceName(gDeviceIdentity.bluetoothName()); - !nameResult) { - ESP_LOGW(kTag, "BT1035 device name set failed"); - } - - if (auto reconnectResult = gBt1035.setAutoReconnect(3U); - !reconnectResult) { - ESP_LOGW(kTag, "BT1035 auto-reconnect set failed"); - } + applyBt1035PostBootSetup(); } gReady = true;