Add BT1035 background boot retry; document intermittent silence findings
BT1035 boot can fail with total UART silence even after the banner-timing fix, on otherwise-identical, correctly-powered hardware. Confirmed via multimeter (VBAT_IN, 1.8V_OUT, SYS_CTRL/RESET, TX all normal) and by observing the same physical module both succeed and fail across different boot attempts in one session, that this is intermittent, not a dead module — the crystal is sealed inside the module and not inspectable or fixable from our side. Since the fault self-clears on a later attempt, mitigate with an indefinite background retry task: if the initial boot() fails, keep retrying with no artificial delay (each attempt already takes ~25-60s) so a temporary failure becomes a bounded, self-recovering delay instead of requiring a manual power cycle. Documented the full diagnostic session, including the ruled-out theories and a possible future ESP32-S31 (native Bluetooth Classic) migration path, in the RF investigation report. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Claude Sonnet 5
+31
-1
@@ -105,10 +105,40 @@ Short version:
|
||||
independently verified correct with a multimeter first — the module and
|
||||
PCB were never at fault. Fixed by waiting up to 25s for the banner
|
||||
(`kBootBannerWaitMs`); boot now succeeds on the first attempt.
|
||||
- **BT1035 — a second, harder failure mode confirmed intermittent, not
|
||||
hardware (2026-08-21).** Distinct from the banner-timing bug above: even
|
||||
with the 25s wait already in place, boot sometimes still gets zero UART
|
||||
bytes at all — no banner, no AT response, silent across all 8 probed
|
||||
baud rates (9600-921600). Root-cause evidence this session: VBAT_IN
|
||||
(3.3V), 1.8V_OUT (1.8V), SYS_CTRL/RESET (~3.27V, matching the firmware's
|
||||
own GPIO readback), and BT1035 TX (idle-HIGH ~3.29V, no short/float) all
|
||||
measured normal with a multimeter. The BT1035's 32 MHz crystal is
|
||||
integrated inside the sealed Feasycom module (confirmed via the module's
|
||||
own datasheet block diagram — no external crystal on our schematic), so
|
||||
it can't be inspected or reworked from our side; a marginal
|
||||
oscillator-startup margin inside the module is the leading suspect.
|
||||
**Decisive evidence it's intermittent, not a dead unit**: the exact same
|
||||
physical module booted cleanly (banner + all AT commands `OK`) on one
|
||||
attempt and went totally silent on the very next attempt, no physical
|
||||
changes in between. A replacement module is therefore not a guaranteed
|
||||
fix — the same defect class could recur on a different unit. Mitigated
|
||||
(not fixed) by an indefinite background retry task
|
||||
(`hardware::bt1035RetryTask` in `main/hardware_bootstrap.cpp`): if the
|
||||
initial `Bt1035Driver::boot()` fails, a background FreeRTOS task keeps
|
||||
calling `boot()` again with no artificial delay between attempts (each
|
||||
attempt already takes ~25-60s on its own) until it succeeds, while the
|
||||
rest of the system (tuner, Wi-Fi, web UI) stays fully usable in the
|
||||
meantime. Turns a permanent-until-manual-power-cycle failure into a
|
||||
bounded, self-recovering delay. See
|
||||
`docs/si4684-rf-investigation-report.md` (2026-08-21 entry) for the full
|
||||
session narrative, including a UART TX/RX loopback test attempt that was
|
||||
inconclusive (bridging the ESP32's own TX/RX pins from cold boot caused
|
||||
an unrelated, reproducible, harmless early-boot hang, not yet explained).
|
||||
- **Still open**: intermittent multi-second HTTP unresponsiveness under
|
||||
load; DAB signal quality still antenna-limited; 24 KB `nvs` partition
|
||||
may be undersized (`saveProfile()` `store_failed` seen intermittently,
|
||||
error code never captured).
|
||||
error code never captured); BT1035 intermittent total-silence boot
|
||||
failures (mitigated via background retry, not root-caused — see above).
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -792,3 +792,119 @@ identified. `/api/bluetooth/status` and `/api/bluetooth/paired` correctly
|
||||
report `{"status":"error","reason":"at_timeout"}` while in this state; the
|
||||
rest of the device (tuner, web UI) stays usable per the existing
|
||||
non-fatal-BT1035-boot design.
|
||||
|
||||
## 2026-08-21 update: BT1035 total silence confirmed intermittent (not
|
||||
## hardware); background retry mitigation added
|
||||
|
||||
Follow-up session dedicated entirely to the "total UART silence" BT1035
|
||||
failure mode above. Summary: **confirmed intermittent on genuinely
|
||||
identical hardware, root cause narrowed to the module's internal crystal
|
||||
(not our PCB, not fixable by us), and mitigated (not fixed) with an
|
||||
indefinite background boot retry.**
|
||||
|
||||
**Diagnostic instrumentation (temporary, added then reverted this
|
||||
session)**: added `BT1035 AT TX: <line>` / `BT1035 UART RX RAW: <hex or
|
||||
<empty>>` / `BT1035 AT RESULT: OK|ERROR|TIMEOUT` logging around
|
||||
`Bt1035Driver::transmitAndCollect()`, and temporarily dropped
|
||||
`kBootAttempts` to 1 for single-attempt clarity. This confirmed the
|
||||
failure signature precisely: `AT` is transmitted, zero bytes ever come
|
||||
back (`<empty>`), timeout. Reverted via `git checkout` once the manual
|
||||
diagnosis was done — not kept in the codebase.
|
||||
|
||||
**Multimeter checks, all normal** (scope-level checks — crystal
|
||||
oscillation, power-on transient — remain out of reach without an
|
||||
oscilloscope):
|
||||
- VBAT_IN: 3.3V (datasheet range 3.0-4.2V) ✓
|
||||
- 1.8V_OUT (module's internal regulator): 1.8V ✓ — proves the module's
|
||||
own power management *is* running, it isn't simply unpowered
|
||||
- SYS_CTRL / RESET (post-boot): ~3.27V, matching the firmware's own GPIO
|
||||
readback log (`post-reset: SYS_CTRL=1 RESET=1`) ✓
|
||||
- BT1035 TX pin (module side) to GND: 3.29V, idle-HIGH, no short/float/
|
||||
reversed polarity ✓ (though idle-HIGH alone doesn't prove the module's
|
||||
firmware is executing — some pads default HIGH from reset state alone)
|
||||
- A 10kΩ pull-down the user had added on SYS_CTRL (matching the
|
||||
datasheet's own recommendation for an undriven pin) was checked and is
|
||||
not the cause — the ESP32 GPIO drives push-pull and its own readback
|
||||
confirms it reaches a valid HIGH regardless.
|
||||
|
||||
**Crystal location determined**: the BT1035 datasheet's own block diagram
|
||||
shows "32MHz Crystal" as an internal block of the QCC3056 die, and the
|
||||
DigiRadio schematic netlist (`Netlist_Schematic1_2026-08-07.asc`) has no
|
||||
XTAL_IN/XTAL_OUT pins wired to any external crystal for U11 — confirming
|
||||
the oscillator is sealed inside the Feasycom module, not on our PCB. This
|
||||
is why nothing on our side (layout, load caps, our firmware) can affect
|
||||
it; if the failure really is a marginal oscillator-startup margin, it's a
|
||||
property of that specific physical module unit (or the part's design
|
||||
tolerance in general).
|
||||
|
||||
**Decisive evidence of intermittency, not a dead unit**: across repeated
|
||||
reboots in the same session (physical power-cycles and serial-port-open
|
||||
resets, which also hard-reset this ESP32-S3's native USB-CDC), the
|
||||
identical physical module was observed to boot **completely successfully**
|
||||
at least once — spontaneous banner `+VER=FSC-BT1035,V6.1.1,20240521` +
|
||||
`+DEVSTAT=1`, then `AT` and `AT+AUXCFG=3` both answered `OK` — and to fail
|
||||
completely silently on other attempts, with no physical change in between.
|
||||
This rules out "defective/dead module" as an explanation; ordering a
|
||||
replacement module is therefore *not* a guaranteed fix, since the same
|
||||
physical unit demonstrably works when it works.
|
||||
|
||||
**UART loopback test attempt — inconclusive, logged for future
|
||||
reference.** Tried to isolate ESP32 vs. module by bridging the ESP32-S3's
|
||||
own GPIO40 (BT1035 UART TX)/GPIO41 (BT1035 UART RX) pins with a jumper
|
||||
held by hand on the ESP32 module's castellated pads (no series
|
||||
resistor/test point exists on this net per the schematic netlist — U8.33
|
||||
↔ U11.P$14 and U8.34 ↔ U11.P$13 directly, nothing else). Twice
|
||||
reproducibly, bridging those pins from cold boot caused the ESP32 itself
|
||||
to hang very early in boot (right after the bootloader's "Disabling RNG
|
||||
early entropy source" line, before `app_main()` even starts) — harmless
|
||||
(board recovers fully once the jumper is removed) but unexplained, and it
|
||||
sidesteps the actual test rather than answering it. Not pursued further
|
||||
this session given the practical difficulty of hand-holding a wire onto
|
||||
castellated pads without a proper SMD test hook. If retried: attach the
|
||||
jumper *after* the ESP32 has already booted past that early stage (there's
|
||||
a ~25s window before the BT1035 AT command is actually sent) rather than
|
||||
from a cold boot.
|
||||
|
||||
**Mitigation implemented: indefinite background boot retry.** Since the
|
||||
module's own internal fault (if that's what it is) isn't something we can
|
||||
fix, and since it demonstrably self-clears on a later attempt rather than
|
||||
needing repair, `main/hardware_bootstrap.cpp` now spawns a
|
||||
`bt1035RetryTask` FreeRTOS task whenever the initial `HardwareBootstrap::
|
||||
boot()`'s call to `Bt1035Driver::boot()` fails. The task loops calling
|
||||
`boot()` again with **no artificial delay** between attempts — each
|
||||
attempt already blocks for ~25-60s on its own (the banner wait times
|
||||
`kBootAttempts`, plus an 8-step baud-rate sweep on final failure), so no
|
||||
extra backoff is needed on top — until it succeeds, at which point it runs
|
||||
the same post-boot setup (device name, auto-reconnect) the normal success
|
||||
path does, then exits. The rest of the system (Wi-Fi, tuner, web UI) never
|
||||
blocks on this and stays fully usable throughout. Verified live: after a
|
||||
forced failure (2 attempts + baud sweep, ~62s), the retry task started
|
||||
immediately, the HTTP server and heartbeat came up normally in parallel,
|
||||
and the retry task began a fresh attempt right away without any pause.
|
||||
|
||||
**Open going forward**: root cause of the intermittent total-silence mode
|
||||
is still not identified — this session's diagnosis exhausted what's
|
||||
possible with a multimeter alone. Real progress would need either an
|
||||
oscilloscope on SYS_CTRL/RESET/crystal across several boots to correlate
|
||||
success/failure with power-on timing jitter, or a large-N automated
|
||||
reboot-cycle statistic (attempted this session via a `pyserial` script,
|
||||
but the ESP32-S3's native USB-CDC re-enumerating on every hardware reset
|
||||
made a fully unattended multi-cycle script unreliable — a naive read loop
|
||||
silently produced a false "0/5 success" result once across a reconnect
|
||||
window). A future attempt at that statistic needs to detect the USB path
|
||||
disappearing/reappearing and reopen the port, or use a separate
|
||||
hardware UART-to-USB adapter that doesn't disconnect when the target
|
||||
resets.
|
||||
|
||||
**Also discussed this session (not implemented, for a future hardware
|
||||
revision)**: whether a different/newer SoC could eliminate the need for
|
||||
the external BT1035 module entirely. Confirmed via web search that
|
||||
Espressif's new **ESP32-S31** (RISC-V, announced April 2026) has
|
||||
integrated **Bluetooth 5.4 with both LE and Classic (BR/EDR)** support —
|
||||
unlike the ESP32-S3 used today, which is BLE-only at the silicon level
|
||||
(confirmed: no Classic BT/A2DP hardware exists on S3, this is not a
|
||||
firmware limitation). An `ESP32-S31-WROOM-3` module also exists. This
|
||||
would be a significant main-MCU redesign, not a drop-in swap, and its
|
||||
ESP-IDF support maturity/availability wasn't independently verified this
|
||||
session — worth a dedicated evaluation before committing to it for a
|
||||
future hardware revision.
|
||||
|
||||
@@ -32,6 +32,8 @@
|
||||
|
||||
#include "driver/spi_master.h"
|
||||
#include "esp_log.h"
|
||||
#include "freertos/FreeRTOS.h"
|
||||
#include "freertos/task.h"
|
||||
|
||||
namespace hardware {
|
||||
|
||||
@@ -96,6 +98,58 @@ bool gReady = false;
|
||||
busHandle,
|
||||
static_cast<std::uint8_t>(board::pins::Eeprom24aaAddr));
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief applyBt1035PostBootSetup — device name + auto-reconnect, run
|
||||
* once after any successful BT1035 boot (first attempt or a
|
||||
* later background retry).
|
||||
*/
|
||||
void applyBt1035PostBootSetup()
|
||||
{
|
||||
if (auto nameResult = gBt1035.setDeviceName(gDeviceIdentity.bluetoothName());
|
||||
!nameResult) {
|
||||
ESP_LOGW(kTag, "BT1035 device name set failed");
|
||||
}
|
||||
if (auto reconnectResult = gBt1035.setAutoReconnect(3U); !reconnectResult) {
|
||||
ESP_LOGW(kTag, "BT1035 auto-reconnect set failed");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @brief bt1035RetryTask — keep retrying Bt1035Driver::boot() in the
|
||||
* background after the initial boot() attempt fails.
|
||||
*
|
||||
* @dname bt1035RetryTask
|
||||
* @pubstate loops gBt1035.boot() with no artificial delay between
|
||||
* attempts — each call already blocks for tens of seconds
|
||||
* (kBootBannerWaitMs's banner wait, times kBootAttempts), so no
|
||||
* extra backoff is added on top. Exits once boot() succeeds.
|
||||
*
|
||||
* Why: BT1035 boot failure has been observed to be intermittent on
|
||||
* identical, correctly-wired, correctly-powered hardware — the same
|
||||
* physical module has booted successfully and failed silently across
|
||||
* different attempts in the same session, with the crystal oscillator
|
||||
* inside the (sealed, non-serviceable) module the leading suspect. Since
|
||||
* the fault clears on a later attempt rather than needing repair, retrying
|
||||
* indefinitely in the background turns a permanent-until-manual-reboot
|
||||
* failure into a bounded, self-recovering delay.
|
||||
*
|
||||
* @author Michele Bigi
|
||||
* @date 2026-08-21
|
||||
*/
|
||||
void bt1035RetryTask(void* /*arg*/)
|
||||
{
|
||||
ESP_LOGW(kTag, "BT1035 background retry started");
|
||||
while (true) {
|
||||
if (auto result = gBt1035.boot(); result) {
|
||||
applyBt1035PostBootSetup();
|
||||
ESP_LOGI(kTag, "BT1035 background retry succeeded");
|
||||
break;
|
||||
}
|
||||
ESP_LOGW(kTag, "BT1035 background retry attempt failed, trying again");
|
||||
}
|
||||
vTaskDelete(nullptr);
|
||||
}
|
||||
} // namespace
|
||||
|
||||
std::expected<void, HardwareBootError> HardwareBootstrap::boot()
|
||||
@@ -164,18 +218,15 @@ std::expected<void, HardwareBootError> HardwareBootstrap::boot()
|
||||
}
|
||||
|
||||
if (auto btResult = gBt1035.boot(); !btResult) {
|
||||
ESP_LOGE(kTag, "BT1035 boot failed — continuing without Bluetooth");
|
||||
ESP_LOGE(kTag, "BT1035 boot failed — continuing without Bluetooth, "
|
||||
"retrying in background");
|
||||
if (xTaskCreate(bt1035RetryTask, "bt1035_retry", 4096, nullptr, 3,
|
||||
nullptr)
|
||||
!= pdPASS) {
|
||||
ESP_LOGW(kTag, "BT1035 background retry task create failed");
|
||||
}
|
||||
} else {
|
||||
if (auto nameResult =
|
||||
gBt1035.setDeviceName(gDeviceIdentity.bluetoothName());
|
||||
!nameResult) {
|
||||
ESP_LOGW(kTag, "BT1035 device name set failed");
|
||||
}
|
||||
|
||||
if (auto reconnectResult = gBt1035.setAutoReconnect(3U);
|
||||
!reconnectResult) {
|
||||
ESP_LOGW(kTag, "BT1035 auto-reconnect set failed");
|
||||
}
|
||||
applyBt1035PostBootSetup();
|
||||
}
|
||||
|
||||
gReady = true;
|
||||
|
||||
Reference in New Issue
Block a user