Firmware crash logs that make field failures debuggable
How we capture firmware crash logs on Cortex-M and ESP32 devices: what to record, where to keep it across a reset, and how to get it home.

A device resets in a customer’s warehouse at three in the morning, comes back up and carries on. Nobody sees it, and nobody can reproduce it on the bench. Without firmware crash logs, that reset is just a gap in the telemetry; with them, it is a program counter, a fault register and a firmware version you can act on.
This post is for firmware engineers and product leads shipping small fleets of connected devices. It covers what we record when something goes wrong, where we keep it so it survives the reset, and how we get it back to a server without building a heavy platform.
What a useful crash record contains
The temptation is to log everything. In practice a crash record has to fit in a few hundred bytes, be written from a fault handler with the system half-broken, and still be readable months later. We keep it small and structured.
- Reset cause: power-on, brown-out, watchdog, software reset, external pin or fault. This alone splits most field issues into “power problem” and “firmware problem”.
- Program counter and link register at the moment of the fault, so you can map the address back to a function with the matching ELF file.
- Fault status registers, which say what kind of fault it was rather than just where.
- Firmware version and build ID. Ideally a hash of the build, not just a semantic version, because addresses are meaningless against the wrong binary.
- Uptime and a boot counter, to tell a one-off from a device that crashes every forty minutes.
- The running task if you use an RTOS, plus a short trail of recent events: the last few state transitions or log lines.
We deliberately leave out full RAM dumps on small parts. They are valuable, but they are expensive to store and transmit, and the fields above are enough to diagnose most crashes.
Capturing the fault on Cortex-M parts
On Arm Cortex-M3, M4 and M33 devices (much of the STM32 range, the nRF52840 and many others), the core does half the work for you. On exception entry it pushes eight words onto the active stack: R0–R3, R12, LR, the return address (PC) and xPSR. Your HardFault handler needs to find that frame and copy it out before doing anything clever.
The handler checks bit 2 of the EXC_RETURN value in LR to learn whether the main stack or the process stack was in use. That decides whether it reads MSP or PSP. Under FreeRTOS, a fault in a task will normally be on the process stack. This step is usually written as a few lines of assembly, because the C compiler’s own prologue can disturb the registers you want.
Then read the fault registers in the System Control Block:
| Register | What it tells you |
|---|---|
CFSR | The combined MemManage, BusFault and UsageFault status: unaligned access, divide by zero, undefined instruction, imprecise bus error and so on |
HFSR | Whether a lower-priority fault escalated into a HardFault |
MMFAR / BFAR | The faulting data address, valid only when the matching flag in CFSR is set |
Note that Cortex-M0 and M0+ parts do not have the configurable fault registers, so on those you get the stacked frame and not much else. That is still enough to find most null-pointer and stack-overflow crashes.
Assertions and watchdogs count too
Not every failure is a hardware fault. A failed assertion, a detected stack overflow or a watchdog expiry should write the same record format with a different reason code. A watchdog reset leaves no stacked frame, so we have the watchdog’s early-warning interrupt (where the part has one) or a software supervisor record the current task and PC just before the reset. We covered the supervisor side in our post on firmware watchdog design that recovers devices in the field.
Where to keep firmware crash logs across a reset
The fault handler runs in a hostile environment. The heap may be corrupt, the stack may be the thing that overflowed and interrupts may be unreliable. Writing to flash from there is possible but risky: an erase can take tens of milliseconds and some parts stall the bus while doing it.
Our default is a two-stage approach:
- Write to a no-init RAM region in the handler. Reserve a small section in the linker script that the startup code does not zero. Fill it with the record, add a magic number and a CRC, then trigger a system reset. SRAM contents generally survive a warm reset, though not a power loss, and some bootloaders clear RAM, so check yours.
- Move it to flash after boot. Early in startup, check the magic and CRC. If they are valid, copy the record into a small ring buffer in flash with the system healthy, then clear the RAM copy.
The flash ring needs the same care as any log: fixed-size slots, a sequence number and wear spread across sectors. Our post on flash wear levelling for settings and logs goes through the layout. Keeping the last eight to sixteen records is plenty for most products.
Reset cause comes from the hardware rather than your own code. STM32 parts expose reset flags in the RCC (in RCC_CSR on many families); read them early and clear them, or you will see stale flags on the next boot. On ESP32 with ESP-IDF, esp_reset_reason() gives you the same information, RTC_NOINIT_ATTR places variables in memory that survives a software reset, and the built-in core dump feature can write a full dump to a flash partition if you have room for it.
Getting the records home
A crash log that never leaves the device only helps when someone returns the unit. We treat upload as a normal, low-priority message type.
- On the next successful connection, send any unsent records, oldest first, and mark them sent only after the server acknowledges.
- Keep the payload binary and compact on LoRaWAN or other constrained links. A record of 60–120 bytes is realistic if you send the PC, LR, fault registers, reason, version hash and uptime, and leave the event trail for Wi-Fi or BLE devices.
- Rate-limit uploads. A device in a crash loop should not flood the uplink or flatten its battery reporting the same fault.
On the server, the important step is symbolication: matching the build hash to the archived ELF and turning addresses into file and line with addr2line or an equivalent. That means your CI must keep every released ELF, not just the binary image you shipped over the air.
Reading the data across a fleet
One crash is a bug report. A hundred crashes are a priority list. Group records by firmware version and faulting function, and the picture usually becomes clear quickly: one stack overflow in a rarely used code path, one brown-out pattern that tracks battery level, one driver that misbehaves on a single hardware revision.
The reset-cause breakdown is worth putting on whatever dashboard the team already watches. A rise in brown-out resets after a hardware change is a power-design question, not a firmware one, and catching that early saves a lot of time spent in the wrong code.
What we would do on a new product
If you are starting firmware today, build crash logging in during the first weeks, not after the first field failure. It is a few hundred lines of code and costs little flash. Our checklist:
- A HardFault handler that captures the stacked frame from the correct stack pointer.
- Fault registers, reset cause, build hash, uptime and boot count in a versioned, CRC-protected record.
- A no-init RAM region for the handler and a wear-aware flash ring for persistence.
- The same record written for assertions, stack overflows and watchdog warnings.
- Rate-limited upload with acknowledgement, and archived ELF files for every release.
- A deliberate fault test on the bench (a write to a null pointer, a forced stack overflow) before each release, to prove the whole chain still works.
That last point matters most. Crash logging is code that only runs when things go wrong, which makes it the easiest code in the product to break without noticing.