# IoT alerting that operators do not learn to ignore

> Practical IoT alerting design for connected fleets: offline detection, hysteresis, grouping and routing so operators act instead of muting.

- Author: Milo Perović, Founder & lead engineer
- Published: 2026-11-18
- Topic: Software

Most connected products get IoT alerting bolted on a week before the first pilot: a threshold here, an email there, and a channel that fills with notifications nobody reads by the end of the month. The fleet is small, so it feels fine. Then it grows, and the alert that mattered arrives buried under two hundred that did not.

This post is for founders, product managers and engineers who are about to put devices in front of real operators. It covers how we think about alert design for a fleet: what to alert on, how to avoid false alarms, where the logic should live and how to route alerts so the right person acts on them.

## Why IoT alerting goes wrong on real fleets

Alerts fail in two directions. Too sensitive, and operators learn that most notifications are noise, so they mute the channel or filter it into a folder. Too lax, and a device sits offline for a week before a customer reports it. Both come from rules written against single readings rather than what a person needs to do. Flaky cellular links, cold batteries and reboots after OTA updates make naive rules fire constantly on a healthy fleet.

The test we apply to every alert is simple: **if this fires at 3am, what should someone do?** If the honest answer is "look at it tomorrow", it is not an alert. It is a report, a dashboard tile or a daily digest.

## Separate device health from process alerts

We split alerts into two families early, because they have different owners and different urgency.

- **Device health**: offline devices, low battery, repeated reboots, failed OTA updates, storage nearly full, clock drift. These usually belong to the team that builds and supports the product.

- **Process or domain alerts**: a freezer above temperature, a tank below level, a door left open, a machine drawing abnormal current. These belong to the customer or operator using the product.

The operator does not care that firmware 1.4.2 failed to install on one unit, and your support engineer cannot do anything about a warm freezer.

## Detecting offline devices without false alarms

"Device offline" is the alert every fleet needs and the one most often done badly. The naive version fires when a single expected message is missing. On a cellular device reporting every five minutes, that will trigger many times a day across a fleet.

A more robust approach looks like this:

- **Base it on a heartbeat, not on data.** A device with nothing to report should still send a small periodic status message so silence means something.

- **Use a multiple of the reporting interval.** We typically start at three to five missed intervals before marking a device offline, then tune from real connectivity data. Battery devices that report hourly or daily need correspondingly longer windows.

- **Use the broker as a hint, not the verdict.** With MQTT, a Last Will message is published by the broker when a client disconnects without a clean goodbye, typically once the keep-alive period has expired (around 1.5 times the keep-alive interval). It is a useful early signal, but a device that reconnects thirty seconds later should not page anyone.

- **Add a stale state before offline.** A three-level status of online, stale and offline gives dashboards something honest to show without raising an alert for every short gap.

Topic and message structure matter here too. If heartbeat, status and data share a clean hierarchy, the offline rule can subscribe to exactly what it needs. We covered that in more depth in our post on [MQTT topic design for a growing device fleet](https://miloperovic.com/journal/mqtt-topic-design-for-a-growing-device-fleet/).

## Thresholds, hysteresis and debounce in IoT alerting rules

A fixed threshold on a raw value is the second biggest source of noise. A temperature hovering around 8.0 °C will cross an 8.0 °C limit dozens of times an hour, and each crossing becomes a notification.

Three techniques remove most of this without hiding real problems:

- **Hysteresis**: raise at one value and clear at another, for example alert above 8 °C and clear only below 7 °C. The gap should be larger than the sensor's noise and its stated accuracy.

- **Duration**: require the condition to hold for a period, such as ten minutes above the limit. Choose the period from the physics of the process. A freezer door opening for a delivery is normal; forty minutes above limit is not.

- **Rate of change**: some problems show up as a slope long before they cross a threshold, such as a battery voltage falling faster than its usual curve or a pressure creeping upwards overnight.

Sending a "resolved" message when a condition clears is as important as raising it, because it tells the operator they can stop worrying and gives you clean data on how long incidents last.

Keep the raw readings. When a customer asks why an alert did or did not fire, you need the underlying series to answer. Our post on [storing IoT time-series data without overbuilding](https://miloperovic.com/journal/storing-iot-time-series-data-without-overbuilding/) covers retention and downsampling that keep this affordable.

## Routing, grouping and escalation

Once rules are sensible, the next problem is volume. A site losing its internet uplink will take every device on that site offline at once. Twenty separate alerts for one router is noise; one alert saying "site X: 20 devices offline" is useful.

We group by the thing a person would fix: site, gateway, customer account or firmware version. A spike of failed OTA updates on one release should be one incident for the firmware team, not hundreds of device alerts.

Routing should follow ownership and urgency:

- **Critical**: push notification or SMS to an on-duty person, with escalation to a second person if nobody acknowledges within a set time.

- **Warning**: in-app notification and email, no escalation.

- **Informational**: a daily or weekly digest, never a real-time message.

Let customers set their own process thresholds and contacts, within limits you control. And always provide a way to mute a single noisy device temporarily with an expiry, so nobody has to mute the whole channel.

## Where the alert logic should live

Most products split it across three places.

LocationGood forTrade-offs

DeviceFast local action, reporting faster when something is wrong, working without connectivityRules change only with a firmware or configuration update; harder to see across the fleet

GatewaySite-level grouping, local sirens or relays, buffering during outagesAnother piece of software to update and monitor

CloudFleet-wide rules, offline detection, grouping, routing and per-customer configurationDepends on connectivity; latency of seconds to minutes

A pattern we like for battery devices is "report by exception": the device samples often but transmits rarely, and switches to frequent reporting when a local threshold is crossed. The cloud still owns the actual alert, but it receives the data it needs quickly without the device paying the radio cost all the time. Offline detection must always live outside the device, for the obvious reason that a dead device cannot report its own death.

## What we would do before the first fleet goes live

A short checklist we run through before a pilot:

- List every alert with its owner, its severity and the action someone should take. Delete anything without an action.

- Implement a heartbeat and a three-state online, stale, offline model, tuned to each device class's reporting interval.

- Add hysteresis and a minimum duration to every threshold rule, and send a resolved message on clear.

- Group alerts by site, gateway and firmware version before they reach a person.

- Give each alert a link straight to the device or site view with the relevant chart already open.

- Log every alert raised, acknowledged and cleared, then review the count per week. If an alert fires often and nobody acts, change the rule.

- Test the alert path itself: unplug a gateway, drain a battery on the bench, push a bad firmware build to a test unit and confirm the right person hears about it.

Good IoT alerting is less about clever rules than about discipline: fewer alerts, each with an owner and an action. Get that right during the pilot and the fleet can grow without the alert channel turning into noise.
