Skip to content

False positives and flapping

A monitoring system's credibility rests on one property: when it says "down", things are actually down. Every false alarm spends a little of that credibility, and a team that stops trusting its alerts has, in effect, no monitoring at all. Understanding why single checks mislead — and how thresholds fix it — is the core of tuning.

Why a single failed check misleads

A check is one packet exchange across the public internet, and any of these can sink it without the target being down:

  • Packet loss. Any hop between the probe and the target can drop a packet; a lost reply looks identical to a dead server.
  • Brief saturation. A backup job, a traffic spike, or a busy uplink can push one response past the 10-second timeout — once.
  • Momentary server hiccups. Service restarts, log rotation, garbage-collection pauses: seconds-long stalls on an otherwise healthy system.

Probe a healthy system every minute for a month and some isolated failures are near-certain. That's the medium talking, not the target.

Consecutive-failure thresholds

The standard defense is to open an incident only after N consecutive failures. Random noise almost never lines up several times in a row at one-minute spacing; a real outage does so trivially. The cost is detection time — with a threshold of 3 and one-minute checks, the incident opens about three minutes in.

In UpAll this is the Failures before incident field: 1 to 10, default 3. Two details worth knowing:

  • The incident's opening timestamp is backdated to the first failure of the streak — the threshold delays the alert, not the recorded downtime.
  • Every failed check is still stored and still counts against uptime, threshold or not. The threshold filters alerts; it doesn't hide data.

Set 1 only where a single failure genuinely matters and the path is very clean. Go higher (5+) for targets on jittery links — a branch office behind a congested connection — where 3 still produces noise. See check intervals and thresholds for tuning, and monitoring intervals for how the interval multiplies the detection delay.

Flapping

Flapping is a monitor that oscillates — down, up, down, up — in quick succession: an overloaded server that collapses under load, recovers as the load sheds, then collapses again; an intermittent WAN link; failing hardware. Untreated, flapping is the worst noise source there is, because each cycle opens and resolves an incident and emails your whole team twice.

Thresholds damp flapping mechanically: a down-streak shorter than N never opens an incident, so quick dips disappear from the incident log while remaining visible in the uptime bars and check history. But if a monitor flaps through a threshold of 3 — repeatedly sustaining three-plus minutes of failure — that's no longer noise. It's a genuinely unstable service, and the fix is investigating the target, not raising the threshold until the signal disappears.