Incidents and severity
An incident is the industry's unit for "something went wrong": a bounded episode with a start time, an end time, and a record of what happened in between. Turning a stream of raw failed checks into incidents is what makes monitoring operationally useful. You don't reason about 400 failed probes — you reason about "the mail server was down from 09:12 to 09:47."
Incidents also carry the math. Downtime minutes are the summed durations of incidents, and those minutes are exactly what uptime percentages and SLA compliance are computed from. An incident record with honest timestamps is the difference between an argument and an answer.
Severity ladders
Operations teams rank incidents on a severity ladder — SEV1 through SEV4, P1 through P4; the labels vary but the purpose doesn't: triage. When everything alerts with equal urgency, on-call engineers go numb and real emergencies drown. UpAll uses a three-step ladder (see incident severities):
| Severity | Meaning | Typical example |
|---|---|---|
| Outage | Not working | a monitor down after consecutive failed checks |
| Degraded | Working, but impaired | an expired certificate; a service you've flagged as running badly |
| Info | No impact yet — needs attention | a certificate entering its last 30 days |
The ladder tells your team how to react: Outage means drop everything, Degraded means fix it today, Info means schedule it before it becomes something worse. It also keeps client communication honest — a slow afternoon is not an outage, and calling it one erodes the meaning of the word.
The incident lifecycle
Every incident follows the same arc: it opens when a problem is detected or declared, it stays open while the problem persists (investigation, communication), and it resolves when service is back to normal. The resolved record — start, end, duration — is what feeds reports and history.
In UpAll, incidents come from three sources:
- Automatic: when a monitor fails its configured number of consecutive checks (default 3), an Outage incident opens. Its opening timestamp is backdated to the first failure of the streak, so the recorded duration reflects the real outage, not the detection delay. It resolves automatically on the next successful check — the full mechanics are in how incidents open and close.
- Certificate: an Info incident 30 days before a certificate expires, a Degraded incident on expiry, each resolving automatically when the certificate is fixed — see certificate incidents.
- Manual: staff can open incidents for anything automated checks can't see — a degraded third-party dependency, emergency work in progress — with a bilingual title and body, any severity, an optional client attribution (leave it blank to keep the incident internal), and a one-click Resolve. See manual incidents.
Whatever the source, opening and resolving an incident emails every active staff member in the workspace, so nobody depends on happening to look at a dashboard.