Skip to main content

Incidents

Status: new. The in-app incident lifecycle ships with this release — manual + auto creation, triage/assign/resolve, and a timeline are live. The notification fan-out (assignee alert on open, resolution notice, daily digest), resolution auto-link to the first passing run, and bidirectional Jira/ServiceNow status sync are planned follow-ups under #411.

A data-quality failure is an incident. Until now PLACEHOLDER Cloud only fired outbound alerts when a checkpoint failed — the triage (who is on it, what the status is, when it was resolved) happened in some other tool. An incident is the first-class record that keeps that lifecycle inside the product.

Two model rows back it: an Incident is the triage record (status, severity, assignee, linked failed run, notes); an IncidentEvent is one entry on its append-only timeline (a manual comment, or an auto-logged status change).

Lifecycle

open ───▶ acknowledged ───▶ resolved
  • open — the default on creation. An auto-opened incident starts here.
  • acknowledged — someone has picked it up. The transition stamps acknowledged_at.
  • resolved — done. The transition stamps resolved_at and is where you record a resolution note.

Every transition appends an IncidentEvent of kind status_change carrying {from, to}, so the timeline is a complete audit of how the incident moved.

Data model

Incident (per-tenant):

ColumnWhat it stores
titleShort human label.
severityinfo / warning / error / critical — the same four-level ladder alerts use.
statusopen / acknowledged / resolved.
assignee_user_id / assignee_emailWho owns it. assignee_email is denormalised so the list renders without a cross-schema join.
validation_result_idThe failed run that triggered it (NULL for a manual incident with no linked run; SET NULL on result delete so history survives).
checkpoint_idThe checkpoint whose run failed. Used to dedupe auto-creation.
root_cause / resolution_noteFree-text investigation + resolution notes.
sourcemanual (an operator escalated a failure), auto (opened by the worker from a failing run), or alert (opened by an alert rule whose action is Create incident — see #656).
created_at / acknowledged_at / resolved_atLifecycle timestamps.

IncidentEvent (per-tenant, append-only):

ColumnWhat it stores
kindcreated / status_change / comment / assigned.
user_id / user_emailThe author (NULL for system / auto entries). For a manual comment this is taken from the authenticated principal — never the request body — so one user can't post as another.
bodyThe comment text (NULL for a status-change entry).
metadata{from, to} for a status-change entry.

No column on either table is a secret, so — unlike an alert destination — nothing is encrypted at rest, and the API returns the rows verbatim.

Creation

  • Manual. An operator opens an incident from the Incidents page (title + severity, optional root cause). source = "manual".
  • Auto (first cut). When a checkpoint run fails, the worker opens an incident automatically (source = "auto", severity graded from the result), alongside the existing alert fan-out. This is idempotent: at most one open auto-incident exists per checkpoint at a time, so a checkpoint that keeps failing does not spawn duplicates — re-runs land against the one open incident. A passing run never opens one.
  • Alert action. An alert rule whose channel is Create incident opens an incident when it fires (source = "alert"), respecting the rule's checkpoint scope, trigger, and minimum severity. See Alerts → Create incident. It shares the single-open-incident-per-checkpoint dedup with the auto path above (the throttle spans both sources), so a repeated failure appends a timeline entry to the open incident rather than opening another.

Isolation

Incident carries workspace_id and is enforced by the workspace-isolation listener, so a member of one workspace cannot read another workspace's incidents. IncidentEvent has no workspace_id — it is reached only through its parent incident (the same shape as a result comment) and is org-isolated via the tenant schema's RLS.

Permissions

ActionPermissionRoles
List / view / read timelinecan_view_incidentviewer, editor, admin, owner
Create / triage / resolve / delete / commentcan_manage_incidenteditor, admin, owner

A viewer sees incidents but the mutate controls render disabled — mirroring the server-side gate.

  • Alerts — the outbound notification an incident complements (an incident is the in-app triage record; an alert is the push to Slack/PagerDuty/etc.).
  • Validation results — the failed run an incident links to.
  • How-to: Manage incidents.