Skip to main content

Manage incidents

Status: incidents are hidden from the app for now (off the sidebar and command palette); existing incidents stay reachable by direct link. See Incidents.

Status: new. The triage/assign/resolve lifecycle, timeline, and one-way push to an external ServiceNow/PagerDuty ticket (#155) ship now. Notifications (assignee alert on open, resolution notice, daily digest of open incidents), resolution auto-link to the first passing run, and external-ticket bidirectional sync are planned under #411.

Goal​

You'll open an incident for a data-quality failure, triage it (acknowledge + assign), discuss it on the timeline, and resolve it with a note — all inside Measured Cloud. See Incidents for the data model.

Prereqs​

  • A role that can manage incidents: editor, workspace admin, or super admin. A viewer can read incidents but every mutate control is disabled.

How incidents start​

  • Automatically. When a validation run fails (including an asset's or a single rule's Run, which runs the asset's default validation), the workspace's default incident rule opens an incident (source = auto) linked to that alert and run. Only one open incident exists per validation at a time, so a validation failing on every run does not flood you — re-runs add a timeline entry to the one open incident. To stop these incidents, pause the default rule on the Alerts page.
  • Manually. Open one yourself for a failure you want to track.

Steps​

  1. Navigate to Incidents. The list shows each incident's title, severity, status, assignee, age (time since it opened), and source. It refreshes every 30 seconds.

  2. Open a new incident (optional). Click New incident, give it a title and severity, choose the data asset it is about (and, optionally, one of its rules), add a root-cause note if you like, then Create incident. It starts in open and takes the asset's (or rule's) team.

  3. Open the detail panel. Click any row. You'll see the severity/status badges, the assignee, the root-cause and resolution notes, the lifecycle controls, and the timeline.

  4. Acknowledge. While the incident is open, click Acknowledge to move it to acknowledged. This stamps the acknowledged time and logs a status_change entry on the timeline.

  5. Assign a triage owner. Use the Assignee dropdown to pick who owns triage, or Unassigned to clear it. The list offers the incident's workspace members; the picker is gated on the manage permission (disabled for a viewer). Only the user is chosen — the owner's email is denormalised server-side (never a value you type), the list's Assignee column updates, and the change logs an assigned entry on the timeline. If a triage rotation applies to the incident, a Next in rotation line shows who's next and an Assign to next button hands them the incident (advancing the round-robin).

  6. Comment. Type into the timeline's comment box and click Comment. Your email (from your authenticated session — never something you type) is recorded as the author.

  7. Resolve. Add a resolution note (optional) and click Mark resolved. This stamps the resolved time, records the note, and logs the final status_change. A resolved incident is read-only in the lifecycle panel.

  8. Push to an external ticket (optional). In the detail panel's External ticket section, pick one of your workspace's ServiceNow or PagerDuty alert destinations and click Open ticket. Measured Cloud reuses that destination's stored credentials to open one upstream ticket, records the ticket's id / URL / status on the incident, and logs an external_ticket timeline entry. See Link an incident to an external ticket below for the details.

  9. Delete (optional). The Delete control (red, in the detail footer) removes the incident and its timeline. It's gated on the same manage permission and stays red even when disabled for a viewer.

Every incident carries a human-facing incident number (INC-<8 hex>) shown in the list and the detail header — the built-in fallback reference, so an incident is always quotable even if you never link it to another system. When you do want the incident tracked in ServiceNow or PagerDuty, push it:

  1. Configure a ServiceNow or PagerDuty alert destination in Settings → Alerts (this is where the encrypted credentials live — the incident push reuses them, so there's no second place to store a secret).
  2. Open the incident's detail panel, and in External ticket choose that destination and click Open ticket (gated on the manage permission).
  3. The result appears as a badge:
    • ServiceNow returns the INC… number, a click-through URL into the record, and the ticket state — the badge links straight to it.
    • PagerDuty triggers an Events-API incident but returns no click-through URL, so the badge shows the dedup key as plain text (no link).

This is push-only. Opening the ticket is a one-way, one-time action:

  • An incident links to at most one external ticket — a second push is refused.
  • There is no status sync-back: resolving the incident here does not close the upstream ticket, and closing the upstream ticket does not resolve the incident. Bidirectional sync is out of scope (tracked under #155).

The push never exposes the destination's credentials — only the non-secret upstream id / URL / status are stored or shown.

Round-robin triage rotations​

Instead of picking an owner by hand every time, configure an on-call rotation — an ordered list of people the incident cycles through. Set one up on the Triage rotations settings page (gated on the manage permission; a viewer who deep-links there sees a role banner, not a broken form). While incidents are hidden in the UI (#144), the Settings nav doesn't list this page — open it directly at /settings/triage-rotations.

A rotation is scoped to one of three levels, and the most specific one wins when an incident resolves whose rotation to use:

  • Asset — for incidents from a failing rule group, matched against the group's member assets (most specific).
  • Team — matched against the incident's derived team (its validation's team, or its legacy checkpoint's).
  • Workspace — the fallback for every incident in the workspace (least specific).

An empty rotation is treated as "not configured" and is skipped, so the next-broadest scope wins. If no rotation applies, the incident just has no "next" — you assign by hand as before.

To use a rotation on an incident:

  1. In the detail panel, next to Assignee, read Next in rotation — the person the rotation would hand this incident to next (a read-only peek; it does not advance anything).
  2. Click Assign to next to assign that person and advance the round-robin, so the following incident's peek shows the next name. This is gated on the manage permission (disabled, never hidden, for a viewer) and logs an assigned timeline entry marked via: rotation.

Rotation members are drawn from the same assignable-members pool as the manual picker, and each member is validated as an assignable member of the workspace when you save. A member who later leaves the workspace is skipped on the next assign (the rotation advances past them). Editing a rotation's list preserves the round-robin position by person where possible.

Need a shareable record without an external system? The incident detail can be exported (GET …/incidents/{id}/export) as the incident plus its full timeline — the built-in "ticket" you can print or attach to a report.

What's logged​

Every action writes an audit row (resource_type = "incident" / "incident_event") and an IncidentEvent on the timeline, so the full triage history is reconstructable.

What's not here yet​

The pieces below are tracked under #411 and are not in this release:

  • Notifications — no email to the assignee when an incident opens, no resolution notice, no daily digest of open incidents.
  • Resolution auto-link — you record a resolution note by hand; the incident is not automatically linked to (or closed by) the first passing run after resolution.
  • External ticket status sync-back — you can push an incident to a ServiceNow/PagerDuty ticket (see above), but incident status is not synced back from that ticket, and resolving the incident does not close the upstream ticket. The link is one-way. (The outbound ticketing alert channels are a separate feature; see Alerts.)
  • SLA breach alerting — the age shown in the list is a display nicety; there is no SLA timer that fires when an incident ages out.
  • Auto-assign on open — a newly auto-opened incident is not assigned from the rotation automatically; the rotation is a manual Assign to next action (with a "who's next" peek). Auto-assign-on-open is a planned follow-up.