Alerts

An alert is how PLACEHOLDER Cloud tells you something went wrong. Two model rows back it: an AlertRule is the configuration ("on every failure of checkpoint X, post to Slack"); an AlertEvent is the occurrence — one row per dispatched message.

Data model

AlertRule (per-tenant):

Column	What it stores
`checkpoint_id`	Scope (NULL = workspace-wide rule).
`trigger`	`on_failure` / `on_success` / `always` / `on_anomaly`.
`channel`	`create_incident` (internal) / `email` / `slack` / `teams` / `webhook` / `pagerduty` / `jira` / `linear` / `discord` / `mattermost` / `twilio_sms` / `datadog` / `jira_ticket` / `servicenow` / `github_issues` / `opsgenie`.
`destination`	A JSONB blob with channel-specific config (Slack webhook URL, email addresses, PagerDuty routing key…). Sensitive fields are encrypted at rest. Empty (`{}`) for the internal `create_incident` action — it opens a native incident rather than notifying an external service.
`enabled`	Toggle.
`min_severity`	`info` / `warning` / `error` / `critical`. Rule fires only when the validation-result severity reaches this level or higher. Default: `error`. This is the four-level alert scale, derived from result pass/fail percentages — distinct from the six-level anomaly severity scale described in Anomalies.

AlertEvent (per-tenant):

Column	What it stores
`alert_rule_id`	Which rule fired.
`validation_result_id`	The failing result that triggered the rule (NULL for anomaly-driven events).
`anomaly_event_id`	The anomaly that triggered the rule (NULL for result-driven events).
`status`	`pending` / `sent` / `failed`.
`sent_at` / `error`	Delivery metadata.
`external_ticket_id`	Upstream record identifier when the channel is ticketing-shaped (Jira issue key, ServiceNow `INC…` number, GitHub `#N`, Opsgenie alias). NULL for chat channels.
`external_ticket_url`	Operator-clickable URL into the upstream service. NULL for chat channels.
`external_ticket_status`	Upstream's state string forwarded verbatim. NULL for chat channels.

Channels

Channel	Status
Create incident (internal action)	shipped (#656)
Email (SMTP)	shipped — dispatcher in `worker/.../dispatch_alert.py`
Slack incoming webhook	shipped — dispatcher in `worker/.../dispatch_alert.py`
Microsoft Teams	shipped (FEATURE-15)
Generic webhook (signed)	shipped (FEATURE-16)
PagerDuty	shipped (FEATURE-17)
Jira — legacy (channel `jira`)	shipped (FEATURE-28) — superseded by `jira_ticket` below. Per #727 it is no longer offered for new rules; the Edit modal still surfaces it (labelled Jira (legacy)) for rules already on it, so existing rules keep working
Linear	shipped (FEATURE-28)
Discord	shipped (issue #81)
Mattermost	shipped (issue #81)
Twilio SMS	shipped (issue #81)
Datadog Events	shipped (issue #81)
Jira (NotifierAdapter, channel `jira_ticket`)	shipped (FEATURE-82) — the recommended Jira channel, shown in the UI simply as Jira
ServiceNow	shipped (FEATURE-82)
GitHub Issues	shipped (FEATURE-82)
Opsgenie	shipped (FEATURE-82)

All channels share the same dispatch contract — a NotifierAdapter subclass in dq_shared/notifiers/ validates the destination at write time, redacts bearer secrets on read, and builds the channel-native payload at dispatch time. Adding a new channel is one new file plus its tests. The one exception is the internal Create incident action below, which writes to the local database rather than POSTing anywhere and so has no NotifierAdapter.

Create incident (internal action)

The create_incident channel (#656) is the one internal action: when the rule fires it opens a native Incident in the app instead of notifying an external service. In the alert-rule modal it sits at the top of the channel picker, separated from the external channels by a divider — internal first, external below.

A fired create_incident rule:

opens an Incident with source = "alert" (the Incidents list's Source column shows it, distinct from a hand-created manual incident or the checkpoint-failure auto one), severity mapped from the result severity, a title derived from the failure (e.g. orders-checkpoint failed — 2 of 5 expectation(s) failed), and validation_result_id / checkpoint_id linking back to the triggering run;
carries no destination — destination is {} and no bearer secret is stored;
respects the rule's checkpoint scope, trigger, and min_severity like every other channel (it produces an AlertEvent, then the dispatcher opens the incident rather than POSTing).

Dedup / throttle. At most one open incident exists per checkpoint across both auto-create sources (alert and the auto checkpoint-failure path). A repeated failure of the same checkpoint while an open/acknowledged incident already covers it appends a timeline entry to that incident rather than opening a new one — so a 2-minute schedule can't spawn thousands. Resolution stays a manual action; auto-resolve-on-pass is a possible later option.

Ticketing channels

The Jira (jira_ticket), ServiceNow, GitHub Issues, and Opsgenie channels share the NotifierAdapter abstraction with the chat channels above but differ in one important way: a fired alert creates an upstream record with its own lifecycle, and the worker stamps the resulting identifier / URL / status onto the AlertEvent row (the three external_ticket_* columns). A follow-up that resolves the ticket when the checkpoint recovers is on the roadmap (bidirectional sync — out of scope for the initial release).

Jira (NotifierAdapter)

Channel value: jira_ticket. Required destination fields: site, project_key, email, api_token (encrypted). Optional: issue_type (default Task), labels (whitespace-bearing labels are silently dropped — Jira rejects those), custom_fields passthrough dict for required project-specific fields (story points, components, parent epic). POSTs to https://{site}/rest/api/3/issue with HTTP Basic auth (email:api_token).

This is the canonical Jira channel — the UI lists it simply as Jira, and it is the only Jira option offered when creating a new rule. The older FEATURE-28 jira channel above also creates an issue (same POST /rest/api/3/issue) but is a functional subset, so to avoid the confusing duplicate the new-rule picker called out in #727 no longer lists it. The Edit modal still shows it (labelled Jira (legacy)) for a rule already on the jira channel, so existing rules keep working — the jira_ticket adapter additionally tracks the created issue back onto the AlertEvent and supports custom_fields.

ServiceNow

Channel value: servicenow. Required: instance (subdomain like acme, or fully qualified acme.example.com), username, password (encrypted). Optional: table (one of incident / problem / cmdb_ci, default incident), urgency / impact (default 2), assignment_group sys_id, custom_fields passthrough. POSTs to https://{instance}.service-now.com/api/now/table/{table}. The response's result.number (the human-facing INC0012345) is persisted as external_ticket_id.

Authentication (#402) is selected by auth_mode:

auth_mode: "basic" (default) — HTTP Basic auth with username / password. This is the original shipped behaviour; destinations that omit auth_mode use it.
auth_mode: "oauth" — ServiceNow OAuth (Resource-Owner-Password-Credentials grant). Adds two required fields: client_id (the registered OAuth application's client id — non-secret, returned on read) and client_secret (encrypted at rest as client_secret_encrypted, redacted to "configured" on read). At dispatch the worker first POSTs to https://{instance}/oauth_token.do with grant_type=password plus client_id / client_secret / username / password to mint a bearer token, then calls the Table API with Authorization: Bearer <token>. ServiceNow's OAuth API only offers the password grant for first-party integrations, so the username/password are still required — the client credentials scope the token to a registered OAuth application rather than authenticating directly. A missing client_id or client_secret is rejected with a 400 at write time, not a 500 at dispatch.

GitHub Issues

Channel value: github_issues. Required: owner, repo, token (encrypted — a classic PAT, fine-grained PAT, or GitHub App installation token). Optional: labels, assignees, milestone (integer). POSTs to https://api.github.com/repos/{owner}/{repo}/issues with Authorization: Bearer <token> and the API-version pin (X-GitHub-Api-Version: 2022-11-28). The body is markdown-rendered; the deep-link to the validation result auto-links.

Opsgenie

Channel value: opsgenie. Required: api_key (encrypted). Optional: region (us default, or eu for the EU API host), priority (P1-P5, default P3), responders (pass-through list of responder objects), tags. POSTs to https://api.opsgenie.com/v2/alerts with Authorization: GenieKey <api_key> — not Bearer <api_key> (Opsgenie's documented format; the wrong auth scheme is silently 401'd). The alias is keyed on checkpoint_id so consecutive failures of the same checkpoint coalesce into one open alert — same dedup posture as the PagerDuty channel above.

Generic webhook

A generic webhook channel POSTs a JSON envelope to an operator-supplied URL — handy for SIEM ingestion, ops dashboards, or any receiver that doesn't speak Slack-block-kit. Configure two fields on the destination:

url (required): where to POST. Stored encrypted at rest; the API only ever returns the sentinel string "configured" on read.
signing_secret (optional): if set, every POST carries an X-DQ-Signature: sha256=<hex> header where the signature is the HMAC-SHA256 of the request body bytes. Also stored encrypted at rest.

Payload shape

{
  "event": "validation_result",
  "result_id": "<uuid>",
  "checkpoint_id": "<uuid|null>",
  "rule_suite_id": "<uuid|null>",
  "success": false,
  "failed_expectations": 3,
  "total_expectations": 12,
  "trigger": "on_failure",
  "run_at": "2026-05-13T12:34:56+00:00",
  "deep_link": "https://dq.example.com/results/<uuid>"
}

deep_link is only present when the operator has configured PUBLIC_BASE_URL. The body is serialised with json.dumps(payload, separators=(",", ":")) so the bytes you sign are the bytes the receiver sees — re-serialising with default whitespace would land on a different signature.

Verifying the signature

The receiver should compute the HMAC over the raw body bytes before any JSON parsing, then compare with hmac.compare_digest:

import hashlib
import hmac

def verify(body_bytes: bytes, header: str, secret: str) -> bool:
    if not header.startswith("sha256="):
        return False
    expected = hmac.new(secret.encode(), body_bytes, hashlib.sha256).hexdigest()
    return hmac.compare_digest(header.split("=", 1)[1], expected)

Reject the delivery if verify(...) returns False. The absence of the header means the rule is unsigned — accept those only if your deployment expects unsigned deliveries.

Dispatch flow

The worker writes a ValidationResult (or AnomalyEvent).
The worker evaluates every matching AlertRule in the same job.
For each match, an AlertEvent is written and a dispatch_alert job is enqueued.
The dispatcher posts to the channel, with retry-with-backoff up to 3 attempts.
On success: status = sent, sent_at set. On terminal failure: status = failed, error populated.

Operations you can do

Create / edit / delete a rule — Alerts route in the UI. Deleting a rule shows an Undo toast for a few seconds before the delete is sent, so an accidental delete is one click to reverse. (Schedule and rule-set deletes work the same way.)
Send a test alert — fires a synthetic payload to the configured destination without touching real data.
See per-event delivery status — every AlertEvent shows its status and last error.
Mute a rule by toggling enabled to false; events still get logged but aren't dispatched.

Who can change alert rules. Reading alert rules needs can_view_alert; creating, editing, pausing, or deleting one needs can_manage_alert — granted to editor, admin, and owner (the same tier that authors schedules). A viewer sees the rules and their delivery status but the Add / Edit / Pause / Delete controls render disabled with a "Requires …" tooltip, mirroring the server's 403. Because each rule's destination is a bearer secret (a webhook URL, API token, or routing key), the create / edit paths additionally require a recent password proof (step-up auth).

PagerDuty

The PagerDuty channel POSTs an Events API v2 trigger event to https://events.pagerduty.com/v2/enqueue. The destination stores one field — the routing key (PagerDuty also calls this the integration key), a 32-character hex string found in PagerDuty under Services → Integrations → Events API v2. The routing key is encrypted at rest under the tenant DEK on the same path Slack uses (HARDEN-15) and never echoed back to the UI.

Deduplication. The dedup_key PLACEHOLDER Cloud sends is dq-cloud-checkpoint-<checkpoint_id>. PagerDuty uses this to coalesce consecutive failures of the same checkpoint into a single open incident rather than minting a new page on every run. This is what on-call rotations typically want — one page per broken pipeline, not one page per scheduled run while the pipeline stays broken. The trade-off is that a fresh failure on an already-open incident does not re-page anyone until the prior incident is resolved (manually in PagerDuty, or, once it ships, via a resolve event on recovery — see FEATURE-17 for the open follow-up).

Workspace-wide rules (no checkpoint_id) fall back to dq-cloud-rule-<rule_id> for the dedup key so multiple events from the same rule still coalesce.

The payload's payload.summary includes the checkpoint name and the failed-expectation count, payload.severity is error, and payload.custom_details carries the asset, expectation counts, and — if PUBLIC_BASE_URL is set — a deep_link back to the failing ValidationResult in the DQ Cloud UI.

Jira

The Jira channel (FEATURE-28) opens an issue every time a rule fires. The destination stores six fields:

Field	What it stores
`site`	The Jira Cloud host, e.g. `acme.atlassian.net`. The worker builds `https://{site}/rest/api/3/issue` from this.
`project_key`	The target project's short key, e.g. `DQ`.
`issue_type`	The issue type to create (defaults to `Task`).
`email`	The Atlassian account email used as the HTTP Basic username.
`api_token`	The matching API token (the HTTP Basic password). Encrypted at rest under the tenant DEK; never echoed back to the UI.
`labels`	Optional list of labels to apply on create. Labels containing whitespace are dropped silently (Jira rejects them).

Token instructions

Mint an API token under id.atlassian.com → API tokens. Atlassian Cloud documents the HTTP Basic auth scheme — username is your account email, password is the API token — under Basic auth for REST APIs.

Payload shape

The worker POSTs the v3 Issue REST payload. description is an Atlassian Document Format (ADF) object — not the plain-text body the v2 REST API took — because Jira Cloud's v3 endpoint refuses string descriptions.

{
  "fields": {
    "project": {"key": "DQ"},
    "summary": "DQ Cloud · checkpoint FAILED",
    "description": {
      "type": "doc",
      "version": 1,
      "content": [
        {"type": "paragraph", "content": [{"type": "text", "text": "Checkpoint status: FAILED\nTotal expectations: 12\nFailed: 3\n..."}]},
        {"type": "paragraph", "content": [
          {"type": "text", "text": "Open in DQ Cloud: "},
          {"type": "text", "text": "https://dq.example.com/results/...", "marks": [{"type": "link", "attrs": {"href": "https://dq.example.com/results/..."}}]}
        ]}
      ]
    },
    "issuetype": {"name": "Task"},
    "labels": ["data-quality"]
  }
}

The deep-link paragraph is only present when PUBLIC_BASE_URL is set.

Linear

The Linear channel (FEATURE-28) opens an issue via Linear's GraphQL API. The destination stores three fields:

Field	What it stores
`team_id`	Linear team UUID. Find it under Settings → Workspace → Teams → API.
`api_key`	Linear personal API key. Encrypted at rest under the tenant DEK; never echoed back to the UI.
`priority`	Optional override for the issue priority (0 = no priority, 1 = urgent, 4 = low). When unset, priority tracks the failure severity.

Token instructions

Mint a personal API key under Linear Settings → API. Linear's auth header is the bare key — not Bearer <key>, contrary to most OAuth-style APIs. The DQ Cloud dispatcher sends Authorization: <api_key> accordingly. Wrapping with Bearer is the single most common bug; Linear answers 401 in that case.

Payload shape

The worker POSTs a GraphQL issueCreate mutation to https://api.linear.app/graphql:

{
  "query": "mutation IssueCreate($input: IssueCreateInput!) { issueCreate(input: $input) { success issue { id identifier url } } }",
  "variables": {
    "input": {
      "teamId": "00000000-0000-0000-0000-000000000000",
      "title": "DQ Cloud · checkpoint FAILED",
      "description": "Checkpoint status: FAILED\nTotal expectations: 12\nFailed: 3\n...\n\nOpen in DQ Cloud: https://dq.example.com/results/...",
      "priority": 2
    }
  }
}

The Open in DQ Cloud: line is only appended when PUBLIC_BASE_URL is set.

Severity mapping

When the operator hasn't pinned a priority, the dispatcher derives it from the validation-result severity (see Severity routing below):

Severity	Linear priority
`info` (all passed)	4 — Low
`warning` (>0% failed)	3 — Medium
`error` (>50% failed)	2 — High
`critical` (all failed)	1 — Urgent

Severity routing

Every AlertRule carries a min_severity column (added by FEATURE-18). When the worker evaluates rules against a ValidationResult it first grades the result on a four-level scale:

Result shape	Severity
All expectations passed (or a passing zero-expectation run)	`info`
>0% of expectations failed	`warning`
>50% of expectations failed	`error`
Every expectation failed (or a failing zero-expectation run)	`critical`

The ordering is info < warning < error < critical. A rule fires only when the computed severity is at least the rule's min_severity.

Routing by severity

This lets a single checkpoint fan its noisy and its on-call traffic to different destinations — one rule per channel, each filtered by the lowest severity that warrants a wake-up:

Channel	`trigger`	`min_severity`	Why
PagerDuty	`on_failure`	`critical`	Page on-call only when the whole checkpoint is broken.
Slack `#data-warnings`	`on_failure`	`warning`	Catch partial failures during business hours.
Email (digest mailbox)	`always`	`info`	Audit trail; nothing actionable.

The min_severity default is error, which preserves the pre-FEATURE-18 behaviour for any rule written before this feature shipped — partial failures (warning) and informational info events are skipped unless the operator opts in.

Discord

The Discord channel (issue #81) POSTs to a Discord Incoming Webhook URL. Mint one under Server Settings → Integrations → Webhooks → New Webhook, then paste the URL into the rule's destination.webhook_url. The URL is encrypted at rest under the tenant DEK on the same path Slack uses; the API only ever returns the sentinel string "configured" on read.

The payload is Discord's native shape — a single embeds[0] block with title / description / color (and url when PUBLIC_BASE_URL is set). The severity colour maps to the four-level alert scale: blue for info, amber for warning, red for error, deep red for critical. Operator-supplied mention strings (e.g. <@USER_ID>, <@&ROLE_ID>) flow through destination.mentions and render in the top-level content field so they actually ping.

Mattermost

The Mattermost channel (issue #81) targets a self-hosted Mattermost's Slack-compatible Incoming Webhook endpoint. Configuration mirrors Discord — paste the webhook URL into destination.webhook_url; it's encrypted at rest. The payload is the Slack-style attachments envelope (with color, title, title_link, fallback, and text) rather than Block Kit because Mattermost's Block Kit support is partial across versions.

Twilio SMS

The Twilio SMS channel (issue #81) POSTs to /2010-04-01/Accounts/{sid}/Messages.json with HTTP Basic auth (account_sid as the username, auth_token as the password). The destination stores four fields:

Field	What it stores
`account_sid`	Twilio Account SID (`AC…`). Not a secret in isolation — it appears in the URL — but kept on the destination for symmetry.
`auth_token`	The matching auth token. Encrypted at rest under the tenant DEK; never echoed back to the UI.
`from_number`	The Twilio-owned `From` number in E.164 format.
`to_number`	The recipient number in E.164 format. SMS is point-to-point; fan-out to multiple recipients requires multiple alert rules.

SMS bodies are limited to 1600 chars (Twilio splits long messages into multiple segments and bills per segment); the adapter truncates at 1500 to leave headroom for an "Open: <url>" tail that always survives the truncation. Twilio rejects requests with delivery errors via 4xx — those re-raise so ARQ's retry-with-backoff kicks in.

Datadog Events

The Datadog Events channel (issue #81) POSTs to /api/v1/events with the DD-API-KEY header. The destination stores three fields:

Field	What it stores
`api_key`	Datadog API key (under Organization Settings → API Keys). Encrypted at rest under the tenant DEK.
`site`	Datadog region host. Default `api.datadoghq.com` (US1); other supported values are `api.datadoghq.eu` (EU1), `api.us3.datadoghq.com`, `api.us5.datadoghq.com`, `api.ap1.datadoghq.com`. The value must be a bare host starting with `api.` — pasting a full URL or a typo'd prefix is rejected at write time.
`tags`	Optional list of `key:value` tag strings appended to every event. The dispatcher always emits `source:dq-cloud` and `trigger:<rule.trigger>` in addition to whatever's configured here.

Datadog Events have only three alert types (info / warning / error); our four-level scale maps critical to error because there's no tier above that. Operators who want true urgency tiers should use Datadog Monitors and target a webhook channel back into Datadog's notification system.

Data model​

Channels​

Create incident (internal action)​

Ticketing channels​

Jira (NotifierAdapter)​

ServiceNow​

GitHub Issues​

Opsgenie​

Generic webhook​

Payload shape​

Verifying the signature​

Dispatch flow​

Operations you can do​

PagerDuty​

Jira​

Token instructions​

Payload shape​

Linear​

Token instructions​

Payload shape​

Severity mapping​

Severity routing​

Routing by severity​

Discord​

Mattermost​

Twilio SMS​

Datadog Events​

Data model

Channels

Create incident (internal action)

Ticketing channels

Jira (NotifierAdapter)

ServiceNow

GitHub Issues

Opsgenie

Generic webhook

Payload shape

Verifying the signature

Dispatch flow

Operations you can do

PagerDuty

Jira

Token instructions

Payload shape

Linear

Token instructions

Payload shape

Severity mapping

Severity routing

Routing by severity

Discord

Mattermost

Twilio SMS

Datadog Events