Anomalies

An anomaly is a statistical deviation in a tracked metric — row count, null percentage, distinct count — relative to that asset's recent profile runs. Anomalies are how PLACEHOLDER Cloud catches problems your rule suites didn't explicitly assert against.

Data model

AnomalyEvent lives in the per-tenant schema.

Column	What it stores
data_asset_id	FK to the affected asset.
profile_run_id	FK to the profile run that triggered detection.
metric_name	What deviated — row_count, null_percent:email, schema_drift, volume_drift, etc.
severity	info / low / medium / warning / high / critical. This is the six-level anomaly scale, derived from the deviation's z-score and floor rules — distinct from the four-level alert min_severity in Alerts.
baseline_value	The rolling-window baseline mean (or, for schema drift, zero — the diff lives in notes).
observed_value	What the latest profile recorded.
z_score	The deviation expressed as standard deviations from the baseline mean.
acknowledged_at	When a human acknowledged the event (NULL if open).
notes	Free-text or structured payload — for schema drift, the column-level diff JSON; for volume drift, {observed, baseline_mean, baseline_stddev, z_score}.

How detection works

The worker's profile_run job invokes the anomaly detector after writing a new ProfileRun. The detector:

1. Loads the prior N profile runs for the asset (default N = 14).
2. Computes a baseline mean + standard deviation per tracked metric.
3. Compares the new value; emits an AnomalyEvent with a z-score-derived severity when the deviation exceeds threshold.
4. For row counts specifically, a relative-drop floor also fires: any drop ≥ 25 % below the baseline mean is at least high, regardless of z-score.

In addition, the dedicated volume-drift detector (FEATURE-13) emits metric_name = "volume_drift" events once the asset has accumulated at least VOLUME_DRIFT_WINDOW_RUNS completed runs (default 10). It fires when either the z-score exceeds 3 or the observed row count is more than 50 % off the baseline mean. Severity is critical when the observed value falls outside [0.5×, 1.5×] of the mean (the "10× spike / 90 % drop" pipeline-breakage cases), warning for smaller-but-statistically-unprecedented shifts inside that band.

The detector is also the source of schema-drift signals (FEATURE-12) and per-column completeness signals — metric_name = "null_rate:<column>" events fire once the asset has accumulated at least COMPLETENESS_DRIFT_WINDOW_RUNS completed runs (default 10), with the same z-score / relative-shift heuristics the volume detector uses. Shipped in FEATURE-14.

Operations you can do

• List anomalies — Anomalies route shows open events, filterable by severity, asset, and metric.

• Acknowledge an anomaly — set acknowledged_at so it drops out of the open list.

• Explain an anomaly with AI — click any anomaly to open the explain panel; the LLM summarises the deviation in natural language with a link to the underlying numbers.

  Status: partial (PARITY-9) — the explain panel UI is in flight; the backend hook is in place.

• Route anomalies to alerts — AlertRule.trigger = "on_anomaly" fires the rule when a matching anomaly is written. See Alerts.

Tuning

The volume-drift baseline window is operator-tunable via the VOLUME_DRIFT_WINDOW_RUNS env var; the completeness-drift window via COMPLETENESS_DRIFT_WINDOW_RUNS (see Configuration). Per-asset overrides — different thresholds for high-cadence vs. weekly assets — remain on the roadmap.