Profile runs

A profile run is a statistical snapshot of a data asset taken at a point in time. PLACEHOLDER Cloud uses profile runs for three things: surfacing column metadata in the UI, seeding AI rule suggestions, and detecting anomalies against a rolling baseline.

Data model

ProfileRun lives in the per-tenant schema.

Column	What it stores
data_asset_id	FK to the asset that was profiled.
row_count	The true asset-wide row count — a real SELECT COUNT(*) (SQL datasources) or the full-frame length (files). This is the headline "Rows".
sampled_row_count	How many rows the column statistics were actually computed over (the bounded representative sample). Distinct from row_count so the UI can say "profiled 50,000 of 4.2M rows" honestly.
schema_hash	A stable hash of the column-name + column-type set. Used by schema-drift detection (FEATURE-12).
column_stats	A JSONB blob: per-column type, null %, distinct count, min/max/mean, top values, and a per-column sampled flag. See Column statistics by type.
status	pending / running / completed / failed.
run_at / completed_at	Timestamps.

The worker's profile_run job is the only thing that writes to this table. The API only reads.

True count vs sampled count

row_count and sampled_row_count are two different numbers and the distinction matters. Earlier builds set row_count to the sample size — so a 4-million-row table that was sampled at 10,000 rows reported "10,000 rows", and the per-column Unique % (distinct ÷ count) was computed against that wrong denominator. The profiler now issues a real dialect-quoted SELECT COUNT(*) for the true count and records the sample size separately. The true count feeds volume-drift and anomaly baselines; the sampled count is the denominator for the per-column Unique % and the likely-unique-key marker.

Column statistics by type

column_stats reports a useful statistic for every column type, not just numerics (earlier builds left date and string columns showing three blank dashes):

Column type	min / max	mean
Numeric	minimum / maximum value	arithmetic mean
Date / datetime	earliest / latest value (ISO-8601 string)	—
String / other	shortest / longest value length	average value length

An all-null numeric column reports null for these statistics rather than failing the whole profile.

What profile runs power

• Column dropdowns in the rule-suite editor — when you author a not_null rule, the column picker is the latest profile's column list.

• AI rule suggestions — the LLM gets the column stats as context and proposes plausible rules.

• Anomaly detection — a rolling window of profile runs is the baseline. Statistical deviation in row_count, null_percent, or other tracked metrics produces an AnomalyEvent.

• The asset detail UI — recent profile runs render as a sparkline of row count + a per-column health card table (see below).

  Status: partial (PARITY-3) — the profile-run UI is in flight.

The column health card

The per-column table on the data-source / data-asset detail page is built for a non-technical data steward — it leads with a plain-language verdict, not a grid of raw numbers (Refs #523, profiling Wave B). Expanding an asset's row shows, for each column:

• A health verdict — a small Healthy / Watch / Problem indicator with a one-line plain-language summary ("4% missing", "looks unique", "all one value"). It's derived from the served stats: a key-ish column with any nulls or any column more than half-missing reads problem; elevated nulls (>10%) or a near-constant single-value column read watch; everything else is healthy. No raw jargon (cardinality / p99 / z-score) appears in this primary view.
• A completeness bar — a visual fill for 1 − null_rate. The exact missing % stays in the bar's tooltip and the Details grid.
• A range / values cell — the humanised min/max for the column's type (numeric range, date span, or string character-length range) plus the average, so date and string columns are no longer three blank dashes.
• Top-value mini-bars — for categorical columns, the served top_values render as tiny ranked bars (most-frequent widest) inside the per-column Details expander (alongside the raw numeric grid), not on the primary row. The profiler serves values in frequency order without per-value counts, so the bars are an honest ordering cue, not a fabricated count.
• The likely-key badge — the #405 key marker (see below) stays next to the column name.
• An inline drift badge — a subtle marker next to the column name on any column whose metric is currently abnormal for this asset, read from the existing anomaly events. A column is matched to an anomaly two ways: per-column null-rate drift via the null_rate.<column> metric name, and FEATURE-14 completeness_drift events via the column field in their JSON notes payload. The badge is a link — clicking it opens the anomaly feed deep-linked to this asset (/anomalies?asset=<id>), so the steward lands on that asset's anomalies rather than the whole-org list.
• An "Add a check" action — a one-click guardrail that pre-fills a sensible rule from the profile (not_null when there are no nulls, unique for a key-like column, between [min, max] for numerics, in set from the top values for a low-cardinality column) and hands it to the rule-suite editor pre-added — no new endpoint, the rule is derived entirely from the served stats.

A sampled affordance on the table flags the whole table as sample-based whenever any column was profiled over a sample, so Unique %, ranges, and averages read as estimates. The full numeric stat grid (distinct, Unique %, exact null %, mean, std dev) — and the top-value mini-bars — live behind the per-row Details expander so the primary view stays scannable on a wide table; a search box plus health filter and sort keep a 60-column table usable.

Histograms, column-to-column correlation, and PII detection are intentionally deferred to a later "advanced" view — the health card stays glanceable.

Likely-unique-key marker

The column metrics table flags likely unique-key columns with a key badge next to the column name (#405). The flag is a heuristic derived at read time from the sampled profile: a column is marked when, over the sample, its distinct count equals the sampled row count and it has no nulls. The table also shows a Unique % cell (distinct count ÷ sampled row count). The denominator is the sampled count, not the true row_count — the distinct count is itself a sample statistic, so comparing it to the true (much larger) count would mark every column non-unique on a sampled table.

Because the marker is computed from the sample — not from a declared schema constraint — read it as "this column looks like a unique identifier in the data we sampled", not as "the database declares this a primary key / unique constraint". A column can be unique in the sample but not in the full table, and vice versa. The same column list (with the marker) shows on both the data source detail page and the dedicated data-asset detail page.

Status: planned (Refs #405) — capturing true primary-key / unique constraints from information_schema (or each dialect's equivalent) during discovery, and the optional auto-suggestion of an expect_column_values_to_be_unique rule for detected key columns, are follow-ups. Today the marker is a sampled estimate only.

Operations you can do

• Run profile now — enqueues a profile job; the UI polls until completion. Useful right after onboarding a new asset.
• Auto-profile — every asset gets profiled once a day by the worker's 3 AM UTC cron job. (HARDEN-6 makes that cadence configurable.)
• View historical profiles — the asset detail page lists the last N runs.

Connector coverage

Profiling runs against every SQL connector that validation supports — Postgres, Snowflake, Redshift, BigQuery, Databricks, MSSQL, ClickHouse, Oracle, SAP HANA, and Trino — using the same datasource dispatch and DSN-building the validation runner uses. (Earlier builds dispatched by substring and silently dropped every connector whose type string didn't contain sql/postgres/snowflake into the file branch, where the profile always failed.) File / object-store assets are profiled with pandas.

Agent-mode datasources

Profiling an asset that lives behind the on-prem agent is not yet supported. The cloud worker must never dial a firewalled datasource, so a profile request for an agent-mode asset is rejected at the trigger endpoint (HTTP 422) and the worker independently refuses to run one.

Status: planned — profiling through the agent (so agent-mode assets get the same column stats + anomaly detection) is a follow-up.

Sampling

For large tables, the profile job samples rather than scanning everything. The sampling cap is PROFILE_SAMPLE_MAX_ROWS (default 10,000; see Configuration), with a hard upper bound so a misconfigured cap can't drag a multi-million-row table into worker memory. The sample is a representative random sample (the dialect's ORDER BY RANDOM() / RAND() / NEWID() / DBMS_RANDOM.VALUE), not an unordered front-of-table slice — so a key- or partition-clustered table doesn't bias the stats. The sample query is built with SQLAlchemy Core, so the row cap compiles to each dialect's spelling — LIMIT n on Postgres / MySQL / Snowflake, TOP (n) on MSSQL, FETCH FIRST n ROWS ONLY on Oracle — rather than a bare LIMIT (which is a syntax error on MSSQL and Oracle).

Numerics, nulls, and uniqueness counts are computed against the sample — read them as estimates, not ground truth. The true row count (row_count) is always exact; only the per-column statistics are sampled, and each column carries a sampled flag so the UI can label them.