Profile runs

A profile run is a statistical snapshot of a data asset taken at a point in time. Measured Cloud uses profile runs for three things: surfacing column metadata in the UI, seeding AI rule suggestions, and detecting anomalies against a rolling baseline.

Data model

ProfileRun lives in the per-tenant schema.

Column	What it stores
data_asset_id	FK to the asset that was profiled.
row_count	The true asset-wide row count — a real SELECT COUNT(*) (SQL datasources) or the full-frame length (files). This is the headline "Rows".
sampled_row_count	How many rows the column statistics were actually computed over (the bounded representative sample). Distinct from row_count so the UI can say "profiled 50,000 of 4.2M rows" honestly.
schema_hash	A stable hash of the column-name + column-type set. Used by schema-drift detection (FEATURE-12).
column_stats	A JSONB blob: per-column type (real declared source_type + raw pandas dtype), null %, distinct count, min/max/mean, top values, and a per-column sampled flag. See Column statistics by type and Column type.
status	pending / running / completed / failed / cancelled (a user stopped it — see Stop a profile run under Operations you can do).
run_at / completed_at	Timestamps.

The worker's profile_run job writes the results. The API creates the pending row when you start a run, and the only other API write is a Stop, which sets status to cancelled.

True count vs sampled count

row_count and sampled_row_count are two different numbers and the distinction matters. Earlier builds set row_count to the sample size — so a 4-million-row table that was sampled at 10,000 rows reported "10,000 rows", and the per-column Unique % (distinct ÷ count) was computed against that wrong denominator. The profiler now issues a real dialect-quoted SELECT COUNT(*) for the true count and records the sample size separately. The true count feeds volume-drift and anomaly baselines; the sampled count is the denominator for the per-column Unique % and the likely-unique-key marker.

Column statistics by type

column_stats reports a useful statistic for every column type, not just numerics (earlier builds left date and string columns showing three blank dashes):

Column type	min / max	mean
Numeric	minimum / maximum value	arithmetic mean
Date / datetime	earliest / latest value (ISO-8601 string)	—
String / other	shortest / longest value length	average value length

An all-null numeric column reports null for these statistics rather than failing the whole profile.

Column type (source_type vs dtype)

Each column carries two type fields, and the UI shows the first:

• source_type — the column's real declared type in the source (VARCHAR, DATE, NUMBER(38, 2), TIMESTAMP, …). For SQL datasources the profiler reflects this from the source catalog (INFORMATION_SCHEMA.COLUMNS on Snowflake / BigQuery / Redshift / MSSQL, pg_catalog on Postgres, ALL_TAB_COLUMNS on Oracle, and so on) via a dialect-agnostic introspection, so it generalises across every SQL engine — not just Snowflake. This is the type the column health card displays.
• dtype — the internal pandas dtype the profiler saw while computing the sample statistics (object, float64, int64, …). It's retained for schema-drift detection and debugging, but it is not the user-facing type: pandas represents every text and date column as object and every numeric as float64, so a DATE and a VARCHAR are indistinguishable at the dtype level (Refs #226).

When the real declared type genuinely isn't available — a file/object-store asset with no catalog, or a reflection that the driver rejects — source_type falls back to a friendly label mapped from the dtype (object → text, float64 → number, int64 → integer, bool → boolean, datetime64 → datetime). The fallback can't recover DATE-vs-VARCHAR (both are object), so it's a last resort — but the raw internal object / float64 string is never surfaced as a column's type. Profile runs recorded before this field existed have no stored source_type; the API backfills the friendly label from dtype at read time, so older assets stop showing object / float64 without needing a re-profile.

What profile runs power

• Column dropdowns in the rule editor — when you author a not_null rule on an asset's Rules card, the column picker is the latest profile's column list.

• AI rule suggestions — the LLM gets the column stats as context and proposes plausible rules.

• Anomaly detection — a rolling window of profile runs is the baseline. Statistical deviation in row_count, null_percent, or other tracked metrics produces an AnomalyEvent.

• The asset detail UI — recent profile runs render as a sparkline of row count + a per-column health card table (see below).

  Status: partial (PARITY-3) — the profile-run UI is in flight.

The column health card

The per-column table on the data asset's page is built for a non-technical data steward — it leads with a plain-language verdict, not a grid of raw numbers (Refs #523, profiling Wave B). It shows, for each column:

• A health verdict — a small Healthy / Watch / Problem indicator with a one-line plain-language summary ("4% missing", "looks unique", "all one value"). It's derived from the served stats: a key-ish column with any nulls or any column more than half-missing reads problem; elevated nulls (>10%) or a near-constant single-value column read watch; everything else is healthy. No raw jargon (cardinality / p99 / z-score) appears in this primary view.
• A completeness bar — a visual fill for 1 − null_rate. The exact missing % stays in the bar's tooltip and the Details grid.
• A range / values cell — the humanised min/max for the column's type (numeric range, date span, or string character-length range) plus the average, so date and string columns are no longer three blank dashes.
• Top-value mini-bars — for categorical columns, the served top_values render as tiny ranked bars (most-frequent widest) inside the per-column Details expander (alongside the raw numeric grid), not on the primary row. The profiler serves values in frequency order without per-value counts, so the bars are an honest ordering cue, not a fabricated count.
• The likely-key badge — the #405 key marker (see below) stays next to the column name.
• An inline drift badge — a subtle marker next to the column name on any column whose metric is currently abnormal for this asset, read from the existing anomaly events. A column is matched to an anomaly two ways: per-column null-rate drift via the null_rate.<column> metric name, and FEATURE-14 completeness_drift events via the column field in their JSON notes payload. The badge is a link — clicking it opens the anomaly feed deep-linked to this asset (/anomalies?asset=<id>), so the steward lands on that asset's anomalies rather than the whole-org list.
• An "Add a rule" action — a one-click guardrail that pre-fills a sensible rule from the profile (not_null when there are no nulls, unique for a key-like column, between [min, max] for numerics, in set from the top values for a low-cardinality column) and hands it to the asset's Rules card pre-added — no new endpoint, the rule is derived entirely from the served stats.

A sampled affordance on the table flags the whole table as sample-based whenever any column was profiled over a sample, so Unique %, ranges, and averages read as estimates. The full numeric stat grid (distinct, Unique %, exact null %, mean, std dev) — and the top-value mini-bars — live behind the per-row Details expander so the primary view stays scannable on a wide table; a search box plus health filter and sort keep a 60-column table usable.

Histograms, column-to-column correlation, and PII detection are intentionally deferred to a later "advanced" view — the health card stays glanceable.

Likely-unique-key marker

The column metrics table flags likely unique-key columns with a key badge next to the column name (#405). The flag is a heuristic derived at read time from the sampled profile: a column is marked when, over the sample, its distinct count equals the sampled row count and it has no nulls. The table also shows a Unique % cell (distinct count ÷ sampled row count). The denominator is the sampled count, not the true row_count — the distinct count is itself a sample statistic, so comparing it to the true (much larger) count would mark every column non-unique on a sampled table.

Because the marker is computed from the sample — not from a declared schema constraint — read it as "this column looks like a unique identifier in the data we sampled", not as "the database declares this a primary key / unique constraint". A column can be unique in the sample but not in the full table, and vice versa. The column list (with the marker) shows on the data asset's page.

Status: planned (Refs #405) — capturing true primary-key / unique constraints from information_schema (or each dialect's equivalent) while listing a source's tables, and the optional auto-suggestion of an expect_column_values_to_be_unique rule for detected key columns, are follow-ups. Today the marker is a sampled estimate only.

Operations you can do

• Run profile now — enqueues a profile job. Start it from the Profile button on the asset page header, or the play (▶) button on the asset's row in the data source's assets table. You confirm first, because profiling samples the asset. While the run is queued or in progress, the button becomes a spinning stop control. It keeps spinning after a refresh, and when the run was started somewhere else: on the other page, by the nightly sweep, or by a teammate. A toast confirms when the profile run starts and another when it finishes, cancels, or fails. The finish toast still shows if you've moved to another page. Useful right after onboarding a new asset.
• Stop a profile run — click the spinning stop control, or call POST /api/v1/organizations/{org_id}/workspaces/{ws_id}/profile-runs/{run_id}/cancel (see the API reference). The stop is cooperative. A run still in the queue never starts. A run already sampling finishes that pass, but its statistics are thrown away and drift / anomaly detection is skipped. The run ends cancelled, and the asset's previous profile stays current. Stopping needs the same permission as starting a run (can_edit_asset, editor and above).
• Auto-profile — every asset gets profiled once a day by the worker's 3 AM UTC cron job. (HARDEN-6 makes that cadence configurable.)
• View historical profiles — the asset detail page lists the last N runs.

Connector coverage

Profiling runs against every SQL connector that validation supports — Postgres, Snowflake, Redshift, BigQuery, Databricks, MSSQL, ClickHouse, Oracle, SAP HANA, and Trino — using the same datasource dispatch and DSN-building the validation runner uses. (Earlier builds dispatched by substring and silently dropped every connector whose type string didn't contain sql/postgres/snowflake into the file branch, where the profile always failed.) File / object-store assets are profiled with pandas.

Agent-mode datasources

Profiling an asset that lives behind the on-prem agent is not yet supported. The cloud worker must never dial a firewalled datasource, so a profile request for an agent-mode asset is rejected at the trigger endpoint (HTTP 422) and the worker independently refuses to run one.

Status: planned — profiling through the agent (so agent-mode assets get the same column stats + anomaly detection) is a follow-up.

Sampling

For large tables, the profile job samples rather than scanning everything. The sampling cap is PROFILE_SAMPLE_MAX_ROWS (default 10,000; see Configuration), with a hard upper bound so a misconfigured cap can't drag a multi-million-row table into worker memory. The sample is a representative random sample (the dialect's ORDER BY RANDOM() / RAND() / NEWID() / DBMS_RANDOM.VALUE), not an unordered front-of-table slice — so a key- or partition-clustered table doesn't bias the stats. The sample query is built with SQLAlchemy Core, so the row cap compiles to each dialect's spelling — LIMIT n on Postgres / MySQL / Snowflake, TOP (n) on MSSQL, FETCH FIRST n ROWS ONLY on Oracle — rather than a bare LIMIT (which is a syntax error on MSSQL and Oracle).

Numerics, nulls, and uniqueness counts are computed against the sample — read them as estimates, not ground truth. The true row count (row_count) is always exact; only the per-column statistics are sampled, and each column carries a sampled flag so the UI can label them.