Skip to main content

Rules

A rule is one thing you verify about your data — "email is never blank", "order_total is between 0 and 100,000", "the table has at least one row". Every rule belongs to exactly one data asset: you add, edit, and remove rules on the asset's page, and a rule always runs against that asset's table.

Rules are the building block of everything downstream. A validation bundles the rules of one or more assets into a runnable unit, a schedule runs the validation on a cadence, and each run records a pass or fail per rule in its results.

Under the hood Measured Cloud runs rules with the open-source Great Expectations engine: each rule is one GX expectation (rule_type + kwargs), so the rules you write stay portable.

Where rules live​

Open any data asset — Data Assets → click a row — and its page leads with an AI summary, then Table Health (the latest profile), then the Rules card. The Rules card is the one place you author rules for that asset:

  • Rules are a table, one per row: the plain-English label ("Email is never blank"), its target, its severity, its team, and the validations that run it. Sort and filter from the column headers (the rule's name or target contains some text; a checklist for severity, team or validation, including Runs with the asset only), and turn columns off from Columns.
  • Rows sit in folders by rule type (or by column; Group by → None for one flat list). A folder starts collapsed with its rule count, severity breakdown and any unsaved rows on its header; Expand all opens them. While a filter is on, the folders open so every match shows.
  • Click anywhere on a row to open it in place: the editor, the validations it's in, and its memory. Only a new or edited row is marked Unsaved.
  • A new or edited row carries its own Save. Saving an edited rule updates that one rule in place, so it keeps its identity: the validations it belongs to and its memory stay with it. Removing a saved rule persists straight away when it's the only pending change, so a deletion never gets stranded behind an unsaved draft.
  • A Runs in line names the validations that already run this asset's rules, each linking to the validation.

Anyone with the editor role (or higher) can change rules. Viewers see the same card with every control that changes something (including the AI Suggested lane's buttons) disabled, and a clicked rule opens its settings read-only.

Adding a rule​

The Rules card's Add a rule area offers two routes to the same result — a new rule you review and Save:

  • Add rule — opens the rule gallery (below), where you pick the rule type, then fill in its settings on the new row. Column pickers are populated from the asset's latest profile, so profile an asset before you author rules against it. It's the one way to pick a rule by hand: there's no separate type list or "Browse the gallery" button on the asset page.
  • Describe it to AI — write what you want in plain English ("shipped orders always have a shipping address") and AI drafts a Custom SQL rule for you to review before it's added. This needs an AI engine; see AI features. It appears once the asset has at least one saved rule.

You can also:

  • The rule gallery — a searchable catalog of every rule type (plus Custom SQL and any custom rules registered for your organization, under Custom rules), with a one-sentence description, an example, the settings it takes, and the column types it applies to. The gallery is a dialog, not a page: it opens on top of whatever you're doing. Open it from the Rules card (Add rule), or from the command palette (Cmd+K / Ctrl+K) → Rule Gallery. On an asset's page the rule goes to that asset. Opened anywhere else, the gallery first asks which data asset the rule is for (a searchable list of your workspace's assets, narrowed to your selected team; Browse rules first lets you look before choosing), then takes you to that asset's page once you use a rule. Use this rule adds the rule as an unsaved draft, opened for editing. It is never saved on the spot, because the gallery fills in example settings. The column is pre-filled only when the asset really has the example column; otherwise you pick one. On the asset page, Add rule is disabled for viewers (with a tooltip naming the role it needs); they can still browse the catalog from the command palette, but not use a rule. Old links to /rules/gallery or /expectations/gallery still work: they open the Data Assets list, or the asset named by ?assetId=, with the gallery open.
  • Generate rules with AI — proposes a batch of rules from the asset's profile. Proposals land in a Suggested lane at the top of the card: Accept all saves the batch, or Add one proposal to review it as a draft.
  • Apply template — merge a reusable rule template into this asset (it only ever adds rules, never removes yours). Save as template does the reverse.
  • Import from GX — paste an existing Great Expectations ExpectationSuite; every expectation in it becomes a rule on the asset. See Import from Great Expectations.

Save stays disabled while a rule points at a column the asset's latest profile doesn't report, so a typo can't slip in. (A rule that was saved before the column disappeared is left alone and marked "not in latest profile".)

Which validations a rule is in​

Every rule is always in its asset's default validation (see Validations). You choose which other validations run it:

  • When you add it. The Rules card's New rules join line names the validations every new rule joins — by hand, from the gallery, or from an AI suggestion (Accept all included). Change it with its list icon: tick existing validations, or Create a new validation by name. Each unsaved draft, once opened, shows Joins on save with its own list, which you can change before you save it (its row's Validations column reads them "on save"). The rule and its validations are saved together: if a validation can't take the rule, nothing is saved.
  • After it's saved. A saved rule's Validations column lists the validations that run it besides its asset's default (see Monitor a rule's validations). Open the rule for its In validations chips. The list icon opens Edit validations: tick or untick validations, or create a new one. Only the asset's default validation is locked; a chip for any other validation has a ✕ to take the rule out of it.

A new rule always joins its asset's default validation, the validations you pick for it, and every validation that holds All rules of its asset (see How a validation is built). It never joins a validation that holds only Selected rules of the asset unless you pick it.

Monitor a rule's validations​

The Validations column is the way from a rule to the runs that check it. Each validation that runs the rule is a chip showing its last run's status (a ✓ for Passed, a ✕ for Failed, a spinner while it runs, each with the word for screen readers and in its tooltip) and when that run was. Click a chip to open the validation at its Run history. A rule in more than two validations shows the first two and +N; click +N for all of them.

A saved rule in no validation besides its asset's default reads Runs with the asset only: it still runs every time the asset runs (its default validation), just in no other validation. The list icon beside it adds it to one (it needs the permission to manage validations; without it the icon is disabled). Filter the column from its header to the rules in one validation, or to Runs with the asset only.

Adding a rule to a validation, or taking it out, changes what that validation's next run executes. Changing a rule's validations needs the permission to manage validations, as well as to edit rules.

Rule types​

The catalog covers the everyday checks without any SQL:

RuleWhat it checks
Not nullThe column has no nulls.
UniqueEvery value in the column is distinct.
In set / not in setEvery value is from (or never from) a list you give.
Matches / doesn't match a patternEvery value matches (or never matches) a regular expression.
Value betweenNumeric values fall in [min, max].
Length between / length equalsString lengths fall in a range, or equal N.
Mean, sum, min, max betweenA column aggregate falls in a range.
Row count between / equalsThe table's row count is in a range, or exactly N.
Column count equalsThe table has exactly N columns.
Custom SQLAny condition you can express as a SELECT — see below.

The engine name of each rule (for example expect_column_values_to_not_be_null) is shown on hover, and it is what the API stores.

Severity​

Every rule carries a severity: Info, Warning, or Critical (the default). Severity is how you say how much a failure matters:

  • The result page colour-codes each rule by its severity.
  • Alert rules pick which severities they fire on — for example, page on-call only for Critical, send Warning to a Slack channel.

Re-evaluate severities with AI. If an asset's rules all ended up Critical (say, rules saved before AI rated severities), click Re-evaluate severities on its Rules card. The AI reads each saved rule, and the asset's latest profile when there is one, and proposes a severity for each with a one-line reason. The dialog lists every rule as current → proposed. Each proposed change starts ticked: click Apply all, or untick the ones you disagree with and click Apply selected. Nothing changes until you apply. A rule whose severity you changed after the review started reads "Changed since the review" and isn't overwritten. Starting a review needs the same role as generating rules with AI; applying it needs a role that can edit rules. See AI features.

Custom SQL rules​

When a check needs more than the catalog, write a Custom SQL rule: a SELECT that returns the rows that break the rule. An empty result is a pass.

{
  "rule_type": "unexpected_rows_expectation",
  "kwargs": {
    "query": "SELECT id, amount, status FROM {batch} WHERE amount < 0 AND status = 'shipped'",
    "description": "no shipped order has a negative amount",
    "severity": "critical"
  }
}

{batch} is the only template variable: at run time it becomes the asset's fully-qualified, quoted table name (for example "analytics"."raw_orders"). The result page shows the rendered SQL and up to 10 sample failing rows, so you can reproduce a failure in your own SQL client.

Measured Cloud validates custom SQL before saving it:

  • The query must contain {batch} and start with SELECT or WITH.
  • It may not contain ;, --, or /*.
  • It may not contain a write or side-effecting keyword (INSERT, UPDATE, DELETE, DROP, TRUNCATE, ALTER, CREATE, GRANT, REVOKE, COPY, MERGE, CALL, EXEC, EXECUTE, DO, SET, LOCK, VACUUM, ANALYZE).
  • It may not reference system catalogs or privileged functions (pg_catalog, information_schema, pg_sleep, pg_read_file, dblink, and similar).
  • A description is required, so reviewers know what the rule asserts without reading the SQL.

The keyword filter is defense in depth. The real safety boundary is the least-privilege, read-only database role you give Measured Cloud — see Data sources → Security notes.

Row conditions​

Column-map rules accept an optional row_condition that scopes the rule to a subset of rows. The canonical example: "shipping_address is not null when the status is shipped".

{
  "rule_type": "expect_column_values_to_not_be_null",
  "kwargs": {
    "column": "shipping_address",
    "row_condition": "col(\"status\") == \"shipped\""
  }
}

The condition is not SQL. Measured Cloud hands it to Great Expectations' own condition parser (fixed, because we run against SQL backends), and that parser accepts exactly one comparison in GX syntax:

You wantWrite
equals / not equalscol("status") == "shipped", col("status") != "cancelled" (single quotes around the value work too)
numeric comparisoncol("amount") > 100, col("amount") <= -2.5, col("qty") >= 1e3
date comparisoncol("created_at") >= date("2024-01-01")
is not nullcol("email").notnull()

The rules:

  • col("…") takes the column name in double quotes, with no spaces inside the parentheses.
  • The operator is ==, !=, <, <=, > or >=. Plain = and <> aren't accepted.
  • The value is a number, a quoted string, or date("…"). Quoted values can use letters (including Latin-1 accented letters like é), digits, spaces and ASCII punctuation, but no quote characters.
  • There's no AND / OR, no IS NULL, and no SQL functions. Use a Custom SQL rule (above) for those, or for anything involving joins or other tables.

The API checks the condition on save against the same grammar GX uses. A condition GX can't run is rejected with an error that says what's wrong and, for simple SQL-style text, suggests the rewrite (422 from the asset Rules, template and AI-accept endpoints). For example, status = 'shipped' gets "Did you mean: col("status") == "shipped"". The rule editor shows the same message inline as you type. The condition compiles to a parameterised SQLAlchemy filter (a quoted column name and a bound value), so a value that happens to look like a SQL keyword, such as col("action") == "update", is fine.

Rules saved before this check. Earlier the editor suggested SQL-style conditions like status = 'shipped', and the API accepted them. GX can't parse them, so the run skipped the rule and still reported green. Two things changed:

  • A run now reports such a rule as "Could not run". It appears in the result as a failed rule with the reason, and the run is marked failed. (On an agent-mode datasource the on-prem agent strips error text before it sends results, so the rule shows as failed without the reason.)
  • Operators can list the affected rules without changing anything, using dqc find-broken-row-conditions.

A condition with a trailing AND … was different: GX silently ran only the first comparison. The same command finds those too.

Custom expectation plugins​

Teams with their own Great Expectations expectation classes can run them in Measured Cloud without forking. An operator installs the Python package into the worker image, lists it in EXPECTATIONS_PLUGIN_PACKAGES (see Configuration), and restarts the worker. Registered classes then appear in the rule catalog with a form generated from their parameters. Settings → Plugins shows each package's import status, so a failed import is visible without reading worker logs.

Running rules​

Rules run as part of a validation — manually with Run, or automatically on a schedule. A validation run executes exactly the enabled rules the validation holds. A disabled rule stays in its validations but doesn't run. You can also run them directly while you author:

  • Run this asset's rules — the ▶ button in the asset header runs every rule on the asset now.
  • Run one rule — each saved rule has its own ▶ in the Rules card.

Both are runs of the asset's default validation (a single rule's run is restricted to that rule), so they appear in its execution runs and count toward its health. Both work on agent-mode data sources too: a single rule's run sends only that rule to your on-prem agent.

While a run is going, the Rules card shows each rule's live state (Queued, Running, then Passed or Failed). The engine checks all of an asset's rules in one pass, so while the asset's rules run, every rule's ▶ becomes a spinning stop control. Click any of them to stop the run: the asset in progress is interrupted within a few seconds and records no result. If the asset is running as part of a larger validation run, only that asset's part stops and the validation carries on. An asset running on your on-prem agent finishes there. See Validations → Watch and stop a run.

Running is read-only on your data source: Measured Cloud reads your data and records results, and changes nothing in the source.

History and memory​

  • History on the asset header opens the audit trail of every change to the asset and its rules.
  • Each rule can carry a short memory note and a positive or negative flag — context such as "fails every month-end, expected" that AI explanations take into account. Changing what a rule checks (its type, column or other settings) clears its memory, because the note described the old check; changing only its severity keeps it.

API​

Every rule is its own record with a stable id. Rules are written on their asset, one at a time, and can be read per asset or across the workspace:

GET    /api/v2/organizations/{org_id}/workspaces/{ws_id}/rules
GET    /api/v2/organizations/{org_id}/workspaces/{ws_id}/rules/{rule_id}
GET    /api/v2/organizations/{org_id}/workspaces/{ws_id}/data-assets/{asset_id}/rules
POST   /api/v2/organizations/{org_id}/workspaces/{ws_id}/data-assets/{asset_id}/rules
PATCH  /api/v2/organizations/{org_id}/workspaces/{ws_id}/data-assets/{asset_id}/rules/{rule_id}
DELETE /api/v2/organizations/{org_id}/workspaces/{ws_id}/data-assets/{asset_id}/rules/{rule_id}

GET .../rules lists every rule you can see, newest first. Narrow it with ?data_asset_id= (one asset) or ?validation_id= (the rules a validation holds), and page it with limit and cursor (the response then carries next_cursor; without either you get the full list). GET .../rules/{rule_id} returns one rule with validation_ids, the validations that hold it. Each rule carries id, data_asset_id, rule_type, kwargs, severity, name, description, enabled, position, its memory fields, and timestamps. Reading rules needs can_view_rule (viewer and up). Writing them, including deleting one rule, needs can_edit_rule (editor and up).

The asset's GET lists its rules in order; each carries its id and position next to rule_type and kwargs (the response's suite_id is deprecated and always null). POST with {"data": {"rule_type": "...", "kwargs": {...}}} adds one rule (add "position" to insert it at that place). Add "validation_ids": [...] to put the new rule in those validations too, and "create_validation": {"name": "..."} to create one around it; both are saved with the rule, or not at all. PATCH edits one rule in place and keeps its id, so the validations it belongs to are unaffected. DELETE removes one rule and takes it out of its validations.

PUT (or POST with a rules list) still replaces the asset's whole rule list: {"data": {"rules": [{"rule_type": "...", "kwargs": {...}}]}}. An entry that carries a rule's id updates that rule in place; an entry without one matches an identical existing rule, or is added; rules left out are deleted. An entry can also carry validation_ids and create_validation (entries naming the same new validation share it); the response lists any validations it created in created_validations.

PUT .../data-assets/{asset_id}/rules/{rule_id}/validations with {"data": {"validation_ids": [...]}} sets which validations the rule is in (the asset's default always keeps it; create_validation works here too). Related routes run the rules (.../rules/run, .../rules/{rule_id}/run), apply a template (.../rules/apply-template), and list the validations that run them (.../rules/running). See the API reference.

The retired /api/v1/.../rule-suites endpoints are a read-only alias for one release; their writes return 410 Gone. See Rule suites (retired).