Skip to main content

Agent mode

When the customer's database lives behind a firewall, agent mode runs the validation inside their network. Only redacted metadata (counts, success flags, schema column names, severity) crosses back to Measured Cloud — sample failing rows, comparator values, and rendered SQL never leave the customer's side.

Compare with direct mode, where the cloud worker connects straight to the customer DB with an encrypted DSN.

Architecture​

  1. A user clicks Run on a validation, an asset, or a single rule (or a schedule fires). The API plans the run's passes (validation → asset → datasource); for each asset on an agent-mode datasource it writes a RunQueue row with execution_mode = "agent", carrying the run's rule ids, and does not enqueue a worker job. A single rule's Run sends only that rule to the agent.
  2. The on-prem agent polls GET /api/agent/v1/jobs every few seconds.
  3. The agent calls POST /api/agent/v1/jobs/{id}/claim to grab the row atomically (a Postgres-side compare-and-set on claimed_by_agent_token_id IS NULL).
  4. The claim response carries the dispatch envelope: the data asset's table + schema + dialect, and the expectations for exactly the rules the run includes (its rule-id snapshot; a single rule's run sends one), custom-SQL rules included. No DSN. The agent has its own local mapping from datasource UUID to local connection string.
  5. The agent constructs the same Great Expectations pipeline the cloud worker uses (the runner lives in shared/, imported by both), runs it against the customer DB, and then runs the result through the redactor (shared/src/dq_shared/agent_payload.py).
  6. The agent POSTs the redacted payload to POST /api/agent/v1/jobs/{id}/complete. The API runs the same redactor as a validator — if anything forbidden slipped through, the payload is rejected with 400 and the job stays claimed for retry.
  7. On accept, the API writes a ValidationResult row, marks RunQueue.status = "success" (or "failed"), and fires the existing alert + anomaly evaluation pipeline.

What crosses the boundary​

Measured Cloud receives only:

FieldWhy it's safe
success (overall and per-expectation)Boolean flag, derived from counts.
statistics (evaluated_validations, successful_validations, success_percent)Numeric counts.
expectation_typeThe rule's machine name (expect_column_values_to_not_be_null).
descriptionAuthor-supplied prose from the rule — not data-derived.
severityAuthor-supplied (info / warning / critical; #154 dropped the old error tier, folding it up into critical).
columnsSchema metadata. Column names only — the redactor extracts these from kwargs.column / kwargs.column_list and drops the rest of kwargs.
result.unexpected_count, result.evaluated_countCounts.
result.observed_valueOnly kept if it's a number; strings or arrays are dropped.

Measured Cloud never receives:

  • unexpected_rows — the sample failing rows GX collects.
  • rendered_sql — the {batch}-substituted query the runner executed (it carries both schema names and value-comparator literals).
  • kwargs — the GX expectation kwargs, which include value_set, literal min_value / max_value, the unexpected-rows query, regex patterns, etc. The redactor extracts column names into columns and drops the rest.
  • partial_unexpected_list / unexpected_list / unexpected_index_list — alternate row-sample representations.

The cut is enforced twice:

  1. Agent side — redact_gx_result(payload) walks the full GX result and emits only the allowed-keys subset before the agent submits.
  2. Cloud side — validate_redacted_payload(payload) walks the incoming payload at /complete and raises 400 if any forbidden key is present. So even a regressed agent can't smuggle data past the boundary.

Trust model​

  • The agent authenticates with a dqa_-prefixed token issued in the cloud UI by a super admin or technical admin. The token is hashed at rest; only its SHA256 lives in public.agent_tokens.
  • The token scopes the agent to one org + optional workspace. The cloud rejects job claims whose linked datasource isn't owned by the authenticating token's agent_token_id.
  • The atomic CAS on /claim means two agents (e.g. a blue/green rollover) can't both run the same job. Exactly one wins; the other gets 409.
  • Credentials to the customer DB live only in the agent's local env (DQ_AGENT_DATASOURCE_DSNS). They never enter the cloud control plane and are never carried in the dispatch envelope. The agent's sys.excepthook scrubs the DSN from any uncaught traceback before it lands in stderr, so even a misconfigured DSN can't leak via the /fail error message.

Connection status on the datasources list​

Because an agent-mode datasource can only run a profile or a validation when its agent is actually connected, the agent's health is surfaced inline on each source's group header on the Data Assets page (and in the header of its Manage source window), not just under Settings → Agents:

BadgeMeaning
Agent online (green)The agent checked in within the last 5 minutes.
Agent offline (red)The agent has connected before but not within the last 5 minutes.
Agent pendingThe agent token is registered but has never checked in.
Agent revoked (red)The agent's token was revoked — it can no longer connect.
Direct (neutral)A direct-mode source; no agent is involved.

The "online" window is the same 5-minute last-seen threshold the worker uses to reclaim a stale claim (agent_claim_ttl_seconds, default 300 s), so the badge, the Settings → Agents page, and stale-claim reclamation all agree on what "online" means.

When the agent is offline or revoked, the per-table Profile control in the source's Manage source window is disabled with the tooltip "Agent is offline. Check Settings → Agents." — this stops a user from triggering a run that is guaranteed to fail (or get stuck "running") while the agent is down.

The status rides on the datasource API payload itself (an agent block of {status, last_seen_at, token_name} on the V2 datasources list/get), so every role that can see the datasource sees the badge — it does not require the agent-tokens endpoint (super admin or technical admin only). The block carries only non-secret status fields; the token hash never leaves the server.

Choosing between modes​

Use direct mode when…Use agent mode when…
The cloud can reach the customer DB.The DB is in a private VPC / on-prem network.
The customer is willing to share encrypted credentials with the cloud.The customer can't share credentials.
You want minimum operational surface for the customer.The customer already runs services in their network.
You want zero extra latency between trigger and run.You're OK with the agent's poll interval (default 5 s).

The two modes coexist per-org and per-datasource — one customer can mix direct-mode (their public Redshift cluster) and agent-mode (their internal HR Postgres) under the same workspace.

See also​