Skip to main content

Checkpoints

A checkpoint is the unit of execution. It bundles one or more (asset, rule-suite) pairs and runs them as a single job — manually with Run now or automatically on a schedule.

Data model​

Checkpoint lives in the per-tenant schema.

ColumnWhat it stores
nameHuman label (orders-daily).
checkpoint_configA JSONB blob describing which validation configs (asset + suite + batch definition) to execute and in what order.

A checkpoint references ValidationConfig rows, which in turn reference a DataAsset and a RuleSuite. Multiple suites against the same asset (or multiple assets) belong inside one checkpoint when you want them to succeed-or-fail together.

ScheduledRun is the cron metadata attached to a checkpoint.

RunQueue is the work-queue table — one row per actual run. The worker picks rows from here.

Lifecycle​

  1. You click Run now on a checkpoint, or ScheduledRun.next_run_at <= now().
  2. The API (or the scheduler cron job) writes a RunQueue row with status = pending. If the linked ValidationConfig carries a batch_definition_id, that batch slice rides through with the row (see Batch definitions).
  3. The worker claims the row (status = claimed) and hydrates the checkpoint config — it loads the linked DataAsset, RuleSuite, BatchDefinition, and Datasource, decrypts the datasource credentials, and renders any {batch} placeholders in custom-SQL rules against the actual fully-qualified table name (or the batch-scoped subquery when a non-whole_asset batch is in play).
  4. The worker spawns an isolated subprocess sandbox (DQ_EXECUTOR_BACKEND=local) with a stripped environment and RLIMIT_* caps (DQ_EXECUTOR_MEMORY_MB, DQ_EXECUTOR_MAX_OPEN_FILES, DQ_EXECUTOR_MAX_FILE_SIZE_MB). The subprocess constructs an ephemeral Great Expectations 1.x pipeline — add_postgres (or the matching dialect helper) → add_table_asset → add_batch_definition_whole_table (or the batch-aware variant) → ExpectationSuite → ValidationDefinition → Checkpoint — then runs it against the customer database. The runner emits a single JSON payload on stdout; GX warnings + progress bars are redirected to stderr so they don't poison the payload.
  5. The worker writes a ValidationResult row per config and moves RunQueue.status to success or failed. It then evaluates any matching AlertRule rows.

Operations you can do​

  • Create / edit / delete — Checkpoints → New (UI) or the corresponding REST endpoints. Deleting a checkpoint removes its operational state (any schedule and queued runs) but preserves run history: existing ValidationResult rows and AlertRule rows are detached from the deleted checkpoint (their checkpoint_id is set to null) rather than destroyed, so the audit trail survives. The delete returns 204.
  • Run now — enqueues immediately and polls until completion. Running a checkpoint is read-only on your source: it reads your data and records a result, and changes nothing in the source itself.
  • Add a schedule — cron expression + timezone. Schedule CRUD + the cron worker job are shipped; per-tenant timezones shipped in FEATURE-30.
  • Backfill / run-subset — re-run a checkpoint against a list of batch slices (e.g. the last 30 daily partitions) via POST /api/v1/.../checkpoints/{id}/run-subset with a body of {"batch_definition_ids": [...]}. Shipped in FEATURE-37.

Failures​

When a checkpoint produces a ValidationResult with success = false, PLACEHOLDER Cloud evaluates every AlertRule whose scope matches the checkpoint. Matching rules generate AlertEvent rows; the worker dispatcher sends them out to the configured channels. See Alerts.