Skip to main content

Lakehouse checks (Iceberg + Delta)

PLACEHOLDER Cloud treats Apache Iceberg and Delta Lake tables as first-class datasources. You connect a catalog, point at a table, and write the same rule suites you would for a Postgres or Snowflake asset — PLACEHOLDER Cloud handles the snapshot read for you. Two new expectation types layer on top so you can alert on snapshot-level concerns no row-DB adapter can replicate.

This page covers what's shipped today, the trust model around materialising data into PLACEHOLDER Cloud, and what's planned next.

What's shipped (v1)​

  • iceberg and delta datasource types (issue #88).
  • Connection test that opens the catalog / Delta log and confirms the table is reachable. The Iceberg probe reads the snapshot summary; the Delta probe reads the current transaction-log version. Neither pulls row-level data.
  • Checkpoint runs against the current snapshot. The worker reads it into a pyarrow table via pyiceberg / deltalake, materialises it into an in-process DuckDB tempfile, and runs every standard expectation (row count, column values, custom SQL, …) against the DuckDB-backed GX SQL datasource.
  • Two new expectation types:
    • expect_snapshot_diff_row_count_to_be_within_pct — fails when the current snapshot's row count differs from the previous snapshot by more than threshold_pct percent. First-run passes (nothing to compare against).
    • expect_schema_compatible_with_previous — fails when a column has been dropped or a column type has narrowed since the previous snapshot. First-run passes.
  • Snapshot history table (lakehouse_snapshot_history in each tenant schema) seeded by every checkpoint run, so the two expectations above have something to read.
  • Agent-mode support: when the lakehouse sits behind a firewall, the DQ Agent materialises the snapshot in-process and ships the (redacted) GX result back to the cloud — no row-level data leaves the customer network.

What's planned (v2)​

Status: planned (issue #88, v2 follow-up).

  • Time-travel reads. compare_with_snapshot_at_time / snapshot_id selection on the loader. The loader functions already take a single config dict — v2 grows it without breaking the v1 call sites.
  • Hudi support. Same materialise-to-DuckDB shape; needs a third loader function.
  • Catalog-driven asset discovery. Auto-list every table in a Glue / Nessie catalog so you don't have to register them one by one.
  • Snapshot-pushdown for large tables. v1 materialises the entire current snapshot in memory. v2 will push partition filters into the Iceberg / Delta scan so a checkpoint scoped to "yesterday's partition" doesn't pay for the whole table.

Trust model​

When PLACEHOLDER Cloud runs a checkpoint against an Iceberg or Delta table, the worker (or agent) reads the current snapshot into a pyarrow table held in memory and materialises it into an on-disk DuckDB file under the run's per-checkpoint tempdir. The temp file is removed when the run ends and the in-process DuckDB connection is closed before the next run starts.

What this means in practice:

  • Memory footprint scales with the current-snapshot size. A 10M-row Iceberg table will momentarily hold a 10M-row arrow table in the worker process. The platform refuses to materialise a snapshot whose recorded row count exceeds DQ_LAKEHOUSE_MAX_ROWS (default 10M) — operators with bigger tables either raise that ceiling (and confirm the worker has the memory) or partition the asset via a batch definition.
  • Latency is dominated by the storage read, not the GX run. A snapshot that lives on S3 will pay the S3 throughput tax once per checkpoint. The Iceberg / Delta connection test does NOT pull data — it only reads catalog / transaction-log metadata — so "Test connection" stays fast regardless of snapshot size.
  • Credentials. The catalog and object-store credentials live in the same config_encrypted JSONB column every other datasource type uses. Today (v1) that column is plain JSONB in dev installs and AES-256-GCM-encrypted under HARDEN-2 once that ticket lands; the lakehouse code path inherits the platform-wide posture without a parallel encryption code path.
  • Network isolation. When you can't open an outbound connection from the worker to the catalog and object store, deploy the DQ Agent inside that network and register the lakehouse datasource through the agent. The catalog credentials never reach the cloud.
  • URI scheme allowlist. The loaders accept s3://, s3a://, gs://, gcs://, abfs://, abfss://, az://, and file://. Anything else is rejected at config-time as "scheme not recognised" — the same posture as s3_parquet. Not an SSRF allowlist (the worker's IAM role is still the source of truth for which buckets it can read); just a guardrail against typos like s:// or http:/.
  • Sample size. v1 always reads the full current snapshot. If you only want to validate a sample (e.g. the last day's partition), use the same batch definitions you'd use against a SQL backend — the worker pushes the partition filter into the DuckDB layer, not the Iceberg / Delta scan. v2 will lift the filter into the snapshot reader.

Snapshot-aware expectations​

These two are unique to lakehouses — they compare the current snapshot against the previous one and don't have a natural form against a row-DB.

expect_snapshot_diff_row_count_to_be_within_pct​

Pass when |current - previous| / previous * 100 <= threshold_pct. First run passes unconditionally (seeds the history). Kwargs:

{
  "threshold_pct": 10.0
}

Use this to alert on accidental truncation (a job replaced the partition with empty results) or runaway growth (a backfill duplicated rows).

expect_schema_compatible_with_previous​

Pass when no column has been dropped and no column's type string has changed since the previous snapshot. Type-narrowing detection is conservative — int64 → int32 is a break, int64 → int64 is not. First run passes unconditionally (seeds the history).

Use this to alert on breaking schema evolution before a downstream consumer fails. v2 will replace the string-equality check with a proper type-lattice walk.

When to reach for a lakehouse datasource​

  • Your data already lives in S3 / GCS / Azure as Iceberg or Delta. You want native checks against the table-format snapshot instead of routing through a SQL warehouse view.
  • You're not running a Snowflake / Databricks SQL warehouse in front of the lake — or you don't want to pay the warehouse cost for every checkpoint.
  • You want to alert on snapshot-level concerns (row-count drift snapshot-over-snapshot, schema evolution) that don't make sense against a SQL view.

When not to​

  • Your tables are larger than the worker can fit in memory. Today the materialise step pulls the whole current snapshot into a pyarrow table; the platform enforces a 10M-row ceiling by default (DQ_LAKEHOUSE_MAX_ROWS). Stay on the SQL-warehouse-fronted path or partition the asset until v2 lifts that limitation.
  • You need to compare two arbitrary snapshots (or a snapshot at time T). Time-travel comparisons are the v2 work; v1 only checks the current snapshot against the most recent previously-recorded snapshot.

See also​