Skip to main content

Connect an Apache Iceberg datasource

Goal​

You'll register an Iceberg table (sitting on S3 / GCS / Azure behind a REST, Glue, or Nessie catalog) as a PLACEHOLDER Cloud datasource, confirm the connection, and run a rule suite against the current snapshot. This is the v1 happy path for issue #88.

The runner strategy is documented in lakehouse checks — the worker reads the current snapshot into pyarrow, materialises it into an in-process DuckDB tempfile, and runs your rule suite against that. From the rule-author's perspective the datasource feels exactly like any other SQL backend.

Prereqs​

  • A running PLACEHOLDER Cloud.
  • The worker image has been built with the [iceberg] extra. The published dq-cloud-worker image ships it; if you build your own, install with pip install dq-cloud-worker[iceberg]. The agent image needs the same extra (pip install dq-cloud-agent[iceberg]) when the lakehouse sits behind a firewall.
  • A catalog endpoint your worker (or agent) can reach:
    • REST: a base URI (e.g. https://catalog.example.com) and any bearer token / credential the REST catalog requires.
    • Glue: AWS credentials with glue:GetTable on the configured table.
    • Nessie: the catalog's URI and (where applicable) a Nessie ref name.
  • Read credentials for the underlying object store (S3 / GCS / Azure).
  • The namespace.table identifier — Iceberg's two-segment table name (e.g. sales.orders).

Steps​

  1. Datasources → New → pick Iceberg.
  2. Choose the catalog kind (REST / Glue / Nessie). The form swaps the field set to match.
  3. Fill in:
    • Catalog URI (REST / Nessie).
    • Warehouse path (every catalog kind) — the object-store prefix Iceberg writes table data under (e.g. s3://my-bucket/warehouse/).
    • Object-store credentials — s3.access-key-id / s3.secret-access-key / s3.region (or the Azure / GCS equivalents). Only documented keys are forwarded — typos surface at save-time.
    • Namespace — the first segment of the table identifier (e.g. sales).
    • Table — the second segment (e.g. orders).
  4. Test connection — green check + latency. The probe loads the catalog client, resolves the configured table, and reads the snapshot's total-records summary. It does not pull row data.
  5. Add the table as a data asset and attach a rule suite. Every built-in expectation works — row count, column values, custom SQL, etc. Plus the two lakehouse-specific expectations:
    • expect_snapshot_diff_row_count_to_be_within_pct
    • expect_schema_compatible_with_previous
  6. Run a checkpoint. The worker materialises the current snapshot into pyarrow, registers it inside an ephemeral DuckDB, and executes the rule suite. The result lands in Validation results exactly like any other run. The current snapshot's row count + schema are written to lakehouse_snapshot_history so the next run's snapshot-aware expectations have something to compare against.

Verify​

  • The datasource appears in Datasources with a green badge.
  • The first checkpoint run completes with success: true (or with a deterministic failure that matches the rule). The two snapshot-aware expectations pass on the first run (nothing to compare against) and start firing on the second run.

Caveats​

  • Memory. The v1 runner materialises the entire current snapshot in memory. The platform refuses to materialise a snapshot bigger than DQ_LAKEHOUSE_MAX_ROWS rows (default 10,000,000). Operators with bigger tables either raise the env var (and confirm the worker has the memory headroom) or use a batch definition to scope the check to a partition.
  • Catalog discovery. Auto-listing every table in the catalog isn't shipped yet — register each table individually. See the v2 plan in the concept page.
  • Time-travel. Reading a specific Iceberg snapshot id is the v2 path. v1 always reads the current snapshot.
  • Hive / SQL catalogs. The v1 build pulls in REST + Glue + Nessie + s3fs. Hive metastore + sqlite SQL-catalog kinds aren't in the wheel set; if you need one open a PR or follow the v2 catalog work.
  • Credentials encryption. Catalog + object-store credentials sit in the same config_encrypted JSONB column every other datasource uses. AES-256-GCM at rest under HARDEN-2 is in flight; the lakehouse path inherits whatever posture that ticket lands.
  • REST + OAuth2. REST catalogs that require OAuth2 token refresh aren't fully exercised in v1 — operators on long-lived tokens or service-account creds are the supported path. The form accepts pyiceberg's documented config keys (token, credential) verbatim.

See also​