Skip to main content

Connect a Delta Lake datasource

Goal​

You'll register a Delta Lake table (sitting on S3 / GCS / Azure or a local filesystem) as a PLACEHOLDER Cloud datasource, confirm the connection, and run a rule suite against the current version of the table. This is the v1 happy path for issue #88 — the Delta side of the table-format adapters.

The trust model and runner strategy are shared with Iceberg and documented in lakehouse checks. In short: the worker reads the current Delta version into pyarrow, materialises it into an in-process DuckDB tempfile, and runs your rule suite against that.

Prereqs​

  • A running PLACEHOLDER Cloud.

  • The worker image has been built with the [delta] extra. The published dq-cloud-worker image ships it; if you build your own, install with pip install dq-cloud-worker[delta]. The agent image needs the same extra (pip install dq-cloud-agent[delta]) when the lakehouse sits behind a firewall.

  • A Delta table URI the worker (or agent) can reach. Supported schemes:

    • s3://bucket/path/to/table (also s3a://)
    • gs://bucket/path/to/table (also gcs://)
    • abfs://container@account.dfs.core.windows.net/path/to/table (also abfss://, az://)
    • file:///abs/path/to/table (local filesystem — useful for testing)

    Anything else is rejected at save-time as "scheme not recognised" — the same guardrail the s3_parquet connector ships.

  • Read credentials for the underlying object store. Delta uses delta-rs under the hood, which expects credentials under env-var-style keys. Only documented keys are forwarded — typos drop on save.

    • AWS: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, optional AWS_SESSION_TOKEN, AWS_ENDPOINT_URL.
    • Azure: AZURE_STORAGE_ACCOUNT_NAME plus either AZURE_STORAGE_ACCOUNT_KEY or AZURE_STORAGE_SAS_TOKEN, or a service principal (AZURE_STORAGE_TENANT_ID / AZURE_STORAGE_CLIENT_ID / AZURE_STORAGE_CLIENT_SECRET).
    • GCS: a service-account JSON via GOOGLE_SERVICE_ACCOUNT or GOOGLE_APPLICATION_CREDENTIALS.

Steps​

  1. Datasources → New → pick Delta Lake.
  2. Fill in:
    • Table URI — the full s3://… / gs://… / abfs://… / file://… URI of the Delta table directory (the one containing _delta_log/).
    • Storage credentials — pasted into the form's storage_options block. The form drops any key outside the documented allowlist (see Prereqs).
  3. Test connection — green check + latency + the current Delta version (e.g. delta v42). The probe opens the table's transaction log; it does not pull data.
  4. Add the table as a data asset and attach a rule suite. Every built-in expectation works. Plus the two lakehouse-specific expectations:
    • expect_snapshot_diff_row_count_to_be_within_pct
    • expect_schema_compatible_with_previous
  5. Run a checkpoint. The worker materialises the current version into pyarrow, registers it inside an ephemeral DuckDB, and executes the rule suite. The result lands in Validation results exactly like any other run. The current version's row count + schema are written to lakehouse_snapshot_history so the next run's snapshot-aware expectations have something to compare against.

Verify​

  • The datasource appears in Datasources with a green badge and a version string (delta v42).
  • The first checkpoint run completes with success: true (or a deterministic failure matching the rule). The two snapshot-aware expectations pass on the first run (nothing to compare against) and start firing on the second run.

Caveats​

  • Memory. Same as Iceberg — v1 materialises the entire current version in memory. The platform refuses to materialise a snapshot bigger than DQ_LAKEHOUSE_MAX_ROWS rows (default 10,000,000). Pre-filter very large tables via a batch definition until v2 pushes the filter into the Delta scan.
  • Time travel. Reading a specific Delta version (DeltaTable(..., version=N)) lands in v2 alongside the snapshot-comparison expectations.
  • Credentials encryption. storage_options sits in the same config_encrypted JSONB column every other datasource uses. AES-256-GCM at rest under HARDEN-2 is in flight; the lakehouse path inherits whatever posture that ticket lands.
  • Local filesystem URIs are convenient for testing but won't help you in production. Production deployments should point at object-store URIs and grant the worker / agent role read access via IAM (preferred) or static credentials (acceptable for v1).

See also​