Skip to main content

Connect a Delta Lake datasource

Goal​

You'll register a Delta Lake table (sitting on S3 / GCS / Azure or a local filesystem) as a Measured Cloud datasource, confirm the connection, and run rules against the current version of the table. This is the v1 happy path for issue #88 — the Delta side of the table-format adapters.

The trust model and runner strategy are shared with Iceberg and documented in lakehouse checks. In short: the worker reads the current Delta version into pyarrow, materialises it into an in-process DuckDB tempfile, and runs your rules against that.

Prereqs​

  • A running Measured Cloud.

  • The worker image has been built with the [delta] extra. The published dq-cloud-worker image ships it; if you build your own, install with pip install dq-cloud-worker[delta]. The agent image needs the same extra (pip install dq-cloud-agent[delta]) when the lakehouse sits behind a firewall.

  • A Delta table URI the worker (or agent) can reach. Supported schemes:

    • s3://bucket/path/to/table (also s3a://)
    • gs://bucket/path/to/table (also gcs://)
    • abfs://container@account.dfs.core.windows.net/path/to/table (also abfss://, az://)
    • file:///abs/path/to/table (local filesystem — useful for testing)

    Anything else is rejected at save-time as "scheme not recognised" — the same guardrail the s3_parquet connector ships.

  • Read credentials for the underlying object store. Delta uses delta-rs under the hood, which expects credentials under env-var-style keys. Only documented keys are forwarded — typos drop on save.

    • AWS: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION, optional AWS_SESSION_TOKEN, AWS_ENDPOINT_URL.
    • Azure: AZURE_STORAGE_ACCOUNT_NAME plus either AZURE_STORAGE_ACCOUNT_KEY or AZURE_STORAGE_SAS_TOKEN, or a service principal (AZURE_STORAGE_TENANT_ID / AZURE_STORAGE_CLIENT_ID / AZURE_STORAGE_CLIENT_SECRET).
    • GCS: a service-account JSON via GOOGLE_SERVICE_ACCOUNT or GOOGLE_APPLICATION_CREDENTIALS.

Steps​

  1. Data Assets → Add source → pick Delta Lake.
  2. Fill in:
    • Table URI — the full s3://… / gs://… / abfs://… / file://… URI of the Delta table directory (the one containing _delta_log/).
    • Storage credentials — pasted into the form's storage_options block. The form drops any key outside the documented allowlist (see Prereqs).
  3. Test connection — green check + latency + the current Delta version (e.g. delta v42). The probe opens the table's transaction log; it does not pull data.
  4. Add the table as a data asset and add rules to it. Every built-in rule type works. Plus the two lakehouse-specific expectations:
    • expect_snapshot_diff_row_count_to_be_within_pct
    • expect_schema_compatible_with_previous
  5. Run the asset's rules (or a validation that includes it). The worker materialises the current version into pyarrow, registers it inside an ephemeral DuckDB, and executes the rules. The result lands in Results exactly like any other run. The current version's row count + schema are written to lakehouse_snapshot_history so the next run's snapshot-aware expectations have something to compare against.

Verify​

  • The source appears on Data Assets with a green connection badge and a version string (delta v42).
  • The first run completes with success: true (or a deterministic failure matching the rule). The two snapshot-aware expectations pass on the first run (nothing to compare against) and start firing on the second run.

Caveats​

  • Memory. Same as Iceberg — v1 materialises the entire current version in memory. The platform refuses to materialise a snapshot bigger than DQ_LAKEHOUSE_MAX_ROWS rows (default 10,000,000). For bigger tables, raise that ceiling (and confirm the worker has the memory headroom) until v2 pushes partition filters into the Delta scan. A row condition narrows which rows a rule checks, but the whole version is still read first.
  • Time travel. Reading a specific Delta version (DeltaTable(..., version=N)) lands in v2 alongside the snapshot-comparison expectations.
  • Credentials encryption. storage_options sits in the same config_encrypted JSONB column every other datasource uses. AES-256-GCM at rest under HARDEN-2 is in flight; the lakehouse path inherits whatever posture that ticket lands.
  • Local filesystem URIs are convenient for testing but won't help you in production. Production deployments should point at object-store URIs and grant the worker / agent role read access via IAM (preferred) or static credentials (acceptable for v1).

See also​