Skip to main content

Integrate with DataHub

Status: shipped — DataHub outbound + inbound (issue #89). Atlan, Data.world, and Alation now ship as first-class connectors too (issues #403 / #404 / #415) — see Integrate with Atlan, Integrate with Data.world, and Integrate with Alation.

DQ Cloud ships a first-party connector for DataHub that does two things:

  • Outbound — pushes every ValidationResult to DataHub as an assertion aspect so the dataset page shows a quality badge backed by your DQ Cloud checkpoints.
  • Inbound — walks DataHub's GraphQL catalog every hour, JIT-creates DQ Cloud DataAsset rows for any dataset on a datasource you've already connected, and propagates owner / domain / tag / description.

The connector talks to DataHub's GMS endpoint directly over HTTP — no acryl-datahub SDK dependency in the worker image.

Prerequisites​

  • A DataHub deployment you control (OSS or Acryl Cloud) — DQ Cloud needs the GMS endpoint reachable from the worker process.
  • A Personal Access Token if your DataHub has METADATA_SERVICE_AUTH_ENABLED=true. Local-dev clusters without auth can skip this.

Configure the integration​

The catalog-integration row lives in public.catalog_integrations (one row per (org, kind)). You configure it through the org-owner settings API:

curl -X POST "$BASE_URL/api/v1/organizations/$ORG_ID/catalog-integrations" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "kind": "datahub",
        "endpoint": "https://gms.example.com:8080",
        "auth_token": "eyJhbGciOiJIUzI1NiJ9..."
      }'

The auth token is AES-256-GCM-encrypted under the org's DEK before it hits Postgres; the API never returns it back. The endpoint goes through the same SSRF guard as alert webhooks (no 169.254.169.254, no RFC1918 unless your DATASOURCE_NETWORK_ALLOWLIST permits it).

To rotate the token or change the endpoint, PATCH the row — fields that you omit stay unchanged, an explicit null clears them:

curl -X PATCH "$BASE_URL/api/v1/organizations/$ORG_ID/catalog-integrations/$ID" \
  -H "Authorization: Bearer $TOKEN" \
  -d '{"auth_token": "eyJ...new..."}'

To pause outbound + inbound temporarily without losing the row:

curl -X PATCH "$BASE_URL/api/v1/organizations/$ORG_ID/catalog-integrations/$ID" \
  -d '{"enabled": false}'

What outbound looks like in DataHub​

After every ValidationResult commit, the emit_to_datahub ARQ job posts two aspects against urn:li:assertion:<dataset_urn>::<suite_id>:

  • assertionInfo — the static definition of what's being checked (the rule-suite name + each rule's (rule_type, kwargs)).
  • assertionRunEvent — the run record. Carries pass / total via actualAggValue / expectedAggValue plus the operator-friendly nativeResults breakdown ({total, passed, failed}).

DataHub renders the most recent assertionRunEvent as the dataset's quality badge. Re-emits against the same (asset, suite) overwrite in place — DQ Cloud doesn't duplicate the assertion every run.

The emit is conditional: checkpoint_run only enqueues the emit_to_datahub job when the org has at least one enabled catalog_integrations row, so orgs that haven't opted in pay no extra ARQ overhead per result.

How the asset URN is built​

The outbound emitter and the inbound discoverer agree on the same URN shape:

urn:li:dataset:(urn:li:dataPlatform:<platform>,<name>,<env>)
  • <platform> — postgres, snowflake, bigquery, redshift, mysql, mssql, databricks, athena. Anything else falls through to the lowercased datasource type.
  • <name> — the DQ Cloud asset's name field. The convention is <database>.<schema>.<table> to match DataHub's, but DQ Cloud doesn't enforce three-part names.
  • <env> — PROD by default. Set DataAsset.config["catalog"]["datahub_env"] to STAGING, DEV, etc. to override per-asset; the discoverer match-back respects the env tag so a STAGING URN won't collide with a PROD-tagged asset.

The first time outbound emits successfully for an asset, DQ Cloud stamps the URN onto DataAsset.external_urn["datahub"] so the inbound discoverer can match it back on rule 1 (external URN containment) without falling through to the rule 2 fall-back.

Inbound discovery​

The hourly cron at minute 7 enumerates every enabled integration row and fans out one run_inbound_for_integration job per row. Each job:

  1. Switches to the integration's tenant schema.
  2. Pages through DataHub's GraphQL search(input: {type: DATASET, ...}) query.
  3. For each dataset, applies the three-rule match-back:
    • External URN match — external_urn["datahub"] @> <urn>.
    • (platform, name, env) match — find an asset whose datasource type maps to the same DataHub platform key, whose name matches, AND whose configured env matches.
    • JIT-create — otherwise, create a new DataAsset row on the oldest matching Datasource (one with type == platform). If no matching datasource exists, the dataset is skipped and a datahub_discover_no_matching_datasource log line lands so an operator can see why nothing got picked up.

Each pass overwrites owner_email, tags, description, and writes the domain name into config["catalog"]["datahub_domain"]. DataHub is the source of truth for those fields when the integration is enabled.

To trigger an out-of-cycle walk for one integration (useful when you've just connected a new datasource and want DataHub's assets to land immediately):

curl -X POST "$BASE_URL/api/v1/organizations/$ORG_ID/catalog-integrations/$ID/sync"

The endpoint returns 202 and the walk runs asynchronously. Watch last_synced_at on the GET response to see when it completes — this is scoped to the one integration you specified, so other tenants' rows are untouched.

DataHub deployment notes​

  • Auth disabled (local dev) — leave auth_token unset and DataHub will accept the writes without an Authorization header.
  • GMS behind nginx — the connector uses standard HTTP Bearer auth and the v1 /aspects?action=ingestProposal ingest path, which most reverse-proxy configs pass through unchanged.
  • DataHub Cloud (Acryl) — point endpoint at the GMS URL Acryl gave you (the one ending in .acryl.io); the API surface is identical to OSS DataHub.

Other native catalogs​

The catalog-integration framework now backs four first-party connectors. They share the same catalog_integrations table, the same encrypt-at-rest auth-token handling, the same SSRF guard, and the same "enqueue only when an enabled integration exists" gate — only the per-vendor wire shape differs:

  • Atlan (issue #415) — pushes assetDqStatus + a custom-metadata pass-rate attribute on the table asset. See Integrate with Atlan.
  • Data.world (issue #403) — posts a quality insight to the dataset's activity stream. See Integrate with Data.world.
  • Alation (issue #404) — pushes a DataQuality value (pass/fail status + pass-rate score) to the table object. See Integrate with Alation.

DataHub is the only one with inbound discovery today; the other three are outbound-only (their issues ask only to push results).

Follow-ups​

  • datasetProfile aspect emission — the aspect builder (build_dataset_profile_aspect) and unit tests are wired, but the worker hook that fires it after every ProfileRun write is a small follow-up (one dispatch entry, one _org_has_catalog_integration gate).
  • Atlan inbound discovery — the outbound emitter ships now; an inbound walk (mirroring DataHub's discoverer) is a follow-up.