Skip to main content

Integrate with Atlan

Status: shipped (outbound) — issue #415. DQ Cloud pushes validation results to Atlan as a quality signal. Inbound asset discovery (walking Atlan to JIT-create DQ Cloud assets) is a planned follow-up; today the connector is outbound-only.

DQ Cloud ships a first-party connector for Atlan that pushes every ValidationResult to the matching Atlan table asset:

  • sets Atlan's built-in assetDqStatus attribute (Passed / Failed) — the coloured quality pill the asset page renders, and
  • writes a DQ pass-rate custom-metadata attribute (plus pass/total counts, the suite name, and the last-run timestamp) under a named custom-metadata set.

The connector talks to Atlan's metadata API directly over HTTP — no pyatlan SDK dependency in the worker image.

Prerequisites​

  • An Atlan instance and its base URL (e.g. https://tenant.atlan.com), reachable from the worker process.
  • An API token (Atlan → Admin → API tokens) with permission to update entities and custom metadata.
  • The connection qualifiedName for the assets you want stamped. Atlan roots every asset's qualifiedName at its connection (default/<connector>/<epoch>); DQ Cloud can't derive the epoch, so you supply the connection qualifiedName per asset (see below).

Configure the integration​

Create the catalog_integrations row through the org-admin settings API (or the Settings → Catalog integrations page):

curl -X POST "$BASE_URL/api/v1/organizations/$ORG_ID/catalog-integrations" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "kind": "atlan",
        "endpoint": "https://tenant.atlan.com",
        "auth_token": "<atlan-api-token>"
      }'

The auth token is AES-256-GCM-encrypted under the org's DEK before it hits Postgres; the API never returns it back. The endpoint goes through the same SSRF guard as the other catalog connectors.

Map a DQ Cloud asset to an Atlan asset​

For each asset you want pushed, set the Atlan connection qualifiedName on the asset's catalog config:

  • config["catalog"]["atlan_connection"] — the connection qualifiedName, e.g. default/postgres/1700000000. DQ Cloud appends the asset's dotted name (db.schema.table → db/schema/table) to build the full asset qualifiedName.
  • config["catalog"]["atlan_cm_set"] — (optional) the custom-metadata set name the pass-rate attribute lands under. Defaults to DQ Cloud.

An asset without atlan_connection is skipped (a structured emit_to_catalog_atlan_no_connection log line lands so you can see why) — DQ Cloud won't invent a qualifiedName it can't verify.

What it looks like in Atlan​

After every ValidationResult commit, the emit_to_catalog ARQ job upserts the table entity via POST /api/meta/entity/bulk:

  • assetDqStatus flips to Passed or Failed (the pill on the asset page),
  • the named custom-metadata set gets dq_pass_rate (0–100), dq_passed, dq_total, dq_status, dq_suite, and dq_last_run.

Re-emits against the same qualifiedName overwrite in place — Atlan resolves the GUID server-side, so DQ Cloud never duplicates the asset.

Verify​

  • In the Settings → Catalog integrations page, the Atlan row shows Enabled and (after a run) a recent quality push.
  • On the Atlan asset, the quality pill reflects the most recent run and the custom-metadata panel shows the pass rate.