Skip to main content

Integrate with Data.world

Status: shipped (outbound-only) — issue #403. DQ Cloud pushes validation results to a Data.world dataset's activity stream. The connector does not read assets back from Data.world.

DQ Cloud ships a first-party connector for Data.world that posts a quality insight to a dataset's activity stream after every ValidationResult:

  • a titled insight (DQ Cloud quality check: <asset>) whose body carries the pass/fail status, pass/total counts, pass-rate percentage, the suite name, and the run timestamp, and
  • (optional) a dataset-level annotation — a dq-passed / dq-failed tag plus a one-line summary — so consumers can filter the catalog for quality-failing datasets.

The connector talks to the Data.world REST API directly over HTTP — no Data.world SDK dependency in the worker image.

Prerequisites​

  • A Data.world account and the API root (https://api.data.world).
  • A read/write API token (Data.world → Settings → Advanced).
  • The owner (your org or user agentid) and the dataset slug the quality events should land on.

Configure the integration​

Create the catalog_integrations row through the org-admin settings API (or the Settings → Catalog integrations page):

curl -X POST "$BASE_URL/api/v1/organizations/$ORG_ID/catalog-integrations" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
        "kind": "data.world",
        "endpoint": "https://api.data.world",
        "auth_token": "<data-world-api-token>"
      }'

The auth token is AES-256-GCM-encrypted under the org's DEK before it hits Postgres; the API never returns it back.

Map a DQ Cloud asset to a Data.world dataset​

For each asset you want pushed, set the target dataset on the asset's catalog config:

  • config["catalog"]["dataworld_owner"] — the dataset owner (agentid), e.g. acme.
  • config["catalog"]["dataworld_dataset"] — the dataset slug, e.g. sales.

An asset missing either is skipped (a emit_to_catalog_dataworld_no_dataset log line lands). The asset's name rides into the insight body as the table reference, so many DQ Cloud assets can feed one Data.world dataset's feed.

What it looks like in Data.world​

After every ValidationResult commit, the emit_to_catalog ARQ job posts to POST /v0/insights/{owner}/{dataset}. The insight shows up on the dataset's activity stream with the structured quality body. The optional dataset annotation (PATCH /v0/datasets/{owner}/{dataset}) adds the dq-<status> tag.

Verify​

  • In Settings → Catalog integrations, the Data.world row shows Enabled.
  • On the Data.world dataset, the activity feed shows a new DQ Cloud quality check insight after the next checkpoint run.