Skip to main content

Ingest dbt artifacts

Goal​

You'll feed PLACEHOLDER Cloud the two JSON files dbt already produces — manifest.json and run_results.json — and have it register a DataAsset per dbt model, then import the last run's per-model outcomes as synthetic validation results. No re-typing model names; no parallel inventory to keep in sync.

Prereqs​

  • A datasource pointing at the same warehouse dbt builds into. The datasource must already pass Test connection.
  • A recent dbt build (or dbt run + dbt test) that produced target/manifest.json and target/run_results.json in your project.
  • Workspace editor role or higher.

Steps​

Option A — from the UI​

  1. Open the datasource detail page. Navigate to Datasources → <your warehouse>.
  2. Click "Ingest dbt artifacts". A file picker opens.
  3. Select both files in one shot: target/manifest.json and target/run_results.json. The button matches files on filename (manifest, run_results) so either order works.
  4. Wait for the toast. A success banner reports N assets registered, M results imported. The data-assets table on the same page refreshes with the new rows.

Option B — from CI (curl)​

# Run dbt and produce the artifacts.
dbt build --project-dir ./my_dbt_project

# Ingest them. ORG, WS, DS are UUIDs from the PLACEHOLDER Cloud UI;
# DQK is a personal API key with workspace-editor or higher.
curl -X POST \
  -H "Authorization: Bearer ${DQK}" \
  -H "Content-Type: application/json" \
  --data @<(jq -n \
    --arg ds "${DS}" \
    --slurpfile m ./my_dbt_project/target/manifest.json \
    --slurpfile r ./my_dbt_project/target/run_results.json \
    '{datasource_id: $ds, manifest: $m[0], run_results: $r[0]}') \
  "https://placeholder.example.com/api/v1/organizations/${ORG}/workspaces/${WS}/dbt/ingest"

The endpoint responds with:

{
  "data": {
    "assets_registered": 12,
    "assets_updated": 0,
    "results_imported": 12,
    "rule_suite_id": "9d3b...",
    "unmatched_run_results": []
  }
}

Re-running the same curl after a subsequent dbt build is idempotent — model names that already exist get refreshed in place (description, tags, table location), not duplicated. The counts shift from assets_registered to assets_updated on the second call.

Verify​

  • The new assets appear under the datasource's Data Assets table. Each one is now a target for rule suites, profile runs, and rule-suggestion generation.
  • Validation results from the dbt run land under Results — each one tagged with kind: dbt_run in the JSONB result body so the UI can distinguish them from authored-checkpoint results.
  • A synthetic rule suite named dbt:<datasource_id> shows up under Rule Suites. It owns the imported validation results; it has no rules of its own (the suite is a labelling row, not a runnable suite). See Rule suites — synthetic suites.

Caveats​

  • The endpoint reads the artifacts in-process — at very large project sizes (thousands of models per call) the request may take a few seconds. The bound is the dbt project size; typical projects (tens to low hundreds of models) finish in well under a second. A future async path will accept the artifacts and respond 202 Accepted immediately.
  • dbt's status enum is preserved verbatim in result.dbt_status. success and pass map to ValidationResult.success=true; everything else (error, fail, skipped, warn, runtime error, …) maps to success=false.
  • run_results[*] entries that don't match any model in the manifest (typically a stale target/run_results.json from before a model rename) come back in unmatched_run_results. Re-run dbt build to refresh the artifacts.
  • The asset's primary key is (datasource_id, alias or name) — mirroring dbt's own physical-table resolution. If you manually created an asset with the matching name before ingesting, the ingest will adopt it rather than duplicating.
  • The dbt tags array is intersected onto PLACEHOLDER Cloud's DataAsset.tags (FEATURE-7) — search and filter by tag work as if you'd applied them by hand.
  • The rate limit is 6 ingests per minute per org (RATE_LIMIT_DBT_INGEST_PER_MINUTE). Loop your CI on a sensible cadence; don't hammer.

What's not included​

  • Column-level lineage from dbt's catalog.json. The two-file ingest covers asset registration and run-status rollup; deeper lineage is a separate ticket.
  • dbt Cloud API polling (auto-pull on every job run). Today, you push artifacts to the endpoint after every dbt build from your own runner.