Skip to main content

Run the on-prem agent demo

The on-prem agent demo stands up a believable customer environment on your laptop: five tables in a local Postgres database, a drift loop that keeps writing fresh rows (and the occasional bad one), and the DQ Agent pointed at the hosted dev cloud. The first 60 seconds of UI look like a real production deployment instead of a 7-row toy.

This is a demo runbook only — none of the components are intended for production use.

What you get​

ComponentWhat it does
customer_demo Postgres database5 tables — customers (~200), products (~80), orders (~900), order_items (~2,300), events (~3,000) — seeded deterministically.
Drift loop daemonInserts a small batch of valid rows every 30-60 s. 5-10% of ticks insert one bad row that violates exactly one expectation. 3% of ticks corrupt an existing row and heal it 60 s later.
DQ Agent containerPicks up checkpoint runs from the hosted cloud, executes them locally against customer_demo, and posts metadata back.
Demo compositionProvisions 5 data assets + 5 rule suites (~25 expectations) + 5 checkpoints + 5 schedules with staggered crons (*/2, */3, */4, */5).

End-state: a dashboard with a >95% pass rate, a 3-5% sliver of runs showing 1-2 failed expectations, and a visible "anomaly recovered" arc on the timeline whenever the corrupt+heal flow fires.

Prereqs​

  • Docker (for the local Postgres + the agent container).
  • uv (the workspace package manager — see the top-level CLAUDE.md for setup).
  • A hosted DQ Cloud dev tenant you control, with:
    • An agent token (dqa_…) issued from Settings → Agents.
    • An API key (dqk_…) for the demo provisioning script.
    • A workspace id and an org id (visible in the Settings URL).

Environment variables​

VariableRequired forNotes
DQ_CLOUD_BASE_URLagent + demo_setupe.g. https://dev.placeholder.example.com.
DQ_CLOUD_AGENT_TOKENagentThe dqa_… token.
DQ_CLOUD_API_KEYdemo_setupA dqk_… personal API key.
DQ_CLOUD_ORG_IDdemo_setupUUID of your hosted org.
DQ_CLOUD_WORKSPACE_IDdemo_setupUUID of the workspace the agent token is scoped to.
DQ_CLOUD_DATASOURCE_IDdemo_setupUUID returned when the agent registered customer_demo (see step 3).
CUSTOMER_DEMO_DSNseed + drift loopOptional — defaults to postgresql://postgres:postgres@localhost:5432/customer_demo.
DQ_AGENT_DATASOURCE_DSNSagentJSON object mapping datasource UUID → local DSN. See step 4.

1. Stand up local Postgres​

The same compose file the rest of the dev environment uses:

make dev

Postgres lands on :5432 with postgres / postgres as the default credentials. The seed script creates customer_demo for you on first run.

2. Seed customer_demo and start the drift loop​

make demo-up

The first time you run it make demo-up will:

  1. Create the customer_demo database (if missing).
  2. Drop and rebuild the five tables, inserting the seeded row counts.
  3. Start the drift loop as a background process. PID + log live in .demo/drift_loop.{pid,log}.
  4. If DQ_CLOUD_BASE_URL and DQ_CLOUD_AGENT_TOKEN are set, start the agent container too.

Tail the drift log to confirm it's alive:

tail -f .demo/drift_loop.log

You should see entries like:

INFO drift_loop valid_insert order_id=901 customer_id=27
INFO drift_loop valid_insert order_id=902 customer_id=143
WARNING drift_loop bad_insert kind=order expectation=orders.status in (...) customer_id=84
INFO drift_loop heal_applied order_id=412

3. Register customer_demo with the cloud​

From inside the agent container (one-shot registration):

docker exec dq-demo-agent python -c "
import httpx, os
r = httpx.post(
    os.environ['DQ_AGENT_CLOUD_URL'] + '/api/agent/v1/datasources',
    headers={'Authorization': f'Bearer {os.environ[\"DQ_AGENT_TOKEN\"]}'},
    json={'name': 'customer_demo', 'type': 'postgres'},
)
print(r.status_code, r.json())
"

Copy the returned datasource UUID. Save it as DQ_CLOUD_DATASOURCE_ID.

4. Wire the agent's local DSN​

The agent needs to know how to reach customer_demo from inside its container. Restart it with DQ_AGENT_DATASOURCE_DSNS populated:

docker rm -f dq-demo-agent
DQ_AGENT_DATASOURCE_DSNS='{
  "<datasource-uuid>": "postgresql://postgres:postgres@host.docker.internal:5432/customer_demo"
}' \
  make demo-up

host.docker.internal resolves to the host machine on macOS and Linux (with the --add-host=host.docker.internal:host-gateway flag the Makefile passes). If you're on a host without that resolver, use the literal 172.17.0.1 (Docker bridge default) or your VPN-side IP.

5. Provision the hosted side​

Run the demo composition to create the five assets, suites, checkpoints, and schedules:

DQ_CLOUD_API_KEY=dqk_...   DQ_CLOUD_BASE_URL=https://dev.placeholder.example.com \
DQ_CLOUD_ORG_ID=...        DQ_CLOUD_WORKSPACE_ID=... \
DQ_CLOUD_DATASOURCE_ID=... \
  uv run python -m infra.terraform.compositions.services.demo_setup

The script is idempotent — re-running it updates existing rows by name rather than duplicating them, so it's safe to use as part of a "reset the demo" routine.

Within a couple of minutes the dashboard starts to fill in:

  • Five checkpoint cards, each ticking on its own cron.
  • A "Pass rate" gauge climbing past 95%.
  • A timeline with the occasional red dot, plus the recover-after-60s arc whenever the drift loop's corrupt+heal flow fires.

Stopping​

make demo-down       # stop drift loop + agent container, keep data
make demo-reset      # stop daemons AND re-seed customer_demo from scratch

make demo-down is non-destructive — the customer_demo data stays, so restarting with make demo-up resumes where you left off.

Troubleshooting​

could not translate host name "host.docker.internal". Your Docker build doesn't add the host gateway automatically. The Makefile passes --add-host=host.docker.internal:host-gateway, which works on Docker 20.10+; older Engines need a manual --network host swap or the literal bridge IP.

relation "customers" does not exist. The drift loop ran before the seed. Run make demo-reset to drop + reseed, then make demo-up again.

Pass rate stays at 100%. The drift loop hasn't injected a bad row yet — bad inserts are 5-10% of ticks, so on average one every ~10 ticks (5-10 minutes). Tail .demo/drift_loop.log and grep for bad_insert to confirm; once one fires, the next checkpoint run for that table picks it up. If bad_insert lines are present but the UI still shows green, check that the agent container is up and not throwing agent_poll_failed warnings (docker logs dq-demo-agent).

Missing required env vars from demo_setup.py. All five DQ_CLOUD_* env vars are required. The script lists which one is missing — set it and re-run.

Re-registering the datasource returns 409. The hosted API prevents the same agent token from registering two datasources with the same name. Either deregister the old one (DELETE /api/agent/v1/datasources/<id> from inside the agent container) or use a different name.

See also​