Skip to main content

Deploy the on-prem agent

The on-prem agent runs Great Expectations against a database the cloud can't reach. See Agent mode for the architecture; this page is the operator runbook.

Prereqs​

  • An owner account in PLACEHOLDER Cloud — agent tokens are issued from the Settings → Agents screen.
  • A machine inside the customer network with outbound HTTPS to the cloud control plane and inbound access to the customer DB. Docker optional but recommended.
  • The connection string for each datasource the agent will validate.

1. Issue an agent token​

In PLACEHOLDER Cloud, go to Settings → Agents → Issue token. Name it after the machine you'll run on (e.g. prod-vpc-01). Copy the dqa_… token that appears — the token is shown only once.

2. Deploy the agent container​

The agent ships as a Docker image. Pull, set the three env vars, run:

docker run -d \
  --name dq-agent \
  --restart unless-stopped \
  -e DQ_AGENT_CLOUD_URL=https://placeholder.example.com \
  -e DQ_AGENT_TOKEN=dqa_… \
  -e DQ_AGENT_DATASOURCE_DSNS='{}' \
  ghcr.io/tomplace/dq-cloud-agent:latest

The first start will fail to find a datasource — that's expected; you'll wire DSNs in step 4.

3. Register your datasources​

The agent calls the cloud to register each local datasource. From the container, post a registration:

docker exec dq-agent python -c "
import httpx, os
r = httpx.post(
    os.environ['DQ_AGENT_CLOUD_URL'] + '/api/agent/v1/datasources',
    headers={'Authorization': f'Bearer {os.environ[\"DQ_AGENT_TOKEN\"]}'},
    json={'name': 'prod-orders', 'type': 'postgres'},
)
print(r.status_code, r.json())
"

The response includes the datasource's UUID. Copy it.

You can also do this from the cloud UI — Settings → Agents → <your agent> → Register datasource — which posts on your behalf.

4. Map the datasource UUID to a local DSN​

The agent owns the DSN, not the cloud. Stop the container, set DQ_AGENT_DATASOURCE_DSNS to a JSON object keyed by the UUID, and restart:

docker stop dq-agent && docker rm dq-agent
docker run -d \
  --name dq-agent \
  --restart unless-stopped \
  -e DQ_AGENT_CLOUD_URL=https://placeholder.example.com \
  -e DQ_AGENT_TOKEN=dqa_… \
  -e DQ_AGENT_DATASOURCE_DSNS='{"<datasource-uuid>": "postgresql://dq_ro:****@private.db.local:5432/orders"}' \
  ghcr.io/tomplace/dq-cloud-agent:latest

Multiple datasources go in one JSON object. The DSN format is whatever SQLAlchemy + the dialect driver accepts — postgresql://, snowflake://, bigquery://, etc.

5. Run a checkpoint​

In the cloud UI, build a rule suite against the agent-mode datasource and click Run now. Within DQ_AGENT_POLL_INTERVAL_SECONDS (default 5 s) the agent picks up the job, runs the validation locally, and posts redacted metadata back. The result appears in the Results list with no "view failing rows" panel — the on-prem cut by design.

If you watch docker logs dq-agent you'll see one log line per cycle:

agent_started cloud_url=https://placeholder.example.com datasources=[...]
agent_claim_won run_queue_id=... datasource_id=...
agent_complete_ok success=True expectations=4

Configuration​

Env varDefaultWhat it does
DQ_AGENT_CLOUD_URLhttps://placeholder.example.comBase URL of the cloud control plane.
DQ_AGENT_TOKEN(required)dqa_… issued in the cloud UI.
DQ_AGENT_DATASOURCE_DSNS{}JSON object: datasource UUID → local DSN.
DQ_AGENT_POLL_INTERVAL_SECONDS5Sleep between empty-queue polls.
DQ_AGENT_HEARTBEAT_INTERVAL_SECONDS60Keep-alive interval while a job runs.
DQ_AGENT_REQUEST_TIMEOUT_SECONDS30HTTP timeout for each cloud call.
DQ_AGENT_RUN_TIMEOUT_SECONDS600Wall-clock cap on a single GX run.

Operations​

  • Logs. The agent writes structured (JSON-friendly) lines to stderr. Ship them into your SIEM as you would any container's stderr.
  • Health. The agent is headless — there's no /healthz to probe. Liveness is "the container is running"; readiness is "the cloud is reachable", which the poller surfaces as agent_poll_failed warnings if not. Use your orchestrator's process check.
  • Scaling. A single agent processes one job at a time. Run more containers under the same token to scale horizontally — the cloud's atomic claim makes contention safe.
  • Rotation. To rotate the token, issue a new one in the cloud UI, restart the container with DQ_AGENT_TOKEN set to the new value, then revoke the old one. The redactor / claim flow is stateless on the agent side so no draining is needed.
  • Upgrading. Pull a newer image tag and replace the container. In flight jobs that hadn't reached /complete get reclaimed by another agent after the claim TTL (default 5 minutes); make sure another agent is running before stopping the only one mid-run.

See also​