Skip to main content

Deploy with Docker

PLACEHOLDER Cloud ships three runtime images:

ImageBuilt fromWhat it runs
dq-cloud-apiDockerfile.apiFastAPI app + Alembic migrations
dq-cloud-workerDockerfile.workerARQ worker + the GX 1.x checkpoint runner (psycopg2 bundled for Postgres)
dq-cloud-uiDockerfile.uiReact static bundle served by nginx, reverse-proxying the API

All three are multi-stage. The Python images are based on python:3.12-slim-bookworm and resolve deps with uv, scoped per workspace member so the API image doesn't carry Great Expectations and the worker image doesn't carry FastAPI's HTTP plumbing. The UI image is a node:22-alpine build stage piped into nginx:1.27-alpine.

Postgres and Redis are not packaged. Use the local compose (below) for a self-contained dev stack, or wire production deploys to managed RDS / ElastiCache / Cloud SQL.

Local — one command​

make compose-up

That brings up Postgres + Redis + API + worker + UI behind a single network, runs the public Alembic migrations as a one-shot init container, runs seed_dev.py once for a dev org, and serves the UI at http://localhost:3000. Log in as admin@dqcloud.dev / devpassword.

To wipe state:

make compose-down
docker volume rm dqcloud-local_dq_postgres_data    # if you want a clean DB

The compose file lives at infra/docker-compose.yml. It uses dev-only literal secrets — never run that compose with real customer data.

Production self-host (single host)​

infra/docker-compose.prod.yml is the production-shaped stack — bundled Postgres + Redis (or swap for managed), api + worker + ui, healthchecks per service, persisted volumes, optional Caddy TLS edge, and a first-run admin seed driven by FIRST_ORG_NAME / FIRST_ADMIN_EMAIL / FIRST_ADMIN_PASSWORD in .env. Full walkthrough: Self-host with Docker Compose.

cp .env.example .env
# fill in SECRET_KEY, ENCRYPTION_KEY, POSTGRES_PASSWORD, FIRST_ADMIN_*, DQC_PUBLIC_HOST
docker compose -f infra/docker-compose.prod.yml --env-file .env --profile edge up -d

Key shape:

  • Postgres and Redis are bundled but optional. Drop the two services and point DB_URL / REDIS_URL at managed RDS / Aurora / Cloud SQL / ElastiCache / Memorystore / Upstash when you outgrow single-node.
  • Images come from the registry. Default registry is GHCR (ghcr.io/<owner>/dq-cloud-<service>:<tag>); pin to a SHA-stamped tag (sha-<commit>) for rollbacks. Build locally instead by uncommenting the build: block under each service in the compose file.
  • Secrets live in .env. For multi-tenant production posture, wrap ENCRYPTION_KEY under a KMS KEK by setting ENCRYPTION_KMS_KEY_ID instead of the raw key — see Rotate a secret key.
  • Workers scale horizontally. ARQ pulls jobs from Redis with BLPOP, so docker compose up -d --scale worker=N (or kubectl scale deployment worker --replicas=N) just works.
  • TLS at the edge is optional. Bring Caddy up with --profile edge for automatic Let's Encrypt; omit the profile when you already have an ALB / Cloudflare / nginx ingress fronting the host.

A minimal production env file looks like:

IMAGE_REGISTRY=ghcr.io/your-org
IMAGE_TAG=sha-abc1234

DB_URL=postgresql+asyncpg://dq:password@db.internal:5432/dq
REDIS_URL=rediss://:password@redis.internal:6379

SECRET_KEY=...                      # openssl rand -hex 32
ENCRYPTION_KEY=...                  # python -c 'import os,base64; print(base64.urlsafe_b64encode(os.urandom(32)).decode())'
ENCRYPTION_KMS_KEY_ID=arn:aws:kms:us-east-1:1234:key/abcd  # prod-only; remove ENCRYPTION_KEY if you set this

CORS_ALLOWED_ORIGINS=https://dq.example.com
COOKIE_SECURE=true
DEPLOYMENT_REGION=us-east-1

Build images yourself​

make build-images                                # tags :local for all three
docker build -f Dockerfile.api -t dq-cloud-api:my-tag .
docker build -f Dockerfile.worker -t dq-cloud-worker:my-tag .
docker build -f Dockerfile.ui -t dq-cloud-ui:my-tag .

The first build is ~3 minutes (uv resolves the dep tree, then npm install + vite build for the UI). Subsequent builds hit the layer cache and finish in seconds when only source changes.

CI publishes them for you​

The docker-images workflow builds and pushes all three images to GHCR:

  • Every push to main → :main + :sha-<commit> tags.
  • Every v* git tag → :v1.2.3, :1.2, and re-stamped :latest.
  • PRs build but don't push (forks can't push to your namespace).
  • A registry-side buildcache halves the second-run wall-clock.

To pull a published image:

docker pull ghcr.io/your-org/dq-cloud-api:main
docker pull ghcr.io/your-org/dq-cloud-worker:main
docker pull ghcr.io/your-org/dq-cloud-ui:main

Replace your-org with the GitHub repo owner; replace main with the tag you want. If your packages are private, run docker login ghcr.io first with a GitHub token that has read:packages.

Image hardening notes​

  • All three runtime images run as a non-root user (dqcloud, UID 10001).
  • tini is PID 1 in the Python images so SIGTERM from k8s reaches uvicorn / arq cleanly.
  • The API image carries a HEALTHCHECK against /healthz. The UI image carries one against nginx's own /healthz. The worker doesn't have an HTTP server — rely on the orchestrator's process check + the worker's structlog heartbeats.
  • Build context is filtered by .dockerignore so node_modules, .venv, and .git never end up in the image.
  • The runtime images do not include source-mapped JS or .pyc caches with debug info — production builds.
  • Provenance and SBOM attestations are attached at push time by the GitHub Actions workflow (provenance: true, sbom: true in docker/build-push-action).