Skip to main content

Deploy PLACEHOLDER Cloud to production

Goal​

You'll move a working PLACEHOLDER Cloud out of docker compose -f infra/docker-compose-dev.yml up and onto a real production deployment — managed Postgres, managed Redis, the API behind a load balancer, the worker on its own instance, the UI as a static bundle.

This page is opinionated on shape, not on tools. We don't ship Kubernetes manifests or a Terraform module today; the steps below are the principles you apply with the tools your team already operates.

Heads up: PLACEHOLDER Cloud now ships three production-ready Docker images (dq-cloud-api, dq-cloud-worker, dq-cloud-ui) plus a reference infra/docker-compose.prod.yml. See Deploy with Docker for the image build / pull / run mechanics and the CI workflow that publishes them to GHCR. This page covers the production posture — secrets, KMS, CORS, load-balancing — that applies regardless of how you ship the images.

Prereqs​

  • You've run PLACEHOLDER Cloud locally and understand the moving parts (see Self-host and Architecture).
  • You have managed Postgres 16 (Aurora, Cloud SQL, Supabase, …) and managed Redis 7 (ElastiCache, Memorystore, Upstash, …) available.
  • You can run a container image.

Steps​

1. Generate production secrets​

Generate fresh values for both — never reuse the dev ones.

# JWT signing
openssl rand -hex 32

# Datasource / LLM encryption
python -c "import base64,os; print(base64.urlsafe_b64encode(os.urandom(32)).decode())"

Store both in a secret manager (AWS Secrets Manager, GCP Secret Manager, Vault, sealed-secrets…). Do not put them in the deployment image.

2. Provision Postgres​

  • A single Postgres 16 instance is enough to start. Reserve ~25 % of capacity for headroom; tenant schemas accumulate quickly.
  • The API process needs CREATE SCHEMA privileges on the database so dqc provision-tenant can create per-org schemas.
  • Enable point-in-time recovery (or at least daily snapshots) — the per-tenant data lives only here.

3. Provision Redis​

  • One Redis 7 instance. PLACEHOLDER Cloud uses it for ARQ (worker queue) and the auth-token cache. The cache TTL is short (5s after HARDEN-7), so memory usage stays small.
  • Persistence (appendonly yes) is recommended but not required — losing the Redis state means in-flight jobs need to be re-enqueued.

4. Configure environment​

Pull every var from reference/configuration.md and set production values. The ones that always change:

VarWhat to set
SECRET_KEYThe hex generated in step 1.
ENCRYPTION_KEYThe base64 generated in step 1.
DB_URLManaged-Postgres connection string.
REDIS_URLManaged-Redis URL.
CORS_ALLOWED_ORIGINSYour UI host(s). No wildcards.
TRUSTED_PROXY_IPSThe CIDR of your load balancer's egress.
DATASOURCE_NETWORK_ALLOWLISTThe CIDRs holding your customers' datasources.
SMTP_*Your transactional mail provider, if you want email alerts.

5. Run migrations​

From a one-off pod / shell / job:

alembic upgrade head

For each new tenant, run initial provisioning:

dqc provision-tenant <org_id>

For every subsequent deploy that touches tenant tables, roll the per-tenant Alembic branch forward:

# One tenant
dqc migrate-tenant <org_id>

# Every tenant — exits non-zero if any tenant fails, so safe for CI
dqc migrate-all-tenants

If you need to rotate per-tenant DEKs (the AES-256-GCM keys protecting datasource and alert-destination secrets), dqc rotate-encryption-key <org_id> or dqc rotate-all-encryption-keys handle the re-encryption pass. See reference/cli.md for the full subcommand surface, and drain writes for the tenant before rotating — a concurrent writer holding the old DEK can persist a ciphertext the new DEK can't decrypt.

6. Deploy the API​

  • One stateless Python container running uvicorn dq_cloud_api.main:app --host 0.0.0.0 --port 8000. Scale horizontally.
  • Behind your usual L7 load balancer with TLS termination.
  • Health-check path: /health.
  • Set TRUSTED_PROXY_IPS to the LB's source range so X-Forwarded-For is honoured.

7. Deploy the worker​

  • One container running arq dq_cloud_worker.main.WorkerSettings.
  • Same image as the API works fine — the entry-point differs.
  • Scale based on queue depth. Start with one replica; add more if arq:queue:dq_cloud accumulates jobs.
  • Until HARDEN-19 lands, run the worker container with readOnlyRootFilesystem: true, drop all Linux capabilities, and apply a NetworkPolicy that allows egress only to your configured datasource hosts.

8. Deploy the UI​

  • cd ui && npm run build produces a static bundle under ui/dist/.
  • Serve it from S3 + CloudFront, Cloudflare Pages, Netlify, or any static-asset CDN.
  • Configure the UI's API base URL to point at the API's load balancer.

Some managed reverse proxies — Google Cloud IAP is the canonical case, and a handful of corporate WAFs do this too — strip Set-Cookie headers from upstream responses by default. The visible symptoms after deploy:

  • Login appears to work (200 OK, access token in the body) but DevTools → Application → Cookies shows no dq_refresh or dq_csrf for the origin.
  • Logout clicks do nothing (the POST /auth/logout body returns 403 because the CSRF cookie isn't there to echo).
  • Step-up auth for sensitive flows (LLM-config edits, agent-token rotation, AI bulk-accept) permanently fails for the same reason.
  • Silent refresh never fires; users get hard-logged-out the first time their 15-minute JWT expires.

The fix is two layers — apply both:

  1. Configure the proxy to pass Set-Cookie through. Under Google Cloud IAP this is set on the backend service: under "Cookies" / "Session affinity" make sure IAP isn't claiming the cookie surface. Behind nginx, our infra/nginx.conf.template now states proxy_pass_header Set-Cookie; explicitly so an operator overlay can't drop it; double-check any custom proxy_hide_header directive in the chain. Behind an AWS ALB / GCP LB, confirm there's no policy stripping Set-Cookie.

  2. Set AUTH_ALLOW_BODY_TOKEN_FALLBACK=true on the API. This is the operational pressure valve: with the flag on, /auth/login (and refresh, accept-invite, the SSO callback) also return the raw refresh token in the JSON body, and /auth/refresh + /auth/logout accept it in the request body when the cookie is absent. The SPA stashes the body-borne values in memory only. CSRF protection on /auth/refresh and /auth/logout accepts a signature-valid bearer access token as alternative proof of intent when the dq_csrf cookie is missing, since the bearer is in SPA memory and an attacker page on another origin can't read it. Note the bearer is accepted even when it has expired (issue #391): /auth/refresh is by definition called once the access token has lapsed, so requiring an unexpired bearer there deadlocked refresh under a cookie-stripping proxy — the symptom was "session expires on nearly every page navigation". Expiry only ever gated authentication (still enforced everywhere else, and the refresh token presented alongside is validated separately); it never needed to gate the CSRF intent proof. Leave the flag off if you control the proxy and can fix it at the proxy layer — body-borne refresh tokens are XSS-exposed in a way the cookie isn't.

10. Operator-facing security checklist​

Walk the open HARDEN-* tickets in .github/ISSUES/ and decide which apply to your deployment. The unmissable ones for production:

  • HARDEN-1 — restrict CORS (no *).
  • HARDEN-2 — strict ENCRYPTION_KEY validation.
  • HARDEN-3 — auth-endpoint rate limiting is shipped; confirm AUTH_LOGIN_RATE_LIMIT_PER_MINUTE / AUTH_REFRESH_RATE_LIMIT_PER_MINUTE are at the values you want (defaults 10 / 60) rather than 0 (test-only).
  • HARDEN-19 — sandbox the checkpoint subprocess at the deployment layer.
  • HARDEN-20 — security headers + body-size limit.

Verify​

  • https://placeholder.example.com/health returns {"status":"ok"}.
  • Log in as a seed user, run a checkpoint, see a result.
  • Watch arq:queue:dq_cloud — the queue should drain promptly.
  • Watch the per-tenant Postgres schemas grow as expected; no cross-tenant rows.

Things that aren't ready​

  • Helm chart / Terraform module aren't published. The single-host path is Self-host with Docker Compose; multi-host operators translate the env table to their orchestrator of choice.
  • Region-pinned managed PLACEHOLDER Cloud isn't a thing — self-host only. See FEATURE-34.