Deploy Measured Cloud to production

Goal

You'll move a working Measured Cloud out of docker compose -f infra/docker-compose-dev.yml up and onto a real production deployment — managed Postgres, managed Redis, the API behind a load balancer, the worker on its own instance, the UI as a static bundle.

This page is opinionated on shape, not on tools. We don't ship Kubernetes manifests or a Terraform module today; the steps below are the principles you apply with the tools your team already operates.

Heads up: Measured Cloud now ships three production-ready Docker images (dq-cloud-api, dq-cloud-worker, dq-cloud-ui) plus a reference infra/docker-compose.prod.yml. See Deploy with Docker for the image build / pull / run mechanics and the CI workflow that publishes them to GHCR. This page covers the production posture — secrets, KMS, CORS, load-balancing — that applies regardless of how you ship the images.

Prereqs

• You've run Measured Cloud locally and understand the moving parts (see Self-host and Architecture).
• You have managed Postgres 16 (Aurora, Cloud SQL, Supabase, …) and managed Redis 7 (ElastiCache, Memorystore, Upstash, …) available.
• You can run a container image.

Steps

1. Generate production secrets

Generate fresh values for both — never reuse the dev ones.

# JWT signing
openssl rand -hex 32

# Datasource / LLM encryption
python -c "import base64,os; print(base64.urlsafe_b64encode(os.urandom(32)).decode())"

Store both in a secret manager (AWS Secrets Manager, GCP Secret Manager, Vault, sealed-secrets…). Do not put them in the deployment image.

2. Provision Postgres

• A single Postgres 16 instance is enough to start. Reserve ~25 % of capacity for headroom; tenant schemas accumulate quickly.
• The API process needs CREATE SCHEMA privileges on the database, because signup creates each new org's tenant_<org_id> schema (as do dqc provision-tenant and dqc migrate-all-tenants).
• Enable point-in-time recovery (or at least daily snapshots) — the per-tenant data lives only here.

3. Provision Redis

• One Redis 7 instance. Measured Cloud uses it for ARQ (worker queue) and the auth-token cache. The cache TTL is short (5s after HARDEN-7), so memory usage stays small.
• Persistence (appendonly yes) is recommended but not required — losing the Redis state means in-flight jobs need to be re-enqueued.

4. Configure environment

Pull every var from reference/configuration.md and set production values. The ones that always change:

Var	What to set
SECRET_KEY	The hex generated in step 1.
ENCRYPTION_KEY	The base64 generated in step 1.
DB_URL	Managed-Postgres connection string.
REDIS_URL	Managed-Redis URL.
CORS_ALLOWED_ORIGINS	Your UI host(s). No wildcards.
TRUSTED_PROXY_IPS	The CIDR of your load balancer's egress.
DATASOURCE_NETWORK_ALLOWLIST	The CIDRs holding your customers' datasources.
SMTP_*	Your transactional mail provider, if you want email alerts.

5. Run migrations

From a one-off pod / shell / job:

alembic upgrade head

New tenants are provisioned automatically. Signup and the first-admin seed create each org's tenant_<org_id> schema through the tenant Alembic chain, stamped at head, before the org itself is created. You don't need a manual step per org. dqc provision-tenant <org_id> remains available to repair an org that exists without its schema.

For every subsequent deploy that touches tenant tables, roll the per-tenant Alembic branch forward:

# One tenant
dqc migrate-tenant <org_id>

# Every tenant — exits non-zero if any tenant fails, so safe for CI
dqc migrate-all-tenants

When an org's schema is missing, migrate-all-tenants recreates it and reports it, but only if the org's audit log shows it never had tenant data. If the audit log shows tenant data, the schema may have been lost, so the command recreates nothing and fails that org with a possible data loss message. It also refuses a schema that has tables but no version stamp. Both refusals exit non-zero with recovery instructions. Every missing schema also emits a tenant_schema_missing event, a structured log entry with severity ERROR. Alert on it with a log-based alert on this filter:

resource.type="cloud_run_job" AND jsonPayload.event="tenant_schema_missing" AND severity>=ERROR

See reference/cli.md.

Orgs created during the rollout. migrate-all-tenants runs here, before the new API version serves traffic (step 6). While the new version rolls out, the previous one keeps serving requests, signup included. An org that signs up on the previous version gets its tenant schema built by that version's tenant chain, so it is stamped at the previous head. If that happens after migrate-all-tenants has read its list of orgs, the org stays at the previous head: the new API then serves it with tenant tables that lack the new revisions, and its tenant requests can fail until its schema is upgraded. Re-running dqc migrate-all-tenants once the rollout has finished brings such an org to head. It is safe to re-run, because orgs already at head are left alone. The next deploy's migrate-all-tenants does the same if you don't. For a single org, use dqc migrate-tenant <org_id>.

If self-service signup is reachable from the internet, keep its rate limits on:

• AUTH_SIGNUP_RATE_LIMIT_PER_HOUR: every signup attempt, per IP.
• AUTH_SIGNUP_ORG_CREATE_PER_IP_RATE_LIMIT_PER_DAY: org-creating signups, per IP (per /64 for IPv6).
• AUTH_SIGNUP_ORG_CREATE_RATE_LIMIT_PER_DAY: org-creating signups, one global cap.

Each org-creating signup builds a whole tenant schema, and every schema adds work to migrate-all-tenants on each deploy. The per-IP buckets key on the client address, so set TRUSTED_PROXY_IPS behind a load balancer. Once the global cap is reached, every new-org signup is refused until its window rolls over. The first refusal in each window emits one signup_org_create_cap_reached event, a structured log entry with severity ERROR from the API service. Alert on it with a log-based alert on this filter:

resource.type="cloud_run_revision" AND jsonPayload.event="signup_org_create_cap_reached" AND severity>=ERROR

The event carries the cap (limit), the window (window_seconds) and how long until it rolls over (retry_after_seconds). It never carries the email or the client IP. If real sign-up volume, not abuse, tripped it, raise AUTH_SIGNUP_ORG_CREATE_RATE_LIMIT_PER_DAY. See reference/configuration.md.

If you need to rotate per-tenant DEKs (the AES-256-GCM keys protecting datasource and alert-destination secrets), dqc rotate-encryption-key <org_id> or dqc rotate-all-encryption-keys handle the re-encryption pass. See reference/cli.md for the full subcommand surface, and drain writes for the tenant before rotating — a concurrent writer holding the old DEK can persist a ciphertext the new DEK can't decrypt.

6. Deploy the API

• One stateless Python container running uvicorn dq_cloud_api.main:app --host 0.0.0.0 --port 8000. Scale horizontally.
• Behind your usual L7 load balancer with TLS termination.
• Health-check path: /health.
• Set TRUSTED_PROXY_IPS to the LB's source range so X-Forwarded-For is honoured.

7. Deploy the worker

• One container running arq dq_cloud_worker.main.WorkerSettings.
• Same image as the API works fine — the entry-point differs.
• Scale based on queue depth. Start with one replica; add more if arq:queue:dq_cloud accumulates jobs.
• Until HARDEN-19 lands, run the worker container with readOnlyRootFilesystem: true, drop all Linux capabilities, and apply a NetworkPolicy that allows egress only to your configured datasource hosts.

8. Deploy the UI

• cd ui && npm run build produces a static bundle under ui/dist/.
• Serve it from S3 + CloudFront, Cloudflare Pages, Netlify, or any static-asset CDN.
• Configure the UI's API base URL to point at the API's load balancer.

9. Reverse proxies that strip Set-Cookie (issue #117)

Some managed reverse proxies — Google Cloud IAP is the canonical case, and a handful of corporate WAFs do this too — strip Set-Cookie headers from upstream responses by default. The visible symptoms after deploy:

• Login appears to work (200 OK, access token in the body) but DevTools → Application → Cookies shows no dq_refresh or dq_csrf for the origin.
• Logout clicks do nothing (the POST /auth/logout body returns 403 because the CSRF cookie isn't there to echo).
• Step-up auth for sensitive flows (LLM-config edits, agent-token rotation, AI bulk-accept) permanently fails for the same reason.
• Silent refresh never fires; users get hard-logged-out the first time their 15-minute JWT expires.

The fix is two layers — apply both:

1. Configure the proxy to pass Set-Cookie through. Under Google Cloud IAP this is set on the backend service: under "Cookies" / "Session affinity" make sure IAP isn't claiming the cookie surface. Behind nginx, our infra/nginx.conf.template now states proxy_pass_header Set-Cookie; explicitly so an operator overlay can't drop it; double-check any custom proxy_hide_header directive in the chain. Behind an AWS ALB / GCP LB, confirm there's no policy stripping Set-Cookie.

2. Set AUTH_ALLOW_BODY_TOKEN_FALLBACK=true on the API. This is the operational pressure valve: with the flag on, /auth/login (and refresh, accept-invite, the SSO callback) also return the raw refresh token in the JSON body, and /auth/refresh + /auth/logout accept it in the request body when the cookie is absent. The SPA stashes the body-borne values in memory only. CSRF protection on /auth/refresh and /auth/logout accepts a signature-valid bearer access token as alternative proof of intent when the dq_csrf cookie is missing, since the bearer is in SPA memory and an attacker page on another origin can't read it. Note the bearer is accepted even when it has expired (issue #391): /auth/refresh is by definition called once the access token has lapsed, so requiring an unexpired bearer there deadlocked refresh under a cookie-stripping proxy — the symptom was "session expires on nearly every page navigation". Expiry only ever gated authentication (still enforced everywhere else, and the refresh token presented alongside is validated separately); it never needed to gate the CSRF intent proof. Leave the flag off if you control the proxy and can fix it at the proxy layer — body-borne refresh tokens are XSS-exposed in a way the cookie isn't.

10. Operator-facing security checklist

Walk the open HARDEN-* tickets in .github/ISSUES/ and decide which apply to your deployment. The unmissable ones for production:

• HARDEN-1 — restrict CORS (no *).
• HARDEN-2 — strict ENCRYPTION_KEY validation.
• HARDEN-3 — auth-endpoint rate limiting is shipped; confirm AUTH_LOGIN_RATE_LIMIT_PER_MINUTE / AUTH_REFRESH_RATE_LIMIT_PER_MINUTE are at the values you want (defaults 10 / 60) rather than 0 (test-only).
• HARDEN-19 — sandbox the validation-run subprocess at the deployment layer.
• HARDEN-20 — security headers + body-size limit.

Verify

• https://dq.example.com/health returns {"status":"ok"}.
• Log in as a seed user, run a validation, see a result.
• Watch arq:queue:dq_cloud — the queue should drain promptly.
• Watch the per-tenant Postgres schemas grow as expected; no cross-tenant rows.

Things that aren't ready

• Helm chart / Terraform module aren't published. The single-host path is Self-host with Docker Compose; multi-host operators translate the env table to their orchestrator of choice.
• Region-pinned managed Measured Cloud isn't a thing — self-host only. See FEATURE-34.