Configuration reference
Every PLACEHOLDER Cloud setting comes from environment variables, loaded by api/src/dq_cloud_api/config.py. The canonical template is .env.example — this page is the prose version. Both must stay in sync; if you ship a new env var, update both.
Core
| Variable | Default | Description |
|---|---|---|
SECRET_KEY | (no default — must be set) | HS256 signing key for JWTs — the trust anchor for every user session. Generate with openssl rand -hex 32 (or python -c "import secrets; print(secrets.token_urlsafe(48))"). When DEBUG is off the app refuses to boot if this is empty, a known dev/placeholder value (change-me, test-only-secret, …), or shorter than 32 characters — a guessable key lets anyone forge a token for any user. Rotate per Rotate a secret key. |
ADDITIONAL_JWT_KEYS | (empty) | Comma-separated list of extra keys to accept for verification (not signing). Used during SECRET_KEY rotation. |
DB_URL | postgresql+asyncpg://dqcloud:dqcloud@localhost:5432/dqcloud | Postgres connection string. Must use asyncpg. |
REDIS_URL | redis://localhost:6379 | Redis URL for ARQ + token cache + auth rate limiting. |
ENCRYPTION_KEY | (no default — must be set) | AES-256-GCM key for encrypting datasource credentials and LLM API keys. Must be a URL-safe base64 string decoding to exactly 32 bytes. The app refuses to start otherwise. In KMS-backed deployments (ENCRYPTION_KMS_KEY_ID set) this serves as the env-fallback wrapping key. |
ENCRYPTION_KEY_PREVIOUS | (empty) | Comma-separated list of prior ENCRYPTION_KEY values, accepted on decrypt only. When the env-fallback key is rotated in place (e.g. a Cloud Run env-var swap), every tenant's DEK is still wrapped under the old key and would otherwise fail to unwrap (surfacing as Incorrect padding on every pre-existing datasource credential and LLM key). Listing the prior key(s) here lets the wrapper fall back to them so the fleet keeps working until you run dqc rotate-all-encryption-keys to rewrap each DEK under the new key. Each value must decode to 32 bytes, same as ENCRYPTION_KEY. Drop these once the rewrap has run. A malformed value fails startup. See Rotate a secret key. |
ENCRYPTION_KMS_KEY_ID | (empty) | AWS KMS KeyId (or ARN) used as the Key Encryption Key that wraps each tenant's DEK (HARDEN-16). Empty selects the env-fallback EnvKEK (reads ENCRYPTION_KEY). Requires the dq-cloud-api[kms] extra for boto3. |
ENCRYPTION_DEK_CACHE_TTL_SECONDS | 300 | How long a tenant's plaintext DEK is held in process memory before forcing a KMS round-trip. Higher values reduce KMS load; lower values reduce the time window a leaked process memory image exposes key material. |
CORS_ALLOWED_ORIGINS | http://localhost:3000 | Comma-separated origins allowed to call the API. No wildcards in production. |
PUBLIC_BASE_URL | (empty) | External base URL of the deployment (e.g. https://dq.example.com). Used to build deep-links in outbound alerts (Slack, Teams, generic webhook, PagerDuty custom_details.deep_link) and to build SAML / OIDC redirect URIs. Empty disables the deep-link button on alert messages and breaks SSO callbacks. |
DEPLOYMENT_REGION | (empty → self-hosted) | Operator-declared region label surfaced by GET /api/v1/deployment and rendered in the UI footer (FEATURE-34). Free-form — suggested values are us-east-1 / eu-west-1 / apac-northeast-1 / self-hosted or any other label that describes where this deployment actually runs. No auto-detection: PLACEHOLDER Cloud is self-host-first, so the value is whatever you put here. |
DATASOURCE_NETWORK_ALLOWLIST | (empty) | Comma-separated CIDR list for the SSRF guard. Any outbound destination whose host resolves to a private / loopback / link-local / metadata IP (including IPv4-mapped / 6to4 / NAT64 IPv6 forms of those) must match a listed CIDR or PLACEHOLDER Cloud refuses to connect. The guard covers direct-mode datasource DSNs — now including trino:// and databricks://, whose coordinator/workspace host:port is dialed directly by the driver (a genuine private-DNS cluster is re-admitted by listing its CIDR here) — checked at save and before the worker dials them; per-org LLM base_url endpoints (custom / Ollama / Azure OpenAI); the worker's outbound webhook / catalog-sync / alert-channel / DataHub egress; and the pandas-branch http(s) profile-asset path. HTTP egress is pinned to the vetted IP and re-checked on every redirect hop, so a DNS-rebind between validation and connect cannot reach an internal address. Agent-mode datasources connect from the on-prem agent and are out of scope. See HARDEN-12. |
DATASOURCE_TEST_TIMEOUT_SECONDS | 30 | Upper bound (seconds) on the synchronous datasource connection-test (the Test connection button in the UI). The default leaves room for a serverless warehouse (Snowflake, Databricks SQL) to cold-start from an auto-suspended state — those first-probe resumes routinely exceed 5s, so a tighter value makes a valid connection fail its first test (#637). A datasource that actively refuses the connection (TCP RST) or fails DNS resolution still fails fast on the driver's own connect error; a firewalled host that silently drops packets produces no error, so the probe waits out this configured ceiling. This is only the upper bound. Independent of DQ_EXECUTOR_TIMEOUT so tuning the worker doesn't shrink the UI ping. |
HTTP hardening (HARDEN-20)
| Variable | Default | Description |
|---|---|---|
MAX_REQUEST_BODY_MB | 10 | Reject request bodies larger than this with a 413 before the handler runs. |
TRUSTED_PROXY_IPS | (empty) | Comma-separated CIDRs of upstream proxies that may set X-Forwarded-* headers. Leave blank without a real proxy — enabling it without one lets any client forge their source IP. |
Worker
| Variable | Default | Description |
|---|---|---|
DQ_EXECUTOR_BACKEND | local | local (subprocess) or kubernetes (planned in HARDEN-19). |
DQ_EXECUTOR_TIMEOUT | 300 | Per-checkpoint subprocess wall-clock limit in seconds. |
DQ_EXECUTOR_MEMORY_MB | 2048 | Subprocess address-space rlimit (MB). Ignored on macOS. |
DQ_EXECUTOR_MAX_OPEN_FILES | 1024 | RLIMIT_NOFILE cap on the checkpoint subprocess (HARDEN-19). |
DQ_EXECUTOR_MAX_FILE_SIZE_MB | 100 | RLIMIT_FSIZE cap on the checkpoint subprocess (HARDEN-19). |
DQ_WORKER_CONCURRENCY | 4 | How many jobs the worker processes in parallel. |
DQ_WORKER_POLL_INTERVAL | 10 | Seconds between ARQ poll cycles. |
PROFILE_RUN_STALE_SECONDS | 1800 | Staleness threshold for orphaned profile runs (#502). The worker's reap_stale_profile_runs cron runs every minute and marks any profile_runs row still status='running' whose execution started (started_at, the moment the job flipped it to running — falling back to run_at only for legacy rows written before that column existed) is older than this many seconds as failed (rows are never deleted). Thresholding on execution-start rather than enqueue time means a job that waited out a queue backlog before starting is not reaped while it is still healthily in flight. This is the safety net for a worker that dies mid-job — SIGKILL / OOM / pod eviction / redeploy — between writing running and the terminal status, which the in-process handling in the profile_run job (#490) structurally cannot catch. 30 minutes comfortably exceeds a normal profile run; raise it if you legitimately profile very large assets. Validated at config load to be at least 60 seconds — a lower value would reap healthy in-flight runs every sweep. |
GLOBAL_AUDIT_EVENT_RETENTION_DAYS | 90 | Bounded retention for public.global_audit_events (#144). The worker's prune_global_audit_events cron runs daily at 03:30 UTC and deletes rows older than this many days. The table accepts traffic from anonymous probes, so its growth scales with attempt volume rather than active customer count — without a prune it accumulates indefinitely on an internet-exposed deployment. Set to 0 to disable the prune (the cron remains registered but becomes a no-op). |
AUDIT_SIEM_V2_SIGNING | true | AUDIT-3.1 (#256) replay protection for SIEM webhook deliveries. When on, each dispatch stamps an X-DQ-Cloud-Timestamp header (RFC 3339 UTC, Z suffix) AND signs HMAC(secret, f"{timestamp}.{body}") in X-DQ-Cloud-Signature instead of HMAC(secret, body). Receivers MUST reconstruct the signed string as timestamp + "." + raw_body before verification AND reject requests where abs(now - timestamp) exceeds a freshness window (5 minutes is the recommended default) to defeat replay. This is a BREAKING change to the AUDIT-3 v1 wire format; flip to false for one release if you need to update receivers in lockstep before turning it back on. |
Auth
| Variable | Default | Description |
|---|---|---|
JWT_ALGORITHM | HS256 | The signing algorithm. RS256 after HARDEN-13. |
JWT_EXPIRE_MINUTES | 15 | Access-token lifetime. |
REFRESH_TOKEN_EXPIRE_DAYS | 30 | Refresh-token lifetime (and the maximum delay between forced re-login). |
REFRESH_REUSE_GRACE_SECONDS | 0 | Grace window for the refresh-token reuse detector (off by default → strict reuse detection). Replaying an already-rotated refresh token normally revokes the whole token family (stolen-then-rotated signal). When set > 0: within this many seconds of a token's rotation — and only when its family still has exactly one live successor — /auth/refresh treats the replay as a benign tab/reload rotation race and rotates the live successor instead of revoking. The primary [#619] fix is client-side (cross-tab single-flight refresh), which removes the reproducible races; enable this grace (e.g. 10) only for the narrow hard-reload cookie-propagation residual on a non-cookie-stripping deployment, accepting the slight reuse-detection weakening it implies. |
JWT_PRIVATE_KEY_PEM | (empty) | PEM-encoded RSA private key, signer-only. Set to enable RS256 issuance; leaving empty keeps HS256 on for the rollover window (HARDEN-13). |
JWT_PUBLIC_KEY_SET | [] | JSON-serialised JWKS-shaped list of public keys for verification. First entry is the signing kid; verifiers short-circuit on the JWT's kid header. Supports rotation: add new public, deploy, switch private, deploy, drop old. |
JWT_KMS_KEY_ID | (empty) | Optional KMS Key ID for signing (instead of JWT_PRIVATE_KEY_PEM). Empty = read PEM from env. |
JWT_PUBLISH_JWKS | false | If true, exposes GET /.well-known/jwks.json carrying JWT_PUBLIC_KEY_SET. Off by default so the public set isn't exposed unless an operator opts in. |
TOKEN_CACHE_TTL_SECONDS | 5 | How long a dqk_ / dqa_ token → Principal lookup is cached in Redis. A revoked token is accepted for at most this long. Raise only after measuring auth-path load; see HARDEN-7. |
AUTH_LOGIN_RATE_LIMIT_PER_MINUTE | 10 | Per-IP + per-IP-email login attempts (the per-(IP, email) sub-bucket is capped at limit // 2, floored at 1 for non-zero limits). Also drives the matching split bucket on POST /auth/elevate. 0 disables both halves entirely (tests only) — see issue #143 for the per-(IP, email) floor that previously leaked through. |
AUTH_REFRESH_RATE_LIMIT_PER_MINUTE | 60 | Per-IP refresh-token exchanges. 0 disables (tests only). |
Per-org rate limits (HARDEN-18)
Per-tenant token-bucket limits on abuse-prone mutation endpoints. Buckets are keyed rl:org:{org_id}:{bucket} so one org over-shooting can't starve the others. Each value is requests per minute; 0 disables that bucket (useful in tests / single-user self-host). Over-limit hits return 429 with a Retry-After header and increment the rate_limit_rejections_total{bucket="..."} Prometheus counter.
| Variable | Default | Bucket |
|---|---|---|
RATE_LIMIT_PROFILE_RUNS_PER_MINUTE | 60 | profile_runs — POST /api/v1/.../profile-runs |
RATE_LIMIT_AI_SUGGESTIONS_PER_MINUTE | 30 | ai_suggestions — POST /api/v1/.../ai-suggestions |
RATE_LIMIT_DATASOURCE_TEST_PER_MINUTE | 20 | datasource_test — POST /api/v2/.../datasources/{id}/test |
RATE_LIMIT_DATASOURCE_DISCOVER_PER_MINUTE | 20 | datasource_discover — GET /api/v2/.../datasources/{id}/discover |
RATE_LIMIT_AGENT_TOKENS_PER_MINUTE | 10 | agent_tokens — POST /api/v1/.../agent-tokens |
RATE_LIMIT_PERSONAL_API_KEYS_PER_MINUTE | 10 | personal_api_keys — POST /api/v1/organizations/{org_id}/personal-api-keys (each call mints a new dqk_ bearer for the calling user; also step-up gated). See Create an API token. |
RATE_LIMIT_CHECKPOINT_RUN_PER_MINUTE | 60 | checkpoint_run — POST /api/v1/.../checkpoints/{id}/run |
RATE_LIMIT_ALERT_RULES_PER_MINUTE | 20 | alert_rules — POST /api/v1/.../alert-rules |
RATE_LIMIT_INVITES_PER_MINUTE | 10 | invites — POST /api/v1/organizations/{org_id}/invites (each call mints a credential bridge + can emit an invite email). |
RATE_LIMIT_DATA_DOCS_REBUILD_PER_MINUTE | 5 | data_docs_rebuild — POST /api/v1/organizations/{org_id}/data-docs-config/rebuild (each call walks every ValidationResult and republishes the static site). |
RATE_LIMIT_DBT_INGEST_PER_MINUTE | 6 | dbt_ingest — POST /api/v1/.../dbt/ingest (FEATURE-27). Each call writes one row per dbt model and (optionally) one row per run-result, so per-call cost scales with project size. Generous for the canonical "dbt build && curl" CI cadence; raise it if your CI dispatches many parallel dbt projects to one org. |
RATE_LIMIT_RESULTS_ALL_PER_MINUTE | 60 | results_all — the two "All" results aggregators (GET /api/v1/organizations/{org_id}/results/all and GET /api/v1/.../workspaces/{workspace_id}/results/all; one bucket per org per caller, so concurrent dashboard users in one org don't 429 each other). Read-only, but the first page of every walk runs an unbounded GROUP BY over validation_results (the largest tenant table), so the per-caller cap keeps a tight polling loop from burning DB time. Far above any dashboard's refresh cadence. |
RATE_LIMIT_SCIM_RESOURCE_PER_MINUTE | 60 | scim_resource — the SCIM resource CRUD verbs (POST/PUT/PATCH/DELETE on /scim/v2/Users and /scim/v2/Groups). Caps the provision / deprovision rate a dqs_ bearer can drive so a compromised IdP connector can't mass-mutate memberships in a tight loop (the token-mint route is already capped by RATE_LIMIT_AGENT_TOKENS_PER_MINUTE). Read / list verbs are uncapped. A normal IdP reconcile sits well under this; raise it for a large initial backfill. |
COOKIE_SECURE | true | Marks the dq_refresh + dq_csrf auth cookies Secure (HARDEN-14). Set to false only for local HTTP development; any non-localhost deployment must leave this on. After upgrading past HARDEN-14, in-flight localStorage-stored sessions stop working — users sign in once and pick up the new cookie. |
AUTH_ALLOW_BODY_TOKEN_FALLBACK | false | Opt-in workaround for reverse proxies that silently strip Set-Cookie from the API response (Google Cloud IAP is the canonical offender — symptom: logout / step-up / silent refresh all fail because the dq_refresh + dq_csrf cookies never reach the browser). When true, /auth/login, /auth/refresh, /auth/accept-invite, and the SSO callback also return the raw refresh token in the JSON body, and /auth/refresh + /auth/logout accept it back via the request body when the cookie is missing. The SPA holds the body-borne token in memory only. Leave OFF unless a proxy is known to strip cookies — body-borne refresh tokens are XSS-exposed in a way the cookie isn't. The long-term fix is to configure the proxy to pass Set-Cookie through (see Deploy to production → Reverse proxies that strip Set-Cookie). |
AUTH_STEP_UP_WINDOW_SECONDS | 300 | Step-up auth freshness window (HARDEN-24). Access tokens older than this fail sensitive routes with 401 detail=step_up_required and a WWW-Authenticate: StepUp header; the UI prompts for password and calls /auth/elevate to refresh auth_fresh_at. |
AUTH_MFA_ENABLED | false | Deferred (HARDEN-24) — when set, /auth/elevate will accept a TOTP / WebAuthn proof instead of (or in addition to) the password. The knob exists today so the dependency wiring doesn't have to move when MFA lands. |
INVITE_EXPIRY_DAYS | 7 | How long a team invite link is valid (FEATURE-38). Invites past this age are rejected at accept time. |
Google Identity-Aware Proxy (/auth/iap-exchange)
POST /auth/iap-exchange exchanges a Google IAP identity for a DQ Cloud session. It runs in exactly one of two mutually-exclusive, fail-closed modes — there is no fall-through between them.
| Variable | Default | Description |
|---|---|---|
IAP_AUDIENCE | (empty) | When set, the endpoint runs verified-JWT mode: it requires the signed X-Goog-IAP-JWT-Assertion header and verifies it against Google's JWKS for this audience. A missing or invalid assertion is a hard 401 — the header-trust branch is disabled, so a stripped-JWT request cannot downgrade verified mode into header-trust. Audience format: /projects/<project-number>/global/backendServices/<backend-id>. Set this wherever the load balancer preserves the signed assertion. |
IAP_TRUST_HEADERS | false | Explicit opt-in for header-trust mode (only consulted when IAP_AUDIENCE is empty). When true, the endpoint mints a session from the unsigned X-Goog-Authenticated-User-Email + X-Goog-Authenticated-User-Id header pair. When false (the default) a header-trust request is refused with 401 — turning what was previously a silent default into a conscious operator decision. Only safe when Cloud Run ingress is internal-and-cloud-load-balancing (run.googleapis.com/ingress; the run.app URL must 404 to the public internet) and an IAP-fronted load balancer overwrites those headers on every request. This is the GCP Cloud Run reality where Cloud Run strips the signed JWT assertion before it reaches the container; set this true there. If either precondition fails, any client that can reach the container can impersonate any user — the API logs a loud error at startup when this is on without DEBUG. |
IAP_JIT_PROVISION_DOMAINS | (empty) | Comma-separated email domains whose IAP-authenticated users get auto-created (JIT-provisioned) on first sign-in. Empty refuses every JIT path — only pre-provisioned users can sign in via IAP. |
IAP_DEFAULT_ORG_SLUG | (no default) | Org slug new IAP-JIT users are attached to. Required when IAP_JIT_PROVISION_DOMAINS is non-empty — there is no implicit default (#282); JIT requests 503 until it is set. |
IAP_DEFAULT_JIT_ROLE | editor | Role a freshly JIT-provisioned IAP user is minted at in IAP_DEFAULT_ORG_SLUG (org + first-workspace membership). Defaults to least-privilege editor. A small single-tenant install whose IAP audience is the admin set can set this to admin so the first IAP user lands as an org admin instead of needing a manual promote. Validated against the human-role matrix; an unknown or token-only value falls back to editor. Hardcoding this to editor is what left the IAP-provisioned admin@dqcloud.dev account stuck as an editor (#336/#340/#343) — to fix an already-drifted account, run dqc set-role <email> admin. |
Agent-side environment (dq-cloud-agent)
The on-prem agent reads these from its container env. They are
distinct from the cloud-side .env — never mix the two. See
Deploy the agent for the operator
runbook and Agent mode for the
architecture.
| Variable | Default | Purpose |
|---|---|---|
DQ_AGENT_CLOUD_URL | https://placeholder.example.com | Base URL of the cloud control plane the agent polls. |
DQ_AGENT_TOKEN | (required) | dqa_… token issued in the cloud UI. |
DQ_AGENT_DATASOURCE_DSNS | {} | JSON object mapping datasource UUID → local DSN. The DSN is whatever SQLAlchemy + the dialect driver accepts; never sent to the cloud. |
DQ_AGENT_POLL_INTERVAL_SECONDS | 5 | Sleep between empty-queue polls. |
DQ_AGENT_HEARTBEAT_INTERVAL_SECONDS | 60 | Keep-alive interval while a job runs so another agent doesn't reclaim it. |
DQ_AGENT_REQUEST_TIMEOUT_SECONDS | 30 | HTTP timeout for each cloud call. |
DQ_AGENT_RUN_TIMEOUT_SECONDS | 600 | Wall-clock cap on a single GX run. |
MCP server (dq-cloud-mcp)
The hosted MCP server is a separate service (uvicorn dq_cloud_mcp.server:app, default port 8080) that exposes PLACEHOLDER Cloud as Model Context Protocol tools and proxies every call to the REST API. It reads its own settings from the environment, all prefixed DQ_MCP_ (loaded by mcp/src/dq_cloud_mcp/config.py). These are distinct from the cloud-side .env consumed by the API / worker — the MCP server only needs the URLs and the page-limit knobs below. See Connect an MCP client for the operator/user walkthrough.
| Variable | Default | Description |
|---|---|---|
DQ_MCP_DQ_API_URL | http://localhost:8000 | Root URL of the PLACEHOLDER Cloud REST API this server proxies to (e.g. https://placeholder.example.com). Trailing slashes are tolerated — the client strips them. Doubles as the OAuth issuer_url advertised to MCP clients once OAuth lands (the API is the authorization server that mints dqk_ keys). In Docker Compose this points at the internal api service (http://api:8000); on Cloud Run, at the internal API service URL. |
DQ_MCP_MCP_PUBLIC_URL | http://localhost:8080 | The publicly-reachable URL of this MCP server (e.g. https://mcp.placeholder.example.com). Advertised as the protected resource_server_url in the OAuth metadata so clients know which audience the bearer is for. Set it to the externally-reachable URL clients dial, not the internal one. |
DQ_MCP_REQUEST_TIMEOUT | 30.0 | Per-request timeout (seconds) for outbound calls to the REST API. Must be > 0. |
DQ_MCP_DEFAULT_PAGE_LIMIT | 25 | Page size a list tool uses when the caller omits limit. Must be > 0. |
DQ_MCP_MAX_PAGE_LIMIT | 100 | Hard ceiling on limit; a caller asking for more is clamped to this. Protects both the API and the model's context window from an unbounded page. Must be > 0. |
The MCP server holds no database, no Redis, and no secrets of its own — it authenticates the caller's dqk_… bearer by forwarding it to the API, which enforces all org / workspace / role isolation. So there are no SECRET_KEY / ENCRYPTION_KEY / DB_URL settings here.
AI features
| Variable | Default | Description |
|---|---|---|
DEFAULT_LLM_PROVIDER | (empty = disabled) | Fallback provider when no LLMConfig row exists. One of openai / anthropic / ollama / custom. |
DEFAULT_LLM_API_KEY | (empty) | Bearer credential for the fallback provider. Per-org keys via LLMConfig override this. |
DEFAULT_LLM_BASE_URL | (empty) | Override for Ollama / Azure / OpenAI-compatible custom endpoints. |
DEFAULT_LLM_MODEL | (empty) | Model identifier per provider (e.g. claude-3-5-sonnet, gpt-4o-mini). |
PROFILE_SAMPLE_MAX_ROWS | 10000 | Cap on rows scanned per profile run. Larger means slower + more accurate. |
VOLUME_DRIFT_WINDOW_RUNS | 10 | Rolling window of completed ProfileRuns used to baseline row_count for the FEATURE-13 volume-drift detector. Detector is a no-op until at least this many prior completed runs exist for the asset. |
COMPLETENESS_DRIFT_WINDOW_RUNS | 10 | Rolling window of completed ProfileRuns used to baseline per-column null_rate for the FEATURE-14 completeness-drift detector. Detector is a no-op until at least this many prior completed runs exist for the asset. |
The daily profile sweep runs at 03:00 UTC and is not env-configurable today; HARDEN-6 introduces PROFILE_SCHEDULE_CRON.
Custom expectation plugins (FEATURE-35)
| Variable | Default | Description |
|---|---|---|
EXPECTATIONS_PLUGIN_PACKAGES | (empty) | Comma-separated list of importable Python packages the worker walks on startup for great_expectations.Expectation subclasses. Each discovered class is registered per-org so the rule-suite editor and org settings page can list it. The seven built-in expectation types remain available regardless. A failing import is logged + skipped (never blocks startup) and surfaced via GET /api/v1/organizations/{org_id}/custom-expectations/packages so an operator can see the failure in the settings UI without tailing worker logs. |
OpenTelemetry (HARDEN-22)
| Variable | Default | Description |
|---|---|---|
OTEL_EXPORTER_OTLP_ENDPOINT | (empty) | OTLP/HTTP collector endpoint (e.g. http://otel-collector:4318). Empty disables OTel init entirely — dev / test / CI don't need a collector running. |
OTEL_SERVICE_NAME | dq-cloud-api | Lands on the OTel service.name Resource attribute. |
OTEL_ENVIRONMENT | development | Lands on the OTel deployment.environment Resource attribute. Use to route traces in your backend. |
OTEL_TRACES_SAMPLE_RATIO | 1.0 | ParentBased(TraceIdRatioBased) value. Lower if trace volume becomes an issue. |
GCP_TRACE_ENABLED | false | Export spans to Google Cloud Trace (in addition to any OTLP exporter). Authenticates via Application Default Credentials — on Cloud Run that's the service account, no key file needed. Enables OTel init on its own, so you can run Cloud Trace without an OTLP collector. See LLM tracing. |
GCP_PROJECT_ID | (empty) | Destination project for GCP_TRACE_ENABLED. Empty falls back to the ADC-detected project (the right default on Cloud Run). |
OTEL_GENAI_CAPTURE_CONTENT | false | Privacy opt-in: when true, LLM spans carry the prompt and completion text (gen_ai.input.messages / gen_ai.output.messages), truncated to ~2 KB per attribute with a …[truncated] marker. Off by default because that text is customer-derived data; token counts are always recorded regardless. Platform-global: this flag has no per-org scope — enabling it captures prompt/completion text (which can embed profiled customer data) from every org's LLM calls into the operator's trace backend. Intended for single-tenant / self-hosted deployments; a per-org opt-in would be needed before turning it on in a shared multi-tenant deployment. |
LLM tracing → Google Cloud Trace
Every AI completion the self-hostable data-quality platform makes — rule suggestions, root-cause narratives, the rule "explain" endpoint — flows through a single client, and that client emits one OpenTelemetry span per call following the GenAI semantic conventions. You don't configure anything per-callsite; turning on an exporter is enough.
What each span records:
- Name
chat <model>, span kindCLIENT. - Provider and model —
gen_ai.system(e.g.anthropic,openai,azure_openai,ollama,custom),gen_ai.request.model,gen_ai.request.max_tokens,gen_ai.request.temperature. - Token usage —
gen_ai.usage.input_tokensandgen_ai.usage.output_tokenson success. These are always recorded. - Errors — a failed completion sets the span status to
ERRORand records the exception, so a provider outage or rate-limit shows up in the trace.
Prompt and completion text are not captured by default — that's
customer-derived data. Only the metadata and token counts above leave the
process unless you explicitly set OTEL_GENAI_CAPTURE_CONTENT=true. When you
do, each captured value is truncated to roughly 2 KB (with a …[truncated]
marker) — full prompts can run to tens of KB, and Cloud Trace clips string
attributes at 256 bytes regardless.
Multi-tenant note.
OTEL_GENAI_CAPTURE_CONTENTis a single, platform-global switch — it has no per-org scope. Turning it on records prompt/completion text (which can embed profiled customer data) from every org's LLM calls into the operator's trace backend. Treat it as a single-tenant / self-hosted debugging aid; a shared multi-tenant deployment would need a per-org opt-in before enabling it.
To ship these spans to Google Cloud Trace, set GCP_TRACE_ENABLED=true
(optionally pin GCP_PROJECT_ID; empty uses the ADC-detected project). The
exporter authenticates with Application Default Credentials, so on Cloud Run
the service account is all you need — no key file, no collector. The Cloud
Trace and OTLP exporters are independent: enable either, both, or neither. The
span itself is a no-op when no exporter is configured, so dev and CI pay
nothing.
SMTP (optional — required for email alerts)
| Variable | Default | Description |
|---|---|---|
SMTP_HOST | (empty) | SMTP relay hostname. Without this, the email alert dispatcher logs and skips. |
SMTP_PORT | 587 | Relay port. |
SMTP_USER | (empty) | SMTP auth username. |
SMTP_PASSWORD | (empty) | SMTP auth password. |
SMTP_FROM | dq-cloud@example.com | From: header on outbound alert mail. |
docker-compose-only variables
These are consumed by infra/docker-compose.prod.yml, not by the API / worker processes. See Self-host with Docker Compose for the operator walkthrough.
| Variable | Default | Description |
|---|---|---|
POSTGRES_USER | dqcloud | Superuser created on first boot of the bundled Postgres container. Must match the user portion of DB_URL. |
POSTGRES_PASSWORD | (no default — must be set) | Password for POSTGRES_USER. Picked up on first boot only; rotating it after that requires the usual ALTER USER flow. |
POSTGRES_DB | dqcloud | Database created on first boot. Must match the database portion of DB_URL. |
IMAGE_REGISTRY | ghcr.io/tomplace | Registry the api / worker / ui / agent images are pulled from. |
IMAGE_TAG | latest | Tag pulled for all four images. Pin a SHA-stamped or release tag in production. |
DQC_PUBLIC_HOST | localhost | Public DNS name served by the Caddy TLS edge under --profile edge. Real DNS opts into automatic Let's Encrypt. |
DQC_ACME_EMAIL | (empty) | Email Let's Encrypt sends certificate expiry notifications to. Required when DQC_PUBLIC_HOST is a real DNS name. |
DQC_UI_HOST_PORT | 8080 | Host port the UI container is bound to when Caddy is NOT in front. Ignored under --profile edge. |
DQC_MCP_HOST_PORT | 8081 | Host port the optional mcp service (the hosted MCP server, --profile mcp) is bound to. Defaults to 8081 to avoid clashing with DQC_UI_HOST_PORT. The container always listens on 8080; point your external reverse proxy / LB at this host port and route it directly (the MCP server is not behind the UI's SSO / IAP gate — external LLM clients carry a dqk_ bearer they couldn't get through a browser SSO). |
DQ_MCP_MCP_PUBLIC_URL | http://localhost:8081 | (compose-only default for the mcp profile) The externally-reachable URL of the MCP server clients dial — set it to the public hostname (e.g. https://mcp.<your-domain>), not the internal one. The MCP server itself reads this and the other DQ_MCP_* vars; see MCP server above for the full list. |
FIRST_ORG_NAME | (empty = skip seed) | Org name created by the one-shot seed container on first boot via scripts/seed_first_admin.py — also read by dqc seed, where it is required. |
FIRST_ORG_SLUG | (derived from name) | Optional slug override. |
FIRST_ADMIN_EMAIL | (empty = skip seed) | Admin user created by the seed container / dqc seed. Seeded with the canonical super_admin org role + derived workspace_admin workspace role. |
FIRST_ADMIN_PASSWORD | (empty = skip seed) | Admin password (12+ chars). Rotate via the UI immediately after first login. |
FIRST_LLM_PROVIDER | (empty = skip LLM config) | dqc seed only: provider for the seeded default LLM config (openai / anthropic / azure_openai / ollama / custom). Set together with FIRST_LLM_MODEL. |
FIRST_LLM_MODEL | (empty = skip LLM config) | dqc seed only: model id for the seeded default LLM config. |
FIRST_LLM_API_KEY | (empty) | dqc seed only: provider API key — AES-256-GCM-encrypted under the org DEK before it is stored; never persisted or echoed in plaintext. Omit for Ollama. |
FIRST_LLM_BASE_URL | (empty) | dqc seed only: base URL for ollama / azure_openai / custom providers. |
FIRST_AGENT_TOKEN_NAME | seed-agent | dqc seed only: name of the freshly minted dqa_ agent token (also the re-run idempotency key). |
DQ_AGENT_TOKEN | (empty) | dqa_… token consumed by the optional agent profile. Issue it in the cloud UI; the agent process exits if empty. |
DQ_AGENT_DATASOURCE_DSNS | {} | JSON object mapping datasource UUID → local DSN. Read by the agent profile. |
Verifying audit-SIEM webhook signatures
When AUDIT_SIEM_V2_SIGNING=true (the default), each delivery carries two headers: X-DQ-Cloud-Timestamp (RFC 3339 UTC, Z suffix) and X-DQ-Cloud-Signature: sha256=<hex>. Receivers verify a delivery by:
- Reconstructing the signed payload as the literal string
timestamp + "." + raw_body(concatenate the header value, a single., and the raw request body bytes — do not re-serialise the JSON). - Computing
HMAC-SHA256(secret, signed_payload)and constant-time comparing the lowercase hex against the value aftersha256=. - Rejecting deliveries whose timestamp is more than ~5 minutes from
now(the recommended freshness window) to defeat replay of captured payloads.
When AUDIT_SIEM_V2_SIGNING=false, the X-DQ-Cloud-Timestamp header is omitted and the signature is HMAC-SHA256(secret, raw_body) — the original AUDIT-3 wire format.
Notes
- Every variable is read once at process startup. Restarting after a
.envchange is required. - Production deployments should pull secrets (
SECRET_KEY,ENCRYPTION_KEY,DEFAULT_LLM_API_KEY,SMTP_PASSWORD) from a secret manager rather than baking them into the image. See Deploy to production. - If you add a new env var to
config.py, update.env.exampleand this page in the same commit.