Skip to main content

Rotate a secret key

Goal​

You'll replace SECRET_KEY (used to sign JWTs) on a running PLACEHOLDER Cloud deployment without invalidating active user sessions.

PLACEHOLDER Cloud signs JWTs with SECRET_KEY and verifies them against SECRET_KEY plus every value listed in ADDITIONAL_JWT_KEYS. Rotation works by adding the old key to the verification set, swapping the primary, and waiting until the refresh window elapses.

Prereqs​

  • Operator-level access to the deployment's environment.
  • A way to deploy a config change (a kubectl rollout, a docker compose restart, a systemctl reload — whatever your stack uses).
  • 30 days of patience — that's the default REFRESH_TOKEN_EXPIRE_DAYS value. Active users need this long to organically refresh their session onto the new key.

Steps​

  1. Generate the new key.

    openssl rand -hex 32

    Copy the output. We'll call it $NEW_KEY.

  2. Append the old SECRET_KEY value to ADDITIONAL_JWT_KEYS. Comma-separated. If the env already has ADDITIONAL_JWT_KEYS=ABCD, set it to ADDITIONAL_JWT_KEYS=ABCD,$OLD_KEY.

  3. Set SECRET_KEY to the new value ($NEW_KEY).

  4. Deploy. Every API process picks up the new env and:

    • Signs new tokens with $NEW_KEY.
    • Verifies incoming tokens against $NEW_KEY + every entry in ADDITIONAL_JWT_KEYS.

  5. Wait REFRESH_TOKEN_EXPIRE_DAYS days (default 30). Every active session will have organically refreshed at least once during this window — refresh issues a fresh JWT signed with the new primary key. Sessions older than the refresh window are forced to log in again on their next request, which is fine.

  6. Remove the old value from ADDITIONAL_JWT_KEYS. This is the irreversible step — any JWT still signed with $OLD_KEY is rejected starting now.

  7. Deploy again.

Verify​

  • curl -H "Authorization: Bearer $NEW_TOKEN" https://placeholder.example.com/api/v1/me returns 200.
  • curl -H "Authorization: Bearer $OLD_TOKEN" … returns 401 after step 7.
  • Logged-in users in the browser don't get bounced to the login screen during step 5 — they refresh transparently.

RS256 key rotation​

Production deployments sign JWTs with RS256 (asymmetric) rather than HS256 (symmetric). The procedure mirrors the HS256 flow above, but you're rotating an RSA key pair instead of a single shared secret. The big win: the private signing key never leaves the signer, so a leaked verifier credential can't be used to mint admin JWTs.

The verification side is a JWKS — a JSON-serialised list of public keys, each tagged with a kid. The kid header on a JWT tells a verifier which entry in the JWKS to use, so verification short-circuits to a single key lookup rather than iterating.

Steps​

  1. Generate the new key pair.

    # Private key (signer only — never share, never commit)
    openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:2048 -out new-private.pem
    # Public PEM (we'll convert this to a JWK in step 2)
    openssl rsa -in new-private.pem -pubout -out new-public.pem

  2. Convert the new public PEM to a JWK and append it to the existing JWT_PUBLIC_KEY_SET. Pick a stable kid (e.g. primary-2026-05) — verifiers cache by kid.

    # Quick conversion via python-jose:
    from jose import jwk
    with open("new-public.pem") as fh:
        pub = fh.read()
    jwk_dict = jwk.construct(pub, algorithm="RS256").to_dict()
    jwk_dict["kid"] = "primary-2026-05"
    jwk_dict["use"] = "sig"
    import json; print(json.dumps([jwk_dict]))

    JWT_PUBLIC_KEY_SET is a JSON list. To rotate, prepend the new JWK and keep the old one at the tail: [<new>, <old>]. Convention: index 0 is the active signer.

  3. Deploy. API processes verify tokens against either public key (by kid) but still sign with the OLD private key (because JWT_PRIVATE_KEY_PEM hasn't changed yet).

  4. Swap JWT_PRIVATE_KEY_PEM to the new private key.

  5. Deploy again. New tokens are signed with the new private key, advertised with the new kid. In-flight tokens signed by the old key still verify because the old JWK is still in JWT_PUBLIC_KEY_SET.

  6. Wait REFRESH_TOKEN_EXPIRE_DAYS days (default 30). All active sessions refresh during this window onto the new key.

  7. Drop the old JWK from JWT_PUBLIC_KEY_SET. Only the new key remains.

  8. Deploy a final time.

What about HS256 tokens during an RS256 cutover?​

During the initial HS256 -> RS256 migration, in-flight HS256 tokens have no kid header. The verifier detects the missing kid and falls through to the HS256 path (SECRET_KEY + ADDITIONAL_JWT_KEYS). Keep the legacy SECRET_KEY populated until the refresh window elapses, then clear it.

/.well-known/jwks.json​

Set JWT_PUBLISH_JWKS=true to expose the public key set to downstream services / SDKs that want to verify your JWTs without holding the signing key. Off by default — the endpoint returns 404 unless explicitly enabled.

Rotating the encryption key (HARDEN-16)​

ENCRYPTION_KEY is now a Key Encryption Key (KEK), not the per-customer key. Each tenant owns a Data Encryption Key (DEK) stored encrypted under the KEK in public.organisations.dek_encrypted. Two independent rotation procedures fall out of that split.

DEK rotation (per tenant, offline-tolerable)​

Use this when one customer's DEK might be compromised (suspicious operator access, an export script that overran its scope, etc.). The --new-dek mode walks every ciphertext for that tenant — datasource credentials, alert destinations, LLM API keys — and re-encrypts under a freshly-generated DEK.

# Drain the API + worker for this tenant first (a maintenance window
# announcement, or pause their checkpoints in the UI). The rotation runs
# in a single transaction per tenant; concurrent writers holding the old
# DEK could otherwise persist a ciphertext the new DEK won't decrypt.

uv run dqc rotate-encryption-key <org-uuid> --new-dek

For an annual key-hygiene sweep across every tenant:

uv run dqc rotate-all-encryption-keys --new-dek

Failures are reported per-tenant; the loop keeps going so one stuck org doesn't block the rest of the fleet. Exits non-zero if any tenant failed.

Without --new-dek both commands run in rewrap mode — they re-wrap the existing DEK under the current ENCRYPTION_KEY instead of minting a new one. That's the recovery path for an env-key rotation; see the next section.

KEK rotation (cheap, no ciphertext walks)​

The KEK lives in KMS for production deploys (ENCRYPTION_KMS_KEY_ID) or in env for self-hosted (ENCRYPTION_KEY). Rotation re-wraps the per-tenant DEKs — not the underlying ciphertexts — so it's bounded by the number of tenants, not the volume of stored secrets.

KMS-backed KEK (the production path):

  1. Create a new KMS key (or rotate the existing one via aws kms enable-key-rotation — annual automatic rotation is supported natively).
  2. Point ENCRYPTION_KMS_KEY_ID at the new key.
  3. Run a one-time re-wrap: walk organisations.dek_encrypted, decrypt each under the old KMS key, re-encrypt under the new one. (This is a short ops script using crypto._kek_decrypt + _kek_encrypt.)
  4. Deploy. Customer ciphertexts are unchanged; only the wrapping changed.

Env-fallback KEK (ENCRYPTION_KEY, self-hosted):

Same shape, but the KEK is the env value and the rewrap is a first-class CLI command. The DEKs are wrapped under the current ENCRYPTION_KEY, so when you change that env var every DEK must be re-wrapped under the new value or the fleet can no longer unwrap them (the symptom is Incorrect padding / a decryption failure on every pre-existing datasource credential and LLM key — see issues #301 / #317).

Controlled cutover:

  1. Generate the new key and keep the old one handy: 
    python -c "import base64,os; print(base64.urlsafe_b64encode(os.urandom(32)).decode())"
  2. Deploy with both keys set — the new value as ENCRYPTION_KEY, the old value as ENCRYPTION_KEY_PREVIOUS. The wrapper tries the current key first and falls back to the previous one on decrypt, so nothing breaks during the transition. ENCRYPTION_KEY_PREVIOUS accepts a comma-separated list if you're catching up on more than one prior rotation.
  3. Rewrap every DEK under the new key: 
    uv run dqc rotate-all-encryption-keys
    (Default mode is rewrap — no --new-dek. Customer ciphertexts are untouched; only organisations.dek_encrypted changes.)
  4. Drop ENCRYPTION_KEY_PREVIOUS and redeploy. Every DEK is now wrapped under the new key alone.

If you rotated ENCRYPTION_KEY without setting ENCRYPTION_KEY_PREVIOUS and the fleet is already failing to decrypt, set ENCRYPTION_KEY_PREVIOUS to the old value, redeploy, then follow steps 3–4. Individual datasources whose credentials can't be recovered (the old key is genuinely lost) surface an actionable "re-enter the credentials" message on the connection test — re-enter them from the datasource detail page to re-encrypt under the current key.

Install extra for KMS​

KMS-backed KEK requires boto3. It ships as an optional extra so self-hosted users without an AWS account don't pull it in:

uv add "dq-cloud-api[kms]"
# or
pip install "dq-cloud-api[kms]"
  • HARDEN-4 — JWT signing-key rotation (the underlying mechanism).
  • HARDEN-13 — RS256 + KMS-backed signing keys (the next-generation story; will obsolete the comma-separated env-var approach).
  • HARDEN-16 — per-tenant DEKs under a KMS-backed KEK (this document's encryption-rotation procedures).