Use a secret manager with the Privacy Agent

Status: in development (#560). Secret-manager support (#551) ships with the Privacy Agent; see Deploy the Privacy Agent. The Use a secret reference toggle is in the data source form under Connect via → Privacy Agent.

Goal

You'll connect the Privacy Agent to your secret manager, so connection details live in your HashiCorp Vault, AWS Secrets Manager, Azure Key Vault or GCP Secret Manager. There are three ways to use it, and you can combine them:

Write-through. The agent keeps each data source's connection details as one secret in your secret manager, instead of in its encrypted file on disk. A replacement or second replica of the same agent reads the same secrets, so this is also the high-availability setup.
Secret references. A secret field of a connection (a password, a token, a service-account key, a whole connection string) holds a pointer such as vault:secret/dq/warehouse#password instead of the secret. You never type the secret into Measured Cloud; the agent fetches it when it connects.
Declared data sources. You define data sources in a file on the agent's host, usually with references. They appear in Measured Cloud, read-only, and you change them only in the file.

The agent signs in to each secret manager with your platform's own identity: a Vault Kubernetes or AppRole login, an AWS IAM role, an Azure managed identity, or a GCP service account through workload identity. You don't need an extra secret to reach your secrets.

Prereqs

A v2 Privacy Agent, enrolled (Settings → Privacy Agents).
The agent's id, shown in Settings → Privacy Agents and in the agent's log (privacy_agent_key_fingerprint agent_id=…). The examples below call it AGENT_ID.
Permission to create policies or role assignments in your secret manager.
For AWS, Azure or GCP: the agent image with that backend's extra. The published image includes all of them. To build your own: pip install 'dq-cloud-agent[aws]' (or [azure], [gcp], [vault]). An agent configured for a backend whose extra is missing refuses to start and names the extra.

Steps

1. Pick where connection details are stored

Set DQ_AGENT_CREDENTIAL_BACKEND on the agent:

Value	Where the details go
`local` (default)	The agent's encrypted file store on its persistent volume (`DQ_AGENT_STORE_KEY`).
`vault`	HashiCorp Vault, KV version 2.
`aws`	AWS Secrets Manager.
`azure`	Azure Key Vault (`DQ_AGENT_AZURE_VAULT_URL`).
`gcp`	GCP Secret Manager (`DQ_AGENT_GCP_PROJECT`, or the project of the agent's credentials).

Each data source is one secret named <prefix><data source id>. The default prefix is dq-agent/<AGENT_ID>/ for Vault and AWS, and dq-agent-<AGENT_ID>- for Azure and GCP, whose names don't allow /. Set DQ_AGENT_SECRET_PREFIX to change it; {agent_id} in it is replaced by the agent's id. Keep the agent id in the prefix unless every agent has its own vault or account: two agents that share a prefix overwrite each other's secrets.

Secret references work with any backend, including local. You only need the reading permission of step 2 for them.

2. Give the agent least-privilege access

Grant the agent only what it uses. The write-through part covers the agent's own prefix; the reference part covers the secrets you point at.

HashiCorp Vault

Vault policy dq-agent (KV version 2 mounted at secret/):

# Write-through: the agent's own records
path "secret/data/dq-agent/AGENT_ID/*" {
  capabilities = ["create", "update", "read"]
}
path "secret/metadata/dq-agent/AGENT_ID/*" {
  capabilities = ["read", "list", "delete"]
}
path "secret/metadata/dq-agent/AGENT_ID" {
  capabilities = ["list"]
}

# References: only the secrets your data sources point at
path "secret/data/dq/warehouse" {
  capabilities = ["read"]
}

Bind it to the agent's identity. On Kubernetes:

vault auth enable kubernetes   # once per cluster
vault write auth/kubernetes/role/dq-agent \
  bound_service_account_names=dq-agent \
  bound_service_account_namespaces=dq \
  token_policies=dq-agent token_ttl=1h

and on the agent:

DQ_AGENT_VAULT_ADDR=https://vault.example.com:8200
DQ_AGENT_VAULT_AUTH=kubernetes
DQ_AGENT_VAULT_ROLE=dq-agent

Elsewhere, use AppRole:

vault write auth/approle/role/dq-agent token_policies=dq-agent token_ttl=1h
vault read auth/approle/role/dq-agent/role-id
vault write -f auth/approle/role/dq-agent/secret-id

and set DQ_AGENT_VAULT_ROLE_ID and DQ_AGENT_VAULT_SECRET_ID_FILE (a mounted file; DQ_AGENT_VAULT_SECRET_ID also works). A Vault Agent sidecar that writes a token file works too: DQ_AGENT_VAULT_AUTH=token and DQ_AGENT_VAULT_TOKEN_FILE. The agent logs in again before its token's lease runs out, and once more if Vault refuses a token. Vault Enterprise and HCP namespaces: DQ_AGENT_VAULT_NAMESPACE.

AWS Secrets Manager

Run the agent with an IAM role (an ECS task role, EKS IRSA or Pod Identity, or an EC2 instance profile). The agent uses the AWS default credential chain; set DQ_AGENT_AWS_REGION if the environment doesn't. Policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "WriteThrough",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:CreateSecret",
        "secretsmanager:PutSecretValue",
        "secretsmanager:GetSecretValue",
        "secretsmanager:DeleteSecret",
        "secretsmanager:TagResource"
      ],
      "Resource": "arn:aws:secretsmanager:REGION:ACCOUNT_ID:secret:dq-agent/AGENT_ID/*"
    },
    {
      "Sid": "ListOwnRecords",
      "Effect": "Allow",
      "Action": "secretsmanager:ListSecrets",
      "Resource": "*"
    },
    {
      "Sid": "References",
      "Effect": "Allow",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "arn:aws:secretsmanager:REGION:ACCOUNT_ID:secret:prod/warehouse-*"
    }
  ]
}

ListSecrets can't be limited to a prefix by AWS; it returns names, never values, and only the status page uses it. If you set DQ_AGENT_AWS_KMS_KEY_ID (a customer-managed key for new secrets), also allow kms:GenerateDataKey and kms:Decrypt on that key. The agent deletes a record at once (ForceDeleteWithoutRecovery), like the local store.

Azure Key Vault

Give the agent a managed identity (or AKS workload identity) and use a vault with Azure RBAC permissions:

Write-through: assign Key Vault Secrets Officer on a vault dedicated to the agent, and set DQ_AGENT_AZURE_VAULT_URL=https://<vault>.vault.azure.net.
References: assign Key Vault Secrets User on each referenced secret (the role's scope can be a single secret), or on the vault.

az role assignment create --assignee <identity-object-id> \
  --role "Key Vault Secrets User" \
  --scope "$(az keyvault show -n dq-refs --query id -o tsv)/secrets/warehouse-password"

Key Vault keeps deleted secrets recoverable. If a data source is deleted and its secret's name is used again, the agent recovers the deleted secret and overwrites it, which Key Vault Secrets Officer allows.

GCP Secret Manager

Run the agent as a service account (GKE workload identity, or the VM's or Cloud Run's service account). The agent uses Application Default Credentials.

References: grant roles/secretmanager.secretAccessor on each referenced secret.
Write-through: create a custom role with secretmanager.secrets.create, secretmanager.secrets.get, secretmanager.secrets.list, secretmanager.secrets.delete, secretmanager.versions.add, secretmanager.versions.access, secretmanager.versions.list and secretmanager.versions.destroy, and grant it on the project with an IAM condition that limits it to the agent's prefix:

gcloud projects add-iam-policy-binding acme-prod \
  --member=serviceAccount:dq-agent@acme-prod.iam.gserviceaccount.com \
  --role=projects/acme-prod/roles/dqAgentSecrets \
  --condition='expression=resource.name.startsWith("projects/acme-prod/secrets/dq-agent-AGENT_ID-"),title=dq-agent-prefix'

Each write adds a version and destroys the older ones, so a replaced password doesn't linger. The agent labels its secrets managed-by=dq-privacy-agent.

3. Point secret fields at your secrets

A reference is the whole field value, <scheme>:<locator>, optionally followed by #<field> to pick one key out of a JSON secret:

Scheme	Shape	Example
`vault`	`vault:<mount>/<path>#<field>` (KV version 2; `#<field>` is required)	`vault:secret/dq/warehouse#password`
`aws-sm`	`aws-sm:<name or ARN>[#<field>]`	`aws-sm:prod/warehouse#password`
`azure-kv`	`azure-kv:<vault>/<secret>[/<version>][#<field>]`, or `azure-kv:https://<vault>.vault.azure.net/secrets/<secret>[/<version>][#<field>]`	`azure-kv:dq-refs/warehouse-password`
`gcp-sm`	`gcp-sm:projects/<project>/secrets/<secret>[/versions/<version>][#<field>]` (the version defaults to `latest`)	`gcp-sm:projects/acme-prod/secrets/warehouse#password`

Rules:

Without #<field>, the secret's whole value is used. With it, the secret must be a JSON object and the field a string in it.
The first path segment of a Vault reference is the KV mount.
A field value that starts with vault:, aws-sm:, azure-kv: or gcp-sm: is always read as a reference. If it isn't a valid one, the form (and the agent) says so instead of using it as a password.
A password inside a connection string works too: the form puts the reference into the string percent-encoded, and the agent decodes it, fetches the secret and encodes the secret back in. To write such a string by hand, percent-encode the reference (postgresql://reader:vault%3Asecret%2Fdq%2Fwarehouse%23password@db:5432/sales), or make the whole connection string one secret and reference that.

In Measured Cloud, add the data source through the Privacy Agent, and for each secret field select Use a secret reference and enter the reference. The details are still sealed to the agent, as for typed-in secrets, but they hold no secret.

The agent keeps a fetched secret in memory for DQ_AGENT_REFERENCE_CACHE_SECONDS (default 60; 0 turns the cache off), so a rotated secret is picked up within a minute.

4. (Optional) Declare data sources on the agent

For setup entirely on your side, write a file and point DQ_AGENT_DATASOURCES_FILE at it (YAML, or JSON with a .json name):

version: 1
datasources:
  - key: warehouse                # a stable slug; unique in this file
    name: Sales warehouse
    type: postgres
    workspace_id: 7c0e5f3a-…      # needed unless the agent is pinned to a workspace
    config:
      connection_string: "vault:secret/dq/warehouse#dsn"
  - key: lake
    name: Snowflake lake
    type: snowflake
    workspace_id: 7c0e5f3a-…
    config:                       # Snowflake key pair (see "Connect Snowflake with a key pair")
      auth_mode: key_pair
      account: ab12345.us-east-1
      user: MEASURED_SVC
      private_key: "vault:secret/dq/snowflake#private_key"
      database: LAKE
      schema: PUBLIC
      warehouse: COMPUTE_WH
      role: MEASURED_RO

config has the same keys the Measured Cloud form stores. Literal secrets work, but the agent logs a warning for each one: use references, so the file holds no secret.

The agent reads the file when it starts, and reports each entry's key, name, type and workspace (never its config) to Measured Cloud. Each entry becomes a data source bound to the agent and marked Managed in the agent's config file: its name and connection details can't be changed in Measured Cloud. To change one, edit the file and restart the agent. An entry you remove from the file stays in Measured Cloud as an ordinary agent data source, whose details you can then enter in the form. A data source you delete in Measured Cloud isn't brought back by the next restart; restore it from the recycle bin, or give the entry a new key.

An invalid file stops the agent at start, with a message that names the problem.

Verify

Restart the agent and check its log: privacy_agent_credential_store backend=vault (write-through) (or your backend), and privacy_agent_declared_synced … when you declared data sources. An entry Measured Cloud couldn't take (an unknown workspace, the data source limit) is logged as skipped with the reason.
Open the agent's local status page (http://127.0.0.1:8765 on the agent's host by default). The Credentials column says where each connection's secrets come from: local, secret_manager (with the store named), reference or declared.
Click Test connection for a connection that uses a reference. A reference that doesn't resolve fails with the reference and the reason, for example could not resolve secret reference vault:secret/dq/warehouse#password: Vault read of secret/dq/warehouse: permission denied (HTTP 403). The message never contains a secret value.
In your secret manager, the agent's records sit under its prefix, one per data source.

Troubleshooting

Message	Fix
`the AWS Secrets Manager secret-manager backend needs the 'boto3' package: install the agent with the [aws] extra`	Use the published image, or install the extra.
`Vault needs DQ_AGENT_VAULT_ADDR`	Set the Vault address (also needed for `vault:` references with another backend).
`permission denied (HTTP 403 …)`	The policy or role assignment doesn't cover that path or secret. Compare it with step 2.
`the secret has no field 'password'`	The secret is JSON without that key. Fix the `#<field>` or the secret.
`the secret is not a JSON object, so #password can't be read from it`	Drop `#password` to use the whole value.
`this data source is declared in the agent's config file`	Change it in `DQ_AGENT_DATASOURCES_FILE`, not in Measured Cloud.

Goal​

Prereqs​

Steps​

1. Pick where connection details are stored​

2. Give the agent least-privilege access​

HashiCorp Vault​

AWS Secrets Manager​

Azure Key Vault​

GCP Secret Manager​

3. Point secret fields at your secrets​

4. (Optional) Declare data sources on the agent​

Verify​

Troubleshooting​