Connect a data source through a Privacy Agent

Status: in development (#560). Connecting, testing, adding tables, reading their columns, profiling them and running validations (including on a partition) through a Privacy Agent work today, as do validations of Iceberg / Delta tables and object-storage files. Profiling and reading the columns of those (#554) come later.

Goal

You'll connect a database that Measured Cloud can't reach, through a Privacy Agent running inside your network. You type the connection details into the same form as for a direct source, but they are encrypted in your browser for the agent: Measured Cloud never sees or stores them. Then you'll test the connection, add the database's tables as data assets and look at their columns, without touching the agent's host.

See Agent mode for how it works and what crosses back to Measured Cloud.

Prereqs

• A Privacy Agent that is online and allowed in your workspace. A super admin or technical admin creates it in Settings → Privacy Agents and installs it with the snippet the dialog shows; see Deploy the agent.
• The agent's store key set (DQ_AGENT_STORE_KEY or DQ_AGENT_STORE_KEY_FILE), and its state directory (/var/lib/dq-agent) on a persistent volume. Without a store key the agent refuses to keep connection details.
• The agent's host can reach the database, and you have a read-only database user for it.
• The editor role (or higher) in the workspace.

Steps

1. Go to Data Assets → Add source.

2. Under Connect via, pick Privacy Agent. The workspace's agents are listed with their status. An agent that can't take connection details right now (offline, for example) is shown but can't be picked, with the reason. The first available agent is picked for you.

3. Check the key fingerprint. It's shown under the picker, with the line "Connection details are encrypted for this agent; MeasuredCloud can't read them." It should match the fingerprint the agent prints in its log at every start (privacy_agent_key_fingerprint … fingerprint=…) and the one in Settings → Privacy Agents.

4. Fill in the connection form exactly as for a direct source: name, type, then the connection fields. Host names and addresses are resolved by the agent, inside your network, so a private address is fine here.

5. Click Add through the agent. Your browser seals the details to the agent's key and sends only the sealed box. While the agent opens it, stores the details and tests the connection, the form says Waiting for {agent}…. A connected agent usually answers within a second or two.

   • The agent stored the details: the window moves on to step 2, Add tables. If the agent's connection test failed (a wrong password, say), step 1 says so, with the agent's error; fix it later with Edit connection.
   • The agent couldn't store them: the form shows the agent's error, for example a missing store key. Fix the cause and click Add through the agent again: it re-sends the details for the same data source.

6. Add tables. The agent lists the database's tables and views with their approximate row counts, exactly as for a direct source. Tick the ones you want and click Add N tables. See Add tables from a source.

7. Look at a table's columns. Open the data asset. Until it has a profile, its Table Health card shows the columns the agent read from the database catalog: each column's name, declared type and whether it allows nulls. The rule editor offers the same columns.

8. Profile it. Click Profile on the asset page, or the table's Profile control in Manage source, and confirm. The agent samples the table and sends back statistics only; the controls read Waiting for {agent}… until the profile lands. See Profile runs.

9. See who can read it (Snowflake, Postgres, Redshift). After the profile, the agent reads the table's grants and sends back the accounts that can read it, never data. They're listed under Who can read this table on the asset page. See See who can read your sensitive data.

Manage the source later

Open the source's Manage source window from Data Assets:

• Test connection runs the agent's connection test and shows the result, with Waiting for {agent}… while it runs.
• Edit connection loads the non-secret fields (host, port, database, user …) from the agent. Each secret reads Unchanged: leave it blank to keep the stored value, or type a new one. Saving seals the details to the agent again.
• Refresh tables asks the agent for the table list again. The list is kept for 15 minutes.
• Delete source moves the source and its data assets to the recycle bin and tells the agent to delete its copy of the details. If you restore the source, enter the details again with Edit connection.

While the agent is offline, Test connection and Edit connection are disabled and the window says why.

Verify

• The source's badge on Data Assets names the agent and its status, for example warehouse-agent · Online.
• Test connection shows a green check with the round-trip time.
• The agent's local status page lists the connection, with its last test. See Use the agent's status page.

Troubleshoot

What you see	What to do
"… couldn't store the connection details: … DQ_AGENT_STORE_KEY …"	The agent has no store key. Set DQ_AGENT_STORE_KEY (or DQ_AGENT_STORE_KEY_FILE) and restart it.
"The Privacy Agent's key has changed since the page loaded it …"	The agent re-enrolled with a new key pair. Reload the page, check the new fingerprint, and enter the details again.
"… holds no connection details for this data source; enter them again …"	The agent lost its store (no persistent volume, or a different store key) or the source was restored from the recycle bin. Use Edit connection and enter every field.
Waiting for {agent}… doesn't finish	The agent isn't picking up work. Check that it's running and online in Settings → Privacy Agents. A pending action gives up after its deadline and shows "didn't pick this up in time".
A validation's asset fails with "agent offline"	The run waited for the agent until its deadline (an hour, plus the run's time limit). Start the agent and run the validation again.
A validation's asset fails with "Blocked by this agent's local policy: custom SQL is disabled"	The agent's policy.yaml sets allow_custom_sql: false and the validation has a custom SQL rule. Remove the rule from the validation, or allow custom SQL on the agent.
The table list says the type isn't supported	An Iceberg or Delta data source is one table, so it has no table list: add the table as a data asset directly. Its columns, profile and validations work through the agent.

See also

• Agent mode: the architecture, what the agent sends back, and the trust model.
• Deploy the agent: install and run the Privacy Agent.
• Datasources: direct and agent data sources.