Datasources

A datasource in PLACEHOLDER Cloud is one logical place data lives — a Postgres database, a Snowflake account, an S3 prefix. You connect a datasource once and PLACEHOLDER Cloud reuses that connection to profile assets, run checkpoints, and discover schemas.

Data model

Datasource lives in the per-tenant schema. The columns that affect what you see in the UI:

Column	What it stores
name	A short human label (prod-postgres, analytics-snowflake).
type	The dialect. One of postgres, snowflake, bigquery, redshift, databricks_sql, mssql, trino, clickhouse, hana, oracle, s3_parquet, iceberg, delta.
execution_mode	direct (PLACEHOLDER Cloud opens the connection itself) or agent (an on-prem DQ Agent runs the query).
config_encrypted	Connection parameters, AES-256-GCM-encrypted at rest. Only the worker process and the API decrypt this. NULL for agent-mode datasources.
display_metadata	A small open-ended JSONB blob for non-credential metadata (description, owner email).

When you click Test connection, PLACEHOLDER Cloud decrypts the config, opens a connection, runs SELECT 1 (or the dialect's equivalent), and returns {ok, latency_ms, error} without persisting anything.

Execution modes

• Direct is the default. PLACEHOLDER Cloud's worker process opens an outbound connection to your database. Use this when the worker can reach the database over the network.
• Agent is for datasources behind a firewall. You run the DQ Agent on a machine inside the customer network; the agent registers itself with PLACEHOLDER Cloud using a dqa_… token. The worker hands work to the agent rather than connecting to the database directly. The connection string never leaves the customer network.

Operations you can do

• Create — Datasources → New (UI) or POST /api/v2/.../datasources (API).
• Edit connection — direct-mode sources have an Edit button on the detail page (or PATCH /api/v2/.../datasources/{id} with a config_encrypted.connection_string). The Edit modal shows the same per-dialect structured form as Create — for Snowflake you edit account / user / warehouse / etc., for BigQuery the project / dataset / service-account JSON, and so on; only the raw-DSN dialects (Postgres, MySQL, SQLite) show a single connection-string field. Non-secret fields are pre-populated from the source's non-sensitive metadata; secret fields (password, token, service-account JSON) start blank — leave a secret blank to keep the stored value, or type a new one to rotate it. On save the form re-assembles the connection string the same way Create does and the patch merges server-side, so rotating a password leaves linked assets, rule suites, and checkpoint history intact — no delete-and-recreate. The stored secret is never shown back to the browser. Requires the can_edit_datasource permission (editor and above).
• Delete — direct-mode only, from the detail page or DELETE /api/v2/.../datasources/{id}. A datasource that still has data assets discovered through it is not deletable: the request is blocked with 409 Conflict and a message naming how many assets remain. Remove the data assets first (each asset has its own delete; see Deleting an asset), then delete the datasource — at which point the empty source deletes cleanly and returns 204. This referential-integrity guard (#722) replaces the older silent cascade so a connection delete can't quietly destroy a whole catalog subtree. The UI surfaces the 409 as a "remove its data assets first" toast (errors are never swallowed silently).
• Test connection — green check + latency, or red X + error. Direct-mode only.
• Discover assets — list tables, views, and (for object stores) parquet datasets that this datasource can see. Pick the ones you care about; PLACEHOLDER Cloud creates a DataAsset row for each. Direct-mode only.
• View (agent-mode) — agent-registered sources are read-only: their detail page shows type, mode, and linked assets, but no Edit / Delete / Test / Discover, because the connection and credentials live on-premises. They are still clickable from the list (a View link opens the read-only detail page).

Supported types today

Type	Status
Postgres	shipped (test + discover)
S3 + Parquet	shipped (test only)
Snowflake	shipped (test + discover) — FEATURE-1
BigQuery	shipped (test + discover) — FEATURE-2
Redshift	shipped (test + discover) — FEATURE-3
Databricks SQL	shipped (test + discover) — FEATURE-4
Microsoft SQL Server	shipped (test + discover; SQL auth + Azure AD service principal; see Connect MSSQL; needs unixodbc + msodbcsql18 at deploy time) — FEATURE-5
Trino / Presto	shipped (test + discover) — FEATURE-6
ClickHouse	test + discover + checkpoint; needs the clickhouse worker extra (not in the default image — see callout below) — #399. See Connect ClickHouse / SAP HANA / Oracle.
SAP HANA / Datasphere	test + discover + checkpoint; needs the hana worker extra — #400. See Connect ClickHouse / SAP HANA / Oracle.
Oracle Database	test + discover + checkpoint (covers Oracle ERP / Oracle Agile PLM); needs the oracle worker extra — #401. See Connect ClickHouse / SAP HANA / Oracle.
Apache Iceberg	shipped (test + checkpoint via materialise-to-DuckDB) — issue #88. See lakehouse checks.
Delta Lake	shipped (test + checkpoint via materialise-to-DuckDB) — issue #88. See lakehouse checks.

Status: ClickHouse (#399), SAP HANA / Datasphere (#400), and Oracle (#401) are wired through the platform (selectable type, structured form, connection test, asset discovery, profiling, and validation), but their SQLAlchemy dialect drivers are not bundled in the default worker/API image. Each ships as an opt-in worker extra — clickhouse (clickhouse-sqlalchemy), hana (sqlalchemy-hana + hdbcli), oracle (oracledb, thin mode, no Instant Client). Until your image installs the relevant extra, Test connection and Discover return a "driver not available" message. See Connect ClickHouse / SAP HANA / Oracle for the build step.

Security notes

• The user / role you give PLACEHOLDER Cloud should be least-privilege read-only. PLACEHOLDER Cloud never writes back to your data warehouse.
• Connection strings are AES-256-GCM-encrypted with ENCRYPTION_KEY. Make sure that key is generated correctly (HARDEN-2).
• An SSRF guard rejects datasource URLs that resolve to private / link-local IPs by default; see DATASOURCE_NETWORK_ALLOWLIST in reference/configuration.md.