Skip to main content

Connect a ClickHouse, SAP HANA / Datasphere, or Oracle datasource

Status: the connectors are wired end-to-end (selectable type, structured form, connection test, asset discovery, profiling, validation), but their SQLAlchemy dialect drivers are not bundled in the default worker/API image. Each is an opt-in worker extra — clickhouse (#399), hana (#400), oracle (#401). Until your image installs the relevant extra, Test connection and Discover return a "driver not available" message. The build step is in Prereqs below.

Goal​

You'll add a ClickHouse, SAP HANA (or SAP Datasphere), or Oracle Database instance to DQ Cloud, confirm the connection, and discover its tables so you can author rules against them. This is the same UX as the other warehouse connectors — these three differ only in that their drivers ship as opt-in extras.

These three share a code path: unlike the ADBC-backed connectors (Postgres / Snowflake / BigQuery / Redshift / Databricks / Trino), they have no ADBC driver, so DQ Cloud reaches them through a SQLAlchemy engine built from the same registered dialect Great Expectations uses for validation. One driver path, test-through-validate.

Other warehouses on the same connector family: Postgres, Snowflake / BigQuery, Microsoft SQL Server.

Prereqs​

  • A running DQ Cloud (see Self-host).

  • The worker image built with the extra for the dialect you want. The extras are declared in worker/pyproject.toml:

    • ClickHouse — clickhouse extra (clickhouse-sqlalchemy, providing both clickhouse+native:// over TCP 9000 and clickhouse+http:// over HTTP 8123). Pure-Python, no system packages.
    • SAP HANA / Datasphere — hana extra (sqlalchemy-hana over SAP's official hdbcli DB-API driver). SAP ships binary hdbcli wheels; no system HANA client install needed.
    • Oracle — oracle extra (oracledb, the pure-Python successor to cx_Oracle). Runs in thin mode by default, so no Oracle Instant Client install is required.

    Install them when building the worker (and, if you run the in-process API connection test, the API) image, e.g. uv sync --extra clickhouse --extra hana --extra oracle. The published default image omits them so operators who don't run these warehouses don't pay for the drivers.

  • A reachable endpoint for the warehouse you're connecting.

  • A least-privilege read-only user with SELECT on the schema you want to validate.

If your warehouse is on a private network (10.0.0.0/8, 192.168.0.0/16), opt-in by adding the CIDR to DATASOURCE_NETWORK_ALLOWLIST — see Configuration. The SSRF guard rejects private targets by default; these three connectors carry a real routable host in the DSN, so the guard runs for them unmodified (no scheme bypass).

Steps​

  1. Log in to the UI as a workspace editor or higher.

  2. Navigate to Datasources → New.

  3. Pick the type: ClickHouse, SAP HANA / Datasphere, or Oracle Database.

  4. Fill in the form. Each is a structured per-field form (host, port, database/service, user, password); the password is stored encrypted and never logged.

    • ClickHouse — Transport (native, TCP 9000 / http, 8123), Host, Port (defaults to the transport's port), Database, User, Password.
    • SAP HANA / Datasphere — Host, Port (defaults to 443 for HANA Cloud / Datasphere), Database (the databaseName tenant), User, Password. SAP Datasphere exposes a HANA-compatible SQL endpoint, so use this same type.
    • Oracle — Host, Port (defaults to 1521), Service name, User, Password. The service name is sent as service_name to the oracledb thin-mode driver.

  5. Click "Test connection". DQ Cloud opens a SQLAlchemy connection and runs the dialect's version probe — ClickHouse SELECT version(), HANA SELECT VERSION FROM SYS.M_DATABASE, Oracle SELECT BANNER FROM v$version — and reports a green check with the round-trip latency or a red X with the error.

    • Common error: "driver not available" / "Can't load plugin" — the worker/API image doesn't have the dialect extra installed. Rebuild with the extra (see Prereqs).
    • Common error: authentication failed — credentials are wrong, or the user lacks SELECT on the target schema.
    • Common error: "Datasource address rejected by network allowlist" — the target resolved to a private IP and isn't in DATASOURCE_NETWORK_ALLOWLIST. Add the CIDR and restart the API.

  6. Save the datasource.

  7. Click "Discover assets". DQ Cloud lists the user-owned tables and views, filtering each dialect's system schemas:

    • ClickHouse — queries system.tables, filters system / INFORMATION_SCHEMA.
    • SAP HANA — unions SYS.TABLES and SYS.VIEWS, filters SYS / _SYS_* / PUBLIC.
    • Oracle — unions ALL_TABLES and ALL_VIEWS, filters the data-dictionary + bundled-component owners (SYS / SYSTEM / XDB / CTXSYS / …) so you see only your own application schemas (Oracle ERP / Oracle Agile PLM, etc.).

    Tick the ones you want and click Add selected.

Verify​

  • The datasource shows up under Datasources with a green health badge (once the worker image has the extra).
  • Discovered assets appear under the datasource's detail page.
  • You can now author a rule suite against one of those assets — see Write a rule suite without code.

DSN shapes (for SDK / agent registration)​

The UI assembles these client-side from the structured fields; the SDK and agent-registration endpoints emit them directly under config_encrypted.connection_string. For reference:

DialectDSN
ClickHouse (native)clickhouse+native://<user>:<password>@<host>:9000/<database>
ClickHouse (http)clickhouse+http://<user>:<password>@<host>:8123/<database>
SAP HANA / Dataspherehana://<user>:<password>@<host>:443/?databaseName=<DB>
Oracleoracle+oracledb://<user>:<password>@<host>:1521/?service_name=<svc>

Caveats​

  • The PARITY-1 test endpoint times out after 5 seconds by default; bump DATASOURCE_TEST_TIMEOUT_SECONDS if your warehouse takes longer to acknowledge a connection.
  • Discover results are cached in Redis for 15 minutes. If you create a new table and don't see it, click Refresh in the modal (or wait).
  • Connection details (host, port, database, credentials) are AES-256-GCM-encrypted at rest. The plaintext exists only inside the worker / API process during a job.
  • Profiling and validation route through Great Expectations' generic add_sql datasource — GX 1.x has no dedicated add_clickhouse / add_hana / add_oracle helper, but each registered SQLAlchemy dialect drives add_sql directly.
  • Row-count estimates come from each dialect's catalogue statistics (ClickHouse total_rows, HANA RECORD_COUNT, Oracle NUM_ROWS) and can be stale until the next stats gather — the profiler's explicit COUNT(*) follow-up populates exact numbers.
  • Oracle runs in oracledb thin mode (no Oracle Instant Client). Features that require thick mode (some advanced auth methods) are out of scope for this revision.