Skip to main content

Connect an object-storage datasource (S3 / Azure Blob / GCS)

Goal​

You'll register a raw file dataset — a CSV, Parquet, or JSON object sitting in Amazon S3, Azure Blob (ADLS Gen2), or Google Cloud Storage — as a Measured Cloud datasource, list the objects under a bucket/prefix, and profile + validate them. This is the object-storage connector class from issue #208: a path/prefix + file-format model, complementary to the Delta / Iceberg table-format connectors.

Unlike the SQL connectors, an object-storage source is not a connection string. You give it a bucket (or Azure container) + optional prefix and a file format; the table list shows the matching objects and each object becomes one data asset.

Prereqs​

  • A running Measured Cloud.
  • The worker image built with the [objectstorage] extra. The published dq-cloud-worker image ships it; if you build your own, install with pip install dq-cloud-worker[objectstorage] (pulls in s3fs / gcsfs / adlfs + pyarrow + duckdb). The API image does not need it — the connection-test / table-listing paths import fsspec lazily and fall back to a "run from the worker" message when a provider backend is absent.
  • The worker (or its cloud service account / IAM role) can reach the bucket. Ambient IAM is the preferred production path — grant the worker's role read access and leave the credential fields blank. Static credentials are supported for local dev and non-IAM setups.
  • One object format per datasource. Supported in this increment: CSV, Parquet, JSON (line-delimited JSON / .jsonl / .ndjson also read). Delta and Iceberg tables stay on the Delta / Iceberg connectors.

Steps​

  1. Data Assets → Add source → pick Amazon S3, Azure Blob (ADLS Gen2), or Google Cloud Storage.
  2. Fill in the provider fields:
    • Amazon S3 — Bucket + optional Prefix, File format, optional Region, optional Endpoint URL (for MinIO / S3-compatible stores), and optionally an Access key ID + Secret access key (leave blank to use the worker's ambient IAM role — recommended).
    • Azure Blob (ADLS Gen2) — Storage account + Container + optional Prefix, File format, and either a SAS token or a service principal (Tenant ID / Client ID / Client secret). Leave the secrets blank to use the worker's managed identity.
    • Google Cloud Storage — Bucket + optional Prefix, File format, optional Project, and optionally a Service-account JSON (leave blank to use the worker's ambient service account — recommended).
  3. Test connection — a green check confirms the bucket/prefix is reachable with the supplied (or ambient) credentials. The probe lists the prefix; it does not pull object data. If you run the test from an API build without the provider backend, you'll get a "run the test from the worker" message — that's expected.
  4. Add tables — step 2 of the same window (or the source's Manage source window later). Measured Cloud lists the objects under the bucket/prefix and infers each one's format from its extension (falling back to the datasource's declared format for extension-less objects). Tick the objects you want and click Add N tables; each becomes one data asset whose config carries the object's full path + format.
  5. Add rules to the asset and run them. The worker reads the object (threading your storage credentials into pandas' storage_options), materialises it into an in-process DuckDB table, and runs your rules against it — the same DuckDB strategy the lakehouse connectors use. Profiling reads a bounded sample for column stats and a true row count.

Verify​

  • The source appears on Data Assets as a group header with a green connection badge.
  • The table list shows one row per object under the prefix, each with an inferred format.
  • A profile run completes and shows column stats + a true row count. A run lands in Results exactly like any other data source.

Caveats​

  • Memory / size. The worker reads the whole object to compute the true row count (Parquet / JSON) or streams it (CSV / JSONL). Parquet is bounded by a footer row-count pre-check and CSV / JSONL stream, all under _FILE_READ_MAX_ROWS (5,000,000 rows). A whole-array JSON object has no cheap row count, so it is bounded by a byte pre-check (_JSON_READ_MAX_BYTES, 512 MiB) via fs.info before the read — a giant object refuses to materialise instead of OOMing the worker. Point the asset at a smaller object or a partition (or convert whole-array JSON to line-delimited jsonl, which streams) until the partition-aware path lands.
  • One object = one asset. Glob / Hive-style partitioned multi-file datasets are a follow-up; for now register the individual objects you want to check.
  • IAM preferred. Grant the worker's cloud role read access and leave the credential fields blank. Static keys / SAS / service-account JSON are accepted (useful for local dev and non-IAM stores) but ambient credentials are the production recommendation.
  • Encryption at rest. Static credentials sit in the same config_encrypted JSONB column every other datasource uses — AES-256-GCM encrypted under the org DEK. They are never returned in an API response, never written to display_metadata, and never logged; the list / detail panes show only bucket / container / prefix / format / region / account / endpoint / project.
  • SSRF. The s3:// / gs:// / abfs:// schemes resolve their own endpoints via the cloud SDK. A custom Endpoint URL (MinIO / S3-compatible) is a user-controlled http(s) host, so it is run through the same SSRF guard the DSN connectors use — an internal / metadata address is refused. The endpoint is validated at save time (on create and on update), not just when you click Test connection, so a blocked endpoint can never reach the worker even if the test is skipped. The object path each profile / validation run reads is guarded the same way on the worker before the object is opened.
  • Agent mode. Object-storage listing / profiling through the on-prem agent is not wired yet (agent-mode profiling is unimplemented platform-wide). Use direct mode; agent-mode support is a follow-up.

See also​