Skip to main content

Send lineage with OpenLineage

Goal​

You'll point an OpenLineage client (Airflow's OpenLineage provider, dbt-ol, or any other integration built on the OpenLineage client) at Measured Cloud. Each time a job finishes, Measured Cloud records which of your data assets it read and which it wrote: an upstream-to-downstream link from every input table to every output table.

Nothing new runs in your pipelines. The orchestrator already knows what each task read and wrote; this sends it to Measured Cloud.

Prereqs​

Steps​

1. Build the endpoint​

Every client below takes a base URL and an endpoint path:

  • URL: your Measured Cloud address, with no path: https://dq.example.com.
  • Endpoint: api/v1/organizations/<ORG_ID>/workspaces/<WORKSPACE_ID>/openlineage.
  • API key: the dqsa_… token. The OpenLineage HTTP transport's api_key auth sends it as Authorization: Bearer <token>, which is what Measured Cloud expects.

Keep the path in the endpoint, not the URL. The client joins the two, and a path left on the URL can be dropped.

2a. Airflow​

Install the OpenLineage provider (apache-airflow-providers-openlineage) in your Airflow environment, then set the transport on the scheduler and workers:

export AIRFLOW__OPENLINEAGE__NAMESPACE="airflow-prod"
export AIRFLOW__OPENLINEAGE__TRANSPORT='{
  "type": "http",
  "url": "https://dq.example.com",
  "endpoint": "api/v1/organizations/<ORG_ID>/workspaces/<WORKSPACE_ID>/openlineage",
  "auth": {"type": "api_key", "apiKey": "dqsa_…"}
}'

The same settings go in airflow.cfg under [openlineage] as namespace and transport. Keep the token in your secrets backend rather than in the file.

Operators the provider supports (SQL operators, the warehouse transfer operators and others) report their input and output tables automatically.

2b. dbt (dbt-ol)​

Install openlineage-dbt next to dbt and run your project through its dbt-ol wrapper instead of dbt:

pip install openlineage-dbt

export OPENLINEAGE_URL="https://dq.example.com"
export OPENLINEAGE_ENDPOINT="api/v1/organizations/<ORG_ID>/workspaces/<WORKSPACE_ID>/openlineage"
export OPENLINEAGE_API_KEY="dqsa_…"
export OPENLINEAGE_NAMESPACE="dbt-prod"

dbt-ol build

dbt-ol sends a COMPLETE event for each model, with the model's table as the output and the tables it selects from as the inputs.

If you already ingest dbt artifacts, you have dbt lineage. The two are recorded separately: neither replaces the other's links.

2c. Dagster and other OpenLineage clients​

Dagster, and anything else that emits through the OpenLineage Python client, reads the client's standard configuration. Use the same three environment variables as dbt-ol above, or an openlineage.yml in the working directory (or wherever OPENLINEAGE_CONFIG points):

transport:
  type: http
  url: https://dq.example.com
  endpoint: api/v1/organizations/<ORG_ID>/workspaces/<WORKSPACE_ID>/openlineage
  auth:
    type: api_key
    apiKey: dqsa_…

A client in another language uses its own HTTP transport with the same URL, endpoint and Bearer token.

3. Run a job​

Trigger a DAG, a dbt-ol build or a Dagster run that reads and writes registered tables.

What gets recorded​

  • Only finished runs. A COMPLETE event records lineage. START, RUNNING, FAIL, ABORT and OTHER events are accepted (Measured Cloud answers 200) and change nothing, so a failed run leaves the last good run's lineage in place.
  • One job, one current set of links. A job is its namespace plus its name (airflow-prod / orders_dag.build_summary). Each COMPLETE event replaces that job's earlier OpenLineage links with this run's, so a table the job stopped reading drops out. Links someone added by hand, links from dbt artifact ingest and other jobs' links are never touched.
  • Every input to every output. A run that read orders and customers and wrote order_summary records two links into order_summary.

How a dataset is matched to a data asset​

OpenLineage names a dataset by a namespace (postgres://warehouse:5432, snowflake://acme-analytics, bigquery) and a name (shop.public.orders). Measured Cloud matches it to a data asset in the workspace the endpoint names, or an org-wide one:

  1. The namespace picks the source type. postgres:// and postgresql:// match Postgres sources; mysql://, snowflake://, bigquery, redshift://, databricks://, trino://, oracle://, clickhouse:// and hana:// match their own type; mssql:// and sqlserver:// match SQL Server. Any other namespace (s3://, gs://, kafka://, file://) matches no data asset.
  2. The name splits on dots. The last part is the table and the one before it the schema (a BigQuery dataset, a MySQL database). Anything before those is the database: a Postgres, Snowflake, Redshift or SQL Server database, a BigQuery project, a Databricks or Trino catalog.
  3. The parts are compared, ignoring case, with the data asset's recorded table, schema and database. The table and schema must agree. The database is compared only when both the dataset and the data asset name one.

The host in the namespace isn't compared. Measured Cloud can't read it from your data sources: a direct connection's details are encrypted at rest and a Privacy Agent's never leave your network. Two sources with the same schema and table therefore look alike. See the ambiguous case below.

When a dataset matches:

  • Exactly one data asset: that asset.
  • No data asset, as an input: an external upstream, labelled with the system openlineage and the dataset (postgres://warehouse:5432/shop.public.raw_feed), so you still see where the data came from.
  • No data asset, as an output: nothing is recorded for it.
  • More than one data asset: skipped and reported as ambiguous. Usually the same table is registered on two sources. Recording the database on the assets tells them apart.

A table that is both an input and an output of the same run doesn't get a link to itself.

Verify​

Measured Cloud's response to each event says what it did. OpenLineage clients don't show it, so send one event yourself to check the setup, with your token and IDs:

curl -X POST \
  -H "Authorization: Bearer $DQSA_TOKEN" -H "Content-Type: application/json" \
  "https://dq.example.com/api/v1/organizations/$ORG_ID/workspaces/$WS_ID/openlineage" \
  -d '{
    "eventType": "COMPLETE",
    "eventTime": "2026-10-28T09:30:00Z",
    "producer": "https://example.com/manual-test",
    "schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent",
    "run": {"runId": "0192d6e4-6c1e-7a51-b3a0-5f3c0f1d2e44"},
    "job": {"namespace": "manual-test", "name": "check-setup"},
    "inputs": [{"namespace": "postgres://warehouse:5432", "name": "shop.public.orders"}],
    "outputs": [{"namespace": "postgres://warehouse:5432", "name": "shop.analytics.order_summary"}]
  }'
{
  "data": {
    "events": 1,
    "applied": 1,
    "edges": 1,
    "unmatched": [],
    "ambiguous": [],
    "ignored": []
  }
}
  • applied: COMPLETE events applied. edges: links recorded.
  • unmatched: datasets that matched no data asset, with role input (recorded as an external upstream) or output (not recorded).
  • ambiguous: datasets that matched more than one data asset, with how many (matches).
  • ignored: events that changed nothing, and why: not a COMPLETE event, or not a valid event.

The endpoint also takes a JSON array of up to 500 events. Each is checked on its own, so an invalid one is listed under ignored and the rest still apply. Your test job's links stay until another COMPLETE event for manual-test / check-setup replaces them.

Lineage isn't shown in the app yet. Read a data asset's lineage from the API, which returns its upstream and downstream assets (up to three hops each way) with each link's origin (openlineage, dbt or manual):

curl -H "Authorization: Bearer $DQSA_TOKEN" \
  "https://dq.example.com/api/v1/organizations/$ORG_ID/workspaces/$WS_ID/data-assets/$ASSET_ID/lineage"

Troubleshooting​

  • 401: the token is wrong, revoked or missing. Check the client sends it: Airflow's transport needs "auth": {"type": "api_key", "apiKey": "…"}, and dbt-ol needs OPENLINEAGE_API_KEY.
  • 403: the token can't write here. It needs the Editor role (a Viewer is refused), and a service account pinned to one workspace can only send to that workspace's endpoint.
  • 404: the URL is wrong. Check the endpoint path, and that the base URL has no path of its own.
  • 429: the organization sent more than RATE_LIMIT_OPENLINEAGE_PER_MINUTE events in a minute (default 120). Airflow sends several events per task, so a busy deployment can reach it. Raise the limit on a self-hosted install. The OpenLineage clients log a failed send without failing the job; a dropped COMPLETE event is made good by the job's next one.
  • 413: more than 500 events in one request, or a body over MAX_REQUEST_BODY_MB. Send smaller batches.
  • Datasets in unmatched you expected to match: compare the dataset name with the data asset's table, schema and database. The usual causes: the asset's schema isn't recorded, the asset is in another workspace, or the namespace is one Measured Cloud doesn't map (step 1 of the matching rules).
  • Datasets in ambiguous: the same schema and table are registered on two sources of that type in the workspace. Record the database on those assets, or keep one of them.
  • Nothing recorded, ignored says "only COMPLETE run events change lineage": the job hasn't finished, or it failed. That's expected for START and RUNNING events.

Caveats​

  • Lineage is table-level. Column lineage facets are accepted and ignored.
  • Hosts aren't compared (see the matching rules), so two warehouses with identically named tables in one workspace need the database recorded to tell apart.
  • An Airflow task whose operator doesn't report its tables sends events with no inputs or outputs. Its COMPLETE event records nothing, and clears that task's earlier links.