Skip to main content

Run checks after a Fivetran sync

Goal​

Fivetran tells Measured Cloud when a connector finishes a sync, and Measured Cloud runs the validations for that connector's tables right away. You find a bad load minutes after it lands, not at the next scheduled run.

How it works​

Fivetran sends a sync_end event to a webhook URL in Measured Cloud, signed with a secret that only the two of you know. When the sync succeeded, Measured Cloud runs every validation tagged with the connector:

  • fivetran:<connector id>, the connector's id in Fivetran (a name like mystified_presiding); or
  • fivetran:<connector name>, its name in Fivetran, which is usually its destination schema.

Each tagged validation starts one run, the same as a run you start by hand. Its results and alerts work as usual.

  • A failed sync runs nothing. It may have loaded only part of the data; the next successful sync runs the checks.
  • Fivetran retries a delivery it couldn't make, for up to 24 hours. Measured Cloud recognizes a sync it already handled and doesn't run its checks twice.
  • Runs count toward your plan like any other run.

Prereqs​

  • The super admin or technical admin role in Measured Cloud. Saving the secret asks you to sign in again.
  • A Fivetran API key and secret, with permission to create webhooks. Fivetran creates webhooks through its REST API.
  • Validations for the connector's tables. Every data asset with rules has a default validation that holds all of them.

Steps​

  1. Tag the validations to run. Open each validation, choose Edit, and add the tag fivetran:<connector id> or fivetran:<connector name>, for example fivetran:salesforce. Or use the API:

    curl -X PATCH \
      -H "Authorization: Bearer $MEASUREDCLOUD_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"data": {"tags": ["fivetran:salesforce"]}}' \
      "https://app.measured.cloud/api/v2/organizations/$ORG_ID/workspaces/$WS_ID/validations/$VALIDATION_ID"

  2. Create a secret. Open Settings → Fivetran and click Generate a secret. Copy it, then click Save secret. You can't see it again after you leave the page.

  3. Create the webhook in Fivetran. The settings page shows the command, with your webhook URL and secret filled in:

    curl -X POST https://api.fivetran.com/v1/webhooks/account \
      -u "$FIVETRAN_API_KEY:$FIVETRAN_API_SECRET" \
      -H "Content-Type: application/json" \
      -d '{"url": "<webhook URL>", "events": ["sync_end"], "active": true, "secret": "<secret>"}'

    /v1/webhooks/account covers every connector in your Fivetran account. To cover one destination's connectors only, use /v1/webhooks/group/<group id>.

    Fivetran sends a test delivery when it creates the webhook. Measured Cloud accepts it and runs nothing.

Verify​

  • In Settings → Fivetran, Last delivery shows when Measured Cloud last received a signed event from Fivetran.
  • After the connector's next successful sync, its tagged validations show a new run on their Execution runs card, started by api.

Troubleshooting​

  • Last delivery stays "never": Fivetran isn't reaching Measured Cloud, or it's signing with a different secret. List your webhooks with GET https://api.fivetran.com/v1/webhooks and check the URL. If you generated a new secret, update the webhook with PATCH /v1/webhooks/<webhook id> and the new secret.
  • Deliveries arrive but nothing runs: check the tags. They must match the connector's id or name exactly, after the fivetran: prefix. Also check that the sync succeeded: a failed sync runs nothing.
  • A validation's run doesn't start: it has nothing to run (no enabled rules), or your organization's monthly runs are used up. The other tagged validations still run.
  • To stop: click Turn off in Settings → Fivetran. Fivetran's deliveries are then refused; delete the webhook in Fivetran too.