Skip to main content
Orchestration controls when and how pipelines run. You move from manual execution to repeatable operations by attaching schedules, file or cloud event triggers, signed webhooks, and dependencies between pipelines.

​
Building blocks

Schedules

Run pipelines on cron expressions with explicit time zones.

Triggers

Start runs when objects land in S3, GCS, Azure Blob, or related event feeds.

Webhooks

Invoke pipelines from external systems over HTTPS with optional signatures.

Pipeline chaining

Model multi-step workflows with trigger nodes and dependencies.

​
Design principles

  • Idempotency: Assume retries. Design loads so a second run does not corrupt facts (merge keys, dedupe stages).
  • SLA clarity: Match schedule cadence to business deadlines; faster is not always better if cost or contention rises.
  • Blast radius: Isolate experimental pipelines on separate triggers so a noisy upstream does not block critical loads.

​
Permissions

Creating or editing orchestration rules requires roles that can execute the target pipeline and read the connections those pipelines use. Service principals used for cloud triggers need least-privilege policies on bucket notifications or queue subscriptions.
Orchestration metadata (who triggered a run, from which webhook or schedule) appears in run history for audit.

​
Next steps

1

Choose a trigger style

Pick schedule vs event vs webhook based on how upstream systems signal readiness.
2

Prototype runs

Execute manual test runs, then attach automation once outputs stabilize.
3

Add monitoring

Wire alerts for failures and duration anomalies.