The cron job that emails "success" while the warehouse table is empty has been running for two years. Nobody remembers which server owns it. The new hire finds three overlapping Python scripts that all claim to refresh the same report.
Apache Airflow (~47k GitHub stars, Apache 2.0) is the boring answer that still wins: define pipelines as Python DAGs, let a scheduler run tasks on workers with explicit dependencies, and use a web UI to see what failed at 3 a.m. without SSH archaeology. It is a platform, not a single script. Spotify's Luigi and older Hadoop-era tools solved pieces of the same puzzle; Airflow became the default mental model for "data pipeline as code."
What it is (and what it is not)
Airflow orchestrates work; it usually does not crunch terabytes inside the scheduler process. Tasks should be idempotent, and big data moves happen in Snowflake, Spark, dbt, or whatever operator you plug in. It is not a stream processor, though teams often batch-read from Kafka on a schedule.
Upstream is explicit that Airflow fits slowly changing, repeatable DAG shapes. If every run looks completely different, you will fight the tool. For click-and-connect SaaS automation, look at Activepieces or n8n. Airflow is for engineers who want git-reviewed DAGs, backfills, and provider packages for clouds and databases.
Modern Airflow (3.x stable on Python 3.10+) also shows up in ML and agent pipelines: train, evaluate, deploy steps coordinated as tasks, with growing AI/ML provider coverage in the official registry. The scheduler still runs your DAG; the LLM is just another external system you call from an operator.
What you get self-hosted
- DAGs in Python — version in git, code review, CI tests on parse errors before deploy.
- Scheduler + executors — LocalExecutor for small installs; Celery or Kubernetes executors when you need horizontal workers.
- Web UI — grid and graph views, task logs, backfill date ranges, asset graphs in newer releases.
- Providers — hundreds of hooks/operators (Postgres, S3, Google Cloud, etc.) as separate pip packages with their own semver.
- Metadata database — PostgreSQL (or MySQL) in production; SQLite only for local experiments.
Why self-host?
Job definitions and secrets stay yours. Connection strings, API keys, and task logs sit on infrastructure you control, not a multi-tenant managed Airflow SKU unless you choose that.
Cost at steady volume. A Canadian VPS or small K8s cluster plus Postgres often beats per-DAG managed pricing once nightly ETL and ML retraining are core operations.
Compliance narrative. When auditors ask where orchestration metadata lives, "our Postgres in Beauharnois, workers without inbound SSH" beats a black-box cloud control plane.
Escape hatches. Apache license, massive community, and the option to run the official apache/airflow Docker image or Helm chart without a vendor contract.
Getting started
For local proof of concept, the project documents Docker Compose and constraint-file pip installs so dependency versions match what CI tests. Production paths are typically:
- Docker Compose on a dedicated VM (scheduler, webserver, triggerer, worker, Redis, Postgres).
- Official Helm chart on Kubernetes when you already run a cluster.
pip install 'apache-airflow==3.4.0' \
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-3.4.0/constraints-3.12.txt"Use pinned constraints; bare pip install apache-airflow can land incompatible dependency sets per upstream warnings. Put TLS and auth in front of the web UI; disable the default demo DAGs; store Fernet keys and DB passwords in a secrets manager, not git.
Who should run it
Good fit: analytics teams running dbt or SQL ETL; ML engineers scheduling retraining; platform folks centralizing cron sprawl; anyone who already names DAGs in standups.
Skip it: one-off Zapier replacements; sub-hour real-time pipelines; hosts that cannot run Postgres reliably; teams with zero Python appetite.
Hosting in Canada
Airflow wants always-on scheduler and metadata DB, workers sized to peak task concurrency, and disk for logs. A managed VPS or dedicated box in Beauharnois or Toronto with nightly Postgres backups is a sane minimum before you graduate to K8s.
Talk with Swift Host if you want sizing for Docker Compose Airflow, log retention, and TLS for the web UI on Canadian metal. We are in St. Albert and actually answer the phone.
