Self-Hosted Kestra: YAML Orchestration Between Airflow and Cron

Airflow is the default answer in data shops until someone who is not a Python DAG author needs to change a schedule. Then you either wait on a PR or you grow a second tool for "the other" automations. Kestra's pitch is one control plane: YAML flows for data, infra, business processes, and (as of 2.0) LLM/agent steps, editable from a UI that still writes real files you can git.

Kestra (kestra-io/kestra, ~30k GitHub stars, Apache-2.0, Java) is an open-source orchestrator. Flows are YAML. Triggers can be cron or events (files, Kafka, SQS, webhooks, and a long plugin list). Tasks run Python, Node, Go, shell, SQL, HTTP, Docker, Kubernetes, cloud APIs, or custom plugins. Docs and the hosted story live at kestra.io; the software itself you can run in Docker on your own metal.

What you get in the open-source core

• Flows as YAML — id, namespace, tasks, inputs, outputs, retries, timeouts, error handlers, parallel/sequential steps, subflows, backfills. The UI, no-code editor, and API all edit the same definition.
• Triggers — schedules plus event-driven starts; 2.0 renamed trigger filters to when and added date helpers.
• Plugins — hundreds of connectors; the latest-slim image can pull plugins from Maven Central the first time a flow needs them (KESTRA_PLUGINS_AUTO_INSTALL_ENABLED).
• Git-friendly — push from the UI, or keep flows in CI/Terraform. Draft revisions let you iterate without breaking the revision triggers currently run.
• Scale story — 2.0 moved worker talk to gRPC so workers can sit in another network; task outputs go to storage instead of bloating the execution row.
• AI extras (optional) — Copilot in the editor, agent/RAG task types, MCP server so coding agents can call your flows. You do not have to turn any of that on.

Enterprise Edition adds RBAC, policies, worker groups 2.0, and similar governance. OSS is enough to run real pipelines; do not assume SSO and case management are in the Apache image.

How it compares to tools we already covered

Airflow is Python DAGs, a scheduler fleet, and a metadata DB you will babysit. Kestra is YAML-first with a much more opinionated UI. Dagu is the opposite extreme: one binary, file-backed state, no required database. GoCron and Cronicle are cron-plus-UI, not a plugin marketplace. Pick Kestra when you want Airflow-shaped orchestration without writing operators in Python, and you accept a Java + Docker (or k8s) footprint.

Why self-host?

Flows touch warehouse credentials, production SSH, and sometimes customer extracts. Running Kestra on a Canadian VPS keeps execution history, logs, and secrets in infrastructure you named in the contract. Apache-2.0 is straightforward for internal use. The commercial cloud is there if you want someone else to run the control plane; self-host if the data plane cannot leave the building.

Cost of doing it yourself: Postgres (or the local H2-style store only for experiments), object storage for outputs at any real volume, and workers that can reach Docker or the cloud you actually run jobs on. The quick-start command mounts the Docker socket as root. That is convenient and also full host container control. Do not expose 8080 naked on the internet.

Quick start

docker run --pull=always --rm -it -p 8080:8080 --user=root \
  --name kestra \
  -v kestra_data:/app/storage \
  -v kestra_db:/app/data \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /tmp:/tmp \
  -e KESTRA_PLUGINS_AUTO_INSTALL_ENABLED=true \
  kestra/kestra:latest-slim server local

UI: http://localhost:8080. First flow:

id: hello_world
namespace: dev
tasks:
  - id: say_hello
    type: io.kestra.plugin.core.log.Log
    message: "Hello from Kestra"

Production: follow upstream Compose/Helm, put TLS and auth in front, pin an image tag instead of latest-slim, and read the 2.0 migration notes if you are coming from 1.x (Loop replaced ForEach, among other breaks).

Who it is for

Good fit: data teams tired of Airflow UI for simple YAML jobs; platform folks mixing dbt, Spark, and shell; companies that want Git + a topology view the business can click; anyone wiring LLM calls as tasks next to boring ETL.

Skip it: if crontab or GoCron is enough; if you already run Airflow well and live in Python operators; if you need Dagu's no-database edge box; if "orchestration" for you is Huginn watching RSS.

Hosting in Canada

Kestra wants more RAM than a static site. Size a CloudBox VPS for the JVM plus whatever the workers spawn. Put Postgres and storage on volumes you snapshot. Keep workers near the data (Beauharnois for warehouse jobs, not a laptop VPN). Canadian residency of the control plane does not magically keep AWS task runners in-country; say explicitly where each task type executes.

Bottom line

Kestra is the YAML orchestrator with a grown-up UI and a plugin shelf, sitting between Dagu's simplicity and Airflow's Python platform. Self-host it when the workflows are yours and the metadata should stay on your disks. Start local with the slim image, then move to Compose before you trust it with production backfills.

Talk with Swift Host if you want help sizing a Canadian Kestra box: TLS, Postgres, storage, and an honest look at whether you need workers on the same host. We are in St. Albert and actually answer the phone.