Apache Airflow
Workflow & automationThe long-standing open-source platform for authoring, scheduling, and monitoring data pipelines as code.
Airflow is the established standard for batch data orchestration. Pipelines are defined as code — directed graphs of tasks — and Airflow schedules them, tracks runs, and handles retries and dependencies.
In AI systems it most often sits at the data layer: the recurring jobs that ingest, clean, embed, and refresh the data a model later draws on.
Where it's ideally used
The default when scheduled, dependency-aware data pipelines feed an AI system and a data team already knows the tool.
Where it doesn't fit
Built for batch schedules, not low-latency or event-driven agent workflows — and operationally heavy for a small project.