Get the latest tech news

Apache Airflow 2.10 arrives to usher in a new era of AI data orchestration

The open source Apache Airflow data orchestration technology is adding new features to help enterprises build better data pipelines and understand where data is coming from.

Airflow 2.10 introduces hybrid execution, allowing organizations to optimize resource allocation across diverse workloads, from simple SQL queries to compute-intensive machine learning (ML) tasks. “If you think about why you adopt orchestration from the start, it’s that you want to coordinate things across the entire data supply chain, you want that central pane of visibility, ” Julian LaNeve, CTO of Astronomer, told VentureBeat. Robust lineage information helps address this by providing a clear, auditable trail that shows how engineers sourced, transformed and used the data to train the model.

Get the Android app