The Announcement
Microsoft announced in late 2022 that ADF Studio would support authoring and running Apache Airflow DAGs on managed Airflow infrastructure. You write DAGs in the ADF Studio interface. Microsoft runs the Airflow scheduler and workers. No Airflow cluster to provision or operate.
On the surface: a convenient managed service for Airflow users who want to stay in the Azure ecosystem.
Under the surface: the most interesting ADF strategic announcement in years. Let me explain why.
What This Tells You About Microsoft's View of the Market
Airflow has dominated the Python-native data engineering community for the better part of a decade. If you work in a Python-first data engineering shop, Airflow is probably your orchestrator. The DAG model — Python code defining task dependencies — resonates with software engineers in a way that ADF's drag-and-drop pipeline model doesn't always.
Microsoft could have responded to Airflow's popularity in several ways:
- Ignore it (Airflow is for Python shops, ADF is for enterprise teams — different markets)
- Build an Airflow equivalent inside ADF (a Python DAG authoring experience with ADF's managed infrastructure)
- Integrate Airflow directly — let users write real Airflow DAGs and run them on Microsoft-managed infrastructure
They chose option 3. That's the tell.
Choosing to integrate rather than reinvent is an admission: Airflow has won the Python-native orchestration market, and trying to build a competing DAG model inside ADF's pipeline paradigm would be a losing battle. Better to give Airflow users a managed runtime than to fight them with a second-rate alternative.
I respect this decision. It's honest about where ADF's pipeline model competes well and where it doesn't.
What This Means in Practice
For teams with existing Airflow expertise and existing Airflow DAGs: you can now move to managed infrastructure on Azure without abandoning your DAG investment. Your Python code runs on Microsoft-managed Airflow. You get the operational simplicity of a managed service without giving up the orchestration model your team already knows.
For teams evaluating orchestration for the first time: you have a choice inside ADF Studio. ADF native pipelines or Airflow DAGs. Both run on managed infrastructure. Both connect to Azure services. The choice is about your team's skill set and your workflow complexity requirements.
When Airflow Is the Better Choice
There are scenarios where Airflow's DAG model genuinely outperforms ADF's native pipeline model:
Complex task dependencies. Airflow's DAG model handles arbitrary dependency graphs elegantly. In ADF, expressing "run task C only if tasks A and B both succeeded, but run D regardless, and E only if A succeeded and B failed" requires nested If Condition activities that get visually complex fast. In Airflow, it's a few lines of Python defining DAG edges.
Python-native teams. If your data engineering team thinks in Python, writing DAGs is natural. Writing ADF expressions in a browser-based GUI is a context switch. ADF expressions are powerful but they're not Python, and for engineers who live in Python, the cognitive overhead is real.
Existing Airflow infrastructure. If you're already running Airflow and you want to move to managed infrastructure without rewriting your orchestration layer, ADF Managed Airflow is the path of least resistance.
Dynamic task generation. Airflow's dynamic DAG generation — Python code that programmatically generates task graphs at runtime — has no direct equivalent in ADF's pipeline model. ForEach handles iteration, but genuinely dynamic pipeline topology isn't possible in ADF native pipelines.
When ADF Native Pipelines Are Still the Better Choice
This isn't a concession speech for ADF pipelines. There are scenarios where the native model wins:
High-throughput data movement. The Copy Activity with its parallelism controls, fault tolerance, staging support, and schema mapping is purpose-built for moving data at scale. Airflow tasks that copy data are typically wrappers around other tools. ADF Copy Activity is the tool.
Non-Python teams. If your team doesn't have Airflow expertise and doesn't want to maintain Python DAG code, ADF's visual pipeline model is more accessible. The drag-and-drop interface and expression language have a learning curve, but it's not "learn Python + Airflow operator library."
100+ connectors without custom operators. ADF's connector library covers things that Airflow would require custom operators for. The Salesforce connector, the SAP connector, the ServiceNow connector — these are built into ADF Copy Activity. In Airflow, you're writing or finding a community operator.
Azure-SSIS IR. If you need to run SSIS packages, you're in ADF native pipelines. Airflow has no SSIS integration.
Data Flows. The Spark-backed transformation engine inside ADF has no Airflow equivalent. If you need GUI-driven Spark transformation without managing a Spark cluster, Data Flows are the answer.
The Implicit Message
Microsoft is telling you something with this announcement: ADF native pipelines are not the right tool for all orchestration scenarios. That's true. It was always true. The fact that Microsoft is now saying it implicitly, by integrating a competitor rather than expanding the native pipeline model to cover Airflow's use cases, is more honest than the messaging has been.
This is the right call. Orchestration is not a winner-take-all market. ADF native pipelines and Airflow solve overlapping but not identical problems. Giving teams both options, on managed infrastructure, with the same operational model, is a better answer than forcing everything into one paradigm.
I'll be testing this in production. I'll report back.