The Arc
SSIS. ADF. Fabric. Three names over twenty years. One continuous story about how Microsoft builds and evolves data integration tooling, and about the persistent gap that community ecosystems keep filling because Microsoft keeps leaving the same 20% undone.
I've lived through all of it. I want to write the full story while it's still fresh, before Fabric becomes the new normal and people forget what came before.
SSIS: 2005-2014
SQL Server Integration Services shipped with SQL Server 2005. It was the on-premises ETL workhorse for Microsoft shops for the better part of two decades. The architecture was correct for its era: a Windows service, graphical package authoring in BIDS (later SSDT), rich transformation library, SSIS catalog for deployment and monitoring in SQL Server 2012.
SSIS got to approximately 80% of a complete metadata-driven ETL platform. The remaining 20%: there was no first-party way to programmatically generate SSIS packages from metadata. If you had 200 tables to load, you either built 200 packages manually or you built your own code generation framework.
The community answered with BIML — Business Intelligence Markup Language. BIML is an XML-based language for generating SSIS packages programmatically. You define your transformation logic in BIML templates, provide metadata, and BIML generates the SSIS packages. The BIML ecosystem — Mist, BimlExpress, BimlStudio, the community around varigence.com — grew up entirely because SSIS shipped at 80% and left the metadata-driven gap for someone else to fill.
BIML thrived. Microsoft never closed the gap. The community got a commercial product category out of it.
ADF: 2014-2023
Azure Data Factory launched in 2014 as the cloud-native successor to SSIS's capabilities. The v1 architecture had problems (the slice/availability model), which the v2 redesign (2017-2018) addressed. By 2018, ADF v2 was a genuine enterprise cloud orchestration platform: parameterized pipelines, managed compute, 100+ connectors, git integration, CI/CD via ARM templates.
ADF got to approximately 80-90% of a complete metadata-driven orchestration platform. The remaining 10-20%: same gap as SSIS. No first-party metadata-driven framework. The Lookup → ForEach → parameterized child pipeline pattern works and is clearly the right approach, but Microsoft never shipped documentation, templates, or tooling to make it first-party.
The community answered again: blog series (mine among them), GitHub repositories, conference sessions, consulting frameworks. Every ADF shop that built a metadata-driven framework built their own. Most of them are on GitHub if you look.
ADF thrived. Microsoft eventually absorbed it into Fabric. The community got a professional services category out of it.
Microsoft Fabric: 2023 and Beyond
Fabric launched in preview at Build 2023, GA in November 2023. It's the third generation of the Microsoft unified analytics platform: Power BI + Synapse SQL + Synapse Spark + ADF pipelines + Real-Time Analytics + Data Science in one workspace with one storage layer (OneLake) and one governance model.
Fabric Data Factory gets to approximately... 80-90% of a complete metadata-driven orchestration platform. The same gap. The Lookup → ForEach → parameterized child pipeline pattern works in Fabric Data Factory exactly as it worked in ADF. No first-party metadata framework documentation, templates, or tooling.
The community will answer again. Within 12-24 months, someone will ship the Fabric equivalent of what BIML did for SSIS — a framework layer, probably open-source, possibly commercial, that provides the metadata-driven scaffolding that Microsoft consistently doesn't. The Fabric ecosystem is younger than the ADF ecosystem was in 2022, but the demand is identical and the pattern is proven.
What the Pattern Tells You
Microsoft ships orchestration engines at 80-90% completion and relies on ecosystems to fill the remaining gap. This is not incompetence. It might be deliberate. A first-party metadata-driven framework would reduce the ecosystem of consulting services, open-source frameworks, and commercial tooling that grows up around the gap — and that ecosystem creates stickiness, community, and advocacy that benefits Microsoft more than closing the gap would.
I've stopped expecting Microsoft to close the gap. I've started treating the gap as a feature of the landscape: it exists, it's predictable, and there's value in being the person who can bridge it for clients.
The Skills That Actually Transferred
Across twenty years of SSIS → ADF → Fabric, what actually carried forward?
The pattern did. Metadata-driven ETL — control table, iteration, parameterized templates — exists in every platform. The implementation details change (BIML XML in SSIS, ForEach JSON in ADF, ForEach in Fabric), but the pattern is identical. The engineer who built a metadata-driven SSIS framework in 2008 had an immediate advantage when ADF parameterization shipped in 2017. The engineer who built a metadata-driven ADF framework in 2019 has an immediate advantage with Fabric Data Factory today.
The data modeling fundamentals did. The tables you're loading — facts, dimensions, slowly changing dimensions — don't change because the tool does. SCD Type 2 logic in ADF Data Flows is the same logic as SCD Type 2 in SSIS Slowly Changing Dimension wizard (just executed on Spark instead of SQL Server). The business problem is invariant; the tool is variable.
The operational discipline did. Version control, environment management, parameterized configuration, monitoring, alerting — these practices were necessary in SSIS, necessary in ADF, and necessary in Fabric. The implementation changes; the discipline doesn't.
What I Expect in 2024
Fabric will mature. The preview rough edges will smooth out. The community will build the metadata-driven framework tooling that Microsoft didn't. Microsoft will ship new capability — probably deeper OneLake integration, probably better monitoring, probably something in the real-time analytics space — while continuing to maintain standalone ADF for the shops that haven't migrated yet.
The fundamental question — will Microsoft ever close the metadata-driven gap? — will remain open. My answer: no. The community will do it. The community has always done it. That's how this ecosystem works.
I've been writing about Microsoft data integration tools since 2012. I've watched SSIS peak, ADF mature, and Fabric arrive. The technology improved dramatically over that arc. The gap didn't close. The community filled it. That's the story.
It's been a privilege to be part of it. I expect to be writing about Fabric's community ecosystem in 2024. As always, I'm here to help you navigate whatever comes next.