ADF git integration shipped in 2018. I've been using it since the early access period, and I've onboarded more teams to it than I can easily count. Three years of production use, three years of watching teams get confused by the same things, three years of finding the right explanations. Here's the definitive version.
The Model That Confuses Everyone
ADF git integration has two branches that serve completely different purposes, and conflating them is the single most common mistake I see teams make.
The collaboration branch (I use main) is where developers work. ADF resource definitions -- pipelines, datasets, linked services, triggers, data flows -- are stored here as JSON files. When you're in ADF Studio and you click Save, your changes go to the collaboration branch (or your feature branch, if you're working on a feature branch). This branch is what you manage with normal git workflows: feature branches, PRs, code review, merge.
The adf_publish branch is entirely different. It contains ARM templates generated by ADF's Publish operation. This branch is not for human editing. It does not mirror the collaboration branch. It is an artifact branch -- ADF generates it, your CI/CD pipeline consumes it.
When you click Publish in ADF Studio, ADF reads the current state of the collaboration branch, compiles it into ARM templates, and writes those templates to adf_publish. Your CI/CD pipeline watches adf_publish for changes and deploys when they appear. That's the whole model.
The Failure Mode When Teams Don't Understand This
Someone edits files directly in adf_publish. They push the changes. The CI/CD pipeline deploys them. Then a developer clicks Publish from ADF Studio, and the next deployment overwrites their manual edit. The change is gone.
Or: a developer switches branches in ADF Studio without clicking Save first. Their uncommitted changes are lost -- ADF doesn't warn you before branch switching that you have unsaved work.
Or: the team thinks merging to main triggers a deployment. It doesn't. Only clicking Publish updates adf_publish. Merging to main with no Publish step means the deployed factory is out of sync with the collaboration branch. Teams that don't understand this have deployed stale ARM templates and wondered why their pipeline changes weren't live.
The Publish Step Problem
The Publish step is still manual in 2021. This is the part I can't fully work around.
When a developer merges a PR, the collaboration branch is updated. adf_publish is not. Someone has to go to ADF Studio, confirm they're on the collaboration branch (check the branch indicator in the top bar), and click Publish. There's no webhook, no API trigger, no native way to automate this.
The community workaround is the @microsoft/azure-data-factory-utilities npm package. It exposes a publish command that you can call from an ADO pipeline triggered on merges to main:
- task: NodeTool@0
inputs:
versionSpec: 14.x
- script: npm install -g @microsoft/azure-data-factory-utilities
displayName: Install ADF utilities
- script: |
npx azure-data-factory-utilities publish \
--subscriptionId $(subscriptionId) \
--resourceGroup $(devResourceGroup) \
--dataFactoryName $(devAdfName)
displayName: Publish ADF ARM templates
This works. I've used it. It's also a dependency on a community-adjacent package that Microsoft hasn't committed to as a first-party CI/CD path. Use it with that context in mind.
The Branch Switching Gotcha
If you switch branches in ADF Studio without saving first, your unsaved changes are gone. ADF Studio does not prompt "you have unsaved changes, are you sure?" before switching. It just switches.
The muscle memory fix: before switching branches, always click Save (Ctrl+S on Windows). Saved changes go to the current branch as a commit. Unsaved changes live only in the browser session.
I add this to every team onboarding document I write for ADF projects. It's the gotcha that costs people the most work.
Tools That Help
The VS Code extension for ADF (Azure Data Factory) allows you to edit ADF JSON files locally. For developers who prefer working outside the browser IDE, this gives you a local editing experience with standard git tooling.
The microsoft/azure-data-factory-utilities package for the automated Publish step, as described above.
The Verdict After Three Years
Git integration in 2021 is solid. The collaboration branch model works well for team development. PR workflows, code review, feature branching -- all of this works exactly as you'd expect from any git-backed development workflow.
The manual Publish step is the one genuine rough edge that remains. It's a known gap, it has a community workaround, and it should have been closed by Microsoft years ago. Until they close it, the community workaround is reliable enough for production use.
If you're setting up git integration for an ADF project and want to get the onboarding document right before you start confusing your team, I'm here to help.