ADF CI/CD in 2022: The Automated Publish Finally Works Properly

Shannon Lowder

Shannon Lowder

— 3 min read

The Button That Haunted Every ADF Deployment

If you've been running ADF with git integration, you know the button. The Publish button in the ADF Studio UI. The one that generates the ARM templates in the adf_publish branch. The one that every CI/CD pipeline in every ADF shop eventually had to work around, automate, script, or just live with a human clicking it before every production deployment.

In 2022, that button has a proper automated solution. Let me show you what the current state of the art looks like.

Why the Manual Publish Was a Problem

The ADF git integration model: you author pipelines in a feature branch, merge to main via PR, then someone clicks Publish in the ADF Studio to generate the ARM templates in the adf_publish branch. Those ARM templates are then deployed to test and production environments via Azure DevOps.

The human-clicks-Publish step broke continuous delivery. You couldn't have a fully automated pipeline from PR merge to production deployment because the ADF Publish step required a human to open a browser, log into ADF Studio, and click a button. In some shops, this button was the gatekeeper for every release, clicked by whoever "knew about ADF."

Workarounds over the years included: writing PowerShell scripts that called the ADF REST API to trigger publish, running the publish step from CI on a timer, and (I'm not proud of this) having a developer keep the ADF Studio tab open in their browser and clicking the button as part of the release checklist.

None of these were acceptable.

The 2022 Solution: microsoft/azure-data-factory-utilities

The microsoft/azure-data-factory-utilities npm package provides a validateAndGenerateArmTemplate function that can be called from an Azure DevOps pipeline step without anyone clicking anything. It reads the ADF JSON from the repository, validates it, and generates the ARM templates — the same templates that the Publish button generates, via the same logic, without requiring a browser session.

Here's the relevant ADO pipeline step:

- task: Npm@1
  inputs:
    command: 'install'
    workingDir: '$(Build.Repository.LocalPath)/adf/'

- task: Npm@1
  inputs:
    command: 'custom'
    workingDir: '$(Build.Repository.LocalPath)/adf/'
    customCommand: 'run build export $(Build.Repository.LocalPath)/adf/ /subscriptions/$(subscriptionId)/resourceGroups/$(resourceGroupName)/providers/Microsoft.DataFactory/factories/$(factoryName) "ArmTemplate"'

This runs in CI. No button. No human. Merge to main, ARM templates generate, pipeline continues to deployment.

The Complete CI/CD Pipeline in 2022

Here's the full picture of what a mature ADF CI/CD pipeline looks like today:

Source Control Structure

The ADF git integration connects to a repository with the ADF JSON source in a dedicated folder (typically /adf/). The adf_publish branch is treated as a build artifact — it receives generated ARM templates and is not part of normal code review or history review. I configure my git clients to hide this branch from the default branch list.

PR and Merge

Developers author changes in feature branches. PRs require review. Merge to main triggers the CI pipeline.

CI Pipeline: Validate and Generate

The npm validate-and-generate step runs on merge. Validation catches syntax errors, broken references, and missing parameter definitions. ARM template generation produces the deployment artifacts. If validation fails, the pipeline fails — no ARM templates, no deployment. This catches ADF authoring mistakes before they reach production.

CD Pipeline: Multi-Stage Deployment

Three stages: dev, test, prod. Each stage:

  1. Disable triggers in the target factory (prevents partial-state executions during deployment)
  2. Deploy ARM template with environment-specific parameter file (connection strings, storage account names, integration runtime names, global parameters vary per environment)
  3. Enable triggers in the target factory

The trigger disable/enable step is the one most shops forget. Deploying ARM templates while triggers are running can cause pipeline runs that execute against a partially-deployed factory. Disable first, deploy, enable after.

What's Still Manual

The automated publish solves the button problem. It doesn't solve everything.

Global parameter changes require manual environment parameter file updates. When you add a new global parameter to the factory, the environment parameter files for test and production need to be updated with the environment-specific value before the next publish. This is not automated — it requires a human to recognize that a new global parameter was added and update the corresponding files. Catch this in PR review or in the CI pipeline's parameter file validation step.

ARM template size limits for very large factories. Azure ARM template deployments have a 4MB template size limit. A factory with 200+ resources can approach this limit. The workaround: split the ARM template deployment into multiple templates using the linked template pattern, or use incremental deployment mode carefully. This is a ceiling that most shops don't hit, but if you're approaching it, plan ahead.

The adf_publish branch is verbose. A large factory generates thousands of lines of ARM templates. Code review of the adf_publish branch is not useful — treat it as machine-generated output and don't try to read it. The readable source is the ADF JSON in the main branch. Review that.

The Net Result

In 2022, ADF CI/CD is finally a solved problem for most shops. PR → merge → automated validate/generate → multi-stage deployment to dev/test/prod, fully automated, no human in the loop. It took until 2022 to get here cleanly, but here we are.

If you're still clicking the Publish button in 2022, I'm here to help. There's a better way.

Read more