The client had forty-three source systems. Forty-three databases, flat files, and APIs that all needed to land in their Azure Data Lake before anything downstream could run. Their current approach: build each ADF pipeline by hand. A senior data engineer had spent three weeks on it and was about a
This is the question I get most often in client engagements now. "We're starting a new data platform project. Should we use ADF or Synapse Pipelines?" I've given versions of this answer probably thirty times in the past year. Here's the complete
If you've written ETL in SQL Server, you've written MERGE. It's the standard pattern for incremental loads: insert new rows, update existing rows, optionally delete rows that disappeared from the source. Delta Lake supports the same MERGE semantics, running distributed across your cluster. Here&
A year ago I was a SQL Server and Azure SQL Data Warehouse person who had just watched Microsoft slowly back away from Azure Data Lake Analytics. Now I'm writing PySpark full-time, managing Delta Lake tables, and genuinely excited about data engineering in a way I haven'
Every year I do a rough count of how many ADF pipelines I'm actively maintaining for clients. At the end of 2020, that number is significant enough that I'd feel the loss if ADF disappeared tomorrow. That was not true at the end of 2018. Something
A notebook that you run manually is a prototype. A notebook that runs on a schedule, retries on failure, sends you an email when it breaks, and logs each run's success or failure is a pipeline. In Databricks, Jobs are the mechanism that turns the first thing into
Delta Lake tables accumulate small files fast. Every streaming write that commits a micro-batch produces files. Every append job that runs hourly produces files. Every MERGE that touches 1,000 rows produces files. After a few weeks of steady writes, you have a table with thousands of tiny Parquet files,
SQL Server DBAs learn early to read execution plans. That skill transfers to Spark — the concepts are the same even if the output looks different. This post is the EXPLAIN decoder ring for T-SQL practitioners. Getting the Plan df = spark.read.parquet("abfss://[email protected]/
For two years I've been running Databricks pipelines as separate notebooks orchestrated by dbutils.notebook.run() or scheduled individually via the Jobs UI. It works, but it shows in the monitoring — five separate jobs in the UI, no unified view of the pipeline, no easy way to see
PySpark is the right tool for complex transformations, multi-step pipelines, and anything that benefits from the full DataFrame API. But if your goal is "run a SQL query against a Delta table and see the results," you don't need Python at all. Databricks SQL gives you
Delta Lake's ACID guarantees work well for standard write patterns — appends, overwrites, MERGEs. The concurrency model gets more interesting when you have multiple writers, or when you're running operations like DELETE and UPDATE simultaneously with ongoing streaming appends. Understanding the isolation model tells you what'
Most joins in Spark require a shuffle: both sides get redistributed so that matching rows from each side end up on the same executor. For two large tables, that's unavoidable. But if one side of the join is small enough to fit in memory, there's a