ADF Under the Analytics Umbrella: Where Microsoft Is Going With This

I pay attention to how Microsoft talks about ADF in their marketing materials. Not because I trust marketing, but because product positioning is a leading indicator of roadmap decisions, and roadmap decisions affect the architectures I'm building for clients today. The signal I've been tracking in

Great Expectations vs. tSQLt: Two Frameworks, One Principle

I've been writing about tSQLt since 2012 and Great Expectations since 2017. I've used both across different clients and projects, and I get asked regularly: which one should I use? The answer is almost always: both, depending on where in your stack you are. This post

PySpark GroupBy Aggregations: Every Aggregate You Know From T-SQL, Translated

You've been writing GROUP BY aggregations in T-SQL for years. PySpark has all the same aggregate functions — and a few you probably wished SQL Server had. Here's the translation, plus a few patterns that are unique to the distributed world. The Basic Pattern from pyspark.sql

Adaptive Query Execution in Spark 3.0: What the Optimizer Finally Does Automatically

Spark 3.0 shipped in June, and Databricks Runtime 7.x brought it to production workspaces. The headlining feature — Adaptive Query Execution — has been running in my pipelines for a few months now. Here's what it actually changed versus what I expected. What AQE Changed in Practice The

Checkpoint Files and Idempotency: Getting Streaming Restarts Right

Checkpoint files are the most misunderstood part of a Structured Streaming pipeline. They are not a cache. They are not a nice-to-have. They are the mechanism that makes exactly-once delivery possible. Delete them and you will either duplicate your data, gap your data, or both, depending on where your job

The Shuffle: When Spark Redistributes Data Between Executors and Why It Costs You

If you want to understand why a Spark job is slow, you need to understand the shuffle. The shuffle is the most expensive operation in Spark, and most of the performance tuning work in any non-trivial Spark application comes back to minimizing, eliminating, or managing shuffles. What a Shuffle Is

Lazy Evaluation in Spark: Why Your Code Feels Instant Until the Action Fires

New Spark developers often ask why their code runs so fast, until suddenly it doesn't. They'll chain 10 transformations — filter, select, join, groupBy, withColumn — and each one completes immediately. Then they call .show() and wait 45 seconds. What's happening? Lazy evaluation. None of those

PySpark Window Functions: Translating ROW_NUMBER, RANK, and LAG to Python

If you know SQL Server window functions — ROW_NUMBER, RANK, LEAD, LAG, running totals — you already know the hardest conceptual part of PySpark window functions. The syntax is different but the semantics map directly. Here's the translation. The Import You Need from pyspark.sql import functions as F

Databricks Clusters: Interactive, Job, and How to Not Run Up a Cloud Bill

The first time I got a Databricks invoice that was significantly higher than expected, I went looking for where the compute was going. The answer: an interactive cluster I'd left running overnight while prototyping. Nobody was using it. It was just there, burning DBUs. Here's how

ADF ARM Template CI/CD: The Complete Pattern After Three Years of Refinement

I have been iterating on the ADF CI/CD pattern since 2018 when git integration shipped. I've rebuilt it for five different clients, hit the same problems each time, and landed on a pattern that I now consider stable. Here it is -- the complete picture, including the

What GPT-3 Can (and Can't) Do for a Data Engineer

OpenAI just opened their GPT-3 API to a small beta. I got access last week. The first thing I did was paste a data engineering problem into the Playground — describe a pipeline, ask for the Spark code. The results were simultaneously impressive and wrong in ways that taught me more

PySpark DataFrames for the T-SQL Developer: Your First Week's Vocabulary

The operations you've been running in T-SQL for years exist in PySpark. They just look different. This post is the translation table I wish I'd had during my first week with Databricks — every fundamental T-SQL operation mapped to its PySpark equivalent, with actual examples you can