MQTT to Kafka to Delta Lake: An IoT Ingestion Architecture That Actually Holds Up

I have written about MQTT, Kafka, and Delta Lake separately. This post is the end-to-end: what the full IoT ingestion architecture looks like when you wire them together, what each hop adds, and why the complexity is justified. The scenario: sensors in a manufacturing facility publish readings via MQTT. The

ADF Git Integration Three Years Later: The Complete Picture

ADF git integration shipped in 2018. I've been using it since the early access period, and I've onboarded more teams to it than I can easily count. Three years of production use, three years of watching teams get confused by the same things, three years of

Databricks Jobs vs. SQL Server Agent: Scheduling Pipelines in the Lakehouse Era

SQL Server Agent has been the scheduling backbone of SQL-based ETL for twenty years. Multi-step jobs, success/failure routing, email alerts, retry logic, job dependencies — it's a full pipeline orchestrator built into the database engine. Moving to Databricks means finding an equivalent. Here's how the concepts

Auto-Generating Terraform from a Metadata Config Table

The metadata-driven pipeline generator was producing ADF pipelines and Databricks notebooks from a config table. The missing piece was infrastructure: every new client data source also needed a storage container, a Databricks cluster configuration, a linked service, and sometimes a Key Vault secret reference. Those were still being created by

Great Expectations Checkpoints: Orchestrating Data Quality Across Your Pipeline

Earlier versions of Great Expectations had a concept called "validation operators" — a way to bundle a validation run with actions like saving results and building Data Docs. The API worked but was awkward to configure and reason about. Checkpoints replace it with something cleaner: a named, reusable configuration

Delta Sharing: The Open Protocol for Cross-Organization Data Exchange

Databricks announced Delta Sharing at Data + AI Summit last month, and I've been running it in a test environment since. If you haven't seen the announcement: Delta Sharing is an open protocol for sharing live data across organizations without copying it. Recipient gets a read endpoint,

Real-Time Risk Scoring: Wiring Multiple ML Models Behind One REST Endpoint

The field engineers had a problem: they were making risk decisions with data from three different ML models, each running independently, each returning a different score, and no single system aggregating those scores into a decision. One model scored credit risk. Another scored operational risk. A third — newer, still being

Auto Loader in Databricks: Incrementally Processing Files From Cloud Storage

The standard approach to incremental file ingestion is messy: track which files you've processed in a table, query the directory listing, compare, load only the new ones, update the tracker. It works, but it's boilerplate that every pipeline ends up reimplementing. Auto Loader is Databricks'

Dead Letter Queues in Kafka: Where Rejected Records Go and How You Get Them Back

Every pipeline that reads from Kafka eventually encounters a message it can't process. Wrong schema, missing required field, value that fails a business rule, upstream system sent malformed JSON at 2am on a Tuesday. What you do with that message determines whether your pipeline is robust or just

ADF in 2021: Mature Platform, Persistent Gaps

Seven years since ADF was in preview. Time for an honest inventory. I've been running ADF in production for the better part of that timeline. The platform has changed substantially. My view of it has changed substantially. Here's where things actually stand in 2021 -- what&

Structured Streaming in Databricks: From Batch to Near-Real-Time Without Rewriting Everything

The appeal of Structured Streaming in Databricks is that it uses the same DataFrame API you already know. The query that runs as a batch job on yesterday's data can run continuously on data as it arrives, with minimal changes. This is the architectural convergence that makes it

Consumer Group Lag: The Metric That Tells You When Your Streaming Pipeline Is in Trouble

Consumer group lag is the number of messages between where your consumer has committed its offset and where the end of the partition currently is. Zero lag means you are caught up — every message that has been published has been consumed. Rising lag means you are falling behind. Lag at