I have described the three-stage streaming architecture in a few posts now — raw ingest, deserialize, merge. This post is the implementation detail: what it actually looks like in Databricks notebooks and Workflows, with Delta Lake as the intermediate store between stages. The Target Architecture Three notebooks, three Databricks jobs, three
Databricks gives you a genuine choice between PySpark DataFrames and SQL for almost every operation. Both approaches produce identical Spark execution plans — there's no performance difference. The choice is about readability, team skill set, testability, and context. Here's how I decide. They Produce the Same Plan
Last time, we signed up for Postman, logged into the web portal, and issued a simple GET request to the NHL API for Teams info. This time, we’re going to learn to set some Parameters and configure our client to send credentials to the server for authorizing our access to...
This post started as a tweet From @MMarie, suggesting someone blog about getting started with Postman. I guess I’m lucky I spent so much time with devs who were building application programming interfaces (APIs) on top of databases I was building. I got to know the lingo and technology behind...
The data ingest pipeline was reliable until something upstream changed. A source system would update a row, the pipeline would pick it up on the next run, and whatever was downstream would see the change — but only after the pipeline completed and the Delta table got refreshed. That was fine
A client's security team has disabled public network access on every Azure data resource in production. Azure SQL: private endpoint only. Storage account: no public access. Azure Key Vault: private endpoint required. The network is locked down correctly. Now ADF needs to read from all of them. The
The last year has been the Databricks year for me. New client, new stack — Spark instead of pandas, Delta Lake instead of SQL Server, notebooks instead of SSDT. And the first question I had to answer was whether the data quality discipline I'd built around Great Expectations would
The Hive Metastore is essential infrastructure in a Databricks environment. It's also a system that fails in specific, predictable ways when tables get large or partitioning is done wrong. Understanding those failure modes before you hit them is much better than diagnosing them from a production incident. The
We'd already solved the ADF side with a metadata config table and a generic pipeline. Forty-three sources, one pipeline, parameterized at runtime. The problem was what happened next. Once the raw data landed in ADLS, each source needed a Databricks notebook to clean, conform, and write it to
Rubber duck debugging is one of those techniques that sounds ridiculous until you try it. The premise: explain your problem out loud to an inanimate object. A rubber duck on your desk, a stuffed animal, an empty chair. The act of articulating the problem forces you to structure it — and
The cluster is running a groupBy job. 19 tasks finish in 30 seconds. One task runs for 8 minutes. The cluster is 95% idle while one executor processes an outsized partition. This is data skew, and it's one of the most common reasons Spark jobs are slow despite
Both repartition() and coalesce() change the number of partitions in a Spark DataFrame. They are not interchangeable. Picking the wrong one doesn't cause an error — it just makes your job slower or produces unbalanced output files. Here's the distinction. The Difference repartition(n) triggers a full