The Client Question That Forces Honesty A client called me in January to talk about their new data platform. Greenfield. Azure. No legacy debt. The question: should we build on Azure Data Factory or Synapse Pipelines? Both are on the table. Give us a recommendation. I've been running
The Hive Metastore that ships with Databricks has worked fine for years as a catalog. What it has never had is governance: no column-level access control, no cross-workspace data sharing, no lineage tracking, no centralized policy management. Unity Catalog — announced in 2021 and rolling out through 2022 — is Databricks'
Unity Catalog has been in preview since mid-2021. If you've been watching it and wondering when to actually engage with it, the answer is: now is a reasonable time to start experimenting, especially if you have a new workspace or a non-critical environment where you can work without
Delta Live Tables has been in preview for a few months and I've been watching it closely. The concept is appealing: instead of writing notebook code that explicitly manages the read-transform-write lifecycle, you declare what the data should look like at each stage and DLT figures out the
After seven years of running ADF in production, my mental model of what ADF is has settled. This clarity resolves most of the architectural debates I see teams have about ADF versus other tools. Let me put it plainly. ADF is an orchestrator. It coordinates the execution of tasks across
Dimensional modeling works well when your business questions are known in advance and your schema can be designed around them. It falls apart when the questions change, when you need full historical auditability of every change to every record, or when you're building a centralized enterprise data warehouse
Databricks announced Photon at Data + AI Summit last month, and it's now available in Databricks Runtime 9.1. The pitch: a native vectorized query engine written in C++ that replaces the JVM-based Spark SQL execution for analytical queries. Faster scans, faster aggregations, faster joins — without changing any code.
Spark Structured Streaming can read from Kafka indefinitely — theoretically. The practical limit is how fast the downstream pipeline can process incoming messages relative to how fast the upstream producers are sending them. When those rates diverge, you have a backpressure problem: more data arriving than the consumer can handle, queues
GPT-3's context window is 4,096 tokens. That number sounds generous until you try to use the model for real pipeline work — and then it becomes the most important constraint in your workflow. Four thousand tokens is roughly 3,000 words. That's enough for a well-specified
If you've ever trained a model, gotten good results, then lost track of which hyperparameters you used, you understand why MLflow exists. It's an experiment tracking system — a log of every training run with the parameters, metrics, and artifacts that produced each result. On Databricks, it&
The MLflow Model Registry gives you versioning and stage transitions. The next question is serving: how do you turn a registered model into a REST endpoint that a downstream application can actually call? Databricks has a first-party answer for this now, and it's worth understanding both what it
Your Kafka producer is sending orders. Your consumer is writing them to Delta Lake. Everything works in the dev environment where all the producers are your own Python scripts. Then someone onboards a new upstream service, it starts sending messages with a field renamed, a type changed, a required field