Three Years of Production Data Flows ADF Data Flows went GA in 2019. I've been running production Data Flows for three years. Not toy examples — production pipelines loading dimension tables with SCD logic, conformed zone aggregations, multi-source joins feeding analytical models. The kind of work where Spark startup
Databricks announced Asset Bundles at Data + AI Summit last month. If you've been deploying Databricks workloads by hand — dragging notebooks through the UI, running Databricks CLI commands in CI/CD scripts, or writing custom REST API calls to deploy jobs — Asset Bundles is worth paying attention to. The
I started seriously learning Databricks in 2019 when Microsoft deprioritized Azure Data Lake Analytics. It's now mid-2022. Three years of production use, multiple clients, real production incidents, real performance tuning wins, and a few decisions I'd make differently if I were starting over. Here's
Photon launched on DBR 9.1 last November. I've been running it in production on three different workloads since then — a daily batch aggregation pipeline, a large MERGE-based SCD2 dimension load, and an ad-hoc analytical query layer that analysts run throughout the day. Six months of production data
A Milestone That Stopped Being a Milestone ADF's connector library crossed 100 in 2022. Microsoft announced it. There were blog posts. There was celebration. I didn't write a post about it at the time because, honestly, reaching 100 connectors is not interesting news in 2022. And
One of the friction points when adopting Great Expectations is the blank-page problem: you have a new dataset, you know you should write expectations for it, and you're not sure where to start. What ranges are reasonable for this column? What's the actual null rate? What
Ten years of building data models across different platforms has given me some strong opinions about what works in a lakehouse environment versus what you'll fight against. The lakehouse relaxes a lot of the old constraints — no referential integrity, no NOT NULL enforcement at the engine level, schema-on-read
The metadata-driven pipeline design I've been building toward solves a specific problem: how do you build a pipeline that runs everywhere without rewriting it for each environment? The answer is that the pipeline doesn't know where it's running. The config table knows, and the
Unity Catalog tracks lineage automatically for SQL queries and DataFrames run in notebooks and jobs attached to a UC metastore. You run a query, UC records which tables were read and which were written, and that information shows up in the Catalog Explorer lineage graph. For straightforward table-to-table transformations, this
Unity Catalog's permission model handles table-level and schema-level access cleanly. What it also gives you — that Hive metastore doesn't — is column-level security: the ability to restrict specific columns within a table, and row-level security through dynamic row filters. If you're dealing with PII, financial
Delta Live Tables went GA in February. I've been running it in production for a couple of months now, and the experience is different from the preview. The framework has stabilized, the documentation has caught up to reality, and I have enough production hours on it to tell
The pattern I keep coming back to on client engagements is one that looks the same no matter which cloud, which tooling stack, or which domain. A config table drives everything: what sources exist, how to extract from them, where to land the data, and what transformations to apply. The