A client came to me with a straightforward-sounding problem. They had a Kafka publisher producing over 1,000 topics — one per data source in their manufacturing network — and they needed to ingest all of them into their Databricks environment and make the data available in gold layer Delta tables for
The Announcement Microsoft announced in late 2022 that ADF Studio would support authoring and running Apache Airflow DAGs on managed Airflow infrastructure. You write DAGs in the ADF Studio interface. Microsoft runs the Airflow scheduler and workers. No Airflow cluster to provision or operate. On the surface: a convenient managed
All-purpose Databricks clusters are flexible — attach notebooks, run streaming jobs, train models. They're also sized and managed for those mixed workloads. If your use case is "BI analysts running SQL queries against Delta tables," there's a better-suited compute option: SQL Warehouses. What SQL Warehouses
A data catalog tells you what exists. It doesn't tell you what it means, who owns it, how sensitive it is, whether it's safe to join with another dataset, or what business rules were applied in its transformation. That context — the metadata that makes a table
Let Me Just Say It In 2022, I cannot find a compelling technical reason to choose Azure Data Factory over Synapse Pipelines for a new project, or vice versa. The feature parity is real. The Microsoft differentiation story — different tools for different use cases — is becoming marketing rather than architecture.
The logical extension of "LLM helps me write Spark code" is "LLM writes the pipeline definition from a description." If it can generate a transformation function, can it generate the Airflow DAG that orchestrates the whole thing? I've spent the last few weeks testing
Unity Catalog changed the Databricks data governance story significantly — a single metastore, fine-grained access controls, cross-workspace data sharing, and lineage tracking. It also changed the infrastructure assumptions for Great Expectations integrations, since Unity Catalog tables live in a different place than the Hive Metastore tables that earlier GE Databricks integrations
Unity Catalog's permission model is more expressive than the Hive metastore's — and more complicated. If you're coming from a Hive background where permissions were a simple GRANT SELECT ON TABLE, the UC model requires a mental shift. Here's the complete picture. The
Your Hive metastore has been accumulating tables for years. Hundreds of them, maybe more — bronze ingestion tables, silver conformance tables, gold aggregations, ad-hoc analyst tables, experiment tables that never got cleaned up. Unity Catalog is now on the horizon and you need to move them. The good news: most of
Databricks renamed Jobs to Workflows last year, and the UI redesign that came with it reflects something more fundamental: the platform has a real pipeline orchestration story now, not just a job scheduler. If you're still scheduling individual notebooks as separate jobs and managing dependencies through external orchestrators,
After creating a Unity Catalog metastore, the first practical question is: how do you give it access to your data that already lives in cloud storage? The answer involves two concepts that work together — storage credentials and external locations — and getting them right is the foundation for everything else in
Dimensional modeling gave us a set of conventions that have held up remarkably well for thirty years: fact tables at the center, dimension tables around them, foreign keys enforced by the database, constraints preventing bad data from landing. It works because the database does active work to maintain integrity. Move