ADF has five trigger types in 2021. Most teams use one -- the Schedule Trigger -- for everything. That's fine for simple workloads. It's suboptimal for workloads that would benefit from the other four. Here's the complete map of what each trigger type does
A question I've been asked more than once since moving to the Databricks stack: "We're using Delta Lake — doesn't that already handle data quality?" The answer is: partly. Delta handles one category of data quality problems very well and is silent on
You've trained a model. MLflow captured the run — parameters, metrics, the pickled artifact. It lives in the MLflow tracking server, tagged with the experiment name and a run ID that's forty random characters you'll never remember. Now you need to put it in production,
The first time you connect to a new data source, you're flying blind. You have a table name, maybe a schema diagram if you're lucky, and a contact at the source team who can answer questions "when they get a chance." You need to
Microsoft's ADF documentation covers the building blocks thoroughly: how to create a pipeline, how to parameterize an activity, how to use ForEach. What it doesn't give you is a complete, production-ready metadata-driven framework. The documentation shows you the Lego blocks. It does not show you how
Schema evolution is a persistent source of pain in data engineering. Source systems add columns. Business requirements add metrics. Downstream consumers break when they encounter columns they weren't expecting, or fail to notice that a column they were expecting disappeared. Delta Lake has built-in mechanisms for managing this
After enough metadata-driven pipeline work, you develop strong opinions about what belongs in the config table. Most teams start with the obvious things — source connection info, table names, extract type — and add columns reactively as each new requirement arrives. By the time the table has twenty-five columns and three cross-reference
I ran my first production Mapping Data Flow about two years ago. It was an experiment -- I was watching performance, checking that it handled my edge cases, making sure the Spark execution model didn't do anything surprising at scale. I had a Databricks fallback ready if Data
GitHub Copilot landed in technical preview a couple of months ago. I've had six weeks with it now, and I want to give an honest read rather than a reaction — because the first week reactions I've seen range from "this replaces junior developers" to
After the EDC assessment, the client had a specific problem: the catalog knew about their data assets, but metadata quality was low. Scanner-discovered attributes were incomplete. Business context was missing entirely — no descriptions, no data owners, no sensitivity classifications. The gap wasn't in the tool's capabilities;
PySpark has over 300 built-in functions in pyspark.sql.functions. For everything those don't cover, you have User-Defined Functions. UDFs are the escape hatch when the built-ins don't fit — but they come with real performance costs. Understanding why helps you know when the tradeoff is worth
One of the clients this year runs on Informatica's catalog stack — Enterprise Data Catalog for discovery and lineage, Data Quality for profiling and rules, Axon as the business glossary layer. The expectation when I arrived was that these three tools, integrated together, would provide a complete metadata management