Lakeflow GA: A Data Engineer's Take on Databricks Going All-In on Pipelines

Shannon Lowder

28 Jul 2025 — 1 min read

Lakeflow hit general availability at DAIS, and the scope of what it covers is broader than the old DLT story. End-to-end data engineering: ingestion from external sources, transformation via Delta Live Tables, orchestration, and monitoring — all under one product surface. The declarative pipelines authoring experience in particular is a significant ergonomic improvement over writing raw DLT Python.

I've been working with DLT since it was in preview, so let me give you the honest practitioner take.

What Actually Improved

The expanded ingestion capabilities are the most immediately useful. Lakeflow can now directly ingest from a wider range of sources — including change data capture feeds and streaming sources — with built-in schema inference and evolution handling. Previously, getting CDC data cleanly into DLT required significant custom code or a third-party connector. The declarative ingestion authoring reduces that.

The monitoring improvements are also real. Pipeline health, data freshness, expectation failures, and lineage are now surfaced in a unified view rather than scattered across job logs and DLT event logs. For teams running multiple pipelines, that observability improvement pays off quickly.

What's Still Missing

The debugging experience for failed DLT pipelines is still not great. When an expectation fails or a schema mismatch happens mid-pipeline, finding the specific records and understanding the root cause requires diving into event logs that are more verbose than informative. The error surfaces have improved but they haven't solved the "why did this quarantined record fail" problem elegantly yet.

Also: Lakeflow GA doesn't mean everything that touches pipelines on Databricks is unified under one UI. There's still overlap between Workflows, DLT, and Lakeflow that creates "which tool do I use for this" confusion for new teams. The direction is right; the execution is still catching up. I'm here to help navigate it.

The Context Problem Neither Agent Mesh Nor OpenSharing Solves

I wrote recently about Azure Agent Mesh and OpenSharing — two infrastructure layers that between them cover how enterprises register, discover, share, and execute agents. Between them, they address a lot of the plumbing that has been missing from the enterprise agent stack. But there's a gap neither of

Unity AI Gateway and What a Governed Model Access Layer Actually Buys You

Unity AI Gateway, announced at DAIS this week, is the feature I've been waiting for since Agent Bricks shipped last year. It's a centralized governance layer for model access in Databricks — you configure which models are approved for use in your environment, who can call them,

You Don't Need Fable. You Need a Router.

The performance gap between open-weight models and closed frontier models has spent the last year collapsing faster than anyone predicted. Epoch AI's tracking puts open weights at roughly a three-to-four-month lag behind state-of-the-art closed models on average. For coding tasks, the gap has effectively closed — DeepSeek V3.2

DAIS 2026: Genie One and the Context Problem Databricks Is Solving

The central message from DAIS this week, delivered by Ali Ghodsi in the opening keynote, was direct: AI doesn't have an intelligence problem, it has a context problem. If your CFO can't get an AI system to explain why margins changed, that's not a