What Databricks Still Hasn't Solved

Shannon Lowder

04 Apr 2025 — 2 min read

It's April 2025, two months before DAIS 2025, and I've been running a list of what Databricks hasn't solved from the 2024 announcements. This isn't a complaint — the platform has gotten better every quarter. But honest critique is how you calibrate what to pay attention to at the next Summit versus what's still vaporware.

The Agent Governance Gap

At DAIS 2024, Databricks announced Mosaic AI Agent Framework as a way to build compound AI systems. The framework is real and useful. What's still missing: first-class governance for agents in Unity Catalog. You can't register an agent as a UC object, define its authorized tool access in UC, audit its invocations through UC, or track its lineage in UC. The agents exist outside the governance layer that everything else in the platform runs through.

This matters because enterprises that are serious about deploying agents in production need to answer compliance questions: what tools does this agent have access to, under what conditions can it invoke them, and what's the audit trail. Right now, answering those questions requires custom instrumentation on top of the framework, which means every team answers them differently. UC-native agent governance would standardize this. I expect it to arrive; it's not here yet.

Cross-Account Lineage

Still missing in any coherent form. If you have a dev workspace and a prod workspace in separate Databricks accounts — which is the right security architecture for most enterprises — the lineage chain breaks at the account boundary. A model trained in dev and promoted to prod doesn't have end-to-end lineage from source data to production serving endpoint. You have two separate lineage chains that you have to mentally join.

This is a real limitation for enterprises doing regulatory AI documentation. "What data trained this production model" currently requires looking at both the prod account and the dev account and assembling the answer manually.

Streaming Governance in Unity Catalog

Unity Catalog governs batch Delta tables comprehensively. Streaming sources — Kafka topics, Event Hubs, streaming Delta tables maintained by Structured Streaming — are second-class citizens in the UC permission model. You can't manage Kafka topic access through UC. You can't track Kafka-to-Delta lineage through UC without custom instrumentation. For enterprises running streaming-heavy architectures, this is a daily friction point.

My Questions for Summit 2025

Three things I'm specifically looking for: a demo of UC-native agent governance that shows an agent registered as a UC object with governed tool access. A cross-account lineage story that's more than "we're working on it." And a streaming governance capability that puts Kafka sources and real-time Delta tables in the same UC governance namespace as batch tables. If those three things are on the keynote, DAIS 2025 will have been worth attending. As always, I'm here to help.

The Context Problem Neither Agent Mesh Nor OpenSharing Solves

I wrote recently about Azure Agent Mesh and OpenSharing — two infrastructure layers that between them cover how enterprises register, discover, share, and execute agents. Between them, they address a lot of the plumbing that has been missing from the enterprise agent stack. But there's a gap neither of

Unity AI Gateway and What a Governed Model Access Layer Actually Buys You

Unity AI Gateway, announced at DAIS this week, is the feature I've been waiting for since Agent Bricks shipped last year. It's a centralized governance layer for model access in Databricks — you configure which models are approved for use in your environment, who can call them,

You Don't Need Fable. You Need a Router.

The performance gap between open-weight models and closed frontier models has spent the last year collapsing faster than anyone predicted. Epoch AI's tracking puts open weights at roughly a three-to-four-month lag behind state-of-the-art closed models on average. For coding tasks, the gap has effectively closed — DeepSeek V3.2

DAIS 2026: Genie One and the Context Problem Databricks Is Solving

The central message from DAIS this week, delivered by Ali Ghodsi in the opening keynote, was direct: AI doesn't have an intelligence problem, it has a context problem. If your CFO can't get an AI system to explain why margins changed, that's not a