Unity Catalog and Apache Iceberg: What the Format Convergence Means for Your Stack

Shannon Lowder

08 Jul 2025 — 1 min read

Databricks announced full Iceberg support in Unity Catalog at DAIS — managed Iceberg tables, the Iceberg REST Catalog API, read/write from external engines like Spark, Flink, and Kafka. The format wars between Delta Lake and Iceberg have been running for years. This announcement doesn't end them, but it does change what you have to decide when you're building a new data platform.

Previously the choice was architectural: pick Delta, get Unity Catalog governance and Databricks optimization features; pick Iceberg, get broader engine compatibility but give up some of the Databricks-specific tooling. Now the choice is narrower: it's mostly about which format your existing data already lives in.

What Iceberg in Unity Catalog Actually Enables

The Iceberg REST Catalog API means any engine that speaks Iceberg — Flink, Trino, Kafka, dbt — can read and write Unity Catalog-managed tables without going through a Databricks cluster. That's the real interoperability unlock. Your streaming pipeline in Flink can write directly to a UC-governed Iceberg table and the lineage, access control, and audit log all work the same as if a Databricks job wrote it.

For organizations that have invested in Flink for streaming or Trino for interactive queries, this removes the "we'd use Unity Catalog if only it supported our engine" objection.

The Practical Implication for Existing Delta Tables

If your lakehouse is already built on Delta Lake, there's nothing urgent to do. Delta isn't going anywhere and Databricks will keep optimizing it. The Iceberg support is additive — it lets you bring new workloads and new engines into the governance umbrella without requiring a format migration. The decision framework hasn't changed: use the format your workload naturally produces, govern both through Unity Catalog. I'm here to help think through the edge cases.

The Context Problem Neither Agent Mesh Nor OpenSharing Solves

I wrote recently about Azure Agent Mesh and OpenSharing — two infrastructure layers that between them cover how enterprises register, discover, share, and execute agents. Between them, they address a lot of the plumbing that has been missing from the enterprise agent stack. But there's a gap neither of

Unity AI Gateway and What a Governed Model Access Layer Actually Buys You

Unity AI Gateway, announced at DAIS this week, is the feature I've been waiting for since Agent Bricks shipped last year. It's a centralized governance layer for model access in Databricks — you configure which models are approved for use in your environment, who can call them,

You Don't Need Fable. You Need a Router.

The performance gap between open-weight models and closed frontier models has spent the last year collapsing faster than anyone predicted. Epoch AI's tracking puts open weights at roughly a three-to-four-month lag behind state-of-the-art closed models on average. For coding tasks, the gap has effectively closed — DeepSeek V3.2

DAIS 2026: Genie One and the Context Problem Databricks Is Solving

The central message from DAIS this week, delivered by Ali Ghodsi in the opening keynote, was direct: AI doesn't have an intelligence problem, it has a context problem. If your CFO can't get an AI system to explain why margins changed, that's not a