Long Context vs. Retrieval: The Architecture Decision That Still Matters

Shannon Lowder

24 Sep 2025 — 1 min read

With Llama 4 Scout's 10-million-token context and Claude Sonnet 4.5's 1-million-token beta window, a reasonable question is whether RAG is still necessary. If you can fit your entire knowledge base in a context window, why build the chunking, embedding, and retrieval infrastructure?

It's a fair question. The answer is "it depends," which is usually a cop-out but in this case is actually the right frame.

When Long Context Wins

One-shot analysis on a bounded corpus: feed the whole thing, ask the question, get the answer. Debugging a specific pipeline — here's the full codebase, here's the error, what's wrong — is a strong long-context use case. The setup cost is low (no index to build), the retrieval is perfect (nothing is missed), and the corpus is small enough that the context cost is acceptable.

Ad-hoc exploration: when a data engineer needs to understand an unfamiliar codebase or a new schema, dumping the relevant files into context and having a conversation is faster than building a retrieval index first.

When Retrieval Still Wins

High-volume, repeating queries against a large corpus. If you're routing 10,000 support tickets a day to the right team based on a product catalog, sending the entire catalog in every request is cost-prohibitive regardless of context window size. Retrieval amortizes the embedding cost and keeps per-query cost manageable.

Freshness: a retrieval index can be updated incrementally. A context that includes "the whole knowledge base" needs to be reconstructed when the knowledge base changes. For frequently-updated data — a schema registry, a live product catalog, a monitoring dashboard — retrieval stays current more efficiently.

The practical answer: use long context for bounded, infrequent, high-comprehension tasks. Use retrieval for high-volume, recurring, fresh-data needs. The tools aren't competing — they cover different parts of the problem space. I'm here to help design the right approach for your specific use case.

The Context Problem Neither Agent Mesh Nor OpenSharing Solves

I wrote recently about Azure Agent Mesh and OpenSharing — two infrastructure layers that between them cover how enterprises register, discover, share, and execute agents. Between them, they address a lot of the plumbing that has been missing from the enterprise agent stack. But there's a gap neither of

Unity AI Gateway and What a Governed Model Access Layer Actually Buys You

Unity AI Gateway, announced at DAIS this week, is the feature I've been waiting for since Agent Bricks shipped last year. It's a centralized governance layer for model access in Databricks — you configure which models are approved for use in your environment, who can call them,

You Don't Need Fable. You Need a Router.

The performance gap between open-weight models and closed frontier models has spent the last year collapsing faster than anyone predicted. Epoch AI's tracking puts open weights at roughly a three-to-four-month lag behind state-of-the-art closed models on average. For coding tasks, the gap has effectively closed — DeepSeek V3.2

DAIS 2026: Genie One and the Context Problem Databricks Is Solving

The central message from DAIS this week, delivered by Ali Ghodsi in the opening keynote, was direct: AI doesn't have an intelligence problem, it has a context problem. If your CFO can't get an AI system to explain why margins changed, that's not a