RAG vs Long Context — Which Is Cheaper and Better?

The core trade-off

Long context: simpler to build, no retrieval errors, but costs O(documents) per query. RAG: more complex, can have retrieval failures, but costs O(retrieved_chunks) per query — typically 5–20× cheaper for large corpora.

When long context wins

Small, static document set (< 50K tokens total). Cross-document analysis where every document is relevant. "Needle in a haystack" queries where you don't know which document has the answer. Gemini's 1M context at $1.25/M input makes long-context economical for moderate corpus sizes.

When RAG wins

Large corpus (> 100K tokens of source material). Frequently updated documents (re-embedding is cheaper than full-context re-processing). Known query types where relevant chunks are predictable. Cost-sensitive high-volume use cases where per-query savings compound.

Cost comparison example

100-document knowledge base, 2,000 tokens/doc = 200K tokens total. 10K queries/month. Long context: 200K × $1.25/M × 10K = $2,500/month. RAG (5 chunks, 1K tokens each): 5K × $1.25/M × 10K = $62.50/month + embedding costs (~$2/month). RAG is 40× cheaper at this scale.