RAG vs. CAG: Solving Knowledge Gaps in AI Models

Large language models face a fundamental knowledge problem – they can’t recall information that wasn’t in their training data, whether it’s recent news like Oscar winners or proprietary business data. Two powerful techniques have emerged to address this limitation: Retrieval-Augmented Generation (RAG) and Cache-Augmented Generation (CAG).

Understanding RAG: The Retrieval Approach

RAG operates as a two-phase system designed to fetch relevant knowledge on demand. The process begins with an offline phase where documents are ingested, broken into chunks, and converted into vector embeddings using an embedding model. These embeddings are stored in a vector database, creating a searchable index of knowledge.

When a user submits a query, the online phase activates. A RAG retriever converts the user’s question into a vector using the same embedding model, performs a similarity search of the vector database, and returns the top 3-5 most relevant document chunks. These chunks are then combined with the original query in the LLM’s context window to generate an informed response.

The modular nature of RAG allows teams to swap out vector databases, embedding models, or LLMs without rebuilding the entire system.

CAG: Front-Loading Knowledge

CAG takes a completely different approach by preloading all knowledge into the model’s context window at once. Instead of retrieving information on demand, CAG formats all gathered documents into one massive prompt that fits within the model’s context limits.

The system processes this knowledge blob in a single forward pass, capturing and storing the model’s internal state in what’s called the KV cache (key-value cache). This cache represents the model’s encoded form of all documents, essentially allowing the model to “memorize” the entire knowledge base.

When users submit queries, the system combines the pre-computed KV cache with the question, eliminating the need to reprocess text during generation.

Comparing RAG and CAG

Accuracy

RAG’s accuracy depends heavily on its retriever component – if the retriever fails to fetch relevant documents, the LLM may lack the facts to answer correctly. However, effective retrievers shield LLMs from irrelevant information by providing focused context.

CAG guarantees that relevant information exists somewhere in the knowledge cache, but places the burden on the LLM to extract the right information from a large context. This can lead to confusion or mixing of unrelated information in responses.

Latency

RAG introduces additional retrieval steps that increase response time, including query embedding, index searching, and LLM processing of retrieved text. Each query incurs this overhead, resulting in higher latency.

CAG achieves lower latency once knowledge is cached, requiring only one forward pass of the LLM on the user prompt plus generation, with no retrieval lookup time.

Scalability

RAG can scale to handle massive datasets stored in vector databases, potentially millions of documents, because it only retrieves small relevant portions per query. The LLM never processes all documents simultaneously.

CAG faces hard limits based on model context size, typically 32,000 to 100,000 tokens, accommodating only a few hundred documents at most. Even as context windows grow, RAG maintains scalability advantages.

Data Freshness

RAG supports easy knowledge updates by incrementally adding new document embeddings or removing outdated ones with minimal downtime. The system can always access new information without significant overhead.

CAG requires re-computation whenever data changes, making it less appealing for frequently updated knowledge bases as reloading negates caching benefits.

Real-World Applications

IT Help Desk Bot – CAG Winner

For a system using a 200-page product manual updated only a few times yearly, CAG is optimal. The knowledge base fits within most LLM context windows, information remains relatively static, and caching enables faster query responses than vector database searches.

Legal Research Assistant – RAG Champion

Legal systems requiring searches through thousands of constantly updated cases with accurate citations favor RAG. The massive, dynamic knowledge base would exceed context windows, while RAG’s retrieval mechanism naturally supports precise citations to source materials. Incremental vector database updates ensure access to current information without full cache recomputation.

Clinical Decision Support – Hybrid Approach

Hospital systems supporting doctors with patient records, treatment guides, and drug interactions benefit from combining both techniques. RAG first retrieves relevant subsets from massive knowledge bases, then CAG loads retrieved content into long-context models, creating temporary working memory for specific patient cases. This hybrid approach leverages RAG’s efficient searching with CAG’s comprehensive knowledge access for follow-up questions.

Choosing Your Strategy

Consider RAG when working with large or frequently updated knowledge sources, requiring citations, or operating with limited resources for long context models. Choose CAG for fixed knowledge sets fitting within context windows, prioritizing low latency, or seeking simplified deployment.

Both RAG and CAG represent powerful strategies for enhancing LLMs with external knowledge, each excelling in different scenarios based on scale, update frequency, and performance requirements.