Why Prompt Caching is Making Traditional RAG Obsolete for Small-to-Medium Codebases

The Retrieval-Augmented Generation Tax

For the last two years, Retrieval-Augmented Generation (RAG) has been the undisputed gold standard for letting large language models access private data. If you wanted an AI assistant to write code against your proprietary library, explain your internal documentation, or debug your platform, the playbook was always the same. You chunked your files, ran them through an embedding model, shoved the vectors into a dedicated database, and wrote a retrieval pipeline to fetch the most "relevant" snippets to feed into the LLM context window.

Let’s be entirely honest: this setup is a massive headache.

We have spent countless developer hours arguing over chunk sizes, overlap strategies, and semantic similarity thresholds. We have watched queries fail because a crucial piece of context was split across an arbitrary chunk boundary. We have paid premium subscription fees for vector databases just to index a codebase that could easily fit onto an old USB thumb drive.

This architectural overhead was a necessary tax. Context windows were small, and feeding hundreds of thousands of tokens into an LLM on every single query was a sure-fire way to bankrupt your project.

But the landscape has shifted. With the advent of native prompt caching in models like OpenAI's GPT-4o, Anthropic's /platforms/claude, and Google's /platforms/gemini, the economic and technical arguments for traditional RAG in small-to-medium codebases have completely collapsed.

Enter Prompt Caching

Prompt caching is a simple but incredibly powerful feature. When you send a massive prompt to an LLM provider, the API detects if a large portion of the prefix—such as a massive system prompt, a set of documentation files, or an entire repository—matches a previously processed request. If it does, the model reuses the compiled key-value (KV) states of those tokens from the server's cache.

Instead of parsing, tokenising, and running expensive attention calculations over your entire codebase on every single keystroke, the LLM provider simply reads the cached state.

This keeps your API bills ticking over at a tiny fraction of the cost. Anthropic, for example, offers up to a 90% discount on cached input tokens. Google's Gemini pricing drops drastically for cached content as well. This changes the entire mathematical equation of LLM application architecture.

Why Chunking is Bad for Code

Traditional RAG works relatively well for flat, unstructured text like legal contracts or customer support articles where information is highly localised. If a user asks about a refund policy, you only need the refund paragraph.

Code, however, does not behave like a legal contract. It is a highly interconnected web of dependencies, imports, inherited classes, and structural paradigms.

When you run a traditional RAG pipeline over a codebase, you are essentially taking a beautiful, cohesive tapestry, cutting it into neat 500-token rectangles with a pair of scissors, and throwing them into a drawer. When the user asks, "How do I implement a new payment gateway using our existing database wrapper?", your vector database pulls out the payment gateway schema and maybe a random database helper file.

What it misses is the global architecture. It misses the implicit design patterns, the global error-handling decorators, the middleware stack, and the helper functions declared three directories over. The model is forced to guess, leading to hallucinations, broken imports, and code that simply does not compile.

By leveraging prompt caching, you don't have to chop up your tapestry. You can feed your entire codebase—every single directory, configuration file, and markdown guide—directly into the system prompt. The LLM retains a perfect, holistic view of your entire architecture. It knows exactly how your modules interact because it can see them all at once.

The Cost-Benefit Breakdown

Let’s look at some real-world numbers to see how this plays out in production.

Suppose you have a medium-sized codebase of about 150,000 tokens (roughly 1.2 MB of raw text, which covers a substantial web application).

Under a Traditional RAG Setup: * You pay for embedding API calls. * You pay for a hosted vector database. * Every query retrieves around 4 chunks of 500 tokens (2,000 tokens total) plus system instructions. The catch*: The LLM's answers are frequently incomplete or hallucinated because it lacks the full context of the repository, requiring constant manual prompt refinement.

Under a Prompt Caching Setup: * You load the entire 150,000-token codebase into the context window once as a cached prefix. * Your base cache write cost is slightly higher for the very first query. * For every subsequent query, you pay the cached rate (a 90% discount on input tokens). The result*: The LLM has zero hallucination on imports, understands the architecture perfectly, and returns working code on the first attempt.

If you are actively developing and sending dozens of queries an hour, the cached approach is not only vastly more accurate, but it is also highly cost-competitive. You are trading expensive developer debugging time for a few pennies of optimized API usage.

Where Traditional RAG Still Makes Sense

We are not suggesting that vector databases are dead across the board. If you are building an AI assistant to query millions of customer emails, a legacy database of millions of academic papers, or an enterprise knowledge base spanning tens of gigabytes, RAG is still your only viable path. Current context windows and caching lifetimes (which usually expire after a few minutes of inactivity) cannot handle scale of that magnitude.

But if your target data fits comfortably within a 200,000 to 1 million token limit, building a complex retrieval pipeline is no longer a smart engineering choice. It is unnecessary over-engineering.

How to Transition Your Stack

If you want to ditch your vector DB and simplify your architecture, the path is straightforward: 1. Bundle your codebase: Write a simple script to traverse your local directory, ignore binary files and node modules, and concatenate your files into a single structured XML or Markdown block. 2. Set up cache markers: Use the appropriate API headers or parameters (such as Anthropic's cache_control block) to tell the provider to cache this massive file dump. 3. Query directly: Send your user queries as user messages following this cached system block.

If you run into issues optimizing your system prompt structure, check out our platform-specific guides for troubleshooting at /platforms/claude/articles and /platforms/gemini/articles.

Stop fighting with semantic similarity scores and arbitrary chunk boundaries. Let the models do what they do best: read the whole manual.