RAG or Fine-Tuning: Which One Do You Actually Need?

This is the most common architecture question in applied AI, and it usually arrives phrased as a fair contest between two options.

It is not a fair contest. The answer is almost always RAG, and the useful part of this post is the specific test for the minority of cases where it is not.

The two things, briefly

Retrieval-augmented generation means you leave the model alone and fix the prompt. When a question comes in, you search your own documents for relevant material and paste it into the context window alongside the question. The model reads it and answers. There is a fuller walkthrough in RAG in plain English.

Fine-tuning means you take an existing model and continue training it on your own examples, adjusting its weights so it behaves differently. You get a new model file.

The instinct that fine-tuning is the serious option and RAG is a workaround is understandable and backwards. Fine-tuning is training, so it feels like real machine learning. But the thing that makes it feel serious is exactly the thing that makes it the wrong default.

The distinction that decides it

Here is the rule that resolves nearly every case:

Fine-tuning teaches a model how to behave. RAG gives a model something to know.

Almost everybody asking this question wants the model to know something. They have a product manual, a policy set, a support history, a codebase, and they want answers grounded in it. That is a knowledge problem, and fine-tuning is a poor way to install knowledge.

Three reasons, and the first is the one that ends most discussions.

Facts change and weights do not. Your documentation changes weekly. Updating RAG means re-indexing a document, which takes seconds. Updating a fine-tuned model means assembling a new dataset and running another training job. You have converted a content update into an ML pipeline run, permanently.

You cannot cite a weight. RAG hands you the passages it retrieved, so you can show a source, let a user check it, and see why the answer came out that way when it goes wrong. A fine-tuned model produces an answer from nowhere in particular. For anything a person will act on, that difference matters more than accuracy does.

Fine-tuning does not reliably install facts anyway. Training nudges a distribution across many examples. It is good at shaping style, format and behaviour, because those are patterns that recur. A specific fact that appears in one document is not a pattern, and fine-tuning it in tends to produce a model that has absorbed the vibe of your documents and will confidently invent things in their style. That failure mode is worse than not knowing, because it is fluent.

When fine-tuning is actually right

There is a real set of cases. They share a shape: you want a change in behaviour that is consistent across every request, and describing it in the prompt is either impossible or too expensive.

A house output format that prompting keeps drifting from. A specific JSON shape, a rigid document structure, a domain-specific markup. If you have a large body of examples and prompt instructions are followed most but not all of the time, fine-tuning will lock it in.

A voice or register you cannot specify. Style is genuinely hard to describe and easy to demonstrate. If you have thousands of examples of how your organisation writes, training on them beats any prompt describing the style.

A narrow, high-volume, repetitive task where you want a smaller model. This is the strongest economic case. Fine-tune a small model on a task a large model currently handles, and if it matches quality you have cut cost and latency substantially. The saving scales with volume, which is why the case is about volume and not about capability.

A genuinely unusual domain language. Not jargon, which retrieval handles. Cases where the tokenisation and structure are far from ordinary text.

Notice what is absent: none of these are "the model needs to know about our products."

The test

Ask one question.

Would a knowledgeable new hire, handed the right document, get this right?

If yes, it is a knowledge problem. Use RAG. The person did not need training; they needed the document. That is the entire analogy and it holds remarkably well.

If no, because the task requires a way of working that a document cannot convey, and you have hundreds or thousands of examples of it being done correctly, that is a behaviour problem and fine-tuning is on the table.

A second, cruder test that works: if your answer would change when a document changes, you need retrieval.

The option nobody suggests

Most of the time, the real answer is neither, and it is worth saying loudly because it gets skipped in favour of the more interesting choice.

Put it in the prompt. If your domain knowledge fits in the context window, you do not need retrieval infrastructure or a training pipeline. You need a well-written system prompt. Context windows are large enough now that a surprising amount of company-specific material simply fits, and prompt caching makes the repeated cost of a long stable prefix much smaller than people assume. The economics are in what one LLM call actually costs and the mechanics in tokens and the context window.

Then improve retrieval before you consider training. When a RAG system gives bad answers, the cause is retrieval far more often than generation. The model answered correctly from the wrong passages. Chunking strategy, hybrid keyword and vector search, reranking, and metadata filters all typically produce larger gains than fine-tuning would, at a fraction of the effort. And there is a real cautionary result here: in at least one head-to-head, a plain vector index beat an agentic search loop on both cost and quality, which is a reminder that the sophisticated option is not automatically the better one.

And they compose. These are not exclusive. The mature setup is often a fine-tuned small model for format and voice, with RAG supplying the facts. The question is which problem you are solving, not which technology to commit to.

The order to work in

1. Prompt engineering. Cheapest, fastest, and solves more than people expect.
2. Stuff the context if your knowledge fits. No infrastructure.
3. RAG when it does not fit, or changes often, or you need citations.
4. Improve the retrieval when RAG underperforms. This is where the wins are.
5. Fine-tune for behaviour, format, voice, or to shrink a model on a high-volume task.

Most projects should stop between two and four. If you are reaching for step five to solve a problem from step three, you are about to build an ML pipeline to do the job of a search index.

The one line

RAG for what the model should know. Fine-tuning for how the model should act.

Almost every version of this question is about knowledge, which is why the answer is almost always RAG, and why the interesting engineering is in retrieval rather than in training.

Further reading. RAG in plain English, embeddings: how a computer gets meaning, tokens and the context window and what one LLM call actually costs. On evaluating whether any of this worked, what an eval is. Anthropic's contextual retrieval writeup and OpenAI's fine-tuning guide are both useful primary references on where each technique fits.