Simple Machine Learning
A field journal · v.01
← Back
FiledSeptember 16, 2026·4 min read·By The Editor

Fine-Tuning vs RAG

Fine-tuning changes the model's weights ahead of time; RAG changes the prompt at question time. Knowledge belongs in retrieval, behaviour belongs in training — and most teams reach for the expensive one first.

llmpractical

Both are answers to the same complaint: the model does not know about my stuff. They solve it in completely different places.

Fine-tuning changes the model. RAG changes the prompt. Almost every practical difference follows from that one sentence.

The two shapes

Figure 1 · two different moments
Fine-tuning compared with retrievalTop lane: fine-tuning changes the model's weights ahead of time, and at question time the model answers from memory. Bottom lane: retrieval leaves the model alone and looks documents up at question time, pasting them into the prompt.FINE-TUNING — change the modelahead of timeyour examplesretrain weightsnew modelat question timequestion →model answers from what it absorbedRETRIEVAL (RAG) — leave the model aloneahead of timeindex your documentsat question timequestionfind 5 chunkspaste + ask
The real distinction is when the knowledge arrives. Fine-tuning bakes it into the weights beforehand; retrieval fetches it at the moment of the question and pastes it into the prompt. One changes the model, the other changes the input.

Fine-tuning continues training on your examples, nudging the weights until your patterns are baked in. You end up with a new model. See gradient descent for what "nudging the weights" actually involves — fine-tuning is the same arithmetic, started from a model that already works.

RAG — retrieval-augmented generation — leaves the model untouched. Ahead of time you chop your documents into chunks and embed them. At question time you embed the question, find the closest chunks, and paste them into the prompt above the user's message.

In plain English

RAG is not a clever algorithm. It is an automated copy-paste: find the relevant page, stick it in front of the question, let the model read it. Its power comes from the fact that models are very good at using text you hand them, and unreliable about text they merely absorbed.

Why RAG is the default for facts

Start with retrieval unless you have a specific reason not to.

  • Facts stay current. Change a document, re-index it, done. A fine-tuned model is frozen at its training moment.
  • You get citations. You know which chunks were retrieved, so you can show your sources. Fine-tuning gives you no audit trail whatsoever.
  • Hallucinations drop sharply. The model is reading rather than recalling — the single most effective mitigation available, as covered in why LLMs hallucinate.
  • It is cheap and reversible. No training run, no GPU bill, and you can fix a bad answer by fixing a document.
  • Permissions work. Retrieve only what this user may see. A fine-tuned model has no concept of who is asking.

The costs are real but modest: every request carries retrieval latency and a longer prompt, and the answer is only as good as the search. Most disappointing RAG systems are disappointing retrieval systems — bad chunking, wrong number of results, no reranking — not bad models.

Why fine-tuning still matters

Fine-tuning is poor at teaching facts and good at teaching behaviour.

Use it when you need consistent output shape, a house voice, a specialist vocabulary, or a reasoning pattern the base model keeps drifting away from. A few thousand good examples of the behaviour you want will do more than any amount of prompt instruction.

In plain English

The test: could you achieve it by pasting a document into the prompt? If yes, use RAG. If what you want is a way of responding rather than a piece of knowledge, fine-tune.

Its real costs are the ones that show up later. The model is frozen at training time. Every update means another training run. You need genuinely good examples, and assembling a few thousand is the actual work. And facts taught this way are unreliable — the model absorbs a statistical tendency toward your content, not a lookup table, so it will still invent things with total confidence.

Choosing

Facts the model lacks
RAG. Your docs, your policies, anything that changes. Retrieval, every time.
Behaviour, format, tone
Fine-tuning. Making it always answer as JSON, or in your house style, or with a particular reasoning pattern.
A specialist vocabulary
Fine-tuning, usually — legal, clinical or internal jargon the base model handles badly.
Answers must cite sources
RAG. It is the only one of the two that can point at where an answer came from.
Data changes weekly
RAG. Re-indexing is minutes; retraining is not.
Latency and cost matter most
Fine-tuning. No retrieval step, shorter prompts.

They are not rivals

The strongest systems use both, and this is the part most comparisons miss. Fine-tune the model so it reliably answers in your format and follows your conventions; use retrieval to feed it current facts at question time. Behaviour from training, knowledge from retrieval.

The sensible order:

  1. Prompt engineering first. It is free and it resolves more cases than people expect.
  2. Then RAG, if the problem is missing knowledge.
  3. Then fine-tuning, if the problem is persistent misbehaviour.
  4. Both, if you have a real product and both problems.

Skipping to step 3 is the classic expensive mistake: it is the slowest, priciest option, and it does not fix the problem people usually have.

The short version

  • Fine-tuning changes weights, ahead of time. RAG changes the prompt, at question time.
  • Knowledge → RAG. Behaviour → fine-tuning.
  • RAG gives you citations, fresh data, permissions, and far fewer hallucinations.
  • Fine-tuning gives you format, tone and vocabulary, at the cost of a frozen snapshot.
  • Bad RAG is almost always bad retrieval, not a bad model.
  • Try prompting, then RAG, then fine-tuning — in that order.
← Archive— end of essay —Front page →