BYD TechThe short answer
RAG, retrieval-augmented generation, looks up relevant passages from your own documents and gives them to the model when it answers. Fine-tuning trains the model further so it behaves differently. Use RAG when the model needs to know your facts, and fine-tuning when it needs a consistent style, format or skill. Most business uses are about facts, so most should start with good instructions and RAG.
Key points
- RAG changes what the model knows at answer time. Fine-tuning changes how it behaves.
- For answers from your own content, start with RAG.
- RAG stays current: update a document and the next answer reflects it.
- Fine-tuning suits fixed style, format or narrow tasks, and needs good training examples.
- Try clear instructions first. They solve more than people expect.
Side by side
| RAG | Fine-tuning | |
|---|---|---|
| What it changes | What the model can see when it answers | How the model itself behaves |
| Good for | Answering from your documents and data | A fixed tone, format or narrow skill |
| Keeping current | Update the documents | Train again |
| Can show its sources | Yes, it can cite the passages it used | No |
| What you need | Clean, well-organized content | Many high-quality example pairs |
| Main risk | Retrieving the wrong passage | Baking in errors that are hard to see |
What RAG is
The term comes from a 2020 research paper by Lewis and colleagues, which combined a language model with a retriever that fetches relevant text. In practice it works in three steps: your documents are split into passages and indexed, the passages most relevant to a question are retrieved, and the model answers using those passages.
- The model answers from your content, not only from what it learned in training.
- It can point to the passage it used, so a person can check the answer.
- When a policy changes, you update the document and nothing is retrained.
- Access rules can apply: a user only retrieves documents they are allowed to see.
Most of the quality comes from retrieval. If the right passage is not found, the best model will still answer badly. Anthropic’s write-up on contextual retrieval is a useful description of how much careful indexing improves results.
What fine-tuning is
Fine-tuning continues a model’s training on your own examples, usually pairs of an input and the output you want. The result is a version of the model that tends to respond in that way without being told each time.
- Useful for a strict output format, a house style, or classifying into your own categories.
- It does not reliably teach facts. A model fine-tuned on your manual may still get details wrong.
- It needs many clean examples, and its quality is only as good as they are.
- When your content changes, the tuned model does not.
How to choose
- Start with clear instructions and a few examples in the prompt. Many problems end here.
- If the model lacks your facts, add RAG.
- If it has the facts but the style or format is still wrong after good instructions, consider fine-tuning.
- If you need both your facts and a fixed behavior, combine them: retrieve, then answer with a tuned model.
Whichever you choose, build a test set of real questions with known good answers first. Without it you cannot tell whether a change helped.
What this looks like in a real product
A support agent that answers from your help center is RAG. So is internal search that answers staff questions from company documents. In both, the important work is unglamorous: cleaning up the source content, deciding who may see what, and testing answers against real questions.
That is the approach in our AI agent development work: answers come from your own content, tested on your real tickets. The wider steps are in How to build an AI agent.
Sources
Checked when this guide was written.
