When a business wants an AI that knows its documents, two approaches come up: retrieval-augmented generation (RAG) and fine-tuning. They are often presented as competitors. They are not really the same tool, and for most business-knowledge problems the answer is clearly one of them. Here is how to choose.
What Each One Actually Does
RAG leaves the model as-is and gives it the right documents at question time. When someone asks about your refund policy, the system retrieves the relevant passages from your actual policy and the model answers from them — with citations you can check. The knowledge lives in your documents, not in the model.
Fine-tuning adjusts the model itself by training it further on your data. The knowledge (or more often, the style and format) becomes baked into the model’s weights. There is no retrieval step at question time; the behavior is trained in.
For Business Documents, RAG Usually Wins
Three reasons, and they are decisive for most cases:
- Freshness. Documents change. With RAG, you update the document and the next answer reflects it. With fine-tuning, new information means retraining — your policy from last week is baked into a model that does not know it changed.
- Citations. RAG can point to the source passage, so a person can verify the answer. A fine-tuned model states things with no way to check where they came from — a serious problem when accuracy matters.
- Cost and control. RAG is cheaper to build and change, and it keeps your knowledge in your documents where you can govern access to it. Fine-tuning is a heavier, less reversible commitment.
When Fine-Tuning Earns Its Place
Fine-tuning is not a mistake — it is a specialist tool. It shines when you need the model to adopt a consistent style or format (a specific tone, a structured output, a domain’s phrasing), or to handle a narrow, stable task with high reliability. It is about teaching the model how to respond, not what facts to know. And the two combine: a fine-tuned model that formats responses your way, answering from RAG-retrieved facts, is a genuinely strong pattern.
The Practical Rule
If your goal is “answer accurately from our documents, and let us verify and update them,” start with RAG. If your goal is “always respond in this specific style or structure,” consider fine-tuning — usually on top of RAG, not instead of it. The failure mode to avoid is fine-tuning a model on your knowledge base to make it “know” your business, then discovering it cannot cite sources, cannot be corrected without retraining, and confidently states last quarter’s facts.
Software Depo builds retrieval-grounded AI that answers from your real content with citations and access control — and we will tell you the specific cases where fine-tuning genuinely adds value on top. Tell us what knowledge you want your AI to work from and we will recommend the right approach, honestly.
The Questions That Decide It
The comparison usually gets argued at the level of capability. It is settled faster at the level of four concrete questions about the system you are actually building, because each one rules out an approach on its own.
None of these depend on how large your document set is or how much budget you have. They depend on the shape of the work.
- Does the answer depend on who is asking? Weights hold one set of knowledge for every caller. A retrieval layer sits between the question and the answer, which is the only place an entitlement check can be applied: the candidate set is reduced to what that person is permitted to see, and the model never receives the rest. If two people are supposed to get different answers to the same question, that settles it before any other consideration.
- Does the answer depend on the current state of a system rather than the contents of a document? Order status, license counts, ticket state, open invoices. These are neither retrieval problems nor training problems. They are tool calls against the system of record, and the same grounding principle applies: the model asks for what it needs at question time instead of carrying it.
- Who fixes it when it is wrong? A retrieval defect is usually fixable by the person who owns the content: correct the document, change the chunking, add a synonym, adjust a filter. A fine-tune defect goes back to whoever owns the dataset and the training pipeline, and it is repaired on the schedule of a training run rather than an edit. That is a staffing question as much as an architecture question, and it decides whether the system is still accurate after the people who built it have moved on.
- Can you write the test before you build the system? If nobody can produce a list of real questions with the answers they should get, neither approach can be evaluated and neither can be defended when someone disputes an answer. Both approaches need that list eventually. Only one of them needs it before the first training run.
What Fine-Tuning Costs You Later
The build is the part everyone estimates. The costs that determine whether a fine-tune was a good decision arrive after it works.
Version the dataset the way you version code, and record for every example where it came from and who approved the output as correct. Retraining is re-approval, and an example nobody can source is one nobody can defend when the tuned model reproduces it. Kept that way, the portable asset is the dataset rather than the weights, and standing up the same behavior on a different base becomes a re-run instead of a re-collection.
- The dataset is the project. The training run itself is the cheap, automated part. The expensive part is assembling examples that all resolve the same situation the same way, and consistency across them matters more than raw count. If two examples handle the same situation differently, the model learns the average of the two, and that averaging is invisible until it surfaces as an inconsistent output you cannot trace to any line of code.
- Facts cannot be removed. A figure that gets retracted, or a record covered by a deletion request, is distributed across the weights with no location to edit and no row to drop. Getting it out means retraining from the base checkpoint with the offending examples excluded, then re-running the evaluation that qualified the previous version before the replacement can ship.
- The base model clock runs on the provider's schedule. A tuned checkpoint is bound to the specific base it was trained from. When that base is deprecated or superseded, the tuning does not carry across: you train again against the new one and validate again. Budget that as recurring maintenance, because it arrives whether or not your own requirements changed.
- The savings are per call, not per project. What a small tuned model buys is a short completion in place of a long prompt full of instructions and examples: lower latency and less token spend on every request. That only repays the training and evaluation effort above a call volume where the per-request saving accumulates. Below that volume, the same behavior from a stock model with a carefully written prompt costs less in total.
The Cases Where the Obvious Answer Is Wrong
Four situations where reaching for retrieval, or reaching for it in the standard shape, produces a system that cannot answer the questions people will actually ask.
- The corpus is small and stable. A few dozen pages that change once a quarter do not need an index. Put the material in the prompt, cache it where the provider supports prompt caching, and skip the chunking decisions, the embedding refreshes, and the entire class of bugs where the right passage exists and does not come back.
- The question is an aggregation. Retrieval returns the passages most similar to the question, which is a sample. "How many contracts renew before March", "which sites are missing a signed form", and "what is the total across all of them" cannot be answered from a sample, and a system that tries will produce a confident number derived from the handful of documents it happened to retrieve. Those belong in a query against structured data, with the model writing the query rather than reading the documents.
- The freshness you get is the freshness of your ingestion. Updating a document makes the next answer current once the index has seen the change. If the connector runs as a nightly batch, answers are current as of the last run, which is fine for a policy library and wrong for anything that moves during the day. Match the ingestion trigger to how fast the source actually changes: scheduled for documents, event-driven or read-through at question time for records that change hourly.
- There is no document set. The approach assumes the knowledge is written down somewhere. When it lives in people's heads, in ticket threads, and in three conflicting spreadsheets, neither option has anything to work from. That is a content problem, and it is much cheaper to name it as one at the start than to discover it halfway through a retrieval build.
Piloting the Decision Before You Commit
This comparison is cheaper to settle with measurement than with argument, and nearly all of the work is reusable whichever way it comes out.
- Write the questions first. Collect real questions from the people who will use the system, each with the answer it should get and the document that answer should come from. That list is the evaluation set. It survives every architecture change, and it is the only thing that turns "it seems better" into a number.
- Measure retrieval on its own. Before any generation, check whether the correct passage appears in the top k results for each question. Recall at k is a ceiling on everything downstream: if the passage never comes back, no amount of model work recovers the answer.
- Then measure the answers against the passages that were retrieved. Splitting the two measurements tells you whether a wrong answer is a retrieval failure or a generation failure. The two have entirely different fixes, and early on the retrieval failures dominate.
- Try few-shot before you train. Put five or ten examples of the output you want directly in the prompt and re-run the evaluation. If few-shot moves the metric, training on more examples is likely to move it further. If it does not move the metric at all, the gap is knowledge or retrieval, and training will not close it.
- Hold out a slice of the evaluation set and never train on it. Once you start tuning against a set, the score on that set stops predicting behavior on questions nobody has seen. The held-out slice is what you use to decide whether the tuned version ships.
How Retrieval Fails, and What It Looks Like
Retrieval is the usual answer here, which makes its failure modes the ones you will spend your time debugging. Almost all of them are in the retrieval stage rather than in the model, and they are diagnosable because you can look at what came back.
- Chunk boundaries cut context away. A table row separated from its header, a clause separated from the term it defines, a procedure step separated from the warning above it. The model can only reason over what the chunk contains. Chunking that follows document structure, with overlap between chunks, costs storage and usually earns it back.
- Pure vector search misses exact strings. Embeddings match on meaning, which is what you want for prose and what fails for part numbers, error codes, policy identifiers, and rare acronyms, where the user's term and the document's term have to match literally. Hybrid search that combines lexical scoring with vector similarity covers both classes and is the sensible default to start from.
- Old versions rank as well as current ones. Three years of the same handbook all match the same query, and the answer comes from whichever chunk scored highest. That is fixed in the index with version metadata and effective-date filtering. It is not fixable in the prompt.
- Nothing relevant comes back and the model answers anyway. With no grounding, it fills the gap from pretraining, and the result reads exactly like a grounded answer. Instruct for abstention, evaluate abstention deliberately with questions your documents genuinely cannot answer, and log every empty-retrieval event, because that log is where your missing content shows up first.
- Permission filtering applied in the wrong order. Retrieving the top k and then discarding what the caller may not see can hand a user an empty result while material they are entitled to sits just below the cutoff. The filter belongs in the query, so that ranking happens within the caller's permitted set.
Frequently Asked Questions
Does a long context window make retrieval unnecessary?
Three things do not go away. Per-call token cost: everything in the prompt is paid for on every request, and a corpus large enough to matter is paid for again on the next question. Attention over long inputs is not uniform, so accuracy on material buried in the middle of a very long context degrades, and more tokens is not the same as more reliable. And a context window has no access control: whatever you place in it is visible to whoever asked. The two also compose well, which is the practical answer for most teams: retrieve more generously than you would have, and let the larger window absorb the extra passages.
How many examples does a fine-tune need?
It depends far more on what you are training than on any target number. Tone and formatting are shallow behaviors and shift on relatively few examples. Reliability on a task with edge cases is a coverage problem instead: every variant you expect the model to handle correctly needs examples of that variant, including the ones where the correct output is a refusal or an escalation rather than an answer. If you cannot enumerate the edge cases, you cannot size the dataset.
Can fine-tuning teach a model new facts at all?
Some, unreliably. What it mostly learns from your documents is the shape and vocabulary of your text rather than a lookup table of its contents, so a model trained on your contracts gets better at sounding like your contracts. Genuinely absorbing a domain's facts is closer to continued pretraining: billions of tokens, a full training run, and a team that does this specifically. That is a different project with a different budget from the few thousand examples a supervised fine-tune uses.
What does it actually take to keep a retrieval system running?
The ingestion pipeline is a service, not a script. Connectors re-authenticate, source systems change their export formats, jobs fail quietly, and a document that silently stopped syncing produces answers that look correct and are out of date. The embedding model is a pinned dependency: queries and stored vectors must come from the same model, so replacing it means re-embedding the whole corpus in a coordinated cutover. Reranking adds a second model call per question, and that is where retrieval latency usually goes. Each of these is a reason to re-run the evaluation set.
Should we fine-tune to fix answers that retrieval is getting wrong?
A fine-tune trained on top of a weak retrieval stage learns to compensate for that stage's specific defects: it fills the gaps the retriever leaves and learns which retrieved passages tend to be noise. Those compensations are fitted to a retriever you are going to change. Improve the chunking or add reranking later and the model is correcting for a problem that no longer exists, which shows up as confident answers that ignore passages that are, by then, exactly right.
