Twenty minutes into a PM loop at a Series C health-tech startup, the interviewer leans back and asks:
→ “When should you use RAG versus fine-tuning?”
The candidate has seen this one before. “RAG is better for most cases because it reduces hallucination. Fine-tuning is expensive and you only need it for very specialized tasks.”
The interviewer nods and asks one follow-up. “Okay. What if the model’s answers are correct, but they’re in the wrong format every time?”
Silence. The memorized answer had nowhere to go.
Now picture a different candidate with the same question. They pause for a second and say: “Before I pick one, I want to know whether this is a knowledge problem or a behavior problem. Does the model not know something, or does it know enough and act wrongly? Those two point to different fixes.” The interviewer stops typing and starts listening.
This question sounds like trivia. It isn’t. It tests whether you diagnose before you prescribe.
This question touches a few other types of questions that you should master.
Debugging a bad RAG bot is an AI Product Improvement question.
The evaluation plan comes from AI Evaluation & Metrics (EVAL-AI).
Patient-scoped retrieval and prompt injection overlap with AI Safety & Responsible AI.
For the full STACK-AI framework, start with How to answer AI technical questions, and for the design-shaped version, see the post on designing an agentic AI system.
Table of Content
Why this question is different, and what’s being tested
“RAG versus fine-tuning” is probably the most common AI technical judgment question, which works against you. Interviewers have heard “RAG for knowledge, fine-tuning for style” hundreds of times, and a canned line earns a nod and an immediate follow-up designed to see whether you understand why.
It’s also a “when to use X” question, so no product comes attached. You either ask for one or propose one, and that choice says a lot about your judgment. A candidate who answers in the abstract for ten minutes never gets to show a decision.
And to be clear: nobody expects you to explain LoRA math or how a vector index is built. They expect you to know what each technique physically changes, what it costs and when it breaks. Here’s what the five competencies sound like on this specific question.
What each technique actually changes
RAG changes what the model reads
Retrieval-augmented generation (RAG) looks up relevant text at the moment of the request and puts it in the prompt, so the model answers from that text instead of from memory. The pipeline has six steps:
Ingest. Pull in your documents: policies, help articles, records.
Chunk. Split them into passages of a few hundred words, because a whole 80-page document is too much to hand the model at once.
Embed. Convert each chunk into an embedding, a list of numbers that represents its meaning, so that passages with similar meanings sit close together.
Retrieve. When a question comes in, embed it too and find the chunks closest to it. Production systems usually combine this with keyword search, because embeddings are bad at exact codes and names.
Rerank. Run a second, more careful model over the top 20 or so results to pick the best 5.
Generate. Give the winners to the model with an instruction to answer only from them and cite which passage each claim came from.
The model itself never changes. Update a document, re-index it, and the next answer uses the new version.
Fine-tuning changes how the model behaves
Fine-tuning continues training a model on example input and output pairs (labeled examples), which adjusts its weights, the internal numbers that decide how it responds.
Most teams now use parameter-efficient fine-tuning such as LoRA, which trains a small add-on instead of the whole model, so it’s cheaper and faster than it used to be.
It still needs hundreds to thousands of good examples, an evaluation set and a retraining process every time the behaviour you want changes.
The diagnostic that settles most of these debates
So the first question is always the same. Is this a knowledge problem or a behavior problem?
A knowledge problem means the model doesn’t know something it needs: your insurer’s current coverage rules, a customer’s order history, this week’s pricing. That fix usually lives in retrieval. A behavior problem means the model knows enough but acts wrongly: wrong format, wrong tone, inconsistent structure, too slow or too expensive per call. That fix usually lives in prompting or fine-tuning.
This is why “let’s fine-tune the model on our documents so it knows our policies” makes interviewers wince. Fine-tuning doesn’t give a model dependable recall of specific facts. Even when it absorbs some of them, they go stale the moment a policy changes, the model can’t cite where a fact came from, and you can’t cleanly remove one fact you shouldn’t have trained on.
The option everyone forgets
Before either technique, there’s prompting. If the model already has the capability and the knowledge fits in the prompt, a well-written instruction with a few examples and a structured output format might be the whole answer, and it ships in hours. With today’s long context windows, pasting in a 40-page policy document is a perfectly good first version.
It stops working when one of three things happens: the document set no longer fits, the cost per call climbs because every request carries the whole set, or quality drops because the model misses details buried in the middle of a very long prompt.
💡 Interview Tip
Say this early: “RAG changes what the model reads. Fine-tuning changes how it behaves. So my first question is which of those is actually broken.”
Comparison table - Prompting vs RAG vs Fine-tuning
When a RAG system gives a wrong answer, the most useful move is to figure out which half failed.