Retrieval-augmented generation looks like one system from the outside: a question goes in, an answer comes out. Inside, it is two systems chained together — a retriever that finds context, and a model that writes an answer from it. A single “accuracy” score hides which half is failing.
Half one: did we find the right context?
Retrieval quality is about whether the documents needed to answer the question made it into the context window, and how much noise came with them. Useful measures include whether the relevant passage appears in the top results at all, and what share of the retrieved passages are actually relevant.
Half two: did the model use it faithfully?
Generation quality asks different questions. Is every claim in the answer supported by the retrieved context? Does the answer address what was asked? Are citations pointing to the passages they claim to? An answer can be fluent, relevant and still invent a detail the documents never mention.
Reading the results together
- Good retrieval, poor faithfulness: the model is ignoring or embellishing the context — work on prompts, instructions or the model.
- Poor retrieval, good faithfulness: the model is honest about thin context — improve chunking, embeddings or the query.
- Both poor: start with retrieval; generation cannot be better than its inputs.
Do not forget the unanswerable
Include questions your knowledge base cannot answer. The right behaviour is to say so, and a system that confidently answers them anyway has a hallucination problem that accuracy on answerable questions will never reveal.
Where to start
Take thirty real questions, label which documents should answer each one, and score retrieval and faithfulness separately. That small exercise usually points straight at the biggest improvement available — and gives you a baseline to measure every change against.