Evaluate Retrieval and Answers Separately
Build a representative test set and diagnose context, citation, answer, safety, latency, and cost failures independently.
Learning objectives
- Create a versioned evaluation set from real task patterns
- Select metrics for retrieval, answers, citations, and refusal
- Combine automated graders with calibrated human review
ToolDix original visual
Frame
Name the outcome and constraints.
Build
Try one bounded workflow.
Review
Keep evidence, revise, and share.
Build cases before optimizing
Collect representative tasks, edge cases, adversarial inputs, and unanswerable questions. Each case needs input, approved evidence or reference properties, metadata, and a reason it matters. Keep a holdout set that prompt authors do not repeatedly tune against.
Synthetic questions can broaden coverage, but a human must verify that they are answerable, realistic, and not merely restating one sentence. Include production failures after privacy review; they are often more valuable than idealized examples.
Measure the pipeline in layers
For retrieval, measure whether supporting evidence appears and at what rank. For answers, measure correctness, completeness, relevance, citation entailment, groundedness, and appropriate refusal. Also track latency, token usage, cost, policy violations, and tool failures.
A single score hides tradeoffs. Report a small dashboard and retain case-level results so a change can be traced to the failures it fixed or created.
Calibrate automated graders
Write a rubric with observable criteria and examples. Compare automated grades with blinded human labels on a sample. Measure disagreement and inspect where the grader favors style over truth or repeats model biases. Version the grader prompt and model like production code.
Practice: controlled comparison
Create twenty cases across normal, edge, adversarial, and unanswerable groups. Compare two system versions with identical data. Report retrieval recall, citation support, answer correctness, refusal quality, p95 latency, and average cost. Review every regression, not only the aggregate winner.
Release rule
Define blocking thresholds before seeing the candidate results. A gain in average answer style must not silently trade away citation accuracy, safety, or performance for a critical user group.
Sources and license context
These references informed the lesson. ToolDix adds its own explanation, workflow, and practice rather than reproducing source material.
- Evaluation Best Practices (OpenAI terms apply)
- RAG Evaluation (Repository and notebook licenses apply)
Take it further
Use a primary source to deepen this lesson.
Each recommendation is a direct link to the publisher or author. The study prompt is ToolDix editorial guidance, not copied course content.
Course
Full Stack LLM Bootcamp
Choose one lab and write an evaluation criterion before changing its prompt, retrieval, or model layer.
Open original source
Hands-on lab
RAG Evaluation
Replace the sample corpus with ten approved documents and manually review every generated evaluation question.
Open original source
Hands-on lab
RAG from Scratch with LlamaIndex
Use five short documents, print the retrieved nodes, and reject any answer whose claim lacks supporting context.
Open original source