Evaluation¶
Choosing a blueprint should be backed by numbers from your data. ragpick eval runs
one or more configs over a question set and prints a comparison table.
Dataset format¶
JSON Lines, one question per line. Only question is required:
{"question": "What does error E-1042 mean?", "ground_truth": "Low battery.", "relevant_doc_ids": ["error-codes.md"]}
{"question": "When do we escalate to Tier 2?", "relevant_doc_ids": ["internal-escalation.md"], "groups": ["support"]}
| Field | Purpose |
|---|---|
question |
The query. |
relevant_doc_ids |
Document ids (paths relative to the ingested directory) that answer it. Enables retrieval metrics. |
ground_truth |
Reference answer, stored in the results for review. |
groups |
Run the query as a member of these groups (to evaluate access-controlled content). |
id |
Optional identifier. |
Blank lines and lines starting with # or // are ignored. Errors name the file and line.
Generating a dataset¶
No questions yet? Generate a starter set from your documents:
ragpick generate-eval ./docs -o eval.jsonl --count 30 -c rag.yaml
- With a real LLM generator, each sampled chunk becomes a natural question and a short
reference answer. Offline (
--mode heuristic), questions come from section headings ("How do I reset your password?"), codes ("What does E-1042 mean?") and titles. - Chunks are sampled round-robin across documents (
--seedmakes it reproducible), using the config's loader and chunker, sorelevant_doc_idsmatch whatingestproduces. - Questions from documents with
allowed_groupsget thosegroups, so evaluation runs with the right permissions. - Review the file. Delete vague questions, fix answers, and add the hard cases your users really ask.
Running¶
# compare two configs; each gets a temporary index built from --data
ragpick eval examples/eval.jsonl \
--config examples/naive.yaml --config examples/hybrid.yaml \
--data examples/data --output results.json
Without --data, each config uses its persisted index and builds it from the config's
data path if none exists. --output writes .json (all rows) or .csv (one row per config).
From Python:
from ragpick import Pipeline
from ragpick.eval import evaluate, load_dataset
report = evaluate(Pipeline.from_yaml("rag.yaml"), load_dataset("eval.jsonl"))
print(report.metrics)
Bench: try every blueprint¶
ragpick bench answers "which blueprint should I use?" with data. It runs every stable
blueprint on your documents and questions, using the same models and the same top_k,
and ranks them:
ragpick bench eval.jsonl --data ./docs --config rag.yaml --metric mrr --write best.yaml
- Providers (loader, embedder, store, generator,
evalsettings) come from the base--config(default:./rag.yamlif present, otherwise offline fake components). - Architecture (chunker, keyword index, retriever, reranker) comes from each blueprint.
--metricpicks the winner:hit_rate@k,mrr(default),precision@k, or, with a judge,faithfulness/answer_relevance. Ties go to the lower median latency.-b NAME(repeatable) limits the candidates;--top-koverrides k for all of them. By default only document blueprints are compared;sqlanswers from tables and needs a real LLM, so include it explicitly (-b sql) with a table dataset.--write best.yamlsaves the winner's config: your providers plusblueprint: <winner>. Secrets stay as${ENV_VAR}references.- Each candidate gets a temporary index; your real indexes are never touched.
from ragpick.eval.bench import run_bench
result = run_bench("eval.jsonl", "./docs", base_config="rag.yaml", metric="mrr")
print(result.winner, result.ranking)
Metrics¶
Retrieval (no LLM needed). These are computed over the ranked contexts mapped to their
doc_id, with k = query.top_k:
| Metric | Definition |
|---|---|
hit_rate@k |
1 if any of the top-k contexts comes from a relevant document. |
mrr |
1 / rank of the first relevant context (0 if none). |
precision@k |
relevant contexts in the top k, divided by k. |
LLM-judged. These need a judge (eval.judge in the config or --judge openai:gpt-4o-mini):
| Metric | Question asked of the judge |
|---|---|
faithfulness |
Is every claim in the answer supported by the retrieved contexts? |
answer_relevance |
Does the answer address the question directly? |
Each score is 0–1 with a short reason per row (in the JSON output). All judge prompts live
in ragpick/eval/prompts.py.
Operational:
| Metric | Definition |
|---|---|
latency_p50_ms / latency_p95_ms |
end-to-end query latency percentiles |
total_tokens |
prompt + completion tokens of the generator |
est_cost_usd |
from the price table (USD per 1M tokens); n/a for unknown models |
errors |
rows where the query or judge failed (recorded, not fatal) |
Override or extend prices in the config:
eval:
judge: {type: openai, params: {model: gpt-4o-mini}}
prices:
my-finetuned-model: {prompt_per_1m: 0.30, completion_per_1m: 1.20}
Gating CI on quality¶
Fail the build when retrieval quality drops or answers get slow or expensive:
ragpick eval eval.jsonl -c rag.yaml \
--fail-under hit_rate@k=0.85 --fail-under mrr=0.6 \
--fail-over latency_p95_ms=2000 --fail-over est_cost_usd=0.50
or keep the thresholds next to the config:
eval:
fail_under: {hit_rate@k: 0.85, faithfulness: 0.8}
fail_over: {latency_p95_ms: 2000}
Misses are printed and the command exits with status 1; CLI flags override config
values. A required metric that isn't available (e.g. faithfulness without a judge) also
fails, so a misconfigured job can't pass silently. In GitHub Actions:
- run: pip install ragpick
- run: ragpick eval eval.jsonl -c rag.yaml --data docs --fail-under hit_rate@k=0.85
Tips¶
-
Rank metrics (MRR, precision) assume contexts are ordered by relevance. They're not meaningful for
long_contextwithinclude_all(documents in a stable order): compare it withfaithfulness/answer_relevanceusing a judge. -
If
cacheis enabled, repeated runs reuse answers: costs drop to zero and latencies to near zero. Disable the cache when measuring latency. -
30–100 representative questions catch most regressions. Include the exact-term queries your users really send (codes, IDs), plus some that should return "I don't know".
- Add eval to CI: run
ragpick eval ... --output results.jsonand fail the build ifhit_rate@kdrops. - The
fakegenerator and judge are for plumbing tests. Their answer metrics aren't meaningful, but retrieval metrics are.