The metric trap
Recall@k tells you whether the right document was retrieved — not whether the system used it, ignored it, or blended it with something irrelevant. Across our production RAG deployments, we've seen systems with excellent retrieval scores produce ungrounded answers at rates that would be unacceptable in any other software category.
Three dimensions that matter
We evaluate on groundedness (is every claim traceable to a retrieved source?), refusal quality (does the system decline gracefully when the corpus genuinely lacks an answer, instead of improvising?), and answer utility (would a domain expert judge the response as actually useful, not merely correct?). These three routinely disagree with retrieval metrics — and with each other.
A practical protocol
Our current protocol combines an LLM-as-judge pass calibrated against a small human-labeled gold set, a mandatory “unanswerable” question suite, and per-source attribution checks. It runs in CI, costs cents per evaluation, and has caught regressions that every off-the-shelf metric missed. The full methodology is available to clients as part of our development engagements.