blog/measuring-rag-hallucination-claim-by-claim-hero
← Back to blog

Measuring RAG Hallucination Claim by Claim: Faithfulness, FActScore and Leaderboards

Technology•16 November 2026•TREEO Indonesia•5 min read

"How often does it hallucinate?" sounds like one question. In a retrieval-augmented generation (RAG) system it is several. The retriever may have fetched the wrong passage. The generator may have ignored the right one. The answer may be faithful to a source that is itself wrong. A single score for the final answer cannot tell these apart, and the numbers published for different tools measure different things. This guide explains how the common measures work and how to read them.

A note on evidence: almost everything cited here is a preprint, a vendor guide or a leaderboard page. Only one of the 186 sources in the research behind this piece was peer-reviewed. Figures are attributed so you can weigh them.

Claim-level measurement in numbers

supported ÷ total

how faithfulness is computed after an answer is split into atomic claims

near 50%

accuracy of the best hallucination detectors on the FaithBench benchmark (preprint)

1 in 20

unsupported answers a judge with a 95% catch rate still lets through (simple arithmetic)

Faithfulness is a fraction of claims

The common recipe has three steps: break the answer into atomic claims, check each claim against the retrieved context, and report the share that is supported. A faithfulness score of 1.0 means every claim is supported, and 0.0 means none is. An Indonesian-language RAGAS tutorial describes exactly this calculation.

RAGAS is an open-source framework that implements the idea with large language models as judges. It reports four scores: faithfulness, answer relevance, context precision and context recall. It is described as scoring pipelines without ground-truth answers, although the same Indonesian tutorial notes that context recall does need a reference answer, so check which metrics you enable.

FActScore applies the same logic to long-form text. It splits a generation into atomic facts and measures the fraction supported by a trusted reference source. This matters because a long answer can look strong overall and still contain one wrong number or one invented attribution. A pass/fail verdict on the whole answer would hide it.

A third approach, ARES, trains lightweight judges on synthetic data plus curated human judgments, so each new evaluation is cheaper than calling a large model. The trade-off is setup effort: you need human-labelled examples first.

Faithful is not the same as correct

Faithfulness asks whether the answer is supported by the retrieved context. It does not ask whether the context is true. If a source document contains an error, an answer can score near-perfectly on faithfulness and still be wrong. Faithfulness is a check on the generator, not on the corpus.

Why one score hides the failing stage

A vendor guide on RAG evaluation makes a point that is easy to verify in practice: if you only score the final answer, you cannot tell whether retrieval or generation failed, and the two failures need opposite fixes. Missing evidence calls for better retrieval or chunking. Ignored evidence calls for changes to prompting, model choice or checking after generation.

So report at least two groups of numbers:

StageWhat to askTypical measures
RetrievalWas the needed passage fetched?Recall@k, context precision, context recall
GenerationDoes each claim follow from what was fetched?Faithfulness, answer relevance
Whole answerIs it correct and cited properly?Reference-based accuracy, citation checks, human rubric

A practitioner guide suggests starting with ten to twenty real questions that have known-good answers and source passages. This small golden set becomes the baseline for judging any later change. It also recommends rubric scoring by humans with a measured agreement rate, rather than informal review.

The judge is also a model

LLM judges share blind spots with the generators they grade. A judge asked "is this grounded?" can be fooled by the same plausible phrasing that fooled the generator. Detection error also scales with volume: a judge that catches 95% of unsupported answers still passes one in twenty, which is a lot of misses in a busy system.

The FaithBench preprint shows how hard detection is. Its summaries were selected because detectors disagreed on them, and the authors report that even the best detection models reach near 50% accuracy there. A later preprint, which uses balanced accuracy and F1-macro rather than plain accuracy, compares detectors across four datasets and finds that zero-shot GPT-4o and o3-mini-high perform best, but average balanced accuracy stays below 78% and F1-macro below 72%. In that work, the highest balanced accuracy on FaithBench was 68.8%, and the highest F1-macro, 63.7%, came from the small HHEM model scoring claim by claim. The same paper notes smaller fine-tuned detectors can be competitive with zero-shot LLM judges. Because FaithBench articles were picked using detectors including HHEM, those detectors may look worse on it than they would on ordinary text.

Reading a hallucination leaderboard

The best-known public leaderboard is Vectara's, which uses its HHEM model to score how often a language model hallucinates when asked to summarize a document. Its repository states several limits that are worth keeping in mind:

- It measures factual consistency with the source document in a summarization task, not overall factual accuracy. - A summary counts as hallucinated when the HHEM score falls below 0.5 on a 0 to 1 scale. The reported hallucination rate is 100 minus the factual consistency rate. - The rate is computed only on articles the model actually summarized, so an answer rate and an average summary length are reported alongside. - The current version uses a dataset of more than 7,700 articles from law, medicine, finance, education and technology, which is not public. Vectara's blog says rates for top models rose under this newer, longer and harder dataset, so numbers from different versions are not comparable. - It runs on HHEM-2.3, a commercial model, while HHEM-2.1-Open is the open variant.

The repository argues the leaderboard is a good indicator for RAG and agentic systems that summarize search results. The FaithBench paper reports that its model ranking generally aligns with the leaderboard. Treat that as a reason to use it for rough model shortlisting, not as an acceptance test for your own corpus, your language or your question types.

One more caution on benchmarks generally. Difficulty is often set on purpose. HealthBench Professional, for example, upsamples difficult examples by roughly 3.5 times relative to its candidate pool and says its absolute scores should not be read as real-world rates. Each benchmark has its own design like this, so a headline percentage is a statement about that instrument and task, not a portable "hallucination rate".

A short checklist

1. Fix one named instrument and report every number with it. 2. Score retrieval and generation separately. 3. Decompose answers into claims for the generation score. 4. Keep a small human-labelled set to check your automated judge. 5. Say whether you measured faithfulness or correctness.

Frequently Asked Questions

After an answer is split into atomic claims, 90% of them were judged supported by the retrieved context. It says nothing about whether the context itself is correct.

Turn climate goals into a verified portfolio

TREEO connects the full carbon cycle — eligibility, simulation, real-time monitoring and registry-ready reporting — combining expert consulting with dMRV technology so the evidence exists before anyone asks for it.

Related Articles

Measuring RAG Hallucination: Faithfulness Explained | TREEO Indonesia