blog/relevant-is-not-sufficient-rag-abstention-hero
← Back to blog

Relevant Is Not Sufficient: Evidence Checks, Abstention and Small Verifiers for RAG

Technology•16 November 2026•TREEO Indonesia•5 min read

Retrieval-augmented generation (RAG) is often tuned until the retriever returns passages that look on-topic. That solves one problem and hides another. A passage can name the right entities and still fail to justify the answer. Another can be half of a two-step argument. Two passages can disagree. If the system answers anyway, the result reads as grounded but is not.

This guide covers the idea of a sufficiency check, the evidence on how well models abstain without one, and what is known about the small verifier models used to check answers afterwards. Nearly all the work cited is preprints, a dissertation, or papers from the teams that built the tools, so read the numbers as indications.

Sufficiency and verification

3 labels

SUPPORTED, REFUTED or INSUFFICIENT; only SUPPORTED allows an answer (SURE-RAG preprint)

65% to 91%

how often seven LLMs still answered when the evidence conflicted (evidence sufficiency benchmark preprint)

8 hours to 10 minutes

evaluation time reported after one paper swapped in the HHEM verifier

Relevance answers the wrong question

Relevance asks whether a passage is about the question. Sufficiency asks whether the evidence, taken together, supports a definite answer. The SURE-RAG preprint formalizes this: given a question, an answer and the evidence, assign one of SUPPORTED, REFUTED or INSUFFICIENT, and allow the answer only for SUPPORTED.

The authors argue sufficiency is a property of the whole evidence set, not of single passages. Taking the best-scoring passage over-answers when one passage looks supportive but another hop is missing. Averaging scores dilutes a strong refutation among neutral passages. Their approach scores each claim-passage pair, then aggregates signals such as coverage, disagreement and retrieval uncertainty.

The Sufficient Context line of work, from a separate group, defines the test in plain terms: could a diligent reader craft a definitive answer using only the supplied text? Multi-hop reasoning is allowed, leaps of faith are not. Its autorater, a one-shot prompted Gemini 1.5 Pro with chain-of-thought, scored 93% accuracy and 0.94 F1 on 115 human-labelled multi-hop questions. Combining the sufficiency label with the model's own confidence gave up to ten percentage points better selective accuracy at the same coverage, according to the project's summary. The project also reports that models often hallucinate even when context is sufficient.

What the numbers show, and where they stop

On SURE-RAG's controlled multi-hop benchmark, the calibrated version reached 0.9075 Macro-F1:

MethodMacro-F1
SURE-RAG (calibrated)0.9075
Concat cross-encoder0.8888 ± 0.0109
GPT-4o as judge0.7284
DeBERTa, mean pooling0.6516

The authors call the gap to the cross-encoder narrow: the contribution is a transparent aggregation protocol, not a new reasoning model. Three limits are stated in the same paper or visible in its tables. At 30% coverage, the selective risk (the share of answered cases that are unsafe) falls from 0.2588 for DeBERTa mean-pooling to 0.1642 for SURE-RAG, but at 70% coverage the advantage disappears and slightly reverses. On the natural hallucination set HaluBench the order flips: SURE-RAG scores 0.3343 unsafe-F1 against 0.7389 for GPT-4o, which the authors read as evidence that controlled sufficiency checking and natural hallucination detection are different problems. And the conformal thresholds they tried were too conservative to give useful coverage.

Separate work on abstention without special machinery is sobering. An evidence sufficiency benchmark with five levels, from full support to conflicting evidence, tested seven LLMs from five families. Models answered less as evidence worsened, but under conflict the over-answer rates still ran from 65% to 91%, and chain-of-thought prompting helped some models and not others.

The cost of saying no

Abstention trades coverage for safety. Tuning the threshold produces a real false-abstention rate: questions that were answerable but were refused. A dissertation on selective prediction also observes that adding more context can push a model to respond to false-premise questions, raising hallucination. Anyone claiming "zero hallucination" through abstention should be asked for the residual error rate and the false-abstention rate.

Small verifiers: what is known

A verifier checks a finished answer against its sources. Several small open models exist, and one preprint compares them under the HaluBench protocol:

VerifierSizeAverage macro-F1
HHEM110M57.63
AlignScore-Large355M59.21
MiniCheck-Flan-T5-Large783M62.30
Bespoke-MiniCheck7B68.48
Lynx8B77.25

Read it with care. It is a 2026 preprint, and the authors' own 4B model (79.65) tops the table. Its HHEM figure is far lower than the 82.2% accuracy another paper reports for HHEM with non-fabrication checking on a QA set, probably because of different datasets and metrics, though the sources do not say. That other paper, which put HHEM into an evaluation pipeline, also reports 78.9% recall of fabrications. Put plainly, roughly one in five fabrications passed in that setup (simple arithmetic on the reported figure), and HHEM struggled with localized hallucinations in summaries.

The Lynx authors report their 70B model at 87.4% on their 15,000-sample HaluBench, ahead of GPT-4o at 86.5%, with the 8B version at 82.9%. These are self-reported on a benchmark the same team built, which focuses on question answering; extending Lynx to summarization is listed as future work. So the ranking depends on the task.

On cost, HHEM-2.1-Open is about 0.1 billion parameters under Apache 2.0, and the 2.1 generation removed the 512-token context cap of its predecessor. Methods that sample several answers, such as semantic entropy, cost roughly ten times a plain answer. No source in the research gives per-request latency for MiniCheck, AlignScore or Lynx, and none gives their accuracy outside English. Lynx's own authors say most of HaluBench is English.

A practical order of operations

1. Check sufficiency before generating, and refuse or ask for more evidence when it is not met. 2. Generate with citations so each claim can be checked. 3. Run a cheap verifier over each claim, and escalate numbers and high-stakes claims to a stronger check or a person. 4. Measure the false-abstention rate as well as the hallucination rate.

For how to score each stage, see measuring RAG hallucination claim by claim. For the case for running models you control, see building an AI server we own.

Frequently Asked Questions

Relevant evidence is on topic. Sufficient evidence supports a definite answer when read together. A passage can name the right entities and still not justify the answer.

Turn climate goals into a verified portfolio

TREEO connects the full carbon cycle — eligibility, simulation, real-time monitoring and registry-ready reporting — combining expert consulting with dMRV technology so the evidence exists before anyone asks for it.

Related Articles