blog/deep-research-agents-invented-sources-citation-checks-hero
← Back to blog

When Deep Research Agents Invent Sources: Measuring and Checking Citations

Technology•16 November 2026•TREEO Indonesia•4 min read

A deep research agent runs many searches, reads pages and writes a cited report. The citations are what make the report look trustworthy. They are also where several studies have found trouble. This guide sets out what those studies measured, why a link that opens is not proof of anything, and how to check citations before a report is used.

Most of the evidence below is from preprints and from independent commentary on studies, not peer-reviewed trials. The studies use different definitions, so the figures cannot be added together or ranked against each other.

Reported citation problems

3% to 13%

citation URLs reported as hallucinated (no Wayback Machine record), across commercial models and agents (preprint)

5% to 18%

citation URLs that did not resolve at evaluation time (same preprint)

about 42%

average drop in fact-check accuracy between 2 and 150 tool calls, while link validity stayed above 92% (preprint, via commentary)

Four things "citation accuracy" can mean

The research reviewed here uses at least four constructs. Keep them apart.

ConstructWhat it asksReported figure
Hallucinated URLDoes the address have any record in the Wayback Machine, or was it likely never real?3% to 13% of citation URLs
Non-resolving URLDoes it fail to load today?5% to 18%
Fake referenceAre the authors or title invented?roughly 47% to 50% for two products on one dermatology task (commentary)
Unsupported claimDoes the cited source actually back the statement?fact-check accuracy fell about 42% with research depth

The URL-level study

A preprint by Rao, Wong and Callison-Burch examined citation URLs from 10 models and agents on DRBench, covering 53,090 URLs, and from 3 models on ExpertQA, covering 168,021 URLs across 32 academic fields. It reports that 3% to 13% of URLs were hallucinated and 5% to 18% did not resolve. Deep research agents produced many more citations per query than search-augmented language models, but also hallucinated URLs at higher rates. Independent commentary on the same preprint says one vendor's deep research mode had the highest hallucination rate, 13.3%, while producing 113.1 URLs per query.

Subject matters. Non-resolving rates ranged from 5.4% in business to 11.4% in theology. Some models fabricated every address that failed to resolve, while others had many addresses that once existed and later died. Those two cases need different handling: a dead link may have been real, while an invented one never was.

A working link is not a supported claim

The most useful finding is about the gap between surface and substance. A second preprint, on source attribution, scored citations on three dimensions: does the link work, is the content relevant, and does the source support the statement. It tested 14 models on 130 research queries drawn from DeepResearch Bench and BrowseComp. Commentary on it reports that as tool calls rose from 2 to 150, fact-check accuracy fell about 42% on average across two frontier models, while link validity and relevance stayed above 92% at every depth. The authors conclude there is a disconnect between surface citation quality and factual reliability, that factual accuracy is the weakest dimension across providers, and that more search depth does not give more accurate citations. They recommend monitoring citation quality inside agent pipelines.

A practitioner recounts a concrete case. An AI answer claimed returns above 700% on coaching spend. The citation was real and the number was real, but the study measured productivity gains in large enterprises and did not transfer to the consumer coaching case. That failure survives clicking the link, survives a link checker, and survives a reviewer who spot-checks three citations.

A related benchmark paper warns that fluent writing and citation alignment can hide factual defects, which is why evaluating citations is itself slow expert work. Another found one agent that produced no citations at all on two benchmarks, and a framework that cited often but with poor faithfulness to the sources.

Fabricated references in literature reviews

Studies of generated literature reviews find the same pattern at the reference level. An article in JMIR Mental Health reports that fabrication and accuracy rates varied by disorder and by how specific the prompt was, and found no clear evidence of lower hallucination rates in newer model versions. Earlier work it cites found fabricated citation rates of 27.3% in the natural sciences and 23.4% in the humanities.

Independent commentary on a review of five deep research agents on a dermatology literature task says two of them produced roughly 47% to 50% of references with fake authors or titles. For a third product, about 95% of citations were identifiable and about 70% fully correct, yet over 50% of cited statements contained at least one subtle inaccuracy. The commentary stresses this is a worst-case stress test in a citation-dense field, not a general property of the products.

Outside research agents, a 2024 evaluation from Stanford's RegLab, as summarized in a training document, found that three retrieval-based legal research tools produced incorrect or hallucinated answers 17% to 33% of the time despite vendor claims. See why naive retrieval over regulations fails for that evidence.

A checking routine

1. Resolve every URL. Record the date. For failures, check an archive to see whether the page ever existed. 2. Verify identifiers. Look up each DOI in Crossref or OpenAlex and compare the returned title and authors with the citation. The systematic review guide covers logging sources. 3. Open the source and find the passage. Does it contain the number or statement attributed to it? Note scope: population, setting, year, units. 4. Sample by risk. Check every number that drives a decision. For the rest, sample more as research depth and citation count rise. 5. Log the result. Keep which citations were checked, by whom, and what was found.

Frequently Asked Questions

One preprint reports 3% to 13% hallucinated and 5% to 18% not resolving, across commercial models and agents. Rates vary by system, subject and date, so treat them as a range, not a rule.

Turn climate goals into a verified portfolio

TREEO connects the full carbon cycle — eligibility, simulation, real-time monitoring and registry-ready reporting — combining expert consulting with dMRV technology so the evidence exists before anyone asks for it.

Deep Research Agents: Hallucinated Citations Checked | TREEO Indonesia