blog/systematic-reviews-environmental-evidence-prisma-roses-llm-hero
← Back to blog

Systematic Reviews for Environmental Evidence: PRISMA, ROSES and What AI Screening Can Actually Do

Technology•13 November 2026•TREEO Indonesia•4 min read

A systematic review is a literature review that someone else can repeat. It states the question, the sources, the search strings, the screening rules and the counts, so a reader can see what was found and what was left out. That is the reason decision makers ask for them. It is also why they are slow, and why AI tools are tempting.

Why systematic reviews are hard

67 weeks

average time to complete a systematic review

0.98 and 0.27

recall and precision reported for one LLM screening study

39.5%

pooled sensitivity of one AI search tool in an independent test

Pick the right reporting standard

PRISMA 2020 is the best-known reporting checklist, and it grew out of medical reviews. A 2018 paper on reporting in environmental evidence looked at the earlier QUOROM and PRISMA (2009) standards, noted that they were developed for medicine, and highlighted 12 key problems with applying PRISMA to conservation and environmental management.

The environmental answer is ROSES, short for RepOrting standards for Systematic Evidence Syntheses. The Collaboration for Environmental Evidence announced it on 1 March 2018, and the article introducing it was by Haddaway, Macura, Whaley and Pullin. It offers forms for review and map protocols and reports, plus a flow-diagram template. It was designed for environmental topics but is described as broadly applicable across disciplines. EQUATOR lists an update under development, registered on 28 October 2026, so versions matter.

Two further points on process. A protocol for a Collaboration for Environmental Evidence review must be registered and peer reviewed through its editorial process, so a draft from software is not a substitute. And evaluations of published reviews show why standards matter: one evaluation of more than 1,000 evidence syntheses published between 2018 and 2020 found that most had problems with transparency and replicability.

Frame the question

FrameworkElementsUse it for
PICOPopulation, Intervention, Comparator, OutcomeEffect of an intervention
PECOExposure replaces InterventionEffect of an exposure; recognised by the Collaboration for Environmental Evidence
SPIDERSample, Phenomenon of Interest, Design, Evaluation, Research typeQualitative and mixed-methods questions

Whichever you choose, say why. The PRISMA 2020 expanded checklist asks reviewers to describe any search structure that is not PICO-based.

What must be logged

PRISMA 2020 is specific about the search record:

- Item 6 requires every database, register, website, organisation and reference list searched or consulted, and the date each was last searched. - Item 7 requires the full search strategy for each source, including filters and limits. - Item 8 requires the selection process: how many reviewers screened each record, whether they worked independently, and details of any automation tools. - Item 16a requires results from records identified to studies included, ideally in a flow diagram.

The checklist is published under CC BY 4.0, so the wording can be reused with attribution. If an LLM screened any records, item 8 says it must be reported as an automation tool.

For sources, OpenAlex offers free basic use without a key, and a free key raises the daily budget tenfold, according to its documentation. Crossref can verify DOIs. Whatever you use, log the date.

What AI screening has been measured to do

Screening thousands of titles and abstracts is the step where AI help is most tempting. The evidence is mixed:

Study or resultFinding
Review of prior LLM screening workUp to, and sometimes beyond, master's-student level, but not PhD level
One reported resultRecall 0.98 with precision 0.27
34,528 studies across 24 reviews and 9 LLMsSmall average differences among top models, much larger differences between screening tasks
Ten reviews, four prompting methodsA dynamic few-shot method gave the highest F1
A 2026 workflow study on 1,131 recordsLLMs suit validated, auditable, human-supervised workflows better than autonomous exclusion

High recall at low precision is useful: the model flags nearly everything relevant, and people discard the false alarms. Performance depends on the topic, so validating on your own review beats trusting a headline figure. The 2026 workflow study adds that no single metric describes a screening workflow. Agreement, recovery of eligible records and workload saved are different things, and each should be reported.

Active-learning tools raise a related caution. ASReview can reduce screening burden by up to 95% while keeping recall high, but choosing where to stop is subjective. Commentary on a 2026 study (Kempny and colleagues) calls common stopping habits, such as stopping after 100 consecutive irrelevant records or after 5% of the dataset, highly questionable. The study itself found the extra screening needed ranged from 2% to 61% depending on the dataset.

AI search tools are not a search strategy

An independent study tested whether the AI research tool Elicit could find the studies included in four published reviews: one pharmacology, one surgery and two public health. Its pooled sensitivity was 39.5% (range 25.5% to 69.2%), against 94.5% for traditional searching. Precision was higher, 41.8% against 7.55%. Sensitivity was lowest in public-health reviews and highest in drug-intervention reviews. The authors concluded that the tool gives quick, high-precision results and finds some unique studies, but should be a supplement, not a standalone method.

Treat vendor claims on speed and coverage as claims. The study notes that most evaluations of such tools come from their developers and read like marketing. For the tool Consensus, a research-library guide states that it cannot be used for systematic reviews or meta-analyses because its retrieval is not reproducible or auditable in the PRISMA sense.

A workable routine

1. Choose ROSES or PRISMA and the question framework, and record why. 2. Search several sources, log each with its date, and keep the full strings. 3. Let an LLM pre-screen if you wish, but log it as an automation tool, keep humans in the loop, and validate recall on a sample you screen yourself. 4. Do not let the model exclude records alone. 5. Check every cited reference against its DOI before it enters the report.

Frequently Asked Questions

It is widely used, but a 2018 paper on the 2009 version highlighted 12 key problems in applying PRISMA to conservation and environmental management. ROSES was designed for environmental evidence syntheses.

Turn climate goals into a verified portfolio

TREEO connects the full carbon cycle — eligibility, simulation, real-time monitoring and registry-ready reporting — combining expert consulting with dMRV technology so the evidence exists before anyone asks for it.

Related Articles

Systematic Reviews: PRISMA, ROSES and AI Screening | TREEO Indonesia