Managed RAG
This benchmark covers managed RAG and RAG-as-a-Service products that let a buyer upload or connect private documents and get a query API back.
Benchmark overview
What is included and excluded
Managed RAG evaluates services that turn private documents into a query API, so buyers can compare buying a managed service with building the pipeline themselves across grounding, citation and refresh behaviour.
The buying question is build versus buy: whether a managed service is better than assembling and running the retrieval pipeline yourself. Readers learn whether the service makes private knowledge reachable, stays grounded in what it retrieved, cites the supporting source, and catches up when the source changes.
In scope
- Services where you upload or connect documents and get a query API back.
- Services that only retrieve and services that also generate answers.
- Connectors and automatic updates.
- Retrieval machinery sold as a service, including hybrid search, reranking and metadata filters.
Out of scope
- Raw vector databases.
- Website chat widgets.
- Web search.
- Document parsing on its own.
- Multi-tenant isolation and per-user permissions.
- Structured Document Extraction's querying over already extracted records; here Retrieval works on document text.
Participating tools
How each tool did on the 15 scenarios in this benchmark. Each cell is one scenario, in the same order for every tool and grouped by capability; Pass and Fail come from published Results.
| Tool | Scenario results · 15 scenarios in 5 capabilities | Passed | Failed | Explore |
|---|---|---|---|---|
| FutureSmart Document Intelligence | 12of 15 graded | 3 | View tool → | |
| Denser AI | 9of 9 graded | 0 | View tool → | |
| Dify | Failed: A cleanly scanned document | 7of 8 graded | 1 | View tool → |
| RAGFlow | 6of 6 graded | 0 | View tool → | |
| Vectara | 5of 5 graded | 0 | View tool → | |
| Vectorize.io | 2of 2 graded | 0 | View tool → | |
| 2 more toolsin the roster | No published Results yet for Llama Cloud · Onyx.Each tool gets its own row here once its first Result is published. | |||
Capabilities & scenarios
15 scenarios grouped by 5 capabilities. Open a group to explore its scenarios in this benchmark.
Data Ingestion4 scenarios
The handed-over files become reachable, and the service reports when ingestion did not work.
Capability in this benchmark → · Global definition →
- An ordinary digital text document4 tools with published Results
- A simple, clearly formatted table3 tools with published Results
- A cleanly scanned document2 tools with published Results
- A bad file in the upload2 tools with published Results
Retrieval4 scenarios
The passage that answers the question comes back from the documents when it exists, and a missing answer is discoverable when it does not.
Capability in this benchmark → · Global definition →
- Answer is available in the knowledge base6 tools with published Results
- Answer is not available in the knowledge base2 tools with published Results
- The request uses different wording than the source5 tools with published Results
- Retrieval is scoped to part of the documents1 tool with a published Result
Grounded Generation3 scenarios
The answer stays faithful to the passages the system itself retrieved, including declining when those passages do not support an answer.
Capability in this benchmark → · Global definition →
- The retrieved material supports an answer2 tools with published Results
- The retrieved material doesn't support an answer4 tools with published Results
- The retrieved sources disagree with each other2 tools with published Results
Citations1 scenario
The cited passage genuinely supports the claim the answer makes, even when the answer itself is wrong.
Capability in this benchmark → · Global definition →
- A claim in the answer traced back to its source4 tools with published Results
Knowledge Sync3 scenarios
The indexed state catches up with the source after content is added, changed, or removed.
Capability in this benchmark → · Global definition →
- New content is added to the source4 tools with published Results
- Existing content is changed2 tools with published Results
- Content is removed from the source2 tools with published Results
Results overview
Current published evidence in this benchmark.
Publication availability is separate from test coverage and unpublished research progress.
How the benchmark works
A public summary of the evaluation method. The same defined scope and evidence standard apply to every tool assessed under this version.
Unchanged vendor default
Every tool is tested on its own default setup, unchanged.
Build-it-yourself control
A build-it-yourself pipeline runs as a subject alongside the managed services.
Judge retrieval against the documents
Retrieval is judged against the documents: the supporting passage must come back when it exists.
Judge generation against retrieved material
Grounded generation is judged against what the tool actually retrieved, not against the ideal answer.
Do not score one miss twice
A retrieval miss is not charged again in generation; declining is correct when the retrieved material does not support an answer.
Judge citations by support
Citations are judged by whether the cited passage supports the claim, not by whether the answer is correct.
Correct citation, wrong answer
A correct citation can point to the source of a wrong answer; a citation attached to an invented claim is aggravating.
Unsupported capabilities score zero
A capability a tool does not support scores 0, and the 0 stays in the denominator.
End-to-end answer correctness is separate
End-to-end answer correctness is reported separately as a composite outcome.
Follow the documented refresh path
Each Knowledge Sync test follows the tool's own documented update path to completion; time is recorded, and non-completion counts as non-convergence.
Resources and fixtures
The registered material and systems that create a consistent test environment for this benchmark.
Fenwake test documents for grounded Q&A
A fictional document corpus used to evaluate managed RAG systems.