Vectara in Managed RAG
Scenario-level performance from current published Results.
5 of 15 scenarios published
Vectara passed every graded scenario it has a published Result for.
How Vectara performed
Open a capability to explore its scenarios. Each row reports the test set in its published Result.
Citations1 pass
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| A claim in the answer traced back to its source | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectara cited a passage that supported its answer's claim. It said a named duty engineer would acknowledge the out-of-hours fault within 90 minutes, and the cited passage said the same. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Data Ingestion1 pass3 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| An ordinary digital text document | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectara returned the requested fact in the document's own words. In both configurations, the top result was the passage stating a 22-month recalibration interval, and the next results were different warranty passages; both captures showed HTTP 200. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| A bad file in the upload | No published result | ||
| A cleanly scanned document | No published result | ||
| A simple, clearly formatted table | No published result | ||
Knowledge Sync1 pass2 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Content is removed from the source | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectara stopped using the deleted source and declined to answer the question. Across four repeat queries, the deleted document was absent from the top ten results and the supplier was not cited. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Existing content is changed | No published result | ||
| New content is added to the source | No published result | ||
Retrieval2 pass2 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Answer is available in the knowledge base | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectara returned the answer from the knowledge base for this naturally phrased question. The out-of-hours passage ranked first in three independent captures, including a live notebook run, while the business-hours near-miss stayed lower each time. The corpus view shows the relevant document was present. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The request uses different wording than the source | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedVectara returned the matching passage even though the question used different wording from the source. In both retrieval runs, the annual leave passage was rank 1, and the returned passages did not contain the query's alternative terms. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Answer is not available in the knowledge base | No published result | ||
| Retrieval is scoped to part of the documents | No published result | ||
Grounded Generation3 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| The retrieved material doesn't support an answer | No published result | ||
| The retrieved material supports an answer | No published result | ||
| The retrieved sources disagree with each other | No published result | ||
Reading these Results
Published evidence and test coverage answer different questions.
Which scenarios have a Result?
A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.
What does each Result cover?
Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.
Out of 15 scenarios
The summary counts each of the 15 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether Vectara was tested on it.