Dify in Managed RAG
Scenario-level performance from current published Results.
8 of 15 scenarios published
Where Dify failed
1 of 8 graded scenarios. Open a Result to see the conversation and the evidence.
A cleanly scanned document
Dify did not turn the clean scan into accurate text. The scan had no extractable text, yet the document was marked completed and available with a word count, and the lookup returned other records instead of the scan's fact; the Pipeline view was present but unopened.
View Result →How Dify performed
Open a capability to explore its scenarios. Each row reports the test set in its published Result.
Citations1 pass
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| A claim in the answer traced back to its source | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedDify passed this citation check. The answer said acknowledgement would happen within 90 minutes by a named duty engineer, and the top citation pointed to that same passage. The other citations also came from the same policy. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Data Ingestion2 pass1 fail1 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| A bad file in the upload | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedDify reports both unprocessable uploads as errors. Each file ends in error with its own PDFium message, and neither is completed or silent. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| A cleanly scanned document | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedDify did not turn the clean scan into accurate text. The scan had no extractable text, yet the document was marked completed and available with a word count, and the lookup returned other records instead of the scan's fact; the Pipeline view was present but unopened. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| A simple, clearly formatted table | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedDify passes this simple table lookup scenario. The chat answer returned £487 (excluding VAT), and the supporting passage showed that value in the Extended row under Out-of-hours callout; the retrieval output showed the same table passage. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| An ordinary digital text document | No published result | ||
Grounded Generation1 pass2 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| The retrieved material doesn't support an answer | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedDify handled the unsupported question by saying it did not know and giving no leave amount or policy. The retrieved passages were unrelated to parental leave and instead covered handbook and field-service material. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The retrieved material supports an answer | No published result | ||
| The retrieved sources disagree with each other | No published result | ||
Knowledge Sync1 pass2 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| New content is added to the source | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedDify made newly added source content findable after the update path completed. The new document was created through the ingestion endpoint, indexing reached completed before retrieval, and the retrieval call returned that same document first; two additional segments also pointed to it. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Content is removed from the source | No published result | ||
| Existing content is changed | No published result | ||
Retrieval2 pass2 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Answer is available in the knowledge base | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedDify passed this retrieval task. The answer-bearing passage was first in the results, and all four returned segments came from the same document. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The request uses different wording than the source | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedDify retrieved the supporting passage when asked about paid time off, while the employee handbook calls it annual leave. The passage gives full-time employees 26 days per complete leave year. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Answer is not available in the knowledge base | No published result | ||
| Retrieval is scoped to part of the documents | No published result | ||
Reading these Results
Published evidence and test coverage answer different questions.
Which scenarios have a Result?
A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.
What does each Result cover?
Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.
Out of 15 scenarios
The summary counts each of the 15 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether Dify was tested on it.