Tool in benchmark · Version 1

Dify in Managed RAG

Scenario-level performance from current published Results.

7Passed
1Failed
7No published Result yet

8 of 15 scenarios published

CapabilityScenariosResult
Citations1 pass · 0 fail · 1 of 1 published
Data Ingestion2 pass · 1 fail · 3 of 4 published
Grounded Generation1 pass · 0 fail · 1 of 3 published
Knowledge Sync1 pass · 0 fail · 1 of 3 published
Retrieval2 pass · 0 fail · 2 of 4 published
PassedFailedPublished, different test setNo published Result yet

Where Dify failed

1 of 8 graded scenarios. Open a Result to see the conversation and the evidence.

Data Ingestion

A cleanly scanned document

Dify did not turn the clean scan into accurate text. The scan had no extractable text, yet the document was marked completed and available with a word count, and the lookup returned other records instead of the scan's fact; the Pipeline view was present but unopened.

View Result →

How Dify performed

Open a capability to explore its scenarios. Each row reports the test set in its published Result.

Citations1 pass
ScenarioPublished outcomesTest coverageResult
A claim in the answer traced back to its source
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Dify passed this citation check. The answer said acknowledgement would happen within 90 minutes by a named duty engineer, and the top citation pointed to that same passage. The other citations also came from the same policy.

1/1 assessed1/1 gradablePublished test setView Result →
Data Ingestion2 pass1 fail1 no Result yet
ScenarioPublished outcomesTest coverageResult
A bad file in the upload
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Dify reports both unprocessable uploads as errors. Each file ends in error with its own PDFium message, and neither is completed or silent.

1/1 assessed1/1 gradablePublished test setView Result →
A cleanly scanned document
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

Dify did not turn the clean scan into accurate text. The scan had no extractable text, yet the document was marked completed and available with a word count, and the lookup returned other records instead of the scan's fact; the Pipeline view was present but unopened.

1/1 assessed1/1 gradablePublished test setView Result →
A simple, clearly formatted table
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Dify passes this simple table lookup scenario. The chat answer returned £487 (excluding VAT), and the supporting passage showed that value in the Extended row under Out-of-hours callout; the retrieval output showed the same table passage.

1/1 assessed1/1 gradablePublished test setView Result →
An ordinary digital text documentNo published result
Grounded Generation1 pass2 no Result yet
ScenarioPublished outcomesTest coverageResult
The retrieved material doesn't support an answer
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Dify handled the unsupported question by saying it did not know and giving no leave amount or policy. The retrieved passages were unrelated to parental leave and instead covered handbook and field-service material.

1/1 assessed1/1 gradablePublished test setView Result →
The retrieved material supports an answerNo published result
The retrieved sources disagree with each otherNo published result
Knowledge Sync1 pass2 no Result yet
ScenarioPublished outcomesTest coverageResult
New content is added to the source
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Dify made newly added source content findable after the update path completed. The new document was created through the ingestion endpoint, indexing reached completed before retrieval, and the retrieval call returned that same document first; two additional segments also pointed to it.

1/1 assessed1/1 gradablePublished test setView Result →
Content is removed from the sourceNo published result
Existing content is changedNo published result
Retrieval2 pass2 no Result yet
ScenarioPublished outcomesTest coverageResult
Answer is available in the knowledge base
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Dify passed this retrieval task. The answer-bearing passage was first in the results, and all four returned segments came from the same document.

1/1 assessed1/1 gradablePublished test setView Result →
The request uses different wording than the source
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Dify retrieved the supporting passage when asked about paid time off, while the employee handbook calls it annual leave. The passage gives full-time employees 26 days per complete leave year.

1/1 assessed1/1 gradablePublished test setView Result →
Answer is not available in the knowledge baseNo published result
Retrieval is scoped to part of the documentsNo published result

Reading these Results

Published evidence and test coverage answer different questions.

Publication availability

Which scenarios have a Result?

A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.

Test coverage

What does each Result cover?

Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.

Scenario scope

Out of 15 scenarios

The summary counts each of the 15 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether Dify was tested on it.

Dify in Managed RAG | AI Demos