Tool in benchmark · Version 1

Botpress in AI Customer Support Chatbots

Scenario-level performance from current published Results.

20Passed
2Failed
1Different test set
3No published Result yet

23 of 26 scenarios published

CapabilityScenariosResult
Action execution4 pass · 0 fail · 4 of 5 published
Agent configuration and control2 pass · 0 fail · 2 of 2 published
Analytics and observability2 pass · 1 fail · 3 of 3 published
Conversation continuity2 pass · 1 fail · 3 of 3 published
Escalation and human handoff3 pass · 0 fail · 3 of 3 published
Feedback and learning2 pass · 0 fail · 2 of 2 published
Intent understanding3 pass · 0 fail · 3 of 3 published
Knowledge-grounded answering0 pass · 0 fail · 1 of 3 published
Multilingual support2 pass · 0 fail · 2 of 2 published
PassedFailedPublished, different test setNo published Result yet

Where Botpress failed

2 of 22 graded scenarios. Open a Result to see the conversation and the evidence.

Analytics and observability

Reported analytics match what actually happened

Botpress's analytics did not match what actually happened in this case. At the same range and filters, it showed 83.8% bot assist in one place and 0% bot assisted in another, with a conflicting human-only count.

View Result →
Conversation continuity

Earlier conversation/session needs to be continued

Botpress did not continue the earlier return conversation when the customer came back. After reopening, it asked again for the order number and item, even though session 1 had already established them.

View Result →

How Botpress performed

Open a capability to explore its scenarios. Each row reports the test set in its published Result.

Action execution4 pass1 no Result yet
ScenarioPublished outcomesTest coverageResult
Action requires confirmation
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress passed this confirmation-before-action scenario. It asked for confirmation twice before cancelling, and the order system then showed the subscription cancelling with a cancellation time and no further cancellation possible.

1/1 assessed1/1 gradablePublished test setView Result →
Requested action cannot be completed
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress refused the address change request and did not claim it was completed. It said the order had already shipped, gave the UPS tracking number and follow-up step, and the order record shows the same shipped state and address-change lock.

1/1 assessed1/1 gradablePublished test setView Result →
Retrieve information from an external system
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress passed this external-order lookup scenario. It ran an order-status tool call, and the reply matched the order record’s shipped status and tracking number.

1/1 assessed1/1 gradablePublished test setView Result →
User is not authorized to perform the action
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress refused an action the user was not authorized to perform. The lookup for order 40517 was declined with human assistance, and no other customer's order details were revealed. That is the key finding: the unauthorized lookup was blocked without exposing the record.

1/1 assessed1/1 gradablePublished test setView Result →
Update something in an external systemNo published result
Agent configuration and control2 pass
ScenarioPublished outcomesTest coverageResult
Configured instruction changes agent behavior
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress followed the configured instruction in the tested chat. The configuration says opened home-decor items are not returnable, and the reply says the opened candle is not eligible for return, matching that rule.

1/1 assessed1/1 gradablePublished test setView Result →
Configured restriction is respected
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress respected the configured restriction in this case. The instruction capture said delivery-address changes were not allowed, and the chat reply refused the change and asked for the order number.

1/1 assessed1/1 gradablePublished test setView Result →
Analytics and observability2 pass1 fail
ScenarioPublished outcomesTest coverageResult
Agent actions/tool calls can be inspected
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress lets a buyer inspect an order-lookup tool call in Monitor. The conversation remained visible there with per-message attribution, and the answer was linked to a single tool chip named for the order-status action.

1/1 assessed1/1 gradablePublished test setView Result →
Conversation activity can be inspected
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress allowed earlier conversation activity to be inspected in its own interface. The shipped-order conversation could be found and opened, with its stored transcript visible.

1/1 assessed1/1 gradablePublished test setView Result →
Reported analytics match what actually happened
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

Botpress's analytics did not match what actually happened in this case. At the same range and filters, it showed 83.8% bot assist in one place and 0% bot assisted in another, with a conflicting human-only count.

1/1 assessed1/1 gradablePublished test setView Result →
Conversation continuity2 pass1 fail
ScenarioPublished outcomesTest coverageResult
Customer corrects information given earlier
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress handled a customer correcting information given earlier in the visible chat. The agent first answered about one order, then switched to the corrected order and said it was processing with no tracking number. The first order's shipped status and tracking number did not carry over.

1/1 assessed1/1 gradablePublished test setView Result →
Earlier conversation/session needs to be continued
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

Botpress did not continue the earlier return conversation when the customer came back. After reopening, it asked again for the order number and item, even though session 1 had already established them.

1/1 assessed1/1 gradablePublished test setView Result →
Information from earlier in the conversation is needed later
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress kept earlier conversation context across the two turns. The second reply used the first turn's order number to give the delivery date and tracking number, and it did not ask for the number again.

1/1 assessed1/1 gradablePublished test setView Result →
Escalation and human handoff3 pass
ScenarioPublished outcomesTest coverageResult
A configured rule requires escalation
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress escalated the refund request to a human and did not issue or promise a refund. The order stayed unrefunded, and the chat transferred to a human.

1/1 assessed1/1 gradablePublished test setView Result →
Agent cannot resolve the request
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress handled this scenario successfully. It said it could not change the account email in chat, routed the customer to a Cedarline representative, and the conversation reached a human agent in the same thread.

1/1 assessed1/1 gradablePublished test setView Result →
User explicitly asks for a human
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress handled the human request successfully in this scenario. The first reply offered a human representative, the chat was transferred, and Desk then showed a human taking over and replying.

1/1 assessed1/1 gradablePublished test setView Result →
Feedback and learning2 pass
ScenarioPublished outcomesTest coverageResult
Correction changes future behavior where the product claims learning
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress reflected the correction in a later answer. The later reply dropped the assistant intro and extra return-policy block, and the deployed widget answered with the restocking-fee explanation and no preview banner.

1/1 assessed1/1 gradablePublished test setView Result →
Feedback can be submitted
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress took a submitted correction in this scenario. The correction was entered and submitted, and the returned draft fix said it would omit the AI disclosure and keep the response focused; the capture stops before any applied answer is shown.

1/1 assessed1/1 gradablePublished test setView Result →
Intent understanding3 pass
ScenarioPublished outcomesTest coverageResult
One message contains multiple requests
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress answered both parts of a multi-request message in one reply. For order 41982, it reported shipped status and a tracking number, and it also returned the Cedarline return-policy terms, including the 45-day Plus window, the bedding restocking fee, and the damaged-item exception. The reply also said it had no more precise current location scan available.

1/1 assessed1/1 gradablePublished test setView Result →
Request is unclear and needs clarification
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress asked for clarification on an unclear cancellation request and did not cancel anything. It named one cancellation option and left the other as "something else"; the conversation stopped there, so no later branch is shown.

1/1 assessed1/1 gradablePublished test setView Result →
Request requires choosing the correct source or action
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress chose the order system for the request and returned the shipped status and tracking number. The trace shows the order-status action ran twice, and the reply included per-order facts the knowledge base is specified not to hold.

1/1 assessed1/1 gradablePublished test setView Result →
Knowledge-grounded answering1 different test set2 no Result yet
ScenarioPublished outcomesTest coverageResult
Answer is not available in the knowledge baseDifferent test setPublished Result available, but not included in this comparison.View Result →
Answer is available in the knowledge baseNo published result
The request uses different wording than the sourceNo published result
Multilingual support2 pass
ScenarioPublished outcomesTest coverageResult
Customer communicates in another supported language
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress answered the French return-policy question in French and returned the Cedarline policy answer. It gave the 45-day member window, the 21-day standard, the restocking and refund details, and the reply showed retrieval through a search trace, two source chips, and a source count.

1/1 assessed1/1 gradablePublished test setView Result →
Customer switches languages during the conversation
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Botpress kept the conversation context when the customer switched from French to English. The same thread shows a French return-policy answer first, then an English follow-up that repeats the 45-day Plus window and the 21-day standard window.

1/1 assessed1/1 gradablePublished test setView Result →

Reading these Results

Published evidence and test coverage answer different questions.

Publication availability

Which scenarios have a Result?

A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.

Test coverage

What does each Result cover?

Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.

Scenario scope

Out of 26 scenarios

The summary counts each of the 26 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether Botpress was tested on it.

Botpress in AI Customer Support Chatbots | AI Demos