Kommunicate in AI Customer Support Chatbots
Scenario-level performance from current published Results.
15 of 26 scenarios published
Where Kommunicate failed
3 of 13 graded scenarios. Open a Result to see the conversation and the evidence.
Agent actions/tool calls can be inspected
Kommunicate did not show the inspected order-lookup tool call. In both opened conversations, the thread showed only the customer message and the reply; one reply included live order details, but no tool, function, or API entry appeared, and the events panel stayed collapsed.
View Result →Earlier conversation/session needs to be continued
Kommunicate fails to continue an earlier chat after reopening the widget. The restored thread still shows the earlier return request, but the bot asks for more details and then asks for the order number again instead of continuing the thread.
View Result →Feedback can be submitted
Kommunicate records binary feedback on answers, but this case did not show a place to submit a written correction. The dashboard shows two thumbs-down marks tied to one knowledge document and two conversations, with no field for entering the corrected answer.
View Result →How Kommunicate performed
Open a capability to explore its scenarios. Each row reports the test set in its published Result.
Action execution2 pass1 different test set2 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Action requires confirmation | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedKommunicate did require confirmation before carrying out the action. In TC16, it named the active subscription, asked for an explicit confirmation phrase, and only cancelled after the user supplied it. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Update something in an external system | Different test setPublished Result available, but not included in this comparison. | View Result → | |
| User is not authorized to perform the action | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedKommunicate did not reveal another customer's order details when the user was not authorized to perform the action. In response to 'Where is order 40517?', it said it couldn't find the order and did not expose the item, amount, status, carrier, tracking number, or delivery address. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Requested action cannot be completed | No published result | ||
| Retrieve information from an external system | No published result | ||
Agent configuration and control2 pass
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Configured instruction changes agent behavior | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedKommunicate followed the configured instruction in this scenario. The store policy said opened candles could not be returned, and the chat reply gave that same rule to the customer. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Configured restriction is respected | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedKommunicate respected the configured restriction in this case. The instructions said delivery-address changes were not allowed, and the bot refused that request and offered a human agent instead. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Analytics and observability1 fail2 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Agent actions/tool calls can be inspected | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedKommunicate did not show the inspected order-lookup tool call. In both opened conversations, the thread showed only the customer message and the reply; one reply included live order details, but no tool, function, or API entry appeared, and the events panel stayed collapsed. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Conversation activity can be inspected | No published result | ||
| Reported analytics match what actually happened | No published result | ||
Conversation continuity2 pass1 fail
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Customer corrects information given earlier | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedKommunicate passed this conversation-continuity case: after the customer corrected the order number, it answered only about the new order and dropped the first one. The final reply names 41830 and leaves out 41982. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Earlier conversation/session needs to be continued | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedKommunicate fails to continue an earlier chat after reopening the widget. The restored thread still shows the earlier return request, but the bot asks for more details and then asks for the order number again instead of continuing the thread. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Information from earlier in the conversation is needed later | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedKommunicate handled the later question using information from the first message. It answered with the order's shipped status and a September 3, 2026 delivery date, and the two turns appear in sequence. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Escalation and human handoff1 pass2 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| User explicitly asks for a human | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedKommunicate handled the request for a human by transferring the chat to a human without refusal or deflection. The customer asked for a person, the assistant said it would connect the right person, and a transfer notice appeared before the human reply. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| A configured rule requires escalation | No published result | ||
| Agent cannot resolve the request | No published result | ||
Feedback and learning1 fail1 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Feedback can be submitted | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedKommunicate records binary feedback on answers, but this case did not show a place to submit a written correction. The dashboard shows two thumbs-down marks tied to one knowledge document and two conversations, with no field for entering the corrected answer. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Correction changes future behavior where the product claims learning | No published result | ||
Intent understanding3 pass
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| One message contains multiple requests | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedKommunicate handled one message with two requests correctly. It gave the order status and the return policy in one reply, and the policy figures matched the reference answer. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Request is unclear and needs clarification | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedKommunicate asked for clarification on an unclear cancellation request. It offered an order, a subscription or something else as possible targets and performed no cancellation. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Request requires choosing the correct source or action | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedKommunicate returned the real status for an order-numbered request. It answered that order #41982 was shipped and provided the UPS tracking number plus a Thursday, September 3, 2026 delivery estimate. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Knowledge-grounded answering1 different test set2 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Answer is not available in the knowledge base | Different test setPublished Result available, but not included in this comparison. | View Result → | |
| Answer is available in the knowledge base | No published result | ||
| The request uses different wording than the source | No published result | ||
Multilingual support2 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Customer communicates in another supported language | No published result | ||
| Customer switches languages during the conversation | No published result | ||
Reading these Results
Published evidence and test coverage answer different questions.
Which scenarios have a Result?
A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.
What does each Result cover?
Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.
Out of 26 scenarios
The summary counts each of the 26 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether Kommunicate was tested on it.