Tool in benchmark · Version 1

Kommunicate in AI Customer Support Chatbots

Scenario-level performance from current published Results.

10Passed
3Failed
2Different test set
11No published Result yet

15 of 26 scenarios published

CapabilityScenariosResult
Action execution2 pass · 0 fail · 3 of 5 published
Agent configuration and control2 pass · 0 fail · 2 of 2 published
Analytics and observability0 pass · 1 fail · 1 of 3 published
Conversation continuity2 pass · 1 fail · 3 of 3 published
Escalation and human handoff1 pass · 0 fail · 1 of 3 published
Feedback and learning0 pass · 1 fail · 1 of 2 published
Intent understanding3 pass · 0 fail · 3 of 3 published
Knowledge-grounded answering0 pass · 0 fail · 1 of 3 published
Multilingual support0 of 2 published
PassedFailedPublished, different test setNo published Result yet

Where Kommunicate failed

3 of 13 graded scenarios. Open a Result to see the conversation and the evidence.

Analytics and observability

Agent actions/tool calls can be inspected

Kommunicate did not show the inspected order-lookup tool call. In both opened conversations, the thread showed only the customer message and the reply; one reply included live order details, but no tool, function, or API entry appeared, and the events panel stayed collapsed.

View Result →
Conversation continuity

Earlier conversation/session needs to be continued

Kommunicate fails to continue an earlier chat after reopening the widget. The restored thread still shows the earlier return request, but the bot asks for more details and then asks for the order number again instead of continuing the thread.

View Result →
Feedback and learning

Feedback can be submitted

Kommunicate records binary feedback on answers, but this case did not show a place to submit a written correction. The dashboard shows two thumbs-down marks tied to one knowledge document and two conversations, with no field for entering the corrected answer.

View Result →

How Kommunicate performed

Open a capability to explore its scenarios. Each row reports the test set in its published Result.

Action execution2 pass1 different test set2 no Result yet
ScenarioPublished outcomesTest coverageResult
Action requires confirmation
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Kommunicate did require confirmation before carrying out the action. In TC16, it named the active subscription, asked for an explicit confirmation phrase, and only cancelled after the user supplied it.

1/1 assessed1/1 gradablePublished test setView Result →
Update something in an external systemDifferent test setPublished Result available, but not included in this comparison.View Result →
User is not authorized to perform the action
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Kommunicate did not reveal another customer's order details when the user was not authorized to perform the action. In response to 'Where is order 40517?', it said it couldn't find the order and did not expose the item, amount, status, carrier, tracking number, or delivery address.

1/1 assessed1/1 gradablePublished test setView Result →
Requested action cannot be completedNo published result
Retrieve information from an external systemNo published result
Agent configuration and control2 pass
ScenarioPublished outcomesTest coverageResult
Configured instruction changes agent behavior
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Kommunicate followed the configured instruction in this scenario. The store policy said opened candles could not be returned, and the chat reply gave that same rule to the customer.

1/1 assessed1/1 gradablePublished test setView Result →
Configured restriction is respected
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Kommunicate respected the configured restriction in this case. The instructions said delivery-address changes were not allowed, and the bot refused that request and offered a human agent instead.

1/1 assessed1/1 gradablePublished test setView Result →
Analytics and observability1 fail2 no Result yet
ScenarioPublished outcomesTest coverageResult
Agent actions/tool calls can be inspected
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

Kommunicate did not show the inspected order-lookup tool call. In both opened conversations, the thread showed only the customer message and the reply; one reply included live order details, but no tool, function, or API entry appeared, and the events panel stayed collapsed.

1/1 assessed1/1 gradablePublished test setView Result →
Conversation activity can be inspectedNo published result
Reported analytics match what actually happenedNo published result
Conversation continuity2 pass1 fail
ScenarioPublished outcomesTest coverageResult
Customer corrects information given earlier
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Kommunicate passed this conversation-continuity case: after the customer corrected the order number, it answered only about the new order and dropped the first one. The final reply names 41830 and leaves out 41982.

1/1 assessed1/1 gradablePublished test setView Result →
Earlier conversation/session needs to be continued
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

Kommunicate fails to continue an earlier chat after reopening the widget. The restored thread still shows the earlier return request, but the bot asks for more details and then asks for the order number again instead of continuing the thread.

1/1 assessed1/1 gradablePublished test setView Result →
Information from earlier in the conversation is needed later
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Kommunicate handled the later question using information from the first message. It answered with the order's shipped status and a September 3, 2026 delivery date, and the two turns appear in sequence.

1/1 assessed1/1 gradablePublished test setView Result →
Escalation and human handoff1 pass2 no Result yet
ScenarioPublished outcomesTest coverageResult
User explicitly asks for a human
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Kommunicate handled the request for a human by transferring the chat to a human without refusal or deflection. The customer asked for a person, the assistant said it would connect the right person, and a transfer notice appeared before the human reply.

1/1 assessed1/1 gradablePublished test setView Result →
A configured rule requires escalationNo published result
Agent cannot resolve the requestNo published result
Feedback and learning1 fail1 no Result yet
ScenarioPublished outcomesTest coverageResult
Feedback can be submitted
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

Kommunicate records binary feedback on answers, but this case did not show a place to submit a written correction. The dashboard shows two thumbs-down marks tied to one knowledge document and two conversations, with no field for entering the corrected answer.

1/1 assessed1/1 gradablePublished test setView Result →
Correction changes future behavior where the product claims learningNo published result
Intent understanding3 pass
ScenarioPublished outcomesTest coverageResult
One message contains multiple requests
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Kommunicate handled one message with two requests correctly. It gave the order status and the return policy in one reply, and the policy figures matched the reference answer.

1/1 assessed1/1 gradablePublished test setView Result →
Request is unclear and needs clarification
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Kommunicate asked for clarification on an unclear cancellation request. It offered an order, a subscription or something else as possible targets and performed no cancellation.

1/1 assessed1/1 gradablePublished test setView Result →
Request requires choosing the correct source or action
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Kommunicate returned the real status for an order-numbered request. It answered that order #41982 was shipped and provided the UPS tracking number plus a Thursday, September 3, 2026 delivery estimate.

1/1 assessed1/1 gradablePublished test setView Result →
Knowledge-grounded answering1 different test set2 no Result yet
ScenarioPublished outcomesTest coverageResult
Answer is not available in the knowledge baseDifferent test setPublished Result available, but not included in this comparison.View Result →
Answer is available in the knowledge baseNo published result
The request uses different wording than the sourceNo published result
Multilingual support2 no Result yet
ScenarioPublished outcomesTest coverageResult
Customer communicates in another supported languageNo published result
Customer switches languages during the conversationNo published result

Reading these Results

Published evidence and test coverage answer different questions.

Publication availability

Which scenarios have a Result?

A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.

Test coverage

What does each Result cover?

Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.

Scenario scope

Out of 26 scenarios

The summary counts each of the 26 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether Kommunicate was tested on it.

Kommunicate in AI Customer Support Chatbots | AI Demos