AI Customer Support Chatbots
This benchmark evaluates AI support agents for businesses that need customer questions answered and customer requests handled in connected systems.
Benchmark overview
What is included and excluded
Evaluates AI support agents on grounded answers, intent understanding, conversation continuity, human handoff, connected-system actions, multilingual support, configuration and control, analytics and observability, and feedback-driven learning.
A reader learns how a product handles the core support capabilities buyers care about: grounded answers, intent understanding, continuity across turns and sessions, human handoff, connected actions, multilingual use, configuration, observability, and feedback-driven learning.
In scope
- AI support agents for customer support.
- Grounded answers from the business’s own knowledge base.
- Intent understanding before answering or acting.
- Conversation continuity across turns and sessions.
- Escalation and human handoff.
- Actions in connected systems.
- Multilingual support.
- Agent configuration and control.
- Analytics and observability.
- Feedback-driven learning.
Out of scope
- General RAG chatbots with no support workflow.
- Live chat tools with no AI.
- Voice support agents.
- Prompt-injection resistance.
Participating tools
How each tool did on the 26 scenarios in this benchmark. Each cell is one scenario, in the same order for every tool and grouped by capability; Pass and Fail come from published Results.
| Tool | Scenario results · 26 scenarios in 9 capabilities | Passed | Failed | Explore |
|---|---|---|---|---|
| Botpress | 20of 22 graded | 2 | View tool → | |
| Intercom Fin | 19of 20 graded | 1 | View tool → | |
| Freshdesk Freddy | 15of 17 graded | 2 | View tool → | |
| Zendesk AI | 12of 16 graded | 4 | View tool → | |
| Kommunicate | 10of 13 graded | 3 | View tool → | |
| Respond.io | 5of 5 graded | 0 | View tool → | |
| Tidio Lyro | 1of 1 graded | 0 | View tool → | |
| 7 more toolsin the roster | No published Results yet for Ada · Chatbase · CustomGPT.ai · Decagon · FS Agent (DIY control) · Gorgias · Help Scout.Each tool gets its own row here once its first Result is published. | |||
Capabilities & scenarios
26 scenarios grouped by 9 capabilities. Open a group to explore its scenarios in this benchmark.
Knowledge-grounded answering3 scenarios
The agent answers customer questions from the business’s own knowledge base rather than guessing from model assumptions.
Capability in this benchmark → · Global definition →
- Answer is available in the knowledge baseNo published results
- Answer is not available in the knowledge base6 tools with published Results
- The request uses different wording than the sourceNo published results
Intent understanding3 scenarios
The agent works out what the customer actually wants before it answers or acts.
Capability in this benchmark → · Global definition →
- Request requires choosing the correct source or action6 tools with published Results
- One message contains multiple requests5 tools with published Results
- Request is unclear and needs clarification4 tools with published Results
Conversation continuity3 scenarios
The agent keeps track of earlier context across turns and sessions when later replies depend on it.
Capability in this benchmark → · Global definition →
- Information from earlier in the conversation is needed later5 tools with published Results
- Customer corrects information given earlier5 tools with published Results
- Earlier conversation/session needs to be continued5 tools with published Results
Escalation and human handoff3 scenarios
The agent knows when a conversation should go to a human and hands it over properly.
Capability in this benchmark → · Global definition →
- User explicitly asks for a human6 tools with published Results
- A configured rule requires escalation4 tools with published Results
- Agent cannot resolve the request5 tools with published Results
Action execution5 scenarios
The agent can look things up and make changes in connected systems such as orders, subscriptions, or tickets.
Capability in this benchmark → · Global definition →
- Retrieve information from an external system4 tools with published Results
- Update something in an external system2 tools with published Results
- Requested action cannot be completed4 tools with published Results
- Action requires confirmation4 tools with published Results
- User is not authorized to perform the action5 tools with published Results
Multilingual support2 scenarios
The agent handles customers who do not write in English and can follow language switches during a conversation.
Capability in this benchmark → · Global definition →
- Customer communicates in another supported language3 tools with published Results
- Customer switches languages during the conversation2 tools with published Results
Agent configuration and control2 scenarios
The agent follows configured instructions and respects configured restrictions.
Capability in this benchmark → · Global definition →
- Configured instruction changes agent behavior4 tools with published Results
- Configured restriction is respected4 tools with published Results
Analytics and observability3 scenarios
The operator can inspect conversations and agent actions, and the reported numbers match what actually happened.
Capability in this benchmark → · Global definition →
- Conversation activity can be inspected3 tools with published Results
- Agent actions/tool calls can be inspected5 tools with published Results
- Reported analytics match what actually happened3 tools with published Results
Feedback and learning2 scenarios
The agent accepts corrections and, where the product claims learning, changes future answers.
Capability in this benchmark → · Global definition →
- Feedback can be submitted4 tools with published Results
- Correction changes future behavior where the product claims learning4 tools with published Results
Results overview
Current published evidence in this benchmark.
Publication availability is separate from test coverage and unpublished research progress.
How the benchmark works
A public summary of the evaluation method. The same defined scope and evidence standard apply to every tool assessed under this version.
Scenarios describe conditions, not inputs
A scenario is a condition that makes the capability hard, not an input archetype or a smaller capability.
Simplest direct verification
The first test case under a scenario is the simplest, most natural test that directly verifies it.
Add harder variants from execution
Harder variants are added later from observed product differences; depth is earned from execution, not designed upfront.
Resources and fixtures
The registered material and systems that create a consistent test environment for this benchmark.
Cedarline commerce database
A shared PostgreSQL 16 commerce database used for benchmark evaluations.
Cedarline knowledge base
Versioned customer-support help-centre corpus used as the benchmark KB fixture.
Cedarline order system API
A customer-support order-system API for looking up and changing order and account state.