Tool in benchmark · Version 1

Intercom Fin in AI Customer Support Chatbots

Scenario-level performance from current published Results.

19Passed
1Failed
1Different test set
5No published Result yet

21 of 26 scenarios published

CapabilityScenariosResult
Action execution3 pass · 0 fail · 3 of 5 published
Agent configuration and control2 pass · 0 fail · 2 of 2 published
Analytics and observability3 pass · 0 fail · 3 of 3 published
Conversation continuity3 pass · 0 fail · 3 of 3 published
Escalation and human handoff3 pass · 0 fail · 3 of 3 published
Feedback and learning2 pass · 0 fail · 2 of 2 published
Intent understanding2 pass · 1 fail · 3 of 3 published
Knowledge-grounded answering0 pass · 0 fail · 1 of 3 published
Multilingual support1 pass · 0 fail · 1 of 2 published
PassedFailedPublished, different test setNo published Result yet

Where Intercom Fin failed

1 of 20 graded scenarios. Open a Result to see the conversation and the evidence.

Intent understanding

Request is unclear and needs clarification

Intercom Fin failed this unclear-cancellation case. The account had both a membership and a separate subscription that could be cancelled, and Fin replied with dashboard steps for both instead of asking which one the customer meant.

View Result →

How Intercom Fin performed

Open a capability to explore its scenarios. Each row reports the test set in its published Result.

Action execution3 pass2 no Result yet
ScenarioPublished outcomesTest coverageResult
Requested action cannot be completed
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin refused the address change instead of claiming it worked. It said the order had already shipped, gave next steps, and the order state shows shipped with address changes disabled.

1/1 assessed1/1 gradablePublished test setView Result →
Retrieve information from an external system
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin retrieved the shipped order's live status from the external system. The reply and the logged 200 call match on shipped status, UPS, tracking number, shipped date, and estimated delivery, showing the answer came from that call.

1/1 assessed1/1 gradablePublished test setView Result →
User is not authorized to perform the action
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin refused the unauthorized order lookup and did not reveal the other customer's order details. The signed-in customer's own order came back in full, so the lookup path worked and the refusal was about ownership.

1/1 assessed1/1 gradablePublished test setView Result →
Action requires confirmationNo published result
Update something in an external systemNo published result
Agent configuration and control2 pass
ScenarioPublished outcomesTest coverageResult
Configured instruction changes agent behavior
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin followed the configured instruction in this case. It told the customer opened candles were final sale and did not offer a refund, matching the live guidance for home decor items on all channels.

1/1 assessed1/1 gradablePublished test setView Result →
Configured restriction is respected
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin respected the configured restriction in this case. It declined the delivery-address change, said such changes were not supported after checkout, and offered help with other order or shipping questions instead.

1/1 assessed1/1 gradablePublished test setView Result →
Analytics and observability3 pass
ScenarioPublished outcomesTest coverageResult
Agent actions/tool calls can be inspected
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin shows the agent’s tool call in this scenario. The order lookup is linked to the open conversation and returned successfully.

1/1 assessed1/1 gradablePublished test setView Result →
Conversation activity can be inspected
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin let the user inspect an earlier shipped-order conversation in this case. It showed the Fin-involved list, opened the matching thread, and displayed the order 41982 reply and tracking details.

1/1 assessed1/1 gradablePublished test setView Result →
Reported analytics match what actually happened
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin’s analytics matched the recorded conversation and escalation counts for this period. Its conversation report showed 46 of 46 items, and its escalation report showed 2 of 2, with the comparison supported by the visible query document.

1/1 assessed1/1 gradablePublished test setView Result →
Conversation continuity3 pass
ScenarioPublished outcomesTest coverageResult
Customer corrects information given earlier
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin handled the corrected order lookup correctly. After the customer corrected the number, it answered only about order 41830 with processing status and no carrier or tracking number, and left out order 41982's shipped details.

1/1 assessed1/1 gradablePublished test setView Result →
Earlier conversation/session needs to be continued
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin continued the earlier return conversation after the widget was closed and reopened. The follow-up reply stayed on return timing and did not ask again for the order number or item.

1/1 assessed1/1 gradablePublished test setView Result →
Information from earlier in the conversation is needed later
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin passed this scenario. It answered the later question using the earlier order number and gave the shipping status, delivery date, and tracking number without asking for the number again.

1/1 assessed1/1 gradablePublished test setView Result →
Escalation and human handoff3 pass
ScenarioPublished outcomesTest coverageResult
A configured rule requires escalation
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin handled the refund scenario as required. It did not refund order 41455, said the amount was over the limit, and handed the chat to a human after the customer asked for that.

1/1 assessed1/1 gradablePublished test setView Result →
Agent cannot resolve the request
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin handled the unsupported account email change by declining it and directing the customer to support. It gave email and phone contact details, told the customer what to include, and ended by asking whether that was what they were looking for.

1/1 assessed1/1 gradablePublished test setView Result →
User explicitly asks for a human
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin handled a direct request for a human. It offered a human handoff on the first reply, showed a one-tap connect option, and the same conversation later appeared in the agent inbox with a teammate reply.

1/1 assessed1/1 gradablePublished test setView Result →
Feedback and learning2 pass
ScenarioPublished outcomesTest coverageResult
Correction changes future behavior where the product claims learning
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin changed the answer in a new customer conversation after the correction was saved. The later chat gave the corrected 30-day return window and 45 days for Cedarline Plus, while the earlier answer had said 21 days from the same private sources.

1/1 assessed1/1 gradablePublished test setView Result →
Feedback can be submitted
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin lets a customer submit feedback on a wrong answer. It opened the answer-improvement panel, let a correction be entered as a new snippet in English, and showed the saved 30-day version; another saved correction was also visible.

1/1 assessed1/1 gradablePublished test setView Result →
Intent understanding2 pass1 fail
ScenarioPublished outcomesTest coverageResult
One message contains multiple requests
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin answered both requests in one message. It gave the shipping status and return policy in one reply, and kept the two parts separately legible.

1/1 assessed1/1 gradablePublished test setView Result →
Request is unclear and needs clarification
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

Intercom Fin failed this unclear-cancellation case. The account had both a membership and a separate subscription that could be cancelled, and Fin replied with dashboard steps for both instead of asking which one the customer meant.

1/1 assessed1/1 gradablePublished test setView Result →
Request requires choosing the correct source or action
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin passed this order-status request. It answered from the order system, and the returned shipped status, carrier, tracking number, and delivery address matched what it told the customer.

1/1 assessed1/1 gradablePublished test setView Result →
Knowledge-grounded answering1 different test set2 no Result yet
ScenarioPublished outcomesTest coverageResult
Answer is not available in the knowledge baseDifferent test setPublished Result available, but not included in this comparison.View Result →
Answer is available in the knowledge baseNo published result
The request uses different wording than the sourceNo published result
Multilingual support1 pass1 no Result yet
ScenarioPublished outcomesTest coverageResult
Customer communicates in another supported language
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

Intercom Fin answered the French support question in French. The reply matched the English KB on the return window, eligible condition, restocking fee, non-returnable items, refund timing, and damaged-item path, and it stayed French through the closing line.

1/1 assessed1/1 gradablePublished test setView Result →
Customer switches languages during the conversationNo published result

Reading these Results

Published evidence and test coverage answer different questions.

Publication availability

Which scenarios have a Result?

A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.

Test coverage

What does each Result cover?

Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.

Scenario scope

Out of 26 scenarios

The summary counts each of the 26 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether Intercom Fin was tested on it.

Intercom Fin in AI Customer Support Chatbots | AI Demos