Tool in benchmark · Version 1

FutureSmart Database Agent in AI Database Agents

Scenario-level performance from current published Results.

13Passed
14Failed
1Different test set
0No published Result yet

28 of 28 scenarios published

CapabilityScenariosResult
Analytics & Observability3 pass · 1 fail · 4 of 4 published
Answer Presentation2 pass · 2 fail · 4 of 4 published
Conversational Interaction4 pass · 2 fail · 6 of 6 published
Data Access Control0 pass · 3 fail · 3 of 3 published
Question Answering1 pass · 2 fail · 4 of 4 published
Reporting0 pass · 3 fail · 3 of 3 published
Training3 pass · 1 fail · 4 of 4 published
PassedFailedPublished, different test setNo published Result yet

Where FutureSmart Database Agent failed

14 of 27 graded scenarios. Open a Result to see the conversation and the evidence.

Analytics & Observability

The operator needs to find the questions the agent is failing on

FutureSmart Database Agent failed this scenario: the failures view showed zero failures and no failing questions. In the same window, Input 3 and Input 5—the complaints question and the net-promoter-score question—were both marked answered in the query log, so the questions it could not answer were not surfaced as failures.

View Result →
Answer Presentation

The answer is a trend or a comparison

FutureSmart Database Agent did not present the answer as a trend visual. It showed monthly counts in Table view with Chart available but not selected, and the month labels were raw timestamps.

View Result →
Answer Presentation

The user asks to see the same answer a different way

FutureSmart Database Agent failed to keep the same answer when asked for a different presentation. The follow-up returned the same table and said it could not render a chart, while an earlier turn in the same conversation showed a bar chart for the same data; the later turn's values also differed from the earlier chart.

View Result →
Conversational Interaction

The follow-up is ambiguous

FutureSmart Database Agent did not ask for clarification when the follow-up was ambiguous. Turn 2 named the two rankings it had chosen and repeated the same comparison instead of asking which was meant.

View Result →
Conversational Interaction

The user changes direction mid-conversation

When the user changed direction, FutureSmart Database Agent dropped the earlier filter but did not give the unfiltered result the scenario calls for. In the run, it returned a 16-row by-state breakdown instead of the single overall cancellation rate, so the answer did not match the unfiltered result.

View Result →
Data Access Control

An excluded column is asked for directly

FutureSmart Database Agent could not be executed for this scenario because it cannot exclude a single column. Its only exclusion control works on whole tables on both connections, and no other settings screen offers column-level control. No question was asked, so the agent's handling of a direct request for an excluded column was not observed.

View Result →
Data Access Control

An excluded table is needed to answer the question

FutureSmart Database Agent failed the excluded-table scenario. It said it could not answer, but the chat trace shows it generated SQL, read the excluded returns table, and showed a 0 result.

View Result →
Data Access Control

Two users are given different data scope

FutureSmart Database Agent failed to keep the two accounts' answers within their own data scopes. The open account answered plus, while the restricted account answered from another table, gave bedding_and_bath, and its notice said it read the excluded table.

View Result →
Question Answering

The connected data cannot answer the question

FutureSmart Database Agent fails when the connected data does not contain the answer. It answered the question with a count from its activity log, returned 0, and did not say it could not answer from the available data.

View Result →
Question Answering

The user's wording doesn't match how the values are stored

FutureSmart Database Agent failed to handle wording that did not match stored values. It answered with no results and treated that emptiness as a fact about the stored orders; its SQL only searched literal variants of the user's wording, not the stored payment value. The evidence also notes a matching gift-card value exists, with 2 orders totaling $984.06.

View Result →
Reporting

The report is run again after the data has changed

FutureSmart Database Agent failed this scenario. The pinned question returned 57 for the 30-day window, but the console sweep shows no way to save an answer as a report or rerun one, so the result cannot refresh after rows are added.

View Result →
Reporting

The user wants several findings in one place

FutureSmart Database Agent failed to keep the assembled findings consistent. It kept revenue at USD 20,253.47, but the combined report changed the category answer from storage_and_organization to outdoor_and_patio and the cancellation rate from 12.0% to 13.19%.

View Result →
Reporting

The user wants to keep an answer they just got

FutureSmart Database Agent did not let a user save an answer and reopen it later. The answer view showed viewing, filtering, charting, CSV export, thumbs, and copy controls, but no saved-answer destination.

View Result →
Training

The user supplies example questions and the queries they trust

FutureSmart Database Agent did not generalise the trusted example pair. The exact wording returned 42, while the reworded question returned 41 from a new delivery-date query; before training, the same question was refused.

View Result →

How FutureSmart Database Agent performed

Open a capability to explore its scenarios. Each row reports the test set in its published Result.

Analytics & Observability3 pass1 fail
ScenarioPublished outcomesTest coverageResult
A user reports a bad answer and the operator has to find it
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent can find a complained-about exchange from a vague clue. The query log shows the target row with its chat and session IDs among five queries, and the opened Activity page shows the same exchange read-only with the question, complete answer, and share-link attribution.

1/1 assessed1/1 gradablePublished test setView Result →
Reported analytics match what actually happened
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent counted every question that was run, including the two the database could not answer, so the total did not drop unanswered work. The dashboard shows 6 answered and 0 unanswered, and the query log lists the same six rows, including the two unanswerable questions and a separate hi check.

1/1 assessed1/1 gradablePublished test setView Result →
Someone needs to see what people have been asking
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent shows what people have been asking: its Activity view reports 5 of 5 shown for a split between two people. Three questions come from one account and two from the other, and the query log separately names the account for the net-promoter-score question.

1/1 assessed1/1 gradablePublished test setView Result →
The operator needs to find the questions the agent is failing on
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent failed this scenario: the failures view showed zero failures and no failing questions. In the same window, Input 3 and Input 5—the complaints question and the net-promoter-score question—were both marked answered in the query log, so the questions it could not answer were not surfaced as failures.

1/1 assessed1/1 gradablePublished test setView Result →
Answer Presentation2 pass2 fail
ScenarioPublished outcomesTest coverageResult
The answer is a plain fact or a short list
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent answered this case in plain prose rather than only as a chart or one-cell table. It said there were 100 orders in total, and the nearby count grid repeated the same number.

1/1 assessed1/1 gradablePublished test setView Result →
The answer is a trend or a comparison
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent did not present the answer as a trend visual. It showed monthly counts in Table view with Chart available but not selected, and the month labels were raw timestamps.

1/1 assessed1/1 gradablePublished test setView Result →
The answer needs a summary and its detail together
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent passes this scenario. It gave the August total (USD 20,253.47) and broke it down by all seven product categories in a paged table and a bar chart.

1/1 assessed1/1 gradablePublished test setView Result →
The user asks to see the same answer a different way
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent failed to keep the same answer when asked for a different presentation. The follow-up returned the same table and said it could not render a chart, while an earlier turn in the same conversation showed a bar chart for the same data; the later turn's values also differed from the earlier chart.

1/1 assessed1/1 gradablePublished test setView Result →
Conversational Interaction4 pass2 fail
ScenarioPublished outcomesTest coverageResult
Customer corrects information given earlier
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent answered the correction with June only. It reported 20 orders, used June-only SQL, and did not carry forward the July answer.

1/1 assessed1/1 gradablePublished test setView Result →
The agent offers what to ask next
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent offered two follow-up questions after its August 2026 revenue answer, and both were specific to that answer and answerable from the database. The chips ask about August versus July and the highest-revenue month so far.

1/1 assessed1/1 gradablePublished test setView Result →
The follow-up is ambiguous
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent did not ask for clarification when the follow-up was ambiguous. Turn 2 named the two rankings it had chosen and repeated the same comparison instead of asking which was meant.

1/1 assessed1/1 gradablePublished test setView Result →
The next question refers to the previous answer
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent handled the follow-up for the same five customers returned before. It returned those five names again and reported order counts of 10, 9, 8, 6, and 5, with a total of 38.

1/1 assessed1/1 gradablePublished test setView Result →
The user changes direction mid-conversation
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

When the user changed direction, FutureSmart Database Agent dropped the earlier filter but did not give the unfiltered result the scenario calls for. In the run, it returned a 16-row by-state breakdown instead of the single overall cancellation rate, so the answer did not match the unfiltered result.

1/1 assessed1/1 gradablePublished test setView Result →
The user narrows what they just asked
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent passed this narrowing scenario: it kept the earlier 90-day window and added the Texas filter, and its answer reported 7 orders shipped to Texas. The first answer used a 10-row limit without ordering, and the follow-up table used placeholder column labels.

1/1 assessed1/1 gradablePublished test setView Result →
Data Access Control3 fail
ScenarioPublished outcomesTest coverageResult
An excluded column is asked for directly
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent could not be executed for this scenario because it cannot exclude a single column. Its only exclusion control works on whole tables on both connections, and no other settings screen offers column-level control. No question was asked, so the agent's handling of a direct request for an excluded column was not observed.

1/1 assessed1/1 gradablePublished test setView Result →
An excluded table is needed to answer the question
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent failed the excluded-table scenario. It said it could not answer, but the chat trace shows it generated SQL, read the excluded returns table, and showed a 0 result.

1/1 assessed1/1 gradablePublished test setView Result →
Two users are given different data scope
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent failed to keep the two accounts' answers within their own data scopes. The open account answered plus, while the restricted account answered from another table, gave bedding_and_bath, and its notice said it read the excluded table.

1/1 assessed1/1 gradablePublished test setView Result →
Question Answering1 pass2 fail1 different test set
ScenarioPublished outcomesTest coverageResult
The answer is available in the connected dataDifferent test setPublished Result available, but not included in this comparison.View Result →
The connected data cannot answer the question
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent fails when the connected data does not contain the answer. It answered the question with a count from its activity log, returned 0, and did not say it could not answer from the available data.

1/1 assessed1/1 gradablePublished test setView Result →
The question uses a term the company defines itself
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent answered a company-defined revenue question by stating the definition it used instead of giving only a bare number. The answer block and generated SQL use the same 90-day, not-refunded definition, and the capture shows the matching USD total.

1/1 assessed1/1 gradablePublished test setView Result →
The user's wording doesn't match how the values are stored
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent failed to handle wording that did not match stored values. It answered with no results and treated that emptiness as a fact about the stored orders; its SQL only searched literal variants of the user's wording, not the stored payment value. The evidence also notes a matching gift-card value exists, with 2 orders totaling $984.06.

1/1 assessed1/1 gradablePublished test setView Result →
Reporting3 fail
ScenarioPublished outcomesTest coverageResult
The report is run again after the data has changed
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent failed this scenario. The pinned question returned 57 for the 30-day window, but the console sweep shows no way to save an answer as a report or rerun one, so the result cannot refresh after rows are added.

1/1 assessed1/1 gradablePublished test setView Result →
The user wants several findings in one place
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent failed to keep the assembled findings consistent. It kept revenue at USD 20,253.47, but the combined report changed the category answer from storage_and_organization to outdoor_and_patio and the cancellation rate from 12.0% to 13.19%.

1/1 assessed1/1 gradablePublished test setView Result →
The user wants to keep an answer they just got
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent did not let a user save an answer and reopen it later. The answer view showed viewing, filtering, charting, CSV export, thumbs, and copy controls, but no saved-answer destination.

1/1 assessed1/1 gradablePublished test setView Result →
Training3 pass1 fail
ScenarioPublished outcomesTest coverageResult
The agent used the wrong definition and the user corrects it
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent kept the corrected revenue definition on the repeated question and the reworded one. After the customer gave the rule in chat, both later answers excluded cancelled orders and returned USD 51,848.02 from one row.

1/1 assessed1/1 gradablePublished test setView Result →
The database's names are cryptic and the user documents them
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent answered the follow-up from the documented table and column. The follow-up query used flg_x2 = true on returns and returned 7; the training turn showed 7 false rows and 7 true rows.

1/1 assessed1/1 gradablePublished test setView Result →
The user sets a standing rule for all future answers
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

FutureSmart Database Agent passed this scenario. The rule was set once, and both later answers still excluded cancelled orders in their SQL and wording.

1/1 assessed1/1 gradablePublished test setView Result →
The user supplies example questions and the queries they trust
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

FutureSmart Database Agent did not generalise the trusted example pair. The exact wording returned 42, while the reworded question returned 41 from a new delivery-date query; before training, the same question was refused.

1/1 assessed1/1 gradablePublished test setView Result →

Reading these Results

Published evidence and test coverage answer different questions.

Publication availability

Which scenarios have a Result?

A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.

Test coverage

What does each Result cover?

Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.

Scenario scope

Out of 28 scenarios

The summary counts each of the 28 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether FutureSmart Database Agent was tested on it.

FutureSmart Database Agent in AI Database Agents | AI Demos