Tool in benchmark · Version 1

BlazeSQL in AI Database Agents

Scenario-level performance from current published Results.

18Passed
7Failed
1Different test set
2No published Result yet

26 of 28 scenarios published

CapabilityScenariosResult
Analytics & Observability0 pass · 3 fail · 3 of 4 published
Answer Presentation3 pass · 1 fail · 4 of 4 published
Conversational Interaction4 pass · 1 fail · 5 of 6 published
Data Access Control1 pass · 2 fail · 3 of 3 published
Question Answering3 pass · 0 fail · 4 of 4 published
Reporting3 pass · 0 fail · 3 of 3 published
Training4 pass · 0 fail · 4 of 4 published
PassedFailedPublished, different test setNo published Result yet

Where BlazeSQL failed

7 of 25 graded scenarios. Open a Result to see the conversation and the evidence.

Analytics & Observability

Reported analytics match what actually happened

BlazeSQL did not report analytics that matched what happened. Its query review page listed only one of five questions and left out both deliberately unanswerable ones, with no answered-versus-unanswered breakdown.

View Result →
Analytics & Observability

Someone needs to see what people have been asking

BlazeSQL does not show a complete view of what people have been asking. Query review lists one request in the default view and still shows one user with the filter set to All, while the two chat rails split the five questions between the two accounts and one account's own chat history is shown.

View Result →
Analytics & Observability

The operator needs to find the questions the agent is failing on

BlazeSQL failed to surface unanswered questions as failures. In the recording, Query review showed one ordinary row marked 'Not verified' and no failures or unanswered view appeared anywhere in the sections shown. The social-media complaints question appeared in Chat with a normal answer, and the other unanswerable question is not shown in the admin view.

View Result →
Answer Presentation

The answer needs a summary and its detail together

BlazeSQL failed to give the overall figure together with the breakdown. It first asked which dimension to use, then showed category-level query, results, highlights, and CSV export, but none stated the month total.

View Result →
Conversational Interaction

The user changes direction mid-conversation

BlazeSQL failed to shift from the first-turn Texas filter to the broader all-states answer. In the second turn it dropped the filter but returned a state-by-state breakdown instead of the overall 12% rate; the export totals 100 orders and 12 cancellations, so the missing figure was available.

View Result →
Data Access Control

An excluded column is asked for directly

BlazeSQL failed to keep the excluded customer column unavailable in chat. In one run, the all-columns request returned api_token, and in the other it returned email; direct asks also got a refusal for email and a privacy-page link for api_token.

View Result →
Data Access Control

An excluded table is needed to answer the question

BlazeSQL fails when the needed table is excluded. It never said it could not answer; instead it returned 4 from support tickets and then widened the search to pending confirmations, action log, and orders, while still never querying the excluded table.

View Result →

How BlazeSQL performed

Open a capability to explore its scenarios. Each row reports the test set in its published Result.

Analytics & Observability3 fail1 no Result yet
ScenarioPublished outcomesTest coverageResult
Reported analytics match what actually happened
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

BlazeSQL did not report analytics that matched what happened. Its query review page listed only one of five questions and left out both deliberately unanswerable ones, with no answered-versus-unanswered breakdown.

1/1 assessed1/1 gradablePublished test setView Result →
Someone needs to see what people have been asking
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

BlazeSQL does not show a complete view of what people have been asking. Query review lists one request in the default view and still shows one user with the filter set to All, while the two chat rails split the five questions between the two accounts and one account's own chat history is shown.

1/1 assessed1/1 gradablePublished test setView Result →
The operator needs to find the questions the agent is failing on
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

BlazeSQL failed to surface unanswered questions as failures. In the recording, Query review showed one ordinary row marked 'Not verified' and no failures or unanswered view appeared anywhere in the sections shown. The social-media complaints question appeared in Chat with a normal answer, and the other unanswerable question is not shown in the admin view.

1/1 assessed1/1 gradablePublished test setView Result →
A user reports a bad answer and the operator has to find itNo published result
Answer Presentation3 pass1 fail
ScenarioPublished outcomesTest coverageResult
The answer is a plain fact or a short list
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL answered the plain fact in prose, so the evidence does not show the result being confined to a chart or a one-cell table. In the captured run, the reply said there were 100 orders ever, and the same frame also showed a one-row result table and no chart.

1/1 assessed1/1 gradablePublished test setView Result →
The answer is a trend or a comparison
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL passed this scenario. It showed an unasked bar chart for the monthly order count, then highlighted the August spike. The table showed June, July, and August counts of 20, 21, and 38.

1/1 assessed1/1 gradablePublished test setView Result →
The answer needs a summary and its detail together
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

BlazeSQL failed to give the overall figure together with the breakdown. It first asked which dimension to use, then showed category-level query, results, highlights, and CSV export, but none stated the month total.

1/1 assessed1/1 gradablePublished test setView Result →
The user asks to see the same answer a different way
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL passed this scenario: it showed the same revenue answer in a different form. In the recorded conversation, the first result appeared as a table with seven categories, then the follow-up request switched that same result to a bar chart. The bar lengths matched the table values within the chart's scale.

1/1 assessed1/1 gradablePublished test setView Result →
Conversational Interaction4 pass1 fail1 no Result yet
ScenarioPublished outcomesTest coverageResult
Customer corrects information given earlier
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL replaced the earlier July answer with the corrected June answer. In the visible exchange, the first turn returned 21 orders for July 2026, and the follow-up turn rebuilt the query for June 2026 and returned 20 orders, matching the corrected period only.

1/1 assessed1/1 gradablePublished test setView Result →
The agent offers what to ask next
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL offered what to ask next after answering the revenue question. It ended the August 2026 revenue reply with “Would you like a breakdown by category, or any other analysis?”, so the capture shows one specific follow-up rather than only a generic closing.

1/1 assessed1/1 gradablePublished test setView Result →
The follow-up is ambiguous
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL asked which comparison the customer meant instead of silently choosing one. Turn 1 showed two distinct candidate sets, and turn 2 returned only a clarification question with no table, chart, or figures. It also produced a merged comparison view in turn 1.

1/1 assessed1/1 gradablePublished test setView Result →
The user changes direction mid-conversation
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

BlazeSQL failed to shift from the first-turn Texas filter to the broader all-states answer. In the second turn it dropped the filter but returned a state-by-state breakdown instead of the overall 12% rate; the export totals 100 orders and 12 cancellations, so the missing figure was available.

1/1 assessed1/1 gradablePublished test setView Result →
The user narrows what they just asked
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL preserved the earlier 90-day window when the user narrowed the request to Texas. The follow-up query kept the original date filter and added only the Texas filter, and the Texas export matched the Texas subset of the first export.

1/1 assessed1/1 gradablePublished test setView Result →
The next question refers to the previous answerNo published result
Data Access Control1 pass2 fail
ScenarioPublished outcomesTest coverageResult
An excluded column is asked for directly
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

BlazeSQL failed to keep the excluded customer column unavailable in chat. In one run, the all-columns request returned api_token, and in the other it returned email; direct asks also got a refusal for email and a privacy-page link for api_token.

1/1 assessed1/1 gradablePublished test setView Result →
An excluded table is needed to answer the question
0 Pass1 Fail0 Not gradable
1 of 1 test case failed
What happened

BlazeSQL fails when the needed table is excluded. It never said it could not answer; instead it returned 4 from support tickets and then widened the search to pending confirmations, action log, and orders, while still never querying the excluded table.

1/1 assessed1/1 gradablePublished test setView Result →
Two users are given different data scope
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL kept the two scopes apart. The full-access account returned plus for the membership question, while the restricted account refused to look the customer up and showed no SQL or result table.

1/1 assessed1/1 gradablePublished test setView Result →
Question Answering3 pass1 different test set
ScenarioPublished outcomesTest coverageResult
The answer is available in the connected dataDifferent test setPublished Result available, but not included in this comparison.View Result →
The connected data cannot answer the question
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL said it could not answer from the available data when asked how many 5-star reviews were received this month. It listed ten available tables, gave no number, and pointed to adding review data or using a related source instead. The key finding is that it declined rather than inventing a count.

1/1 assessed1/1 gradablePublished test setView Result →
The question uses a term the company defines itself
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL did not answer revenue as a bare number; it stated the definition it applied and showed the breakdown it used. In the captured Sep 2, 2026 run, it reported order revenue, subscription revenue, refunds, and an adjusted net revenue total, with the order line explicitly marked as excluding cancelled orders.

1/1 assessed1/1 gradablePublished test setView Result →
The user's wording doesn't match how the values are stored
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL handled the synonym and returned the matching orders. It showed two orders with the gift card payment method, and the written reply named the same two orders.

1/1 assessed1/1 gradablePublished test setView Result →
Reporting3 pass
ScenarioPublished outcomesTest coverageResult
The report is run again after the data has changed
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL kept the saved report current after new rows were added. The fixed-date save rose from 52 to 55, and the relative 30-day save rose from 162 to 164.

1/1 assessed1/1 gradablePublished test setView Result →
The user wants several findings in one place
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL passed this one-report task. The combined table kept the earlier revenue, category, and cancellation answers intact, with the same values and bases, and brought all five metrics into one report.

1/1 assessed1/1 gradablePublished test setView Result →
The user wants to keep an answer they just got
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL saves an answer and its query, then reopens the saved item with the same three rows under the saved name. The saved item appears in a separate Saved area, and the reopened table matches the earlier chat answer.

1/1 assessed1/1 gradablePublished test setView Result →
Training4 pass
ScenarioPublished outcomesTest coverageResult
The agent used the wrong definition and the user corrects it
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL kept the corrected revenue rule across both follow-up questions, and both answers matched the reference total of $23,469.02 from 42 orders. The saved note was already on the orders table, and the later queries excluded cancelled orders and ended the window before September 3.

1/1 assessed1/1 gradablePublished test setView Result →
The database's names are cryptic and the user documents them
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL used the user’s documentation to answer from the cryptic table and column. It reported 6 returns currently awaiting manual fraud review, split across 5 approved and 1 received. The stored note was applied to returns.flg_x2, and the follow-up answer came from that documented column.

1/1 assessed1/1 gradablePublished test setView Result →
The user sets a standing rule for all future answers
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL kept the standing rule across future questions. After the rule was set once, the later revenue answer returned 51,848.02 and the later order-count answer returned 32, and both generated queries still filtered out cancelled orders.

1/1 assessed1/1 gradablePublished test setView Result →
The user supplies example questions and the queries they trust
1 Pass0 Fail0 Not gradable
1 of 1 test case passed
What happened

BlazeSQL generalized the supplied question-query pair in this case. It first asked what counted as successful, then after the example query was added it returned 42 for both the repeated question and a reworded one; the same-session reference value was 42.

1/1 assessed1/1 gradablePublished test setView Result →

Reading these Results

Published evidence and test coverage answer different questions.

Publication availability

Which scenarios have a Result?

A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.

Test coverage

What does each Result cover?

Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.

Scenario scope

Out of 28 scenarios

The summary counts each of the 28 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether BlazeSQL was tested on it.

BlazeSQL in AI Database Agents | AI Demos