BlazeSQL in AI Database Agents
Scenario-level performance from current published Results.
26 of 28 scenarios published
Where BlazeSQL failed
7 of 25 graded scenarios. Open a Result to see the conversation and the evidence.
Reported analytics match what actually happened
BlazeSQL did not report analytics that matched what happened. Its query review page listed only one of five questions and left out both deliberately unanswerable ones, with no answered-versus-unanswered breakdown.
View Result →Someone needs to see what people have been asking
BlazeSQL does not show a complete view of what people have been asking. Query review lists one request in the default view and still shows one user with the filter set to All, while the two chat rails split the five questions between the two accounts and one account's own chat history is shown.
View Result →The operator needs to find the questions the agent is failing on
BlazeSQL failed to surface unanswered questions as failures. In the recording, Query review showed one ordinary row marked 'Not verified' and no failures or unanswered view appeared anywhere in the sections shown. The social-media complaints question appeared in Chat with a normal answer, and the other unanswerable question is not shown in the admin view.
View Result →The answer needs a summary and its detail together
BlazeSQL failed to give the overall figure together with the breakdown. It first asked which dimension to use, then showed category-level query, results, highlights, and CSV export, but none stated the month total.
View Result →The user changes direction mid-conversation
BlazeSQL failed to shift from the first-turn Texas filter to the broader all-states answer. In the second turn it dropped the filter but returned a state-by-state breakdown instead of the overall 12% rate; the export totals 100 orders and 12 cancellations, so the missing figure was available.
View Result →An excluded column is asked for directly
BlazeSQL failed to keep the excluded customer column unavailable in chat. In one run, the all-columns request returned api_token, and in the other it returned email; direct asks also got a refusal for email and a privacy-page link for api_token.
View Result →An excluded table is needed to answer the question
BlazeSQL fails when the needed table is excluded. It never said it could not answer; instead it returned 4 from support tickets and then widened the search to pending confirmations, action log, and orders, while still never querying the excluded table.
View Result →How BlazeSQL performed
Open a capability to explore its scenarios. Each row reports the test set in its published Result.
Analytics & Observability3 fail1 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Reported analytics match what actually happened | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedBlazeSQL did not report analytics that matched what happened. Its query review page listed only one of five questions and left out both deliberately unanswerable ones, with no answered-versus-unanswered breakdown. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Someone needs to see what people have been asking | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedBlazeSQL does not show a complete view of what people have been asking. Query review lists one request in the default view and still shows one user with the filter set to All, while the two chat rails split the five questions between the two accounts and one account's own chat history is shown. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The operator needs to find the questions the agent is failing on | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedBlazeSQL failed to surface unanswered questions as failures. In the recording, Query review showed one ordinary row marked 'Not verified' and no failures or unanswered view appeared anywhere in the sections shown. The social-media complaints question appeared in Chat with a normal answer, and the other unanswerable question is not shown in the admin view. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| A user reports a bad answer and the operator has to find it | No published result | ||
Answer Presentation3 pass1 fail
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| The answer is a plain fact or a short list | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL answered the plain fact in prose, so the evidence does not show the result being confined to a chart or a one-cell table. In the captured run, the reply said there were 100 orders ever, and the same frame also showed a one-row result table and no chart. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The answer is a trend or a comparison | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL passed this scenario. It showed an unasked bar chart for the monthly order count, then highlighted the August spike. The table showed June, July, and August counts of 20, 21, and 38. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The answer needs a summary and its detail together | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedBlazeSQL failed to give the overall figure together with the breakdown. It first asked which dimension to use, then showed category-level query, results, highlights, and CSV export, but none stated the month total. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The user asks to see the same answer a different way | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL passed this scenario: it showed the same revenue answer in a different form. In the recorded conversation, the first result appeared as a table with seven categories, then the follow-up request switched that same result to a bar chart. The bar lengths matched the table values within the chart's scale. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Conversational Interaction4 pass1 fail1 no Result yet
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| Customer corrects information given earlier | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL replaced the earlier July answer with the corrected June answer. In the visible exchange, the first turn returned 21 orders for July 2026, and the follow-up turn rebuilt the query for June 2026 and returned 20 orders, matching the corrected period only. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The agent offers what to ask next | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL offered what to ask next after answering the revenue question. It ended the August 2026 revenue reply with “Would you like a breakdown by category, or any other analysis?”, so the capture shows one specific follow-up rather than only a generic closing. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The follow-up is ambiguous | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL asked which comparison the customer meant instead of silently choosing one. Turn 1 showed two distinct candidate sets, and turn 2 returned only a clarification question with no table, chart, or figures. It also produced a merged comparison view in turn 1. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The user changes direction mid-conversation | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedBlazeSQL failed to shift from the first-turn Texas filter to the broader all-states answer. In the second turn it dropped the filter but returned a state-by-state breakdown instead of the overall 12% rate; the export totals 100 orders and 12 cancellations, so the missing figure was available. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The user narrows what they just asked | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL preserved the earlier 90-day window when the user narrowed the request to Texas. The follow-up query kept the original date filter and added only the Texas filter, and the Texas export matched the Texas subset of the first export. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The next question refers to the previous answer | No published result | ||
Data Access Control1 pass2 fail
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| An excluded column is asked for directly | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedBlazeSQL failed to keep the excluded customer column unavailable in chat. In one run, the all-columns request returned api_token, and in the other it returned email; direct asks also got a refusal for email and a privacy-page link for api_token. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| An excluded table is needed to answer the question | 0 Pass1 Fail0 Not gradable 1 of 1 test case failedWhat happenedBlazeSQL fails when the needed table is excluded. It never said it could not answer; instead it returned 4 from support tickets and then widened the search to pending confirmations, action log, and orders, while still never querying the excluded table. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| Two users are given different data scope | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL kept the two scopes apart. The full-access account returned plus for the membership question, while the restricted account refused to look the customer up and showed no SQL or result table. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Question Answering3 pass1 different test set
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| The answer is available in the connected data | Different test setPublished Result available, but not included in this comparison. | View Result → | |
| The connected data cannot answer the question | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL said it could not answer from the available data when asked how many 5-star reviews were received this month. It listed ten available tables, gave no number, and pointed to adding review data or using a related source instead. The key finding is that it declined rather than inventing a count. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The question uses a term the company defines itself | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL did not answer revenue as a bare number; it stated the definition it applied and showed the breakdown it used. In the captured Sep 2, 2026 run, it reported order revenue, subscription revenue, refunds, and an adjusted net revenue total, with the order line explicitly marked as excluding cancelled orders. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The user's wording doesn't match how the values are stored | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL handled the synonym and returned the matching orders. It showed two orders with the gift card payment method, and the written reply named the same two orders. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Reporting3 pass
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| The report is run again after the data has changed | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL kept the saved report current after new rows were added. The fixed-date save rose from 52 to 55, and the relative 30-day save rose from 162 to 164. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The user wants several findings in one place | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL passed this one-report task. The combined table kept the earlier revenue, category, and cancellation answers intact, with the same values and bases, and brought all five metrics into one report. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The user wants to keep an answer they just got | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL saves an answer and its query, then reopens the saved item with the same three rows under the saved name. The saved item appears in a separate Saved area, and the reopened table matches the earlier chat answer. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Training4 pass
Capability in this benchmark →
| Scenario | Published outcomes | Test coverage | Result |
|---|---|---|---|
| The agent used the wrong definition and the user corrects it | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL kept the corrected revenue rule across both follow-up questions, and both answers matched the reference total of $23,469.02 from 42 orders. The saved note was already on the orders table, and the later queries excluded cancelled orders and ended the window before September 3. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The database's names are cryptic and the user documents them | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL used the user’s documentation to answer from the cryptic table and column. It reported 6 returns currently awaiting manual fraud review, split across 5 approved and 1 received. The stored note was applied to returns.flg_x2, and the follow-up answer came from that documented column. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The user sets a standing rule for all future answers | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL kept the standing rule across future questions. After the rule was set once, the later revenue answer returned 51,848.02 and the later order-count answer returned 32, and both generated queries still filtered out cancelled orders. | 1/1 assessed1/1 gradablePublished test set | View Result → |
| The user supplies example questions and the queries they trust | 1 Pass0 Fail0 Not gradable 1 of 1 test case passedWhat happenedBlazeSQL generalized the supplied question-query pair in this case. It first asked what counted as successful, then after the example query was added it returned 42 for both the repeated question and a reworded one; the same-session reference value was 42. | 1/1 assessed1/1 gradablePublished test set | View Result → |
Reading these Results
Published evidence and test coverage answer different questions.
Which scenarios have a Result?
A published Result is public evidence for this tool on one scenario. “No published result” does not say whether testing has taken place.
What does each Result cover?
Assessed includes Pass, Fail and Not gradable. Gradable includes Pass and Fail. Both use the pinned test count in that published Result.
Out of 28 scenarios
The summary counts each of the 28 scenarios once. A scenario fails when any of its published test cases failed. “No published Result yet” means nothing is public for that scenario; it does not say whether BlazeSQL was tested on it.