AI Database Agents
This benchmark looks at database agents for business users who want answers from connected relational data without writing SQL.
Benchmark overview
What is included and excluded
AI Database Agents evaluates whether a person who cannot or does not want to write SQL can ask a live relational database for business answers correctly, in the company’s own terms, and honestly when the data cannot answer.
Readers learn how these products behave across question answering, conversation, training, answer presentation, reporting, analytics and observability, and data access control. It is a buyer-facing comparison of behaviour in live database use, not a check on SQL generation alone.
In scope
- asking questions of connected database data conversationally
- how the answer is presented
- whether it can be turned into a report
- whether the agent can be trained
- whether configured data restrictions hold
Out of scope
- database administration
- migrations
- index tuning
- arbitrary writes
- DBA automation
- debugging SQL a user already wrote
- data-engineering pipelines
- authoring BI dashboards from scratch
- question-answering over documents or spreadsheets
Participating tools
How each tool did on the 28 scenarios in this benchmark. Each cell is one scenario, in the same order for every tool and grouped by capability; Pass and Fail come from published Results.
Capabilities & scenarios
28 scenarios grouped by 7 capabilities. Open a group to explore its scenarios in this benchmark.
Question Answering4 scenarios
Answers a question about connected data correctly, using the company’s meaning and the values stored in the database, and says when the data cannot answer.
Capability in this benchmark → · Global definition →
- The answer is available in the connected data2 tools with published Results
- The question uses a term the company defines itself3 tools with published Results
- The user's wording doesn't match how the values are stored5 tools with published Results
- The connected data cannot answer the question5 tools with published Results
Conversational Interaction6 scenarios
Keeps a working session coherent across turns, so context carries forward, follow-ups can narrow or change direction, ambiguity can be raised, and earlier information can be corrected.
Capability in this benchmark → · Global definition →
- Customer corrects information given earlier6 tools with published Results
- The next question refers to the previous answer4 tools with published Results
- The user narrows what they just asked4 tools with published Results
- The user changes direction mid-conversation3 tools with published Results
- The follow-up is ambiguous5 tools with published Results
- The agent offers what to ask next5 tools with published Results
Training4 scenarios
Uses company-specific information to improve later behaviour and to generalise beyond the exact example given.
Capability in this benchmark → · Global definition →
- The agent used the wrong definition and the user corrects it2 tools with published Results
- The database's names are cryptic and the user documents them3 tools with published Results
- The user supplies example questions and the queries they trust4 tools with published Results
- The user sets a standing rule for all future answers3 tools with published Results
Answer Presentation4 scenarios
Chooses and changes the form of the answer, such as prose, a table, a chart, or a combination, when that better fits the question.
Capability in this benchmark → · Global definition →
- The answer is a plain fact or a short list5 tools with published Results
- The answer is a trend or a comparison5 tools with published Results
- The answer needs a summary and its detail together6 tools with published Results
- The user asks to see the same answer a different way6 tools with published Results
Reporting3 scenarios
Turns analysis into a durable artifact that can be reopened or run again later without changing its meaning.
Capability in this benchmark → · Global definition →
- The user wants to keep an answer they just got4 tools with published Results
- The report is run again after the data has changed5 tools with published Results
- The user wants several findings in one place6 tools with published Results
Analytics & Observability4 scenarios
Lets the buyer inspect what people asked and what happened, and checks that reported numbers match reality.
Capability in this benchmark → · Global definition →
- Reported analytics match what actually happened3 tools with published Results
- Someone needs to see what people have been asking5 tools with published Results
- A user reports a bad answer and the operator has to find it3 tools with published Results
- The operator needs to find the questions the agent is failing on3 tools with published Results
Data Access Control3 scenarios
Respects configured restrictions on which tables, columns, and rows the agent may use when answering.
Capability in this benchmark → · Global definition →
- An excluded table is needed to answer the question5 tools with published Results
- An excluded column is asked for directly3 tools with published Results
- Two users are given different data scope4 tools with published Results
Results overview
Current published evidence in this benchmark.
Publication availability is separate from test coverage and unpublished research progress.
How the benchmark works
A public summary of the evaluation method. The same defined scope and evidence standard apply to every tool assessed under this version.
Answer, not SQL
The benchmark grades the answer, never the SQL; SQL is captured only as instrumentation.
Scenario is a condition
A scenario is a condition, not a stimulus; changing the stimulus creates a test case, not a new scenario.
Specifications first
Test cases are specifications, not runnable tests, and concrete literals are bound later at fixture implementation.
Simplest direct verification
Each scenario starts with the simplest, most obvious, natural test that directly verifies it.
Analytics split under test
Analytics & Observability is tested through aggregate and individual groups so execution can show whether they are really separate capabilities.
Resources and fixtures
The registered material and systems that create a consistent test environment for this benchmark.
Cedarline commerce database
A shared PostgreSQL 16 commerce database used for benchmark evaluations.