Does Exa return results for the less famous namesake?
Exa returned ten pages about Grafana Labs’ Loki and none about Marvel’s Loki. The returned documentation also identified the required compactor configuration block for enabling retention.
1 of 1 test case passed
Every test case in this scenario has a result.
The test case
Each test case is judged on its own: Pass, Fail, Not gradable, or Untested. The scenario result above is based on this test case.
Something more famous has the same namePassEvidence
“In Loki, the log aggregation system from Grafana Labs, which configuration block does retention_enabled have to be set in for old log data to actually be deleted?”
Result 1 Enable log entry deletion by setting `retention_enabled` to `true` in the compactor’s configuration (`-compactor.retention-enabled` on the command line). The tenant’s `deletion_mode` must also be `filter-only` or `filter-and-delete`, which is the case by default. `delete_request_store` also needs to be configured when retention is enabled to process delete requests, this determines the storage bucket that stores the delete requests. Result 2 Retention in Grafana Loki is achieved through the Compactor. By default the `compactor.retention-enabled` flag is not set, so the logs sent to Loki live forever.
Transcribed from Output file (JSON)
- query_text
- In Loki, the log aggregation system from Grafana Labs, which configuration block does retention_enabled have to be set in for old log data to actually be deleted?
- results 1
- { "rank": 1, "url": "https://grafana.com/docs/loki/latest/operations/storage/logs-deletion/", "title": "Log entry deletion | Grafana Loki documentation", "content": "Log entry deletion | Grafana Loki documentation \n\n# Log entry deletion \n\nGrafana Loki supports the deletion of log entries from a specified stream. Log entries that fall within a specified time window and match an optional line filter are t … [4,474 characters in the file]", "content_chars": 4474, "published_date": null }
- results 2
- { "rank": 2, "url": "https://grafana.com/docs/loki/latest/operations/storage/retention/", "title": "Log retention | Grafana Loki documentation", "content": "Log retention | Grafana Loki documentation \n\n# Log retention \n\nRetention in Grafana Loki is achieved through the Compactor. By default the `compactor.retention-enabled` flag is not set, so the logs sent to Loki live forever.\n\n> Note\n> \n> If … [18,971 characters in the file]", "content_chars": 18971, "published_date": null }
- results
- [ { "rank": 1, "url": "https://grafana.com/docs/loki/latest/operations/storage/logs-deletion/", "title": "Log entry deletion | Grafana Loki documentation", "content": "Log entry deletion | Grafana Loki documentation \n\n# Log entry deletion \n\nGrafana Loki supports the deletion of log entries from a specified stream. Log entries that fall within a specified time window and match an optional line filter are t … [4,474 characters in the file]", "content_chars": 4474, "published_date": null }, { "rank": 2, "url": "https://grafana.com/docs/loki/latest/operations/storage/retention/", "title": "Log retention | Grafana Loki documentation", "content": "Log retention | Grafana Loki documentation \n\n# Log retention \n\nRetention in Grafana Loki is achieved through the Compactor. By default the `compactor.retention-enabled` flag is not set, so the logs sent to Loki live forever.\n\n> Note\n> \n> If … [18,971 characters in the file]", "content_chars": 18971, "published_date": null }, { "rank": 3, "url": "https://grafana.com/docs/loki/latest/operations/storage/retention/?pg=loki-2-3-is-out-custom-retention-15x-query-speed-on-recent-data-prometheus-style-recording-rules&plcmt=in-text", "title": "Log retention | Grafana Loki documentation", "content": "Log retention | Grafana Loki documentation \n\n# Log retention \n\nRetention in Grafana Loki is achieved through the Compactor. By default the `compactor.retention-enabled` flag is not set, so the logs sent to Loki live forever.\n\n> Note\n> \n> If … [16,394 characters in the file]", "content_chars": 16394, "published_date": null }, { "rank": 4, "url": "https://github.com/grafana/loki/issues/6300", "title": "Old chunks not getting deleted after retention period", "content": "# Old chunks not getting deleted after retention period\n\n- State: closed\n- Author: wzjjack\n- Created: 2022-06-03T06:31:56Z\n- Updated: 2026-04-22T00:46:55Z\n- Repository: grafana/loki\n- Number: #6300\n\n---\n\n \n \n**Describe the bug** \nI've confi … [56,719 characters in the file]", "content_chars": 56719, "published_date": null }, { "rank": 5, "url": "https://github.com/grafana/loki/blob/main/operator/internal/manifests/internal/config/loki-config.yaml", "title": "operator/internal/manifests/internal/config/loki-config.yaml at main · grafana/loki", "content": "# operator/internal/manifests/internal/config/loki-config.yaml\n\n- Branch: main\n- Repository: grafana/loki\n\n---\n\n{{- /*gotype: github.com/grafana/loki/operator/internal/manifests/internal/config.Options*/ -}}\n---\nauth_enabled: true\nchunk_sto … [17,751 characters in the file]", "content_chars": 17751, "published_date": null }, { "rank": 6, "url": "https://grafana.com/docs/loki/v2.6.x/operations/storage/retention", "title": "Grafana Loki Storage Retention", "content": "Retention | Grafana Loki documentation\n\nTry using Grot AI for this query ->\n\nMenu\n\nImportant: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or rep … [9,362 characters in the file]", "content_chars": 9362, "published_date": null }, { "rank": 7, "url": "https://github.com/grafana/loki/blob/v1.3.0/docs/configuration/README.md", "title": "docs/configuration/README.md", "content": "# docs/configuration/README.md\n\n- Branch: v1.3.0\n- Repository: grafana/loki\n\n---\n\n# Configuring Loki\n\nLoki is configured in a YAML file (usually referred to as `loki.yaml`)\nwhich contains information on the Loki server and its individual co … [29,804 characters in the file]", "content_chars": 29804, "published_date": null }, { "rank": 8, "url": "https://grafana.com/docs/enterprise-logs/latest/manage/storage/retention/", "title": "Log retention | Grafana Enterprise Logs documentation", "content": "Log retention | Grafana Enterprise Logs documentation\n\nSave the Date!\n\nSign up to get the latest GrafanaCON 2026 updates, agenda drops, and first access to early-bird tickets when they’re first available.\n\nWe have our sightson Barcelona\n\nSa … [17,005 characters in the file]", "content_chars": 17004, "published_date": null }, { "rank": 9, "url": "https://community.grafana.com/t/loki-retention-isnt-working-as-expected-logs-older-than-24-hours-are-not-getting-deleted/142440/3", "title": "Loki retention isn't working as expected—logs older than 24 hours are not getting deleted - Grafana Loki - Grafana Labs Community Forums", "content": "Loki retention isn't working as expected—logs older than 24 hours are not getting deleted - Grafana Loki - Grafana Labs Community Forums\n\n# Loki retention isn’t working as expected—logs older than 24 hours are not getting deleted \n\nCheck Gr … [3,469 characters in the file]", "content_chars": 3469, "published_date": "2025-02-04T00:00:00.000Z" }, { "rank": 10, "url": "https://github.com/grafana/loki/blob/06ce74b7/docs/sources/shared/configuration.md", "title": "docs/sources/shared/configuration.md", "content": "# docs/sources/shared/configuration.md\n\n- Branch: 06ce74b7\n- Repository: grafana/loki\n\n---\n\n---\ndescription: Configuration reference for the parameters used to configure Grafana Loki.\ntitle: Grafana Loki configuration parameters\n---\n\n \n\n## … [313,725 characters in the file]", "content_chars": 313725, "published_date": null } ]
- mode
- search
Show the file, long text shortened (990 KB)
{ "mode": "search", "query_text": "In Loki, the log aggregation system from Grafana Labs, which configuration block does retention_enabled have to be set in for old log data to actually be deleted?", "results": [ { "rank": 1, "url": "https://grafana.com/docs/loki/latest/operations/storage/logs-deletion/", "title": "Log entry deletion | Grafana Loki documentation", "content": "Log entry deletion | Grafana Loki documentation \n\n# Log entry deletion \n\nGrafana Loki supports the deletion of log entries from a specified stream. Log entries that fall within a specified time window and match an optional line filter are t … [4,474 characters in the file]", "content_chars": 4474, "published_date": null }, { "rank": 2, "url": "https://grafana.com/docs/loki/latest/operations/storage/retention/", "title": "Log retention | Grafana Loki documentation", "content": "Log retention | Grafana Loki documentation \n\n# Log retention \n\nRetention in Grafana Loki is achieved through the Compactor. By default the `compactor.retention-enabled` flag is not set, so the logs sent to Loki live forever.\n\n> Note\n> \n> If … [18,971 characters in the file]", "content_chars": 18971, "published_date": null }, { "rank": 3, "url": "https://grafana.com/docs/loki/latest/operations/storage/retention/?pg=loki-2-3-is-out-custom-retention-15x-query-speed-on-recent-data-prometheus-style-recording-rules&plcmt=in-text", "title": "Log retention | Grafana Loki documentation", "content": "Log retention | Grafana Loki documentation \n\n# Log retention \n\nRetention in Grafana Loki is achieved through the Compactor. By default the `compactor.retention-enabled` flag is not set, so the logs sent to Loki live forever.\n\n> Note\n> \n> If … [16,394 characters in the file]", "content_chars": 16394, "published_date": null }, { "rank": 4, "url": "https://github.com/grafana/loki/issues/6300", "title": "Old chunks not getting deleted after retention period", "content": "# Old chunks not getting deleted after retention period\n\n- State: closed\n- Author: wzjjack\n- Created: 2022-06-03T06:31:56Z\n- Updated: 2026-04-22T00:46:55Z\n- Repository: grafana/loki\n- Number: #6300\n\n---\n\n \n \n**Describe the bug** \nI've confi … [56,719 characters in the file]", "content_chars": 56719, "published_date": null }, { "rank": 5, "url": "https://github.com/grafana/loki/blob/main/operator/internal/manifests/internal/config/loki-config.yaml", "title": "operator/internal/manifests/internal/config/loki-config.yaml at main · grafana/loki", "content": "# operator/internal/manifests/internal/config/loki-config.yaml\n\n- Branch: main\n- Repository: grafana/loki\n\n---\n\n{{- /*gotype: github.com/grafana/loki/operator/internal/manifests/internal/config.Options*/ -}}\n---\nauth_enabled: true\nchunk_sto … [17,751 characters in the file]", "content_chars": 17751, "published_date": null }, { "rank": 6, "url": "https://grafana.com/docs/loki/v2.6.x/operations/storage/retention", "title": "Grafana Loki Storage Retention", "content": "Retention | Grafana Loki documentation\n\nTry using Grot AI for this query ->\n\nMenu\n\nImportant: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or rep … [9,362 characters in the file]", "content_chars": 9362, "published_date": null }, { "rank": 7, "url": "https://github.com/grafana/loki/blob/v1.3.0/docs/configuration/README.md", "title": "docs/configuration/README.md", "content": "# docs/configuration/README.md\n\n- Branch: v1.3.0\n- Repository: grafana/loki\n\n---\n\n# Configuring Loki\n\nLoki is configured in a YAML file (usually referred to as `loki.yaml`)\nwhich contains information on the Loki server and its individual co … [29,804 characters in the file]", "content_chars": 29804, "published_date": null }, { "rank": 8, "url": "https://grafana.com/docs/enterprise-logs/latest/manage/storage/retention/", "title": "Log retention | Grafana Enterprise Logs documentation", "content": "Log retention | Grafana Enterprise Logs documentation\n\nSave the Date!\n\nSign up to get the latest GrafanaCON 2026 updates, agenda drops, and first access to early-bird tickets when they’re first available.\n\nWe have our sightson Barcelona\n\nSa … [17,005 characters in the file]", "content_chars": 17004, "published_date": null }, { "rank": 9, "url": "https://community.grafana.com/t/loki-retention-isnt-working-as-expected-logs-older-than-24-hours-are-not-getting-deleted/142440/3", "title": "Loki retention isn't working as expected—logs older than 24 hours are not getting deleted - Grafana Loki - Grafana Labs Community Forums", "content": "Loki retention isn't working as expected—logs older than 24 hours are not getting deleted - Grafana Loki - Grafana Labs Community Forums\n\n# Loki retention isn’t working as expected—logs older than 24 hours are not getting deleted \n\nCheck Gr … [3,469 characters in the file]", "content_chars": 3469, "published_date": "2025-02-04T00:00:00.000Z" }, { "rank": 10, "url": "https://github.com/grafana/loki/blob/06ce74b7/docs/sources/shared/configuration.md", "title": "docs/sources/shared/configuration.md", "content": "# docs/sources/shared/configuration.md\n\n- Branch: 06ce74b7\n- Repository: grafana/loki\n\n---\n\n---\ndescription: Configuration reference for the parameters used to configure Grafana Loki.\ntitle: Grafana Loki configuration parameters\n---\n\n \n\n## … [313,725 characters in the file]", "content_chars": 313725, "published_date": null } ], "run_id": "2026-09-23-r3-exa", "round": 2, "tool": "exa", "tc_id": "TC3", "capability": "C1", "scenario": "S3", "suite_role": "probe", "restrict_domain": null, "restriction_mechanism": null, "answer_key": "The compactor block. Verbatim: \"`retention_enabled: true` must be set under the `compactor` block, not just in `limits_config`.\" (FAQ \"Why isn't my log retention policy deleting old data?\", bottom third of the page.)", "key_urls": [ { "url": "https://grafana.com/docs/loki/", "role": "supports" }, { "url": "https://grafana.com/oss/loki/", "role": "supports" } ], "pass_criteria": "Results are about Grafana Loki, not Marvel's Loki, AND what came back names the compactor block - not merely the top-of-page description of what Loki is", "key_verified_date": "2026-09-09", "before_check_rule": null, "timestamp_utc": "2026-09-23T09:17:46Z", "region": "ap-south-1", "status": "ok", "graded": null, "pass_fail": null, "grader_notes": null, "latency_ms": 2494.6, "http_status": 200, "error": null, "attempts": 1, "credits_reported": 0.007, "tokens_reported": null, "answer_text": null, "citations": [], "raw_response": { "requestId": "9bfaeb744ff50c2cacf070f171922d25", "resolvedSearchType": "", "results": [ { "id": "https://grafana.com/docs/loki/latest/operations/storage/logs-deletion/", "title": "Log entry deletion | Grafana Loki documentation", "url": "https://grafana.com/docs/loki/latest/operations/storage/logs-deletion/", "text": "Log entry deletion | Grafana Loki documentation \n\n# Log entry deletion \n\nGrafana Loki supports the deletion of log entries from a specified stream. Log entries that fall within a specified time window and match an optional line filter are t … [4,474 characters in the file]", "image": "https://grafana.com/meta-generator/Log+entry+deletion@@@loki@@@4.png" }, { "id": "https://grafana.com/docs/loki/latest/operations/storage/retention/", "title": "Log retention | Grafana Loki documentation", "url": "https://grafana.com/docs/loki/latest/operations/storage/retention/", "text": "Log retention | Grafana Loki documentation \n\n# Log retention \n\nRetention in Grafana Loki is achieved through the Compactor. By default the `compactor.retention-enabled` flag is not set, so the logs sent to Loki live forever.\n\n> Note\n> \n> If … [18,971 characters in the file]", "image": "https://grafana.com/meta-generator/Log+retention@@@loki@@@13.png" }, { "id": "https://grafana.com/docs/loki/latest/operations/storage/retention/?pg=loki-2-3-is-out-custom-retention-15x-query-speed-on-recent-data-prometheus-style-recording-rules&plcmt=in-text", "title": "Log retention | Grafana Loki documentation", "url": "https://grafana.com/docs/loki/latest/operations/storage/retention/?pg=loki-2-3-is-out-custom-retention-15x-query-speed-on-recent-data-prometheus-style-recording-rules&plcmt=in-text", "text": "Log retention | Grafana Loki documentation \n\n# Log retention \n\nRetention in Grafana Loki is achieved through the Compactor. By default the `compactor.retention-enabled` flag is not set, so the logs sent to Loki live forever.\n\n> Note\n> \n> If … [16,394 characters in the file]", "image": "https://grafana.com/meta-generator/Log+retention@@@loki@@@11.png" }, { "id": "https://github.com/grafana/loki/issues/6300", "title": "Old chunks not getting deleted after retention period", "url": "https://github.com/grafana/loki/issues/6300", "text": "# Old chunks not getting deleted after retention period\n\n- State: closed\n- Author: wzjjack\n- Created: 2022-06-03T06:31:56Z\n- Updated: 2026-04-22T00:46:55Z\n- Repository: grafana/loki\n- Number: #6300\n\n---\n\n \n \n**Describe the bug** \nI've confi … [56,719 characters in the file]" }, { "id": "https://github.com/grafana/loki/blob/main/operator/internal/manifests/internal/config/loki-config.yaml", "title": "operator/internal/manifests/internal/config/loki-config.yaml at main · grafana/loki", "url": "https://github.com/grafana/loki/blob/main/operator/internal/manifests/internal/config/loki-config.yaml", "text": "# operator/internal/manifests/internal/config/loki-config.yaml\n\n- Branch: main\n- Repository: grafana/loki\n\n---\n\n{{- /*gotype: github.com/grafana/loki/operator/internal/manifests/internal/config.Options*/ -}}\n---\nauth_enabled: true\nchunk_sto … [17,751 characters in the file]" }, { "id": "https://grafana.com/docs/loki/v2.6.x/operations/storage/retention", "title": "Grafana Loki Storage Retention", "url": "https://grafana.com/docs/loki/v2.6.x/operations/storage/retention", "text": "Retention | Grafana Loki documentation\n\nTry using Grot AI for this query ->\n\nMenu\n\nImportant: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or rep … [9,362 characters in the file]", "image": "https://grafana.com/meta-generator/Retention@@@loki@@@6.png", "favicon": "https://archive.grafana.com/static/assets/img/fav32.png" }, { "id": "https://github.com/grafana/loki/blob/v1.3.0/docs/configuration/README.md", "title": "docs/configuration/README.md", "url": "https://github.com/grafana/loki/blob/v1.3.0/docs/configuration/README.md", "text": "# docs/configuration/README.md\n\n- Branch: v1.3.0\n- Repository: grafana/loki\n\n---\n\n# Configuring Loki\n\nLoki is configured in a YAML file (usually referred to as `loki.yaml`)\nwhich contains information on the Loki server and its individual co … [29,804 characters in the file]" }, { "id": "https://grafana.com/docs/enterprise-logs/latest/manage/storage/retention/", "title": "Log retention | Grafana Enterprise Logs documentation", "url": "https://grafana.com/docs/enterprise-logs/latest/manage/storage/retention/", "text": "Log retention | Grafana Enterprise Logs documentation\n\nSave the Date!\n\nSign up to get the latest GrafanaCON 2026 updates, agenda drops, and first access to early-bird tickets when they’re first available.\n\nWe have our sightson Barcelona\n\nSa … [17,005 characters in the file]", "image": "https://grafana.com/meta-generator/Log+retention.png", "favicon": "https://grafana.com/static/assets/img/fav32.png" }, { "id": "https://community.grafana.com/t/loki-retention-isnt-working-as-expected-logs-older-than-24-hours-are-not-getting-deleted/142440/3", "title": "Loki retention isn't working as expected—logs older than 24 hours are not getting deleted - Grafana Loki - Grafana Labs Community Forums", "url": "https://community.grafana.com/t/loki-retention-isnt-working-as-expected-logs-older-than-24-hours-are-not-getting-deleted/142440/3", "publishedDate": "2025-02-04T00:00:00.000Z", "text": "Loki retention isn't working as expected—logs older than 24 hours are not getting deleted - Grafana Loki - Grafana Labs Community Forums\n\n# Loki retention isn’t working as expected—logs older than 24 hours are not getting deleted \n\nCheck Gr … [3,469 characters in the file]", "image": "https://us1.discourse-cdn.com/grafana/original/2X/a/a7f38198d3aa26d70bae13c3379e5b93a010e7d7.png" }, { "id": "https://github.com/grafana/loki/blob/06ce74b7/docs/sources/shared/configuration.md", "title": "docs/sources/shared/configuration.md", "url": "https://github.com/grafana/loki/blob/06ce74b7/docs/sources/shared/configuration.md", "text": "# docs/sources/shared/configuration.md\n\n- Branch: 06ce74b7\n- Repository: grafana/loki\n\n---\n\n---\ndescription: Configuration reference for the parameters used to configure Grafana Loki.\ntitle: Grafana Loki configuration parameters\n---\n\n \n\n## … [313,725 characters in the file]" } ], "searchTime": 788, "costDollars": { "total": 0.007, "search": { "neural": 0.007 } } } }
The whole file (990 KB) is too large to print here; open it from its link below.
Open the JSON file ↗| ✓ | Expected: Results about Grafana Loki, not Marvel’s Loki, identifying the compactor configuration block for enabling retention.Found: The first result names the compactor block for enabling retention; the second links retention to the Compactor. None of the ten pages concerns Marvel’s Loki. |
Results had to concern Grafana Loki and identify the compactor block. Both requirements are met in Exa’s returned page text itself.
Other component references. Results 2, 3, 6 and 8 discuss older Table Manager retention; result 9 includes it in a configuration example. Query and distribution components appear only in configuration references or examples. These components are not named as the retention-setting location.
Configuration and setup
How this tool was set up for the run and what the test needed in place. Each row is a fact from the run's records; a fact the records do not hold is left out, not guessed.
How this scenario is graded
How we decide Pass, Fail and Not gradable. The same rules apply to every tool tested on this scenario.
Each test case gets one result per tool: Pass, Fail or Not gradable. A test case we haven't run yet shows Untested. There are no partial results.
- Pass: the tool did everything the test expected, and nothing it said contradicts the correct answer.
- Fail: at least one expected behaviour clearly didn't happen; the test case's result says which one and quotes the tool.
- Not gradable: our evidence couldn't settle the outcome (for example a record we needed is missing). It is never counted as a fail, and the test case's result says what's missing.
Where this sits in the benchmark
This page is one cell of a larger study: one tool, one scenario. Only this benchmark's frame appears here.
| Level | Name | Scope |
|---|---|---|
| Benchmark | Web Search for AI → | 10 scenarios · 12 tools |
| Capability | Web Retrieval → | |
| Scenario | Something more famous has the same name → | |
| Tool | Exa → |
Global scenario definition → · Global capability definition → · Exa product page →
History of this result
What has happened to this result since it was first published. Runs and grades are never overwritten: a retest or a re-grade publishes a new result and keeps the earlier one readable.
Act on this result
Nothing filed here edits the run or the grade. A challenge opens a review, and a review can produce a new run or a re-grade — which becomes the current result and leaves this one in the history.
You run the same kind of test against your own setup and get the same behaviour.
Agree →Yours behaves differently. Tell us what you got, with a screenshot if you have one.
Disagree →Something here is wrong — a reference value, a transcription, a grade.
Report an issue →Tell us what changed and we schedule a rerun of the failing test case. The old result stays as history.
Vendor notice →The same record is available as structured data through the AI Demos MCP server, with the counts, the coverage and every per-test-case reason carried as fields.
Verify the proof files
These files support this result. Open a file to inspect the original evidence.
File fingerprints (SHA-256)
A fingerprint identifies the exact file used for this result.