Send a Pull Request
Fastest path: open a PR adding a row to the TRACKS array at the bottom of
leaderboard.html. Include a short description of the setup and a link to your run logs
or code in the PR body.
Submitted results on ATM-Bench, ATM-Bench-Hard, and the NIAH long-context stress test. Use the tabs to switch boards, the chips to filter by system type, and click any column header to sort.
ATM-Bench-Hard agent runs: what each one scored against what it cost, at API list-price equivalent. A line joins one model's reasoning-effort tiers; a diamond is a single configuration.
The same weights, driven by different coding agents.
- indicates an unreported field.
Memory Model is the LLM used to build the memory store;
Retriever is the embedding model used at query time.
Click a column header to sort or a filter chip to narrow by system type.
When a caption model is not stated, it is Qwen3-VL-2B.
* The two legacy Memexa rows use a DeepSeek-V4-flash judge, not the standard gpt-5-mini judge, and are shown for reference rather than direct QS comparison.
◇ Memexa v2 uses DS-v4.1 descriptions and custom source-grounded cards; QS uses gpt-5-mini. Frozen evidence / per-question score summary.
- indicates an unreported field.
Coding agents use their default configuration unless the model label states a reasoning effort.
Results use SGM unless marked (w/o SGM).
When a caption model is not stated, it is Qwen3-VL-2B.
* The DeepSeek-V4-flash rows use a DeepSeek-V4-flash judge, not the standard gpt-5-mini judge; the baseline reruns are community-run, not the original authors' official numbers.
◇ Memexa v2 uses DS-v4.1 descriptions and custom source-grounded cards; QS uses gpt-5-mini. Frozen evidence / per-question score summary.
Cost is an API-equivalent estimate from saved token counters, not an invoice.
The same axes as the chart at the top of the page, with every configuration on it.
Needle-in-a-haystack sweep on ATM-Bench-Hard (31 questions): each question's gold evidence is hidden among k = 25 / 50 / 100 / 200 distractor memory items, under the SGM (Schema-Guided Memory) setting. Oracle = gold evidence only, no haystack. Approximate context depth is shown in each column header.
Why SGM, not raw? Raw (real images/video) edges out SGM at the Oracle ceiling. But that advantage collapses under realistic conditions: as the haystack fills with distractors, raw degrades and even fails (payload/context limits), and under agentic retrieval the gap is stark — every "w/o SGM" (raw) agent lands far below its SGM run. SGM is the representation that holds up once there is noise under realistic conditions.
We welcome new submissions across all three boards. To keep the leaderboard credible, please include reproduction details (system type, harness, model + version, code or commit, total token cost when applicable).
Fastest path: open a PR adding a row to the TRACKS array at the bottom of
leaderboard.html. Include a short description of the setup and a link to your run logs
or code in the PR body.
Prefer not to send a PR? File an issue with your system type, harness, scores, and a reproduction pointer. We will add the row on your behalf.
Open submission issueThis page was adopted from the Nerfies project page, licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. Many thanks to the Academic Project Page Template.