Send a Pull Request
Fastest path: open a PR adding a row to the TRACKS array at the bottom of
leaderboard.html. Include a short description of the setup and a link to your run logs
or code in the PR body.
Submitted results on ATM-Bench, ATM-Bench-Hard, and the NIAH long-context stress test. Use the tabs to switch boards, the chips to filter by system type, and click any column header to sort.
- indicates the field has not been reported by the submitter.
Memory Model is the LLM used to construct the memory store;
Retriever is the embedding model used at query time.
Click any column header to sort; click a filter chip to narrow by system type.
When a caption model is not stated, it is Qwen3-VL-2B (the default).
* Memexa's QS is measured with a DeepSeek-V4-flash judge (its own answer model), not the gpt-5-mini judge used for every other row, so it is shown for reference and is not directly comparable; Recall is judge-independent and like-for-like.
- indicates the field has not been reported.
All coding agents use their default configuration, including the reasoning effort.
Results use our proposed SGM (Schema-Guided Memory) unless marked (w/o SGM).
When a caption model is not stated, it is Qwen3-VL-2B (the default).
* QS on the DeepSeek-V4-flash rows (Memexa and the A-Mem / MemPalace / HippoRAG2 re-runs) is measured with a DeepSeek-V4-flash judge (the system's own model), not the gpt-5-mini judge used for the other rows — shown for reference, not directly comparable. The three baseline re-runs are community-run at this LLM tier, not the original authors' official numbers.
† Memexa's Recall is reported on the fixed Qwen3-VL-2B captions (like-for-like with other rows), although this configuration answers from Qwen3.6-27B captions.
‡ Memexa's Total Tokens / Cost include the one-time memory build (33.9M tokens / ~$3.80 over all 11,034 items, amortizable across evaluations) plus the 31-question query (0.40M / ~$0.04); other rows' agent costs are per full run.
Needle-in-a-haystack sweep on ATM-Bench-Hard (31 questions): each question's gold evidence is hidden among k = 25 / 50 / 100 / 200 distractor memory items, under the SGM (Schema-Guided Memory) setting. Oracle = gold evidence only, no haystack; judge: gpt-5-mini. Approximate context depth is shown in each column header.
Why SGM, not raw? Raw (real images/video) edges out SGM at the Oracle ceiling. But that advantage collapses under realistic conditions: as the haystack fills with distractors, raw degrades and even fails (payload/context limits), and under agentic retrieval the gap is stark — every "w/o SGM" (raw) agent lands far below its SGM run. SGM is the representation that holds up once there is noise under realistic conditions.
We welcome new submissions across all three boards. To keep the leaderboard credible, please include reproduction details (system type, harness, model + version, code or commit, total token cost when applicable).
Fastest path: open a PR adding a row to the TRACKS array at the bottom of
leaderboard.html. Include a short description of the setup and a link to your run logs
or code in the PR body.
Prefer not to send a PR? File an issue with your system type, harness, scores, and a reproduction pointer. We will add the row on your behalf.
Open submission issueThis page was adopted from the Nerfies project page, licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. Many thanks to the Academic Project Page Template.