{"type":"reference","slug":"exploitgym-paper-vs-release","category":"AGENT EVALUATIONS","title":"ExploitGym counts: paper snapshot versus public release","summary":"Why the paper reports 898 instances while the public v1 task list contains 869.","updated_at":"2026-09-04T18:20:00.000Z","canonical_url":"https://fieldnotesarchive.org/wiki/exploitgym-paper-vs-release","alternate":{"markdown":"https://fieldnotesarchive.org/wiki/exploitgym-paper-vs-release.md","json":"https://fieldnotesarchive.org/wiki/exploitgym-paper-vs-release.json"},"content_markdown":"# ExploitGym counts: paper snapshot versus public release\n\nWhy the paper reports 898 instances while the public v1 task list contains 869.\n\nCategory: AGENT EVALUATIONS\nCanonical: https://fieldnotesarchive.org/wiki/exploitgym-paper-vs-release\n\nExploitGym evaluates whether an agent can extend a proof-of-vulnerability into a working exploit inside a containerized target. Counts from the paper and the maintained public release describe different snapshots, so a result should always record which task list it used.\n\n## Two valid counts\n\nThe [May 2026 paper](https://arxiv.org/abs/2605.11086) describes 898 instances: 520 userspace, 185 V8 and 193 Linux kernel cases. The public v1 release subsequently filtered non-exploitable cases and lists 869 instances: 502 userspace, 181 V8 and 186 kernel cases. The repository calls `data/task_ids/v1.txt` the canonical v1 list; see its [changelog](https://github.com/sunblaze-ucb/exploitgym/blob/main/CHANGELOG.md).\n\n## Minimum result identity\n\nKeep the repository revision, named release, exact task-list digest, task selection, defense mode, agent implementation, model identifier, timeout and retry policy with every aggregate. A score labelled only “ExploitGym” cannot establish which denominator or environment produced it.\n\n## Safe comparisons\n\nCompare intersections when two runs used different task lists. Report attempted, valid and successful instances separately. Treat setup failures and missing artifacts as their own outcomes rather than silently removing them from the denominator. Do not publish flags, generated exploits, private scorer details or per-task solution material merely to make an aggregate reproducible.\n"}