AGENT EVALUATIONS

ExploitGym counts: paper snapshot versus public release

Why the paper reports 898 instances while the public v1 task list contains 869.

ExploitGym evaluates whether an agent can extend a proof-of-vulnerability into a working exploit inside a containerized target. Counts from the paper and the maintained public release describe different snapshots, so a result should always record which task list it used.

Two valid counts

The May 2026 paper describes 898 instances: 520 userspace, 185 V8 and 193 Linux kernel cases. The public v1 release subsequently filtered non-exploitable cases and lists 869 instances: 502 userspace, 181 V8 and 186 kernel cases. The repository calls data/task_ids/v1.txt the canonical v1 list; see its changelog.

Minimum result identity

Keep the repository revision, named release, exact task-list digest, task selection, defense mode, agent implementation, model identifier, timeout and retry policy with every aggregate. A score labelled only “ExploitGym” cannot establish which denominator or environment produced it.

Safe comparisons

Compare intersections when two runs used different task lists. Report attempted, valid and successful instances separately. Treat setup failures and missing artifacts as their own outcomes rather than silently removing them from the denominator. Do not publish flags, generated exploits, private scorer details or per-task solution material merely to make an aggregate reproducible.

Alternate formats: Markdown · JSON

← Reference index