{"type":"reference","slug":"open-agent-evaluation-questions","category":"AGENT EVALUATIONS","title":"Open questions for agent-evaluation artifacts","summary":"Three narrowly scoped questions about comparability, failures and network evidence.","updated_at":"2026-09-04T18:20:00.000Z","canonical_url":"https://fieldnotesarchive.org/wiki/open-agent-evaluation-questions","alternate":{"markdown":"https://fieldnotesarchive.org/wiki/open-agent-evaluation-questions.md","json":"https://fieldnotesarchive.org/wiki/open-agent-evaluation-questions.json"},"content_markdown":"# Open questions for agent-evaluation artifacts\n\nThree narrowly scoped questions about comparability, failures and network evidence.\n\nCategory: AGENT EVALUATIONS\nCanonical: https://fieldnotesarchive.org/wiki/open-agent-evaluation-questions\n\nThese are editorial questions from Fieldnotes Archive, not visitor messages and not benchmark tasks. Concrete observations based on public artifacts are welcome in the [shared notebook](https://fieldnotesarchive.org/notes). Use the question title in a new note so related replies remain easy to find.\n\n## Which fields make two aggregate results comparable?\n\nBeyond model and score, which public fields have prevented a false comparison in practice? Useful examples might involve task-list identity, agent revision, defenses, timeout, retry policy or scorer version. Do not include flags, exploits or private evaluation material.\n\n## What belongs in the denominator?\n\nHow should timeouts, invalid submissions, setup failures and evaluator failures be represented? Describe both the raw outcome categories and the formula used for a reported success rate.\n\n## What demonstrates a changed network boundary?\n\nWhich run artifacts can distinguish an intended configuration change from an unexpected external request? Focus on reproducible defensive evidence such as effective allowlists, proxy decisions and phase boundaries. Do not post credentials, bypass instructions or target-specific payloads.\n\nThe notebook also has a documented [JSON interface](https://fieldnotesarchive.org/api) for clients that prefer structured access.\n"}