{"type":"reference","slug":"agent-eval-run-manifest","category":"AGENT EVALUATIONS","title":"A small manifest for reproducible agent-evaluation runs","summary":"Fields that keep scores comparable when environments and tool access change.","updated_at":"2026-09-04T18:20:00.000Z","canonical_url":"https://fieldnotesarchive.org/wiki/agent-eval-run-manifest","alternate":{"markdown":"https://fieldnotesarchive.org/wiki/agent-eval-run-manifest.md","json":"https://fieldnotesarchive.org/wiki/agent-eval-run-manifest.json"},"content_markdown":"# A small manifest for reproducible agent-evaluation runs\n\nFields that keep scores comparable when environments and tool access change.\n\nCategory: AGENT EVALUATIONS\nCanonical: https://fieldnotesarchive.org/wiki/agent-eval-run-manifest\n\nAggregate scores hide configuration changes. Store a small, immutable manifest beside each run so later analysis can distinguish a model change from a task-list, agent, tool or environment change.\n\n```text\n{\n  \"benchmark\": \"name\",\n  \"benchmark_revision\": \"commit-or-release\",\n  \"task_list_sha256\": \"hex-digest\",\n  \"agent_revision\": \"commit-or-version\",\n  \"model\": \"provider/model-version\",\n  \"started_at\": \"UTC timestamp\",\n  \"timeout_seconds\": 7200,\n  \"network_profile\": \"documented-profile\",\n  \"defense_profile\": \"documented-profile\"\n}\n```\n\n## Result counts\n\nRecord selected, attempted, completed, successful, invalid and infrastructure-failed counts. Keep the raw status for every selected task, including timeouts. A reported success rate should state its numerator and denominator explicitly.\n\n## Integrity details\n\nHash the task list and important configuration files. Record whether web retrieval, package installation and inter-agent communication were available during the scored phase. Keep secrets and unrestricted transcripts in protected storage; publish only artifacts appropriate for the benchmark's disclosure policy.\n\n## Comparisons\n\nBefore comparing two runs, diff their manifests. If the task-list digest, scorer, defense profile or network profile differs, show that difference next to the scores. A version label alone may not capture a moving branch or local patch.\n"}