AGENT EVALUATIONS

A small manifest for reproducible agent-evaluation runs

Fields that keep scores comparable when environments and tool access change.

Aggregate scores hide configuration changes. Store a small, immutable manifest beside each run so later analysis can distinguish a model change from a task-list, agent, tool or environment change.

{
  "benchmark": "name",
  "benchmark_revision": "commit-or-release",
  "task_list_sha256": "hex-digest",
  "agent_revision": "commit-or-version",
  "model": "provider/model-version",
  "started_at": "UTC timestamp",
  "timeout_seconds": 7200,
  "network_profile": "documented-profile",
  "defense_profile": "documented-profile"
}

Result counts

Record selected, attempted, completed, successful, invalid and infrastructure-failed counts. Keep the raw status for every selected task, including timeouts. A reported success rate should state its numerator and denominator explicitly.

Integrity details

Hash the task list and important configuration files. Record whether web retrieval, package installation and inter-agent communication were available during the scored phase. Keep secrets and unrestricted transcripts in protected storage; publish only artifacts appropriate for the benchmark's disclosure policy.

Comparisons

Before comparing two runs, diff their manifests. If the task-list digest, scorer, defense profile or network profile differs, show that difference next to the scores. A version label alone may not capture a moving branch or local patch.

Alternate formats: Markdown · JSON

← Reference index