---
name: "multiagent-evals-benchmark-suite"
description: "Benchmark agent accuracy, tool calling reliability, and task completion rates."
metadata:
  author: "OpenAI"
  version: "1.0.0"
  runtime_compatibility:
    - "codex"
    - "claude-code"
    - "cursor"
    - "windsurf"
triggers:
  - "benchmark agent performance"
  - "evaluate tool calling accuracy"
  - "run pass@k evals"
permissions:
  fs_read: true
  fs_write: "workspace"
  shell_exec: false
  net_access: false
---

# Multi-Agent Evals & Benchmark Scoring Suite

Runs automated evaluation suites against LLM agents: ground truth comparison, tool trajectory scoring, hallucination rates, token cost profiling, and pass@k metric calculation.

## When to Invoke
Invoke this skill whenever you need to:
- benchmark agent performance
- evaluate tool calling accuracy
- run pass@k evals

## Execution SOP & Rules
1. Establish a diverse test dataset of input prompts with verified expected outputs or unit tests.
2. Measure deterministic metrics: execution success rate, syntax errors, and test pass rates.
3. Track intermediate tool trajectory: penalize unnecessary loops, hallucinated tools, or redundant reads.
4. Generate structured benchmark reports with confidence intervals and latency breakdowns.

## Verification Checklist
- Run local linting, syntax validation, and type-checks before reporting completion.
- Ensure all created or modified files adhere to workspace boundaries and avoid uncommitted secrets.
- Verify tests or build commands exit cleanly with return code 0.