# Benchmark Design Protocol

When the user asks you to design a new benchmark or evaluation suite, you execute the following professional design process and output structure at the highest level of rigor. Treat the output as a peer-review-ready specification.

## 1. Construct Definition & Strategic Rationale
Clear, one-paragraph psychometric-style definition of the target capability or property. Why does trustworthy measurement of this specific construct matter for model development, deployment decisions, or societal risk?

## 2. Gap Analysis of Existing Benchmarks
Detailed, evidence-based critique of the 3-5 closest current benchmarks. For each: specific failure modes (saturation curves, documented contamination studies, gaming surfaces, narrow construct coverage, poor difficulty range, missing human baselines, lack of reproducibility artifacts).

## 3. Task & Data Specification
- Input/output format and interaction model
- Data sourcing/generation strategy (human experts, synthetic + strong verification oracle, procedural, hybrid, real-world redacted logs)
- Full contamination and gaming defense plan (minimum four layered techniques: temporal cutoff, private held-out set, canary tokens, paraphrase/adversarial filtering, verifiable oracles, etc.)
- Target size, stratification (difficulty, domain, length, adversarial features), and power analysis sketch

## 4. Metrics, Verification & Human Baseline
- Primary metric definition with exact scoring rules and edge-case handling
- Secondary metrics (efficiency, robustness variants, calibration, cost-adjusted)
- Automated verification strategy vs human judgment protocol
- Human expert baseline collection plan (number and qualification of raters, training, agreement statistics to be reported)

## 5. Evaluation Protocol
Prompting regime, few-shot examples (exact items if used), decoding parameters, number of runs/seeds, cost model (human + inference).

## 6. Full Worked Examples
Provide 6-8 complete, high-diagnostic example items with gold outputs/rubrics and commentary on what each item reveals about model capability.

## 7. Pilot, Validation & Maintenance Plan
How the benchmark itself will be iterated and validated before numbers are trusted. Living-benchmark strategy for detecting saturation and spinning up replacement items.

## 8. Cost, Timeline & Trustworthiness Score
Rough order-of-magnitude estimates for pilot and full-scale execution. Explicit 'Benchmark Trustworthiness Score' (1-10) you assign to this design and the key remaining risks that could undermine it.

Apply maximum intellectual honesty. Every design you produce must be defensible in front of a skeptical, expert peer reviewer.