# 📋 Standard AI Optimization Audit Protocol

## Week 0: Scoping & Access (1-2 days)

- Stakeholder interviews (product, eng, infra, finance)
- Access to logs, traces, cost dashboards, eval sets, model cards
- Definition of primary quality metric(s) and tolerance for regression

## Week 1: Measurement & Mapping

- Reproduce or extend golden test set from production logs (stratified sampling)
- Full-stack profiling (client → API → retrieval → LLM → tools → post-proc)
- Cost attribution by stage and by query class
- Identify heavy hitters (queries or stages consuming disproportionate resources)

## Week 2: Hypothesis Generation & Prioritization

- Apply ICE scoring to 8-15 candidate interventions
- Quick theoretical modeling (e.g., If we improve retrieval precision@5 from 0.61 to 0.81, we can reduce k from 12 to 5 and save X tokens)
- Select 3-5 experiments for immediate execution

## Weeks 3-6: Execute, Validate, Iterate

- Run controlled experiments (shadow or canary)
- Rigorous before/after on both efficiency *and* quality metrics
- Document learnings in the permanent playbook

## Week 6+: Embed & Hand Over

- Production rollout with automated monitoring
- Update runbooks and on-call procedures
- Train team on new techniques and the AOL framework
- Establish quarterly Optimization Review ritual

This protocol ensures no critical step is skipped and creates compounding organizational capability.