ai-benchmarks · methodology review

How we test the models

Three capability tiers, six suites, one runner. Every model receives byte-identical prompts and is scored the same way — that identity is what makes this an eval rather than an anecdote. This page is the plan to approve before we spend compute.

The shape · 3 tiers × 2 suites

Each tier pairs a fast micro-bench (text, cheap, deterministic) with a rich main suite (a real artifact). Same capability, two depths.

TIER 01

Knowledge

Does it know the world, and can it teach?

tier1-physics substring

Particle-physics recall, terse answers

24 held-out · 4 sample

tier1-world substring

World-development literacy (Rosling / Factfulness)

20 held-out · 4 sample

tier1-explain judged

Explain a phenomenon (double-slit, Bell, Higgs…)

6 held-out · 2 sample

TIER 02

Building

Can it write code that works?

tier2-w40k judged

Dice-mechanics CLIs to spec

6 held-out · 2 sample

tier2-game judged

Playable browser minigame

3 held-out · 1 sample

TIER 03

Communication

Can it present, accurately?

tier3-apollo judged

Apollo Marp slide decks

6 held-out · 2 sample

tier3-deck judged

Management-report HTML decks (SVG + photos)

3 held-out · 2 sample

How scoring works · two regimes

Substring-QA tier1-physics

30 runs per question at temperature 1.0; the score is the pass rate over that stochastic sample.
Case-insensitive substring match against accepted surface forms; negative answers override (a confused reply fails even if it contains a right token).
No LLM in the loop — fully deterministic and cheap. This is the SkateBench-compatible path.

Judged the other five

The model produces an artifact (code, a deck, an essay). Score = a weighted sum to 1.0 of three parts:
■ Script checks — deterministic, headless (does the code run, are the facts present, does the game logic pass).
■ LLM judge — a model grades quality from the source, JSON rubric, retry-on-malformed.
■ Human — you score feel/presentation via an arena page; folds in later without re-running.

script checks (objective floor)LLM judgehuman arena

tier1-explain

6040

tier2-w40k

8515

tier2-game

701020

tier3-apollo

5050

tier3-deck

503020

The deterministic floor always carries the majority weight, so a generous judge or a flashy-but-broken artifact can't win. Human weight is reserved for the two suites where feel genuinely matters — game and deck.

Who gets tested · the roster

Model	Provider / access	Runs	Status
Qwen3-Coder-Next	oMLX · local MLX	30	enabled
Gemma-4-26B	oMLX · local MLX	30	enabled
gpt-oss-20b	oMLX · local MLX	30	enabled
qwen2.5-coder-7b	Ollama · local	30	enabled · pilot
GPT-5.5	codex-cli · ChatGPT sub	5	enabled · quota-capped
LFM2-24B · GLM-4.6V-Flash	oMLX · local MLX	30	available, off
GLM-5.2	z.ai · remote API	30	needs key

The rules that keep it honest

Fairness & rigor

Judge = Claude Opus (via claude-cli, your existing auth, no API key), configurable per run with --judge-model. A model never judges its own build; one judge is pinned per table.
30 runs, temp 1.0 for free local models; paid GPT-5.5 capped at 5 to respect quota. Scores are means over runs.
Cache-resume — every run is content-signature keyed, so reruns resume and edited prompts invalidate cleanly.
Human folds in late — arena scores drop into a file and re-fold at report time; no re-running the models.

Contamination control

Held-out majority lives in a private sibling repo; only small public samples ship in the open harness.
Canary GUID stamped in every held-out suite — if it ever appears in a model's output, that's training-leak evidence and the suite version retires.
Frozen prompts — once real runs start, prompt text is locked (cache signatures hard-error on drift); changes bump the version.
Each suite has an eval card (docs/evals/) stating what it measures and, crucially, what it does not.

What a result looks like

Per suite: a leaderboard of models by successRate (mean score ×100), with per-check and per-criterion breakdowns and every artifact kept for replay. Across suites: a tier roll-up — knowledge / building / communication — so a model's shape shows, not just one number. Reports regenerate from cache with zero model calls, and late human scores re-fold in place.

Decisions for you

1 · Which models are in the first real matrix?

Enabled now: Qwen3-Coder-Next, Gemma-4-26B, gpt-oss-20b, qwen2.5-7b, GPT-5.5. Add LFM2 / GLM-4.6V (a flag flip) or GLM-5.2 (needs a z.ai key)?

2 · Sample-first or straight to held-out?

Public samples are fast and safe to re-run; the held-out suites are the real measurement (24 physics Qs + 6 each judged). Recommend one sample pass to sanity-check, then held-out.

3 · Judge model — set to Opus.

Now wired: judge-opus (Claude Opus via your Claude auth) is the default judge, fully configurable via --judge-model (judge-sonnet for a cheaper pass, or any local model). Keep Opus, or swap for specific tables?

4 · GPT-5.5 run count.

Capped at 5 for quota. Raise toward 30 for tighter numbers, or keep 5 and note the wider confidence interval?