Add Haiku 5.5 and Fable pricing; record Haiku/Opus 5.5/Fable 5.1 eval

[mattgodbolt]

Summary

• Adds the missing price entries (claude-haiku-5-5 at the up-to-100k-token tier, claude-fable-5, claude-fable-5-1) so prompt-test can cost runs on those models, with tests.
• Records the evaluation below in CLAUDE.md so the "should we move to a newer or cheaper model?" question doesn't get re-run from scratch. No change to the production model or prompt: Sonnet 5.5 stays.

Eval

Production prompt text for every config; only the model: block differs. 21 cases, Opus 5 adaptive reviewer (same method as #48). Accuracy is the 16 assembly cases per run; the haiku-type cases are excluded because the reviewer marks every haiku "incorrect".

Config	Runs	Correct (asm)	Errors	Latency mean / p95 / max	$/call
Sonnet 5.5 (prod)	3	43/48 (3 reviewer failures, 43/45 reviewed)	2	5.2 / 9.4 / 11.9s	0.0108
Haiku 5.5, thinking: disabled	3	36/48	14	3.1 / 5.2 / 5.7s	0.0005
Haiku 5.5, adaptive medium (API default)	3	41/48	8	5.8 / 9.7 / 17.6s	0.0008
Haiku 5.5, adaptive low	3	not reviewed		5.0 / 9.0 / 18.1s	0.0007
Haiku 5.5, adaptive high	3 (1 reviewed)	12/15	3	8.1 / 15.4 / 18.4s	0.0011
Opus 5.5, low effort	1	16/16	0	6.6 / 11.7 / 13.2s	0.021
Opus 5.5, medium effort	2	30/31 reviewed	1	9.6 / 16.2 / 18.7s	0.026
Fable 5.1	1	15/16	1	10.1 / 21.3 / 37.1s	0.059

• Haiku 5.5 with thinking off is the only faster option (about 40% quicker, 20x cheaper), and it is much less accurate. The errors are failures to trace: a straight-line .L4 called a loop, wrong register arithmetic, a hallucinated imul eax, edi, edi, "-O0 doesn't constant-fold". On the je .L4 / .L4: fall-through case from the TODO it misdescribed the jump in about 5 of 12 samples; Sonnet 5.5, Opus 5.5, Fable 5.1 and Haiku-adaptive got it right every time.
• Haiku 5.5 with adaptive thinking recovers some accuracy but is no faster than Sonnet. At high effort, thinking exhausted max_tokens=4096 and gave an empty response on loop_experienced_assembly in 2 of 3 runs.
• Opus 5.5 is within noise of Sonnet at 2-2.5x the cost and 27-85% more latency. Fable 5.1 reached a 37s max, over the 30s API Gateway ceiling.
• Haiku 5.5 accepts thinking: {type: disabled} (Sonnet 5.5 does not) and rejects sampling parameters; Opus 5.5 and Fable 5.1 cannot disable thinking.

Caveats: small samples (Opus 5.5 low and Fable 5.1 are one run each), and the reviewer itself failed to produce a verdict on 4 of the 80 assembly reviews of the Sonnet 5.5 (3 of 48) and Opus 5.5 medium (1 of 32) outputs (JSON parse failures and max_tokens starvation, on the longest assembly).

Testing

• uv run pytest: 116 passed. uv run pre-commit run --all-files: clean.

🤖 Generated with Claude Code