Repository navigation
Fix eval fixture label maps; add hard test corpus; harden the reviewer - #54
Merged
Merged
Conversation
The prompt-testing harness had several problems that made measurements unreliable, and its cases were too easy to separate models. - ce_api: `labelDefinitions` was rebuilt from each line's `labels`, which lists the labels an instruction *references*, as a 0-based index of the first use. Compiler Explorer returns the real map (label -> 1-based line of its definition) and the explain service receives that in production. Use the API's map, and derive it from definition lines when absent. - Recompute `labelDefinitions` in all existing fixtures, and add test_fixtures.py so they cannot drift again. - Add 32 `hard_*` cases (C, C++, Rust, x86-64, AArch64, RISC-V, Wasm, ARM32) built from real CE output. The old cases are near ceiling for Sonnet 5.5; these leave room to compare models and prompts. - Reviewer: use structured outputs so a verdict can never fail to parse, and give thinking-enabled reviews 16k max_tokens (4096 starved the long listings). Run reviews concurrently (default 6) and skip poem cases, which the reviewer always marks incorrect and which never counted. - Document the corpora, the reviewer's refusal on hard_wasm_sum_loop, and the noise level of reviewer-judged accuracy. No change to production code or prompts. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
mattgodbolt
added a commit
that referenced
this pull request
Oct 11, 2026
## Summary One sentence changes in `app/prompt.yaml`: the length instruction goes from "under roughly 250 words" to a hard cap on bullets and sentences. Output length is nearly all of the latency (a Sonnet 5.5 call takes about 0.4s plus 7.7ms per output token), and the word limit was being ignored: explanations averaged 423 words on the hard corpus. ```diff -Keep the whole explanation under roughly 250 words; only exceed that when the assembly is unusually long or complex, and even then stay brief. +Hard limit: at most 6 bullets or short paragraphs in total, none longer than two sentences, and no headings. Only when the assembly is unusually long or complex may you use up to 10. Lead with the single most important insight, and omit anything the reader could infer from the instruction names alone. ``` Anthropic's latency guide notes that word limits work worse than sentence or paragraph limits, which matches what we saw. ## Eval Exactly the text above (nothing else changed), 2 runs of the full suite, Opus 5 adaptive reviewer, against 2 runs of current `main` on the same corpora. This depends on the fixture and reviewer fixes in #54 (the runs used them), but the prompt change itself does not. | | current | this PR | |---|---|---| | Mean latency | 8.2s | **6.5s** (-21%) | | p95 / max latency | 12.2s / 17.7s | **9.0s / 9.9s** | | Output tokens / words | 1002 / 423 | 791 / 335 | | $/call | 0.0172 | 0.0151 | | Correct, original corpus | 30/32 (94%) | 27/32 (84%) | | Correct, hard corpus | 50/62 (81%) | 58/62 (94%) | | **Correct, both pooled** | **80/94 (85.1%)** | **85/94 (90.4%)** (Fisher p=0.37) | The two corpora move in opposite directions, which is what reviewer noise looks like here (two prompts differing only in JSON formatting scored 81% and 90% on the same hard-case reviews). Adding a second run of the same prompt edit with compacted input gives 165/188 (87.8%) vs 166/188 (88.3%) for baseline, p=1.0. So: no detectable change in accuracy, a clear and low-noise latency gain. The errors on the original corpus are the same kind as baseline's and spread over different cases (an `imul` operand form, a clang peel count, the `edge_undefined_001` UB case that fails for both), not a pattern of dropped content. <details> <summary>Before / after on <code>basic_loop_001</code> (beginner, gcc -O2 loop)</summary> **Before** (333 words, 5.7s): headed sections "Calling convention", "Setup (before the loop)", "The loop (`.L3`)", "Why this is efficient", then an `-O0`/`-O3` comparison. **After** (279 words, 5.3s): > - **The loop counter `i` is gone.** With `-O2`, GCC replaces the index with a pointer that walks through the array. `rdi` (the first argument, `arr`, by x86-64 convention) is advanced by 4 bytes each iteration, and the loop stops when it reaches an "end" address held in `rdx`. > - **Setup computes that end address.** `movsx rsi, esi` widens the 32-bit `n` ... Then `lea rdx, [rdi+rsi*4]` calculates `arr + n*4`. `lea` only does the arithmetic; it does not read memory. > - **The loop body is just four instructions** (`.L3`). ... > - **`xor eax, eax` is the idiom for `sum = 0`.** ... > - **The `n <= 0` case is handled up front.** ... > - **Comparison with `-O0`:** unoptimised code would keep `sum`, `i` and the pointer in stack memory ... Here everything lives in registers. </details> ## Things to know - It removes the headed sections (the beginner "Calling convention" heading, for instance) in favour of bullets; the content is still there inline. If you prefer headings for beginners, an audience-specific override is possible, at some latency cost. - Reviewer is Opus 5, which judges correctness and completeness, not pedagogy. Worth reading a few outputs. - It also reduces cost slightly (shorter outputs). ## Testing - `uv run pytest`: 113 passed. `uv run pre-commit run --all-files`: clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Eval-tooling only; no change to production code or prompts. While looking for ways to improve accuracy and speed I found the prompt-testing harness was giving unreliable measurements and could not separate good models from better ones.
prompt_testing/ce_api/models.pyrebuiltlabelDefinitionsfrom each asm line'slabels, which lists the labels an instruction references, as a 0-based index of the first use. Compiler Explorer returns the real map (label to 1-based line of its definition), and that is what the explain service receives in production. Every enriched fixture therefore fed the model label line numbers that contradicted the listing (e.g..L25claimed at line 5, defined at line 75). The client now passes CE's map through, deriving it from definition lines only if the API omits it. The rule reproduces CE's own map exactly on x86-64 gcc and clang, ARM clang, ARM64 gcc and Rust. All existing fixtures are recomputed (diff is confined tolabelDefinitionsblocks), andprompt_testing/test_fixtures.pyguards against drift.hard_cases.yamladds 32 cases built from real CE output (18 C, 10 C++, 4 Rust; x86-64, AArch64, RISC-V, Wasm, ARM32): jump tables, magic-number division, rotated loops, tails shared via backward jumps, landing pads in cold sections, ABI details, and theje .L4/.L4:artefact from the TODO. On these Sonnet 5.5 scores about 85% and Haiku 5.5 about 70%.max_tokensfor thinking-enabled reviews goes from 4096 to 16000. Reviews now run concurrently (--review-concurrency/--concurrency, default 6), which takes a 48-case file from about 20 minutes to a few. Poem-type cases are skipped: the reviewer always marks them incorrect, so they never counted.Known limitation
hard_wasm_sum_loopcannot be scored: the Opus 5 reviewer's safety classifier declines it (stop_reason: refusal, no output) for every config. The explainer models handle it fine. It is left in the corpus and documented.Testing
uv run pytest: 221 passed (includes the new fixture guards).uv run pre-commit run --all-files: clean.🤖 Generated with Claude Code