Fleet-style megakernel decode for DeepSeek-Coder-V2-Lite on AMD MI300X

Batch-1, BF16, autoregressive decode of DeepSeek-Coder-V2-Lite-Base, run as one persistent GPU kernel. Resident workers pull tasks from an on-device task graph and synchronise through event counters, in the style of AMD's Fleet / fleet-chiplet-megakernel runtime. The stock Hugging Face model runs the prefill; only decode goes through the Fleet path.

Results (one AMD MI300X, ROCm 7.2.4)

metric	result	evidence
Milestone	M4: end-to-end greedy decode of 32 tokens through the persistent megakernel	results/REPORT.md
Time per output token, P50 / P95	3.76 ms / 3.77 ms (iterations ≥ 2)	results/bench/bench.json
Tokens per second	266 (HF eager on the same GPU: 69.35 ms per token)	results/bench/bench.json
Correctness, all 27 layers vs HF eager bf16	every layer within gate, worst layer rel. error 1.6 % (layer 13), logits cosine 0.99997, same next token	results/perf/dag_g1_layers/layer_0-26.json
Free-running decode, 32 greedy tokens	32 / 32 tokens identical to HF (5 timed runs); the teacher-forced logit-cosine gate misses at 4 of 32 steps (expert-routing near-ties, reported as failing)	results/m4/e2e.json
GPU launches	1 persistent-kernel call per 32 tokens; HF eager: 3,853 kernel launches per token	results/bench/hf_baseline.json
Memory traffic / achieved bandwidth	5.24 GB per token / 1.39 TB/s	results/REPORT.md
Design target	2.5 ms per token not reached: 3.76 ms measured	docs/design-final-fixed.md §7

History, levers measured but not merged, where the time goes, and next steps: docs/performance.md.

Run it

On a fresh MI300X VM (Ubuntu 24.04, ROCm 7.2.4), from the laptop (mi300x is the ssh Host):

scripts/deploy.sh mi300x:AMD --git main     # push the committed tree over ssh
ssh mi300x 'cd AMD && mkdir -p results/m0 && nohup scripts/day1.sh > results/m0/day1.out 2>&1 < /dev/null &'   # ~10 min setup

Then on the box, from AMD/:

.venv/bin/python -u scripts/run_e2e.py --event-timing    # 32 greedy tokens vs HF   -> results/m4/e2e.json
.venv/bin/python -u scripts/bench.py                     # P50 / P95, tok/s, bandwidth -> results/bench/bench.json

Without the GPU (CPU only: reference vs HF, graph counts, kernels via clang++, runtime simulator, scripts):

scripts/setup_local.sh
.venv/bin/pytest -q tests

Every script and the full run order: scripts/README.md.

Repository map

folder	contents
fleet_ds/	Pure-torch reference decode step (bit-exact vs HF), task-graph builders (local and Mirage), CPU simulator of the Fleet runtime
kernels/	HIP task kernels (GEMV, attention chunk + merge, RMSNorm, MoE top-k / combine, argmax) and a CPU shim to test them with clang++
fleet_glue/	Patches and the register file that plug the kernels into the Fleet runtime
scripts/	VM setup, Fleet build, milestone runs, benchmark, profiling, report (scripts/README.md)
tests/	CPU tests: reference vs HF, graph counts vs design, kernels vs reference, simulator end to end, glue, scripts
results/	Evidence written by the scripts on the MI300X and the generated report (results/README.md)
docs/	Design document, GPU log, performance detail (docs/README.md)
prompts/	The fixed 1,024-token prompt

Further reading

• results/REPORT.md: every metric with its evidence file and command, generated by scripts/report.py
• docs/design-final-fixed.md: the design, with the decision log in the appendix
• docs/gpu-log.md: every run on the MI300X, each failure and its fix
• ROCm/fleet-chiplet-megakernel#9: upstream fix for the worker/scheduler stream leak on re-initialisation, found here

Licence

All rights reserved; evaluation-only licence for the issuing team. See LICENSE.md.