Qwen3.8-Flash-Next performance on my DGX Spark: every attempt

Qwen3.8-Flash-Next is a 38B MoE that should decode at 26 t/s on a DGX Spark. Mine was doing 9–17 while my agents were actually using it.

This is the whole sequence — every attempt, in order, wins and dead ends. The first three rounds are already on this blog (tuning decode speed); this is the full archive including what failed.

Round One: the kernel was prefetching the wrong thing

Baseline was a flat 9–17 t/s under real load. The model repo claims ~26, Reddit threads claim 41–47. I was parked at the bottom.

The PLE table (48 GiB of embeddings, memory-mapped off the NVMe) was being read at random offsets, and the kernel was prefetching gigabytes the engine would never touch. One posix_fadvise(POSIX_FADV_RANDOM) call in the mmap shim fixed the read-ahead, which was basically everything: cold boot went from ~9.5 to a steady 24.9 t/s once the page cache warmed. I also moved MTP 2→3 and A/B'd CUDA graphs: PIECEWISE won 24.9 vs 23.4 over --enforce-eager, which contradicted the Reddit threads — turns out the "PIECEWISE is a dead end" result was a 2×DGX thing. One node keeps the graphs.

Round Two: the README was right

The repo ships a script that converts the side layers to blockwise fp8 and keeps the experts in NVFP4. One run of prepare-hybrid.sh, a restart, and I got 30.2 t/s — +21% for almost no work. I'd skipped it earlier because it sounded too easy.

The DSpark speculator is compiled into the image already, and it benchmarks faster than MTP. But it needs a separate draft checkpoint, and nobody has published one for Flash-Next on Hugging Face. Parked, with the flag ready in serve.sh the day a drafter shows up.

Round Three: the concurrency wall

This is where the single-stream story ends and the batch story begins. I run LiveBench against the box with 8 parallel requests, and the aggregate sat at ~86–96 t/s no matter what. Server metrics said everything was healthy: 8 running, 0 waiting, KV cache a third full. Every worker crawled at ~10.7 t/s and the GPU drew 37 W of 140. When the software window is fine and the power meter is flat, the machine is waiting on something.

I wrote a probe that fires N concurrent requests and sums the generation counters from /metrics, so the number comes from the server itself. Scaling curve, 1 to 16 concurrent:

1   38.8 tok/s aggregate
2   63.1
4   98.0
8   126.3
16  131.8 (16th queued behind 8 slots)

An asymptote around 130. MTP off made it worse (128/95 vs 142/126) — the drafter accepting ~2.4 tokens per step is worth the attention-metadata rebuilds. Doubling max-num-batched-tokens was flat. Whatever serializes across sequences lives in the QSA kernels and the PLE gather path, and no vLLM flag touches that.

Round Four: a baseline that means something, and the false alarm

Before touching anything else I locked down a real baseline: one canonical verify script (zgx_perf_verify.py) that measures single-stream, determinism, cached TTFT, prefill, the 1–16 scaling curve, KV %, and GPU power, and writes baseline.json. Every tweak from here on gets compared against that exact yardstick. That discipline is the reason I can write any of this down.

First suspect: the scheduler cap. serve.sh was pinned at --max-num-seqs 8. The community evidence said the engine scales past 8 when you let more sequences run (NVIDIA forum: 132–134 t/s aggregate at 16 concurrent; one reproduction: ~267 with CPU offload and 48 streams). Set SEQS=16, restart, quick check.

The quick check looked like a regression. Cached TTFT jumped +52%, prefill collapsed −53%. I reverted to 8.

Then I ran the detailed A/B, matched conditions, 5/5/3 reps plus three TTFT bursts:

The quick run was a cold-cache artifact. The real answer: +50% aggregate throughput for one line in a shell script, TTFT flat. Kept. SEQS=16 is the default now.

Round Five: two community recipes that hurt

The mmap path's real wall, the theory went, is fault-latency serialization — so a fork with 64 gather threads and a batch-assembly prefetch should move more tokens at no hardware cost. I ran both, gated on the same baseline.

WORKERS 32→64: everything regressed. Single-stream 33.3/32.4/30.8 vs 37.0/34.4/32.7, TTFT +34%, prefill −51%, p8 −16%. Reverted.

PLE_PREFETCH backport: single 32.9/32.1/30.1, TTFT +31%, prefill −51%, p8 −10%. Also slower than baseline. Reverted — the backport stays in the tree, disabled, so it's one env var away if the balance ever changes.

Two recipes that worked for someone else's box, and mine got slower. That's the whole game with community performance folklore.

Round Six: the int4 fork that hit 45 t/s and wasn't deterministic

The big one. Saren-Arterius's qwen3.8-Flash-DGX-AutoRound fork — Intel W4A16 AutoRound int4 weights, int8 GPTQ lm_head, fp8 side layers, fp8 PLE, ~71 GiB resident. The fork's own bench claimed ~49 t/s single-stream, workload means 44–60. On paper it's the only ≥40 t/s path on one Spark. My copy measured 45.8 t/s.

Here's the catch, from the fork README's own warning to the measured behavior: it is not deterministic at T=0. Same prompt, same temperature zero, different logits run-to-run. For my LiveBench runs and reproducible evals, that's disqualifying — unless it could be fixed.

I spent a day in the kernels. Isolated tests eliminated the usual suspects: marlin_gemm int4 → bit-identical 8/8. MoE w1 int4 → bit-identical 8/8. per_token_group_fp8_quant → bit-identical 8/8. The divergence came from cutlass_scaled_mm, the fp8 block-scaled GEMM (128×128 block, dynamic per-token-group activations). Aligned like-for-like trace, 6 runs: identical inputs, different outputs. On GB10's sm_121, that kernel is run-to-run non-deterministic. Swapping to the Triton implementation of the same op still jittered — so it's not a Cutlass bug, it's the fp8 tensor-core accumulation order on this silicon. The only deterministic substitution was dequantizing fp8→bf16 and doing a bf16 matmul.

That worked, and it was useless:

The fastest config I found on the whole box — 45.8 t/s, +39% over prod — I cannot use, because the instruction the box executes is not reproducible. A fast deterministic int4 would need upstream cutlass work on deterministic fp8 MMA, which is out of scope for a fork. Int4 stays parked, experimental, documented.

Round Seven: the flag flip that doubled the cache

The int4 research surfaced the next lever: the production engine's QSA code already has FP8 KV support baked in — the image ships src/patch_qsa_fp8_kv.py applied at build time. But serve.sh carried a comment that said "fp8 is refused by the QSA layers." The comment was stale; it predated the patch. I'd been assuming the engine couldn't do this for two days because of a comment.

So the A/B was a pure flag flip on the running config: KV_DTYPE=fp8, everything else identical (hybrid fp8, SEQS=16, MTP=3, PIECEWISE).

Doubling the KV cache in the same 23.5 GiB, determinism intact, at the cost of ~11% aggregate throughput and a worse cached-TTFT — the fp8 KV path halves bytes per token but adds a dequant on the read side.

I kept it. The rationale is capacity headroom: doubled KV means more concurrent long-context sessions, and for my interactive agent traffic, determinism plus headroom beats 11% aggregate I mostly don't saturate. KV_DTYPE=fp8 is the default in serve.sh now, and the stale comment is fixed.

Where it landed

Current production config: hybrid fp8 side layers + fp8 KV cache + SEQS=16 + MTP=3 + PIECEWISE + POSIX_FADV_RANDOM on the PLE mmap. Single-stream ~33 t/s, aggregate ~206 t/s at 16 parallel, KV cache holding twice what it did on Monday.

Still queued: DSpark/DFlash speculative decoding the day a Flash-Next draft checkpoint exists on HF (the flag is ready), native offload A/B, the 2×DGX TP2 question, and a night batch pipeline to actually use the cache headroom.

The summary nobody puts in the README: the single biggest win was a kernel syscall on a memory-mapped table. The single biggest number came from an int4 fork I can't ship because the tensor cores don't agree with themselves. And the current best config was blocked for two days by a stale comment that said it was impossible. Check the comments. Question the folklore. Measure against one baseline, twice.