Qwen3.8-Flash-Next performance on my DGX Spark: every attempt

Qwen3.8-Flash-Next is a 180 B-parameter MoE (512 experts, 10 active per token) that should decode at 26 t/s on a DGX Spark. Mine was doing 9–17 while my agents were actually using it.

This is the whole sequence — every attempt, in order, wins and dead ends. The first three rounds are already on this blog (tuning decode speed); this is the full archive including what failed.

Round One: the kernel was prefetching the wrong thing

Baseline was a flat 9–17 t/s under real load. The model repo claims ~26, Reddit threads claim 41–47. I was parked at the bottom.

The PLE table (48 GiB of embeddings, memory-mapped off the NVMe) was being read at random offsets, and the kernel was prefetching gigabytes the engine would never touch. One posix_fadvise(POSIX_FADV_RANDOM) call in the mmap shim fixed the read-ahead, which was basically everything: cold boot went from ~9.5 to a steady 24.9 t/s once the page cache warmed. I also moved MTP 2→3 and A/B'd CUDA graphs: PIECEWISE won 24.9 vs 23.4 over --enforce-eager, which contradicted the Reddit threads — turns out the "PIECEWISE is a dead end" result was a 2×DGX thing. One node keeps the graphs.

Round Two: the README was right

The repo ships a script that converts the side layers to blockwise fp8 and keeps the experts in NVFP4. One run of prepare-hybrid.sh, a restart, and I got 30.2 t/s — +21% for almost no work. I'd skipped it earlier because it sounded too easy.

The DSpark speculator is compiled into the image already, and it benchmarks faster than MTP. But it needs a separate draft checkpoint, and nobody has published one for Flash-Next on Hugging Face. Parked, with the flag ready in serve.sh the day a drafter shows up.

Round Three: the concurrency wall

This is where the single-stream story ends and the batch story begins. I run LiveBench against the box with 8 parallel requests, and the aggregate sat at ~86–96 t/s no matter what. Server metrics said everything was healthy: 8 running, 0 waiting, KV cache a third full. Every worker crawled at ~10.7 t/s and the GPU drew 37 W of 140. When the software window is fine and the power meter is flat, the machine is waiting on something.

I wrote a probe that fires N concurrent requests and sums the generation counters from /metrics, so the number comes from the server itself. Scaling curve, 1 to 16 concurrent:

1   38.8 tok/s aggregate
2   63.1
4   98.0
8   126.3
16  131.8 (16th queued behind 8 slots)

An asymptote around 130. MTP off made it worse (128/95 vs 142/126) — the drafter accepting ~2.4 tokens per step is worth the attention-metadata rebuilds. Doubling max-num-batched-tokens was flat. Whatever serializes across sequences lives in the QSA kernels and the PLE gather path, and no vLLM flag touches that.

Round Four: a baseline that means something, and the false alarm

Before touching anything else I locked down a real baseline: one canonical verify script (zgx_perf_verify.py) that measures single-stream, determinism, cached TTFT, prefill, the 1–16 scaling curve, KV %, and GPU power, and writes baseline.json. Every tweak from here on gets compared against that exact yardstick. That discipline is the reason I can write any of this down.

First suspect: the scheduler cap. serve.sh was pinned at --max-num-seqs 8. The community evidence said the engine scales past 8 when you let more sequences run (NVIDIA forum: 132–134 t/s aggregate at 16 concurrent; one reproduction: ~267 with CPU offload and 48 streams). Set SEQS=16, restart, quick check.

The quick check looked like a regression. Cached TTFT jumped +52%, prefill collapsed −53%. I reverted to 8.

Then I ran the detailed A/B, matched conditions, 5/5/3 reps plus three TTFT bursts:

The quick run was a cold-cache artifact. The real answer: +50% aggregate throughput for one line in a shell script, TTFT flat. Kept. SEQS=16 is the default now.

Round Five: two community recipes that hurt

The mmap path's real wall, the theory went, is fault-latency serialization — so a fork with 64 gather threads and a batch-assembly prefetch should move more tokens at no hardware cost. I ran both, gated on the same baseline.

WORKERS 32→64: everything regressed. Single-stream 33.3/32.4/30.8 vs 37.0/34.4/32.7, TTFT +34%, prefill −51%, p8 −16%. Reverted.

PLE_PREFETCH backport: single 32.9/32.1/30.1, TTFT +31%, prefill −51%, p8 −10%. Also slower than baseline. Reverted — the backport stays in the tree, disabled, so it's one env var away if the balance ever changes.

Two recipes that worked for someone else's box, and mine got slower. That's the whole game with community performance folklore.

Round Six: the int4 fork that hit 45 t/s and wasn't deterministic

The big one. Saren-Arterius's qwen3.8-Flash-DGX-AutoRound fork — Intel W4A16 AutoRound int4 weights, int8 GPTQ lm_head, fp8 side layers, fp8 PLE, ~71 GiB resident. The fork's own bench claimed ~49 t/s single-stream, workload means 44–60. On paper it's the only ≥40 t/s path on one Spark. My copy measured 45.8 t/s.

Here's the catch, from the fork README's own warning to the measured behavior: it is not deterministic at T=0. Same prompt, same temperature zero, different logits run-to-run. For my LiveBench runs and reproducible evals, that's disqualifying — unless it could be fixed.

I spent a day in the kernels. Isolated tests eliminated the usual suspects: marlin_gemm int4 → bit-identical 8/8. MoE w1 int4 → bit-identical 8/8. per_token_group_fp8_quant → bit-identical 8/8. The divergence came from cutlass_scaled_mm, the fp8 block-scaled GEMM (128×128 block, dynamic per-token-group activations). Aligned like-for-like trace, 6 runs: identical inputs, different outputs. On GB10's sm_121, that kernel is run-to-run non-deterministic. Swapping to the Triton implementation of the same op still jittered — so it's not a Cutlass bug, it's the fp8 tensor-core accumulation order on this silicon. The only deterministic substitution was dequantizing fp8→bf16 and doing a bf16 matmul.

That worked, and it was useless:

The fastest config I found on the whole box — 45.8 t/s, +39% over prod — I cannot use, because the instruction the box executes is not reproducible. A fast deterministic int4 would need upstream cutlass work on deterministic fp8 MMA, which is out of scope for a fork. Int4 stays parked, experimental, documented.

Round Seven: the flag flip that doubled the cache

The int4 research surfaced the next lever: the production engine's QSA code already has FP8 KV support baked in — the image ships src/patch_qsa_fp8_kv.py applied at build time. But serve.sh carried a comment that said "fp8 is refused by the QSA layers." The comment was stale; it predated the patch. I'd been assuming the engine couldn't do this for two days because of a comment.

So the A/B was a pure flag flip on the running config: KV_DTYPE=fp8, everything else identical (hybrid fp8, SEQS=16, MTP=3, PIECEWISE).

Doubling the KV cache in the same 23.5 GiB, determinism intact, at the cost of ~11% aggregate throughput and a worse cached-TTFT — the fp8 KV path halves bytes per token but adds a dequant on the read side.

I kept it. The rationale is capacity headroom: doubled KV means more concurrent long-context sessions, and for my interactive agent traffic, determinism plus headroom beats 11% aggregate I mostly don't saturate. KV_DTYPE=fp8 is the default in serve.sh now, and the stale comment is fixed.

Where it landed

Current production config: hybrid fp8 side layers + fp8 KV cache + SEQS=16 + MTP=3 + PIECEWISE + POSIX_FADV_RANDOM on the PLE mmap. Single-stream ~33 t/s, aggregate ~206 t/s at 16 parallel, KV cache holding twice what it did on Monday.

Still queued: DSpark/DFlash speculative decoding the day a Flash-Next draft checkpoint exists on HF (the flag is ready), native offload A/B, the 2×DGX TP2 question, and a night batch pipeline to actually use the cache headroom.

The summary nobody puts in the README: the single biggest win was a kernel syscall on a memory-mapped table. The single biggest number came from an int4 fork I can't ship because the tensor cores don't agree with themselves. And the current best config was blocked for two days by a stale comment that said it was impossible. Check the comments. Question the folklore. Measure against one baseline, twice.

Update, 2026-09-14 — the box runs a different stack now

Two days after this went up, the box stopped running the engine described above. The abliterated-stack question came up in a Slack thread on 09-09 ("we couldn't get past 30") and by that evening prod was a community kit instead of my hybrid.

Concretely: the single DGX Spark now serves drowzeys/keys-Qwen3.8-flash-next-ablit-Mia-Single-Spark-only — a gated repo, and yes, abliterated, meaning the refusal directions are removed. It is the MiaAI-Lab single-Spark recipe. I expected a rebuild. There wasn't one: almost everything it needs was already in our image, so the port turned out to be one runtime-mounted patch plus a set of launch flags.

What actually changed:

• the MTP drafter head is patched to a reduced 47k vocabulary — this only works together with use_local_argmax_reduction, and setting that flag without the patch is a boot-time ValueError, so the two ship as a pair
• bf16 SSM/GDN recurrent-state cache instead of fp32
• the V2 model runner plus full decode-graph capture
• fp8 KV, 262k context, 8 sequence slots, ~0.79 of the box budgeted to the engine

Same harness, same box, against what prod was running on 09-09:

So the 30 t/s wall is gone, and single-stream is at effective parity with the 48.7 the kit advertises. The "30–50% left on the table" framing in the earlier tuning post is now closed out — the missing pieces were a reduced-vocab draft head and someone who had already tuned it.

Four things I need to be straight about, because three of them are me being inconsistent with my own rules:

1. I gave back the 16-stream aggregate. The kit caps at 8 sequence slots and does not scale past them on this box: 180.2 aggregate at 8 streams, 171.5 at 16. My SEQS=16 win in Round Four does not survive the move. At +36% single-stream I'll take that trade, but it is a trade, not a clean win.

2. I traded determinism, which is the thing I said I wouldn't trade. This stack is not bit-identical at T=0 either. In Round Six I walked away from a fork that was faster than this one — 45.8 t/s single-stream, 232 aggregate at 16 — purely on determinism. I adopted this one anyway, because I wanted the community stack's batch numbers more than I wanted the guarantee. I would rather write that down than let Round Six keep burnishing a principle I stopped applying.

3. The quality number is worse than the last one, and I didn't chase it. The only instruction-following datapoint I have for the new engine is a single LiveBench category run (#30, 8 parallel, story_generation subtask): 88.0. The same model on the previous engine scored 96.0 five days earlier. When I swapped, I wrote that I would re-open the question if instruction-following dropped more than ~2 points. It dropped eight. It is one run and the subtask coverage is not identical to the earlier one, so it is not a clean comparison — but it isn't nothing either, and I never followed it up. That is an open item, not a pass.

4. The rollback got more expensive. Cleaning up the old path freed ~180 GB (the RadixArk fp8hybrid checkpoint, 139G, plus both int4-fork images, 41G) — 3.2 T free now, worth it. But the only way back to the previous engine is a re-download, not a config flip.

Still queued, unchanged: DSpark/DFlash still has no Flash-Next draft checkpoint on Hugging Face, so the flag stays parked; the 2×DGX TP2 question; a night batch pipeline to use the cache headroom. And now also: find out whether that 88.0 was the stack or the test.

Update, 2026-09-21 — prefix caching, measured

One more number belongs in this post, because Round Seven was about the cache and this is the first time I have had a clean counter for it. vLLM exposes prefix-cache queries and hits as counters, so this is a measurement and not an impression: over the current engine run (started 2026-09-18 12:01 UTC, so about three days of real agent traffic), prefix_cache_queries_total reads 12.42 M queried tokens against prefix_cache_hits_total 9.48 M — a 76% prefix hit rate, with the resolved cache config reporting enable_prefix_caching=True, block size 8, fp8 cache dtype and a 1,099,833-token KV pool (max concurrency 4.2). The counters reset when the engine restarts, so the honest figure is the one from the current run rather than a lifetime ratio.

And one correction to the first line of this post: Qwen3.8-Flash-Next is a 180 B-parameter MoE, not 38B. I was reading the "3.8" in the model name as a size. Two independent counts agree on 180 B — Hugging Face's parameter count for the base model (179,999,981,459) and my own unpack-and-add of the quantized shards — so the lede now says 180 B.

Not a result, just a line so it does not get lost: GLM-5.3-Flash abliteration for the second Spark, downloads in flight, nothing measured yet.

Update, 2026-10-02 — the box is on a third stack, and the "declared baseline" is not what serves

Everything in the update above is now history. The MiaAI single-Spark kit I wrote down as production is not what answers on the port any more. The abliterated checkpoint isn't the model on disk either — the models directory holds the EXL3 3.05bpw pack now, and the old stack survives only as a stopped container.

What serves today is turboderp's EXL3 3.05bpw pack through vLLM with the vllm-exl3 plugin, on the same port, 262k context, model id qwen3.8-flash-next. That is Cruz's published DGX Spark recipe — the same repo whose "102.9 tok/s" I went and fact-checked a while back and found to be a four-stream aggregate, not a per-stream number.

How the box got here, in order:

09-23 — the EXL3 detour starts. An EXL3 quant map showed up, so I ran turboderp's packs against the incumbent engine on the same box, the same frozen 140-row kit, with the kit's own unmodified graders:

At equal bit width the engine swap is a wash — 3.05bpw lands on exactly the same kit score as the vLLM stack it replaced. The win is the pack, not the engine: 2.50bpw gives up 1.4 points on the kit and buys 27% more decode while occupying 37 GB less. So I deployed 2.50bpw on the port — 57 GB resident against vLLM's 111 GB, one live client.

Two caveats I should carry rather than bury: the kit covers 140 of its 180 rows, because HumanEval's 40 need a sandbox image that was never published; and the EXL3 lanes ran the official base against an abliterated incumbent, so a matched-base re-run would clean that confound up. Both were in the open-gaps list at the time. Both still are.

09-26 — the baseline I declared, and the one that actually serves. I moved the default to Cruz's own exllamav3 fork at 523ecd3 plus his 3.05bpw pack. It failed to start: status=1, "Start request repeated too quickly", five seconds in — and it is still failed and disabled in systemd today. The same evening the recipe's other path went in, vLLM plus the vllm-exl3 plugin, and that is the one answering now.

It went onto the production port instead of a side port for a reason that had nothing to do with benchmarking: the exllamav3 shim broke my L1 agent. Non-streaming responses were the root cause and L1 was quietly falling back to a hosted model. A side port would have left L1 sitting there broken while I admired the benchmark, so the shim came off the port. Cutover verification was SSE streaming, native tool calls, reasoning separation, /metrics back for Prometheus, 48.8 tok/s decode.

So "the declared baseline" and "the serving engine" turned out to be two different things on this box, and I would rather write that gap down than tidy it out of the story.

09-30 — the rung that beat its own README and still got given back. The third MiaAI recipe is TensorFold. I switched to it, ran a sustained-load driver instead of the quality kit, and it exceeded the recipe's published table at every concurrency level: 86.1 tok/s single-stream on the code prompt with thinking off (the repo says 62.4), 236.3 aggregate at four streams (106.7 published), and 2,281 tok/s prefill at 128k, +3.7%. The 47.4 tok/s I'd earlier reported as "aggregate across 4 streams" was never the engine — the quality kit's frozen runner drains every batch before opening the next, so four streams were genuinely in flight for only 14% of that run.

And I gave it back anyway. TensorFold needs ~98 GiB free and cannot co-exist with the baseline, so measuring it means taking the port down for the duration — 24.5 minutes of live inference. That is a fine price for a measurement and a bad price for a production engine, which is what it was: a measurement. The unit reads failed in systemd today because the restore stopped it — exit 130, disabled — which is the shape a rung leaves behind when it was never a candidate.

Three things I should be straight about.

First, determinism. Round Six of this post is where I walked away from a faster fork purely because it was not bit-identical at T=0. The 09-14 update admitted I took a non-deterministic stack anyway. The box has changed engines twice more since, and I still have not re-measured determinism on the current one. Better to keep that visible than let the principle keep accruing credit it is not earning.

Second, the instruction-following question from the 09-14 update — one LiveBench reading of 88.0 against 96.0 five days earlier, against a threshold I set myself at about two points — is still unanswered. Three engine swaps later, I have not gone back to it.

Third, there is a quality campaign running against the deployed pack right now: a reduced LiveBench-shaped battery across math, instruction-following, code, hard reasoning and tool use. The numbers are not final, so they are not in this post.

Still queued, unchanged: DSpark/DFlash speculative decoding (still no Flash-Next draft checkpoint on Hugging Face), the 2×DGX tensor-parallel question, and a night batch pipeline to actually use the cache headroom.

The lesson that keeps repeating here: I now have one script that moves a single port between five engines, and only one of them fits in the box's memory at a time. That exclusivity means "what is production" is a question I have to re-ask every time somebody publishes a recipe. Mine has changed three times in three weeks, and each time it changed because of a pack or a plugin — not because of anything I tuned myself.