Qwen3.8-Flash-Next on my DGX Spark: tuning decode speed

I've been running Qwen3.8-Flash-Next on my DGX Spark for a few weeks now, and this week I spent three evenings trying to make it go faster. Plenty of performance threads around the Spark and these new Qwen models, so figured I'd share the numbers, including the one result I didn't expect. Short version: one mmap flag was worth more than everything else combined, and then concurrency stopped scaling at a number I did not see coming.

System - DGX Spark (GB10), 128 GB unified memory - Qwen3.8-Flash-Next, 38B MoE, NVFP4 quant + hybrid fp8 side layers - Custom vLLM image, 48 GiB PLE table memory-mapped off the NVMe

Round one: baseline was bad

Prometheus had me at 9-17 tokens/s per request while the box was doing real work. The model repo claims ~26 t/s, Reddit threads say 41-47. I was parked at the bottom.

Three things came out of the threads:

• The PLE table is read over mmap and the kernel was prefetching gigabytes of embeddings it would never touch, because the table is read at random offsets. posix_fadvise with POSIX_FADV_RANDOM fixed that. This was basically everything: cold boot went from ~9.5 to a steady 24.9 t/s once the page cache warmed up.
• MTP 2 to 3 speculative tokens.
• CUDA graphs PIECEWISE vs eager. Every thread said PIECEWISE is a dead end under MTP, but that turns out to be a 2xDGX thing. On one node PIECEWISE won, 24.9 vs 23.4 t/s. Kept the graphs.

Round two: read the README

One script converts the side layers to blockwise fp8 and keeps the experts in NVFP4, a restart, and I got 30.2 t/s. That's +21% for almost no work. I'd skipped it earlier because it sounded too easy.

DSpark, the block-drafting speculator that ships in the image, benchmarks faster than MTP, but it needs a separate draft checkpoint and nobody published one for Flash-Next on Hugging Face. Parked that one, with the flag ready in serve.sh the day a drafter shows up.

Round three: the concurrency wall

This is the one that surprised me. I run LiveBench against the box with 8 parallel requests, which is how I actually use it, and the aggregate sits at ~86-96 tokens/s no matter what. Server metrics say everything's healthy: 8 running, 0 waiting, KV cache a third full, time-to-first-token fine. Every worker crawls at ~10.7 t/s and the GPU draws 37 W. The Spark can pull 120 W+. When the software window looks fine and the power meter is flat, the machine is waiting on something, not working.

I wrote a small probe that fires N concurrent requests and sums the generation token counters from /metrics, so the number comes from the server itself and not from my client timings, then went through suspects.

MTP first. Under the custom QSA attention backend the drafter rebuilds attention metadata between draft steps, which sounds serializing. Turned it off: got worse, 128/95 tok/s vs the 142/126 baseline. The drafter accepting ~2.4 tokens per step is worth more than the rebuilds cost. Also amusing: with MTP off the KV cache grew from 814K to 1.17M tokens, and none of that headroom became speed.

Second: max-num-batched-tokens at 8192. Doubled it. Flat, 139/126.

At that point I ran the scaling curve instead of single data points, 1, 2, 4, 8 and 16 concurrent requests with distinct prompts so the prefix cache couldn't help:

It's an asymptote, around 130 tok/s at the 8-way batch. Past 8 requests they just queue. The engine does roughly 3.3x single-stream and stops. Whatever serializes across sequences lives in the QSA kernels and the PLE gather path, and no vLLM flag touches that. Fixing it is custom kernel work, which is a different project.

Unrelated, same week: my LiveBench runner had a bug. It trims the category name data_analysis to data with a naive string split and tried to load a dataset that doesn't exist, killing a 15-hour run after the first three categories. One-line fix, re-queued only the categories it never got to, and the scoring re-reads the finished answer files at the end, so nothing got re-run.

Where it landed: - MTP=3, 8192 batched tokens, hybrid checkpoint. That stays. - Benchmark is running again on the three categories it missed. - The mmap fix is the keeper. Check what the kernel is doing with your page cache before blaming the model. Every experiment in that first round cost me a ten-minute boot, which is how I know.

Not tested yet: ngram-style drafting for this model if a drafter ever shows up, and proper kernel-level batching in the QSA path. Happy to rerun anything here with flags people want compared.