Running Qwen3.8-27B on a DGX Spark: vLLM vs SGLang

I spent some time getting Qwen3.8-27B running in NVFP4 on a single NVIDIA DGX Spark (GB10, 128 GB unified LPDDR5X), and then benchmarking the two big inference engines against each other: vLLM with MTP speculative decoding vs SGLang with DSpark speculative decoding.

Everything below is reproducible — all commands, configs, and raw results are in the repo: Fayyiz55/qwen38-dgx-spark.

The setup

The DGX Spark is an interesting little box: 128 GB of unified LPDDR5X memory shared between CPU and GPU, at roughly ~225 GB/s of usable bandwidth. That unified memory is exactly why I picked NVFP4 quantization (W4A4): the weights come to ~25 GB, versus ~67 GB for BF16 or ~38 GB for FP8. On a 128 GB box, every GB you don’t spend on weights is a GB available for KV cache and batching.

The headline you should not quote

If you take one thing from this post, take this: there is no single “the tok/s” for this model. Speculative decoding — which both engines use — speeds you up in proportion to how predictable the next tokens are. That means throughput is workload-dependent:

• Code, math, reasoning (structured output): the drafter accepts 3.3–5.6 tokens per step → 40–53 tok/s
• Free-form prose / chat: acceptance drops to ~1.5–2.2 → 18–26 tok/s

Same engine, same config, same box — the content decides the speed. My greedy median across structured workloads was 41.0 tok/s (the reference config’s box measured ~50; the gap is within the boot-to-boot/driver variance that’s documented as “the boot lottery”).

So when someone quotes a headline number for a speculative-decoding setup, ask what they were generating. A coding-assistant workload and a chatbot workload are running different sports.

Head to head: vLLM+MTP vs SGLang+DSpark

For a fair comparison I ran both engines through the same engine-agnostic harness (sglang.bench_serving, which speaks the OpenAI API) on the general/prose regime — deliberately the low end for both:

Verdict: SGLang + DSpark is the pick. Roughly 30% faster single-stream, noticeably lower latency, and 4× the context window. Under heavy concurrency both engines saturate the memory bus and converge (~60 tok/s aggregate), and vLLM stays a perfectly solid, simpler fallback.

The reason SGLang wins single-stream is DSpark itself — a trained block-drafter that accepts more tokens per speculation step than MTP’s built-in draft head. But it’s also temperamental. The tuned launch looks like this (abridged — full command in the repo):

docker run -d --name sglang-qwen38 --gpus all \
  -p 30000:30000 --ipc=host --memory=100g \
  lmsysorg/sglang:qwen38-27b \
  python3 -m sglang.launch_server \
  --model-path RadixArk/Qwen3.8-27B-NVFP4 \
  --speculative-algorithm DSPARK \
  --speculative-draft-model-path RadixArk/Qwen3.8-27B-DSpark \
  --enable-torch-compile --torch-compile-max-bs 4 \
  --num-continuous-decode-steps 2 \
  --mamba-full-memory-ratio 8.26 \
  --mem-fraction-static 0.50

Two of those flags are non-optional on GB10: drop --enable-torch-compile and DSpark’s acceptance collapses to ~1.7 and throughput tanks; and --mem-fraction-static 0.50 is safety-critical, for reasons the next section explains.

The gotcha that will bite you: the GB10 memory leak

This is the part I wish someone had written before I started.

Symptom: after an engine container crashes or gets torn down, memory is not released. free -h shows ~90–100 GiB “used” while nvidia-smi shows no GPU processes and ps accounts for barely 1–2 GiB. The next launch fails with:

ValueError: Free memory on device cuda:0 (18.61/121.69 GiB) on startup is less
than desired GPU memory utilization (0.75, 103.44 GiB).

Cause: on GB10’s unified memory, a CUDA process that exits uncleanly (OOM during engine init, a crash mid-compile) leaks its allocation into the driver. It’s a phantom: invisible to ps and to nvidia-smi (which reports N/A for memory usage on this chip anyway). A --restart policy makes it worse — every crash leaks again, compounding until the box can’t start the model at all.

What doesn’t clear it: docker restart, restarting the Docker daemon, nvidia-smi --gpu-reset (fails — the phantom holder counts as a client).

What clears it: a reboot. That’s it. Graceful docker stop -t 30 leaks less, but not zero.

The survival rules I ended up with:

1. Test new configs without a restart policy. A bad config should fail once and stop — not crash-loop and leak.
2. Keep memory fractions conservative (vLLM 0.60, SGLang 0.50) so a small residual leak doesn’t block the next start.
3. Hard-cap the container (--memory=100g) so a runaway kills the container, not the host.
4. Deploy via systemd with ExecStartPre=-docker rm -f, and manage it with systemctl afterwards — the graceful stop path releases memory more cleanly.

The full runbook (including the reboot-recovery dance) is in the repo’s field notes.

Reproduce it

The repo has the complete commands for both engines, the systemd units, the benchmark harness, and the raw results: Fayyiz55/qwen38-dgx-spark.

Two caveats worth repeating. First, credit where it’s due: the SGLang + DSpark recipe, the tuning flags, and the bench.sh methodology come from hasso5703/dgx-spark-qwen38 — they found the fastest validated GB10 config; my contribution is reproducing it on separate hardware, adding the vLLM comparison, and documenting the operational pitfalls. Second, one more reason to use a real harness: a single curl-timed prompt that only counts visible tokens (and misses the thinking phase) inflates tok/s several-fold. bench_serving and bench.sh count reasoning tokens correctly — a naive timing script doesn’t.

Numbers will vary with driver version, workload, temperature, and DSpark acceptance. Benchmark your traffic shape before drawing conclusions.