Everyone running DeepSeek-V4-Flash on DGX Spark hardware today is doing it with community forks: patched vLLM builds, custom images, source-built kernels. The received wisdom is that the stock NVIDIA NGC container cannot serve this model on GB10 (SM121) because upstream vLLM rejects every sparse-MLA attention backend on that architecture (vllm-project/vllm#45317, still open).
We ran the experiment with only nvcr.io/nvidia/vllm:26.06-py3 on a two-node DGX Spark cluster (TP=2 over the 200G QSFP link). Policy constraint: no forks, no nightlies, no patched kernels. Here is what actually happens.
Finding 1 — the sparse-attention wall is gone in the NGC build
NGC 26.06 is not upstream vLLM. It ships 0.22.1+7b9cb5b7.dev with DeepSeek V4 support that upstream does not have: DeepseekV4ForCausalLM resolves, deepseek_v4_fp8 quantization engages, the compilation config carries deepseek_v4_attention and sparse_attn_indexer ops, and both workers initialize DeepSeek’s fp8_ds_mla KV cache with the FP8 Lightning Indexer on GB10, with no backend-selection abort.
If your mental model of this hardware comes from the upstream issue tracker, that is a surprise. The release notes say only “Support DeepSeek V4” with no Spark caveat either way; the support appears real at the attention layer.
Finding 2 — use InstantTensor, or the loader will OOM the box
The default mmap-based safetensors loader dies on unified memory: page cache and CUDA allocations compete for the same 128 GB pool, and around 90% loaded the kernel OOM killer takes the worker (~694 GB total-vm at death).
NVIDIA’s InstantTensor loader fixes it completely:
pip install instanttensor # supported by the build, not preinstalled
vllm serve deepseek-ai/DeepSeek-V4-Flash --load-format=instanttensor ...
All 69,187 tensors load in about 57 seconds per node, no memory pressure.
Finding 3 — the actual wall: the MoE weight-repack spike
Every attempt died seconds after loading, at MoEPrepareAndFinalizeNoDPEPModular (the Marlin MoE weight repack), with a driver-level NVRM: NV_ERR_NO_MEMORY — and only on the head node. The head carries the API server, engine core, and Ray head alongside its ~80 GB weight shard; the repack’s temporary buffers push it a few GB past the 121 GB usable. The lean worker node passed the same step every run.
The failure is configuration-independent. We tried:
| Attempt | Result |
|---|---|
| 131K ctx, gpu-util 0.80 | OOM at repack (head) |
| 65K ctx, 4096 batched tokens, gpu-util 0.70 | OOM at repack (head) |
| 32K ctx, 2 seqs, gpu-util 0.60 | OOM at repack (head) |
PYTORCH_CUDA_ALLOC_CONF=expandable_segments | worse — load-phase failure |
--enable-expert-parallel | no change |
No serve flag controls that repack allocation. The gap is small — single-digit GB on one node — which makes it exactly the kind of thing a point release fixes.
Where that leaves things
Verdict: blocked by container memory behavior, not by kernels, and not by hardware size. That is a much better place than the community assumption. Watch items:
- NGC 26.07+ — the 26.06 known-issues list for Spark is entirely unified-memory allocation behavior; this class is clearly under active repair
- An NGC vLLM new enough for the official NVFP4 checkpoint (different weight and repack profile)
- Ray-less multi-node — moving the coordinator processes off the GPU node may alone close the gap
For calibration: community fork setups report ~44 tok/s plain decode and 60+ tok/s with DeepSeek’s DSpark speculative decoding on identical hardware. If a point release fixes the repack spike, a frontier-adjacent, MIT-licensed reasoning model on the fully supported vendor path is one config file away.
Notes from Conselara’s applied-AI lab, where we evaluate what secure, on-premises AI infrastructure can actually run before recommending it to anyone else.