gpt-oss-120b, a 117B-parameter / 5.1B-active MXFP4 mixture-of-experts model, runs comfortably on a single NVIDIA DGX Spark (GB10, 128 GB unified memory). At ~63 GB of weights it leaves room for a 131K context window, and its reasoning quality makes it a useful deep-reasoning node alongside a faster mid-size daily driver.

The catch is that the SM121 configuration is unforgiving. Several defaults and forum recommendations either silently corrupt output or refuse to start. This is the setup that actually works on the stock NGC container, and the traps that cost us the most time.

Tested on nvcr.io/nvidia/vllm:26.05-py3 (vLLM 0.20.1), a single DGX Spark, GB10 / SM121.

Download only the MXFP4 shards

The HuggingFace repo ships more than you need. Alongside the 14 MXFP4 safetensors shards (~63 GB) there are original/ and metal/ folders that balloon the full download to ~125 GB. Pull only the MXFP4 weights; grabbing everything wastes ~60 GB and a lot of time you’ll spend deleting it again.

The serving config

Environment:

VLLM_MXFP4_BACKEND=marlin          # mandatory (see below)
VLLM_USE_FLASHINFER_MOE_FP4=0
TIKTOKEN_RS_CACHE_DIR=/root/tiktoken_cache
HF_HUB_OFFLINE=1

Flags:

vllm serve openai/gpt-oss-120b \
  --quantization=mxfp4 \
  --moe-backend=marlin \
  --attention-backend=TRITON_ATTN \
  --gpu-memory-utilization=0.90 \
  --max-model-len=131072 \
  --tool-call-parser=openai

That’s the whole thing. Everything below is why each line is the way it is.

marlin is mandatory

VLLM_MXFP4_BACKEND=marlin is not optional on SM121. The CUTLASS FP4 path corrupts the first Harmony token, which surfaces as a content: null on the first response and broken output thereafter (vLLM issue #37030). Marlin is the only MXFP4 backend that produces correct output on this silicon. It mirrors the broader SM121 rule: CUTLASS FP4 is broken here; use Marlin.

FlashInfer is invalid for gpt-oss

On many models FlashInfer is merely a bad idea on SM121. For gpt-oss it’s a hard stop: the engine fails to initialize with attention sinks not supported. gpt-oss uses attention sinks, the FlashInfer path doesn’t implement them here, and the result is a startup crash, not a silent fallback. --attention-backend=TRITON_ATTN is the only working backend.

The Harmony tokenizer needs an offline pre-stage

gpt-oss uses the o200k_harmony tiktoken vocabulary, which vLLM fetches from a CDN on first load. With HF_HUB_OFFLINE=1 (or on any air-gapped or locked-down host), that fetch fails and the server won’t start.

Pre-stage it. tiktoken caches by the SHA-1 of the vocab URL, so the file has to land at exactly the right name:

# the cache filename is sha1(<vocab CDN url>)
~/tiktoken_cache/fb374d419588a4632f3f557e76b4b70aebbca790

Mount that directory into the container and point TIKTOKEN_RS_CACHE_DIR at it. Once the vocab is cached under the correct hashed filename, the model loads offline cleanly.

Two flags you don’t need (and one that doesn’t exist)

  • Don’t pass --reasoning-parser. vLLM 0.20.1 auto-applies openai_gptoss, and it works correctly with the marlin backend: clean content/reasoning split, non-null. Passing it yourself is redundant.
  • --mxfp4-layers does not exist in NGC 26.05. It comes from a community fork, not the stock container; vllm serve --help confirms it isn’t there. Several faster-throughput writeups depend on it; see the next section.
  • Tool calls use the Harmony format: --tool-call-parser=openai, not qwen3_coder or hermes. If you’re moving an agent over from a Qwen model and tool calls start misbehaving, this is usually why.

Throughput reality

On the stock NGC path, expect ~37–39 tok/s single-stream. That is the ceiling here, and it’s a real one.

The faster numbers you’ll see quoted (~59–63 tok/s single-stream, ~80 with TP=2) require a community vLLM fork and its --mxfp4-layers flag. That’s a legitimate path if you’re willing to maintain a custom build, but it’s off the official NGC container and outside a vetted-driver, no-fork operating policy. On the stock path, ~38 tok/s is what correct output costs, and we take correctness over a fork we have to own.

When to use it

gpt-oss-120b isn’t the right daily driver; a mid-size MoE like Qwen3.6-35B is roughly twice as fast for routine work. Where the 120B earns its slot is harder reasoning and agentic coding, where the extra capability is worth dropping from ~70 to ~38 tok/s. Run it as a second, deeper node, not your default.

For why we run open models like this on-prem in the first place, see Open vs. Closed: Choosing AI Models for the Data Boundary.