DGX Spark GB10 Hardware Reference: SM121 Architecture, Memory, and Networking

Reference for the NVIDIA DGX Spark GB10 Grace Blackwell Superchip (SM121). The DGX Spark shares the Blackwell name with datacenter hardware but is architecturally distinct. A lot of documentation, forum posts, and vLLM flags written for B100/B200 do not apply here. Some actively break things.

SM121 is not datacenter Blackwell

When you see forum recommendations or vLLM flags that say “for Blackwell,” verify they’re for SM121 specifically before using them.

Memory architecture

The 128 GB LPDDR5X pool is shared between CPU and GPU. There is no separate VRAM. This affects everything:

• The Linux page cache competes directly with CUDA allocations. A warm filesystem cache can push CUDA into OOM.
• Never set --gpu-memory-utilization above 0.90. In practice, 0.85–0.87 is the safe range. At long context lengths with large KV caches, you may need to drop further.
• If you hit OOM at container startup, drop the page cache first:

sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'

• Ray’s memory monitor misreads page cache pressure as real memory pressure and will kill vLLM mid-inference. In multi-node configurations, always set:

RAY_memory_monitor_refresh_ms=0

GPU count and tensor parallelism

Each DGX Spark presents as one logical GPU. Tensor parallelism beyond TP=1 requires a Ray cluster spanning multiple units: there is no NVLink or NVSwitch. Inter-node communication runs over NCCL via a direct RoCE interconnect.

Software stack

Driver: Stay on 580.x. Driver 590.x has a confirmed CUDAGraph deadlock on GB10. Pin the package to prevent accidental upgrades:

sudo apt-mark hold nvidia-driver-580

QSFP-DD direct interconnect

For multi-node setups, a direct QSFP-DD cable between two Sparks gives you a high-bandwidth RoCE link without a switch.

Cable: A 400G DAC passive copper cable (e.g., Amphenol NJAAKK-N911, 1m) presents as two independent 200 Gb/s logical interfaces on each machine. Both channels share the same serial number. This is normal for this cable type.

Measured bandwidth (ib_write_bw)

The below-theoretical numbers are a known ib_write_bw artifact: it defaults to a single queue pair and 4096 B MTU. Actual NCCL throughput with multiple QPs approaches the full line rate. This is not a hardware or config problem.

Network configuration: nmcli only

The DGX Spark is fully NetworkManager-based. Netplan with the networkd renderer silently does nothing: there is no /run/systemd/network/ directory. All persistent interface configuration must use nmcli:

sudo nmcli con mod <connection-name> \
  ipv4.addresses 192.168.100.10/24 \
  ipv4.method manual \
  802-3-ethernet.mtu 9000
sudo nmcli con up <connection-name>

Set MTU to 9000 (jumbo frames) on all QSFP-DD interfaces.

RoCE GID index

GID index 3 = RoCEv2 + IPv4 on both machines. Confirm with:

show_gids | grep <interface-name>

This is required for NCCL: NCCL_IB_GID_INDEX=3.

Compute kernel compatibility

What works

What is broken or absent on SM121

For full runtime flag guidance, see the vLLM on DGX Spark SM121 post.

Startup time reference

For a two-node Qwen3-235B-A22B-GPTQ-Int4 cluster:

Single-node (120B model): ~8 min total.

Temperature monitoring

# GPU temperature
nvidia-smi --query-gpu=temperature.gpu --format=csv,noheader

# CPU/SoC temperature
sudo tegrastats --interval 1000

Normal operating range under sustained inference load: GPU 60–75°C, CPU 50–65°C. The GB10 thermal solution handles continuous full-load operation without throttling.