Dual DGX Spark (256GB) for local RAG

What fits

Model	Quant / engine	Measured decode	Source
Qwen3-235B-A22B	GPTQ-Int4, vLLM TP=2 EP=2	17.0 tok/s at batch 1; 36.4 aggregate at batch 4	Conselara Labs
Qwen3-235B-A22B	4-bit GGUF, llama.cpp RPC	~12.5 tok/s (1.8 on one unit)	NVIDIA forum
Qwen3-235B-A22B	NVFP4, TensorRT-LLM	Official two-Spark model; 11.73 tok/s at batch 1 as quoted by Conselara	NVIDIA, Conselara
gpt-oss-120b	vLLM TP=2 (eager mode)	2,641 aggregate tok/s at concurrency 48 (vs 222 with pipeline parallel)	myers-dev
Llama 3.1 405B	AWQ-INT4, vLLM TP=2	Loads only for testing (max-model-len 64); NVIDIA warns there is not enough headroom for production. No published benchmark yet.	NVIDIA vLLM playbook
GLM 5.3 Flash	FP4, vLLM cluster recipe	NVIDIA recipe for 2×128GB; no NVIDIA benchmark. Community reports 24–33 tok/s typical	NVIDIA, shawnbot

How to link the units

One QSFP cable between the ConnectX-7 ports gives a 200GbE direct link; set up IPs and passwordless SSH per NVIDIA Connect two Sparks. Then validate with the NCCL playbook.
vLLM: a Ray cluster with --tensor-parallel-size 2 (playbook, cluster playbook). TensorRT-LLM: MPI multi-node (playbook). SGLang: scale with --tp (playbook). llama.cpp: RPC backend (forum how-to).
Tensor parallel beat pipeline parallel on throughput and latency in every run of one two-Spark study (source).

Recommended local RAG stack

Inference: vLLM TP=2 serving Qwen3-235B-A22B or GLM 5.3 Flash.
Embeddings, reranker, and parsing: run bge-m3, the reranker, and Docling on one node.
Vector DB: Qdrant or Weaviate.
Front-end: Dify, Open WebUI, or AnythingLLM pointed at the OpenAI-compatible endpoint.

Models from our top 10 that fit

GLM-5.3-Flash: Yes at ~4-bit (inferred) unverified
Qwen3.8-27B: Yes
Qwen3.6-35B-A3B: Yes
Nemotron 3.5 Super VL: Yes
gpt-oss-120b: Yes
Gemma 4 26B-A4B: Yes
MiMo-V2.6-Flash: Yes (inferred from 170 GB min) unverified

Spark fit is estimated from published memory sizes, not tested on Spark hardware. See Best local models.

Best for

Teams that want a 235B-class MoE model for harder questions over their documents, or more concurrent users than one unit handles.

Gotchas

Multi-node CUDA-graph capture hung in one owner's TP setup, forcing eager mode (source).
Containers need --device /dev/infiniband and --ulimit memlock=-1, or RDMA fails (source).
Startup takes about 15 minutes for a 235B model across two nodes (source).
Single-stream speed does not double; decode stays bound by memory bandwidth.

Benchmarks are quoted from the linked sources with their settings; we have not re-run them. Where no result exists we say "no published benchmark yet". Last updated 2026-10-24.