Single DGX Spark (128GB) for local RAG

What fits

Model	Quant	Measured decode	Source
gpt-oss-120b	MXFP4	41.1 tok/s (prefill 1,169), Ollama 0.12.6	Ollama blog
gpt-oss-120b	MXFP4	57–60 tok/s pure decode, vLLM 0.19 TP=1	Conselara Labs
gpt-oss-20b	MXFP4	58.3 tok/s, Ollama	Ollama blog
Qwen3 32B	q4_K_M	9.4 tok/s, Ollama	Ollama blog
Llama 3.1 70B	q4_K_M	4.4 tok/s, Ollama	Ollama blog
Llama 3.3 70B	NVFP4	Supported in NVIDIA SGLang playbook; no published benchmark yet	NVIDIA
Qwen3-235B-A22B	~4-bit GGUF	~1.8 tok/s, llama.cpp, 2K context (barely fits)	NVIDIA forum
Llama 3.1 405B	any	Does not fit on one unit	—

How to set it up

No linking needed. Pick an inference engine from NVIDIA's playbooks: Ollama, llama.cpp, vLLM, SGLang, or TensorRT-LLM.

Recommended local RAG stack

Inference: Ollama for simplicity, or vLLM for higher throughput and many users. gpt-oss-120b is the commonly recommended single-Spark model (forum).
Embeddings and reranker: bge-m3 + bge-reranker-v2-m3, as used in owner stacks (source, source).
Vector DB: Qdrant (source) or each app's built-in store.
Front-end: Open WebUI (official playbook), AnythingLLM, or Dify. Use Docling for parsing.

Models from our top 10 that fit

GLM-5.3-Flash: Yes at ~3-bit (inferred from LLMCheck 128 GB Mac note) unverified
Qwen3.8-27B: Yes, BF16 fits
Qwen3.6-35B-A3B: Yes
Nemotron 3.5 Super VL: Yes via NVFP4 (NVIDIA card points Spark users to it)
gpt-oss-120b: Yes
Gemma 4 26B-A4B: Yes

Spark fit is estimated from published memory sizes, not tested on Spark hardware. See Best local models.

Best for

Private Q&A over a personal or team document set, with a 120B-class model, one or a few users, and everything on one quiet box.

Gotchas

Dense 70B+ models are slow; prefer MoE models.
Memory is shared by the LLM, embedder, vector DB, and OS.
Use ARM64/cu130 images (see the hub).

Benchmarks are quoted from the linked sources with their settings; we have not re-run them. Where no result exists we say "no published benchmark yet". Last updated 2026-10-24.