Best local models for document Q&A (October 2026)
Ranked for local document Q&A: capability (Artificial Analysis, BenchLeader), whether it fits 24 GB-512 GB hardware, license, runtime maturity. Nulls = no primary source found.
Last updated Oct 24, 2026. Fields tagged unverified lack a primary source. "No published figure yet" means we found no source, and we don't guess. Spark fit is estimated from published memory sizes, not tested on Spark hardware. Ollama listings may be cloud-only tags; check this for Kimi K3 and DeepSeek V4.1 before pulling.
Spark configs: single 路 dual 路 quad 路 DGX Spark hub
Top 10
| # | Model | Maker | Released | Architecture / params | Context | License | Memory | Runs on | Why ranked here |
|---|---|---|---|---|---|---|---|---|---|
| 1 | GLM-5.3-Flash | Z.ai (Zhipu) | 2026-08-25 | MoE, hybrid sparse+linear attention, native multimodal (image/video) 320B total / 18B active | no published figure yet unverified | MIT | BF16/native: ~306 GiB (Yotta Labs) FP8: no published figure yet Q8: no published figure yet Q4: no published figure yet LLMCheck: needs a 128 GB Mac at 3-bit in llama.cpp | Single Spark (128GB): Yes at ~3-bit (inferred from LLMCheck 128 GB Mac note) unverified Dual Spark (256GB): Yes at ~4-bit (inferred) unverified Quad Spark (~512GB): Yes, near-full precision fits (inferred) unverified 24GB GPU: No 128GB Mac / Strix: Yes at 3-bit (LLMCheck) | Best capability that actually fits a 128-512 GB box, MIT license, mainstream llama.cpp support. The sweet spot for Spark owners. Details 路 Hugging Face |
| 2 | Qwen3.8-27B | Alibaba Qwen | 2026-08-05 | Dense, image+video input 27B (27.8B per HF) total / 27B active | 262,144 native, extensible to ~1M | Apache-2.0 | BF16/native: ~54 GB (Yotta Labs) FP8: no published figure yet Q8: no published figure yet Q4: ~16 GB (Yotta Labs; 24 GB GPU class) | Single Spark (128GB): Yes, BF16 fits Dual Spark (256GB): Yes Quad Spark (~512GB): Yes 24GB GPU: Yes 128GB Mac / Strix: Yes | Strongest model that runs on one 24 GB GPU, Apache-2.0, huge context. Default pick for most local RAG builders. Details 路 Hugging Face |
| 3 | DeepSeek V4.1 Flash | DeepSeek | 2026-10-17 (V4.1 Flash 2026-09-10) | Causal encoder-decoder MoE with Engram memory, native image input 552B (Yotta Labs; HF safetensors total ~763B incl. extras) total / 8B input / 16B output active | 1M | MIT | BF16/native: 510 GB on disk, ~614 GB GPU memory (Yotta Labs) FP8: no published figure yet Q8: no published figure yet Q4: no published figure yet V4-Flash (older) runs at 2-bit on a 128 GB Mac per LLMCheck | Single Spark (128GB): No (use V4-Flash-0731 at 2-bit) Dual Spark (256GB): No Quad Spark (~512GB): Only if quantized, untested unverified 24GB GPU: No 128GB Mac / Strix: No (V4-Flash 2-bit yes) | Top benchmark scores and MIT, but no mainstream local runtime yet and too big for quad Spark natively; V4-Flash-0731 is the practical local DeepSeek. Details 路 Hugging Face |
| 4 | MiniMax M3.1 Flash | MiniMax | 2026-10-17 (HF repo; launch reported Oct 19) | MoE, native multimodal (text/image/video) ~428B total / 23B active | 1M | minimax-community (custom) | BF16/native: no published figure yet unverified FP8: no published figure yet unverified Q8: no published figure yet unverified Q4: no published figure yet unverified | Single Spark (128GB): no published figure yet unverified Dual Spark (256GB): no published figure yet unverified Quad Spark (~512GB): Likely at 4-bit (unverified) unverified 24GB GPU: No unverified 128GB Mac / Strix: no published figure yet unverified | Brand-new, strong on BenchLeader (62.0), smaller than DeepSeek/Kimi; local tooling still immature. Details 路 Hugging Face |
| 5 | Kimi K3 | Moonshot AI | 2026-06-13 (HF) | MoE (16 of 896 experts), Kimi Delta Attention, native vision 2.8T total / ~104B (Yotta Labs) unverified active | 1M | Kimi K3 License (custom) | BF16/native: 1.56 TB MXFP4 (Yotta Labs) FP8: no published figure yet Q8: no published figure yet Q4: 1.56 TB is already ~4-bit Needs 1.6 TB+ aggregate GPU memory | Single Spark (128GB): No Dual Spark (256GB): No Quad Spark (~512GB): No 24GB GPU: No 128GB Mac / Strix: No | Strongest open weights, but not runnable on any home rig, so ranked for awareness, not practicality. Details 路 Hugging Face |
| 6 | Qwen3.6-35B-A3B | Alibaba Qwen | 2026-04-15 | MoE (8 routed + 1 shared) 35B total / 3B active | 262,144 native, extensible to ~1.01M | Apache-2.0 | BF16/native: no published figure yet FP8: no published figure yet Q8: no published figure yet Q4: no published figure yet LLMCheck min RAM 24 GB | Single Spark (128GB): Yes Dual Spark (256GB): Yes Quad Spark (~512GB): Yes 24GB GPU: Yes 128GB Mac / Strix: Yes | Best speed-per-dollar for laptops/24 GB GPUs; #2 LLMCheck local index. Details 路 Hugging Face |
| 7 | Nemotron 3.5 Super VL | NVIDIA | no published figure yet unverified | Hybrid Mamba-2 + MoE + attention, C-RADIO vision encoder 120B total / 12B active | Up to 1M | OpenMDW-1.1 | BF16/native: no published figure yet FP8: no published figure yet Q8: no published figure yet Q4: no published figure yet BF16 deploy: 4x H100 or 2x GB200; NVFP4 card targets DGX Spark | Single Spark (128GB): Yes via NVFP4 (NVIDIA card points Spark users to it) Dual Spark (256GB): Yes Quad Spark (~512GB): Yes 24GB GPU: No 128GB Mac / Strix: no published figure yet | The most document-focused model here, first-party DGX Spark (NVFP4) support, and a matching embed/rerank stack. Details 路 Hugging Face |
| 8 | gpt-oss-120b | OpenAI | 2025-08-04 | MoE, MXFP4 post-trained 117B total / 5.1B active | no published figure yet unverified | Apache-2.0 | BF16/native: no published figure yet FP8: no published figure yet Q8: no published figure yet Q4: Fits a single 80 GB GPU in MXFP4 (model card) | Single Spark (128GB): Yes Dual Spark (256GB): Yes Quad Spark (~512GB): Yes 24GB GPU: No 128GB Mac / Strix: Yes (fits in 80 GB) | Over a year old and behind the Chinese MoEs on benchmarks, but the most plug-and-play 128 GB model with a clean license. Details 路 Hugging Face |
| 9 | Gemma 4 26B-A4B | Google DeepMind | no published figure yet unverified | MoE (8 of 128 experts + 1 shared), hybrid local/global attention, multimodal 25.2B total / 3.8B active | 256K | Apache-2.0 | BF16/native: no published figure yet FP8: no published figure yet Q8: no published figure yet Q4: no published figure yet LLMCheck min RAM 18 GB | Single Spark (128GB): Yes Dual Spark (256GB): Yes Quad Spark (~512GB): Yes 24GB GPU: Yes 128GB Mac / Strix: Yes | Fast, Apache-2.0, strong on 24 GB; r/LocalLLaMA favorite for dialogue. Details 路 Hugging Face |
| 10 | MiMo-V2.6-Flash | Xiaomi | 2026-09-27 | Sparse MoE (8 of 256 experts) 309B total / 15B active | 1M | MIT | BF16/native: no published figure yet FP8: no published figure yet Q8: no published figure yet Q4: no published figure yet LLMCheck min RAM 170 GB; needs 192 GB+ Mac | Single Spark (128GB): No Dual Spark (256GB): Yes (inferred from 170 GB min) unverified Quad Spark (~512GB): Yes 24GB GPU: No 128GB Mac / Strix: No | #1 on LLMCheck local index (Oct 18), MIT, great dual-Spark target. Details 路 Hugging Face |
Spark fit is estimated from published memory sizes, not tested on Spark hardware. Speeds come from the cited sources. Entries marked "hosted, not local" are cloud API speeds, and "ESTIMATE" entries are estimates. Neither is a local measurement.
Best for laptops (under 15B)
| Model | Params | Context | License | Memory | Speed | Note | Sources |
|---|---|---|---|---|---|---|---|
| Gemma 4 12B | 11.95B dense | 256K | Apache-2.0 | ~7 GB at 4-bit (LLMCheck) | 19 tok/s on 32 GB Mac mini M6 (LLMCheck) | Text, image and audio input | source source |
| Gemma 4 E4B | 4.5B effective (8B with embeddings) | 128K | Apache-2.0 | no published figure yet | no published figure yet | Tiny and fast; audio input | source |
| NVIDIA Nemotron 3.5 Bolt E4B | no published figure yet unverified | no published figure yet unverified | no published figure yet unverified | 4GB variant exists (repo name) | no published figure yet | Pairs with Nemotron-3.5-Embed-1B for an all-NVIDIA laptop stack | source |
| MiMo-V2.6-Distill-Qwen-9B | 9B (repo name) | no published figure yet unverified | no published figure yet unverified | no published figure yet | no published figure yet | Distilled from MiMo-V2.6 | source |
Laptop speeds are community measurements on the hardware named in each entry.
Honorable mentions
- gpt-oss-20b (21B/3.6B, 16 GB, 45 tok/s M6)
- GLM-5.3 flagship 753B (744 GB FP8, custom license)
- Qwen3.8-Flash-Next 125B/6B
- DeepSeek V4-Flash-0731 284B/~13B MIT
- Muse Spark Open (Meta, 816B, released 2026-10-15)
- KAT-Coder-V2.5 (#1 LLMCheck Oct 5, coding)
Announced but not released
- Qwen 4 (in training)
- DeepSeek V4.1 Pro (announced)
The RAGShelf weekly
New local RAG tools, releases, and tested setups. One email a week, no spam.