How to run NVIDIA Nemotron 3.5 Bolt locally, and whether it really fits in 24 GB

NVIDIA released Bolt on Hugging Face on October 15, and its official GGUF passed 60,000 downloads in three days. The pitch is easy to like: a model this size, built from the start to fit a high-end consumer GPU instead of a server. This is the hands-on version: what it is, how it scores, how fast it really is, whether it fits your card, how to run it, and where it falls short.

What is Nemotron 3.5 Bolt?

Nemotron 3.5 Bolt is a 74B-parameter vision-language model from NVIDIA that activates 9B parameters per token, released under the OpenMDW-1.1 license. It takes text, images and video and writes text, with up to 256K tokens of context, tool calling and a switchable thinking mode. NVIDIA says it is ready for commercial use. The architecture is a hybrid: Mamba-2 layers mixed with mixture-of-experts and attention layers, plus NVIDIA's C-RADIO vision encoder.

It was not trained from scratch. NVIDIA compressed its larger Nemotron 3.5 Super VL (120B total, 12B active) with an automated architecture search that picked how many experts to keep in each layer, aiming for a footprint under 24 GB. Then it quantized every expert to 2-bit (IQ2_XXS, about 2.06 bits per weight) and used quantization-aware distillation to win back the accuracy. That last step is what separates this from a normal 2-bit download: NVIDIA quantized it and then retrained it at 2-bit to win back accuracy, rather than just rounding the weights down.

How good is it?

On NVIDIA's own benchmarks, the 24GB build beats Qwen3.6-35B-A3B and Gemma 4 26B on most agent and reasoning tests, but not all of them. These are selected rows from NVIDIA's table, with Bolt at its 2-bit size and the others at full precision:

Terminal-Bench 2.1, where a model works in a real shell, is the number coding agents care about most, and Bolt leads it by more than six points. It trails Qwen slightly on PinchBench, trails both Qwen and Gemma on the AA-Omniscience knowledge test, and hallucinates more often than Qwen and NVIDIA's own smaller Lightning model. *For MCP-Atlas, NVIDIA measured Qwen at 51.2 on the 500 public tasks; Qwen's own reported score is 62.8, which would put Qwen ahead. None of these numbers have been reproduced independently yet.

How fast is it?

Slower than the small MoE models it beats on quality, and about three times faster than a dense 27B. NVIDIA's own speed table was measured with llama.cpp on an RTX Spark laptop with 128 GB of memory, on its coding speed benchmark:

So the trade is quality per gigabyte, not speed. On the agent replay test, Bolt took about a third longer than Qwen3.6 to finish the same work. These are NVIDIA's numbers on NVIDIA's laptop. Nobody has published RTX 4090 or 5090 figures we can cite yet.

Does it really fit in 24 GB?

The model fits in 24 GB on its own; the full setup NVIDIA recommends does not. Here is what the files weigh before any context:

Those are decimal gigabytes. A “24 GB” card like the RTX 4090 holds 24 GiB, about 25.8 GB, so in GPU terms all three files come to about 23 GiB: they just squeeze in, with about 1 GiB left for the context cache and runtime buffers. The hybrid Mamba design keeps that cache smaller than a normal transformer's, and NVIDIA says its architecture search targeted a footprint under 24 GB, but NVIDIA's own deployment table is more cautious: its smaller recipe uses about 28 GB and targets an RTX Spark with 64 GB or a discrete GPU with 32 GB or more (65K context), and the larger one uses about 32 GB for the full 256K context (48 GB+ cards or a 128 GB Spark). Ollama's package is listed at 25 GB. A 24 GB card such as the RTX 4090 is not in NVIDIA's table.

• 32 GB card (RTX 5090) or RTX Spark: run NVIDIA's 28 GB recipe as published.
• 24 GB card (RTX 4090): the model alone (about 20.5 GiB) leaves roughly 3.5 GiB. Keep the context short, drop the vision encoder or the drafter if you need more room, or let some layers spill into system RAM and accept slower speeds. We have not tested this and nobody has published numbers we can cite.
• RTX 30-series: NVIDIA lists Ada Lovelace (RTX 40) and Blackwell (RTX 50) GPUs as supported. Ampere cards like the 3090 are not listed.
• Mac: Ollama's page says Apple Silicon and Metal are not currently supported, and NVIDIA lists Windows and Linux only.

How do you run Nemotron 3.5 Bolt with Ollama?

Bolt is in Ollama's official library, added in Ollama 0.40.5 on October 14. Update Ollama, then:

$ollama run nemotron-3.5-bolt

Ollama's 25 GB package includes vision and the drafter, so on a 24 GB card part of it will spill into system RAM once you raise the context. Before you point an agent at it, raise the context window: Ollama's documented default is 4,096 tokens, which a thinking model and a coding agent's tool definitions use up fast. Set it when you start the server, for example to 32K:

$OLLAMA_CONTEXT_LENGTH=32768 ollama serve

On Windows, set it in PowerShell instead, and quit the Ollama desktop app first so the new server picks it up:

PS>$env:OLLAMA_CONTEXT_LENGTH=32768; ollama serve

How do you run Nemotron 3.5 Bolt with llama.cpp?

Use a llama.cpp build from October 15 or later, because that is when support for Bolt's LiLiCorr drafter was merged. NVIDIA builds llama.cpp from source with CUDA on (cmake -B build -DGGML_CUDA=ON). Install the Hugging Face CLI with pip install -U huggingface_hub, then download the three files:

$hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf --local-dir models/nemotron-bolt

$hf download nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUF NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf --local-dir models/nemotron-bolt

Then start the server with NVIDIA's 28 GB recipe, which is for 32 GB+ cards. It runs a 65K context with an 8-bit KV cache and up to six drafted tokens:

$llama-server --model models/nemotron-bolt/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf --mmproj models/nemotron-bolt/mmproj-NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf --batch-size 2048 --ubatch-size 512 --ctx-size 65536 --parallel 1 -ngl 999 --spec-type draft-dflash --spec-draft-n-max 6 --spec-draft-model models/nemotron-bolt/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr.gguf --spec-draft-ngl 999 --cache-type-k q8_0 --cache-type-v q8_0 --temp 1.0 --top-p 0.95 --flash-attn on --jinja

Watch the port. NVIDIA's instructions tell you to use whatever port llama-server prints, and that is the right advice: since build b11521 on October 9, llama-server defaults to 9931 instead of 8080. Add --port if your tools expect a specific one. The server speaks the OpenAI-compatible API at /v1.

What settings should you use?

NVIDIA recommends temperature 1.0 and top_p 0.95. Thinking is controlled through the chat template: pass enable_thinking as true or false in chat_template_kwargs, and add medium_effort for a middle setting. Reasoning and the answer share the same max_tokens budget, and NVIDIA's examples use 16,384, so give it room or answers get cut off. For quick edits, turning thinking off makes it feel much faster.

How do you use it in a coding tool?

Point any OpenAI-compatible coding tool at Ollama or your llama-server and pick the model. NVIDIA built Bolt for local agent harnesses with tool calling, web browsing and terminal use, so this is its intended job. Our guide to running a local LLM for coding covers the editor side.

In Bodega One Code, the simplest route is the Ollama preset with Ollama 0.40.5 or later. If you run your own llama-server with NVIDIA's recipe, add it as a Custom OpenAI-Compatible provider pointing at that server, because Bodega's own managed llama.cpp loads a single GGUF and has no setting for NVIDIA's drafter or vision file. See BYOLLM and Models & Providers. With a local model and air-gap mode on, nothing leaves your machine. We have not benchmarked Bolt in Bodega One Code; it uses the same endpoint any tool uses, and that is all we are claiming.

What are the honest caveats?

• The 24GB label is about the model, not the setup. With vision, the drafter and context, NVIDIA's recipes need 28 to 32 GB.
• Every benchmark is NVIDIA's own. And on MCP-Atlas, Qwen's own reported score is higher than the one NVIDIA measured.
• It is not the fastest option. On NVIDIA's own laptop it generated fewer tokens per second than Qwen3.6-35B-A3B and Gemma 4 26B, and took longer on the agent replay.
• NVIDIA hardware only. Ada or Blackwell GPUs on Windows or Linux. No Mac.
• Check the license. OpenMDW-1.1 is a permissive model license from the Linux Foundation, not Apache 2.0. It allows commercial use, but it has a patent-lawsuit termination clause, so read it before you ship a product on it.

When is something else the better choice?

On a 24 GB card, a dense model that fits with room to spare is the safer pick. Qwen3.8-27B is about 17 GB at Q4 and leaves space for a long context, though NVIDIA's own table shows it is much slower. If speed matters most, Qwen3.6-35B-A3B and Gemma 4 26B were both faster than Bolt in that same table. On a smaller card, our Mellum 2.1 guide covers a fast 12B coding model that fits in 12 GB. Check your hardware with the VRAM calculator, and see What's new in local AI for the weekly picture.

Bolt earns the setup when you have a 32 GB card or an RTX Spark and want the strongest agent scores you can get in that memory. On a 24 GB card it is an experiment worth trying, not a sure thing.

Sources

• Model card, benchmarks, deployment recipes and settings: huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF
• NVIDIA technical blog, including the RTX Spark speed table: research.nvidia.com
• LiLiCorr drafter GGUF: huggingface.co/nvidia
• Ollama library page (25 GB package, Apple Silicon not supported): ollama.com/library/nemotron-3.5-bolt
• Ollama 0.40.5 release notes: github.com/ollama/ollama
• llama.cpp LiLiCorr drafter support, merged October 15, 2026: github.com/ggml-org/llama.cpp/pull/30506
• llama-server default port change to 9931: github.com/ggml-org/llama.cpp/pull/30159
• Ollama context window default: docs.ollama.com/faq