How to run JetBrains Mellum 2.1 locally with Ollama or llama.cpp

JetBrains put Mellum 2.1 on Hugging Face on October 7, and its official GGUF builds passed 27,000 downloads in under a week. The jump from Mellum 2 is the reason: on SWE-bench Verified it went from 2.0 to 47.0, and the architecture and base model did not change. This is the hands-on version: what it is, how it scores, which file to download for your hardware, how to run it with Ollama or llama.cpp, the settings that matter, and where it falls short.

What is Mellum 2.1?

Mellum 2.1 is an open-weights mixture-of-experts model from JetBrains with 12 billion parameters, of which 2.5 billion are active for each token, released under Apache 2.0. It has 64 experts with 8 active per token, 28 layers, and a 131,072-token context. It is a thinking model: it writes its reasoning inside <think> tags before the answer. The full name on Hugging Face is Mellum2.1-12B-A2.5B-Thinking.

The architecture is the same as Mellum 2, which JetBrains open-sourced in June. JetBrains says almost all the new work went into post-training, mostly reinforcement learning. For software engineering, the model trained inside real repositories with a shell and file-editing tools, and was rewarded when the tests passed. JetBrains pitches it as a worker inside agent pipelines: find the root cause of a failing test, draft a fix, check it. It also admits Mellum 2 could not work inside a repository as well as JetBrains wanted, and 2.1 is aimed squarely at that.

How good is it at coding?

It is strong at self-contained coding problems and middling at repo-level agent work, where Qwen3.5-9B still leads. These are JetBrains' own numbers, all run with the same pipeline in thinking mode:

Read it in two halves. On LiveCodeBench, HumanEval+ and the BFCL v4 function-calling test, Mellum 2.1 leads its group. On JetBrains' two other tool-use tests, WorkBench and ToolHop, it does not. On the three agentic benchmarks, where the model works inside a real repository through a shell, Qwen3.5-9B is ahead, by 10 points on SWE-bench Pro. JetBrains ran every model through the same open-source agent harness (Pi v0.73.1) with each model's default sampling, a 114K context and up to 16K tokens per turn. None of this has been independently reproduced yet, and none of it was measured on the 4-bit builds you will download.

So why run it instead of Qwen3.5-9B? Speed. With 2.5B active parameters, it does far less work per token than a dense 9B model. JetBrains reports almost twice Qwen3.5-9B's token throughput under heavy load on one H200. That is a server measurement with many requests at once, not your desktop, but the direction holds: fewer active parameters, faster tokens.

What hardware do you need?

A 12 GB GPU runs the recommended Q4_K_M build with the full 131K context; a 16 GB Mac runs it with a shorter context. JetBrains publishes five GGUF files. Each one is a single file, and the table includes JetBrains' own measure of how close each stays to the unquantized model:

The context is cheaper than you would expect. Only 7 of the 28 layers use full attention; the other 21 look back over just the last 1,024 tokens. By our arithmetic from the model's published config (4 KV heads of 128 dimensions, 16-bit cache), the full 131K context needs about 1.9 GB on top of the weights, and 32K about 0.5 GB. That assumes llama.cpp, which by default sizes the cache of the 21 sliding-window layers to the window; a runtime that kept a full cache for all 28 layers would need about 7.5 GB at 131K. It is an estimate, and runtime buffers add a little more, but it is why 8.1 GB of weights plus the full context just fits in 12 GB. The “good fit” column is our guidance, not JetBrains'. Check your own card with the VRAM calculator, and see the GPU guide for picking hardware.

Short on VRAM? llama.cpp can keep some of the expert weights in system RAM with --n-cpu-moe N, which moves the experts of the first N layers to the CPU. It runs slower, but it runs.

How do you run Mellum 2.1 with Ollama?

Mellum 2.1 is not in Ollama's own model library yet, so you pull it straight from Hugging Face with the hf.co prefix. Update Ollama first; support for a new architecture arrives with the llama.cpp version Ollama bundles. Then:

$ollama run hf.co/JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q4_K_M

The suffix after the colon picks the build, so swap Q4_K_M for Q6_K or Q8_0 if you have the memory. Then fix the one setting that will trip you up. Ollama's documented default context window is 4,096 tokens. A thinking model can spend that on reasoning before it writes a single line of code, and a coding agent's system prompt and tool definitions take a big share too. Raise it for every model with an environment variable before starting the server:

$OLLAMA_CONTEXT_LENGTH=32768 ollama serve

Or, inside an ollama run session, type /set parameter num_ctx 32768. Go higher if your memory allows; 32K is a sensible floor for agent work.

How do you run Mellum 2.1 with llama.cpp?

One command downloads the GGUF and starts an OpenAI-compatible server. llama.cpp has supported Mellum's architecture since June 2, so any recent build works. This is the command from JetBrains' model card, with a port added:

$llama-server -hf JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q4_K_M --ctx-size 131072 --temp 0.6 --top-p 0.95 --top-k 20 --port 8080

Why the port? JetBrains' card says the server answers on http://localhost:8080/v1, which was true when they wrote it. Since build b11521 on October 9, llama-server defaults to port 9931. On a current build without --port, a client pointed at 8080 finds nothing. Pass the port you want, or point your tool at 9931.

Two more things are already handled by default in current builds. Tool calling needs llama-server's Jinja chat templates, and those are on. The <think> block is split out of the answer into a separate reasoning field (the --reasoning-format default is auto), so a client that supports it shows reasoning and answer separately.

JetBrains also lists LM Studio as a supported runtime for the GGUF builds. Search for “Mellum2.1” in its model browser and pick the same Q4_K_M file. vLLM users can serve the full-precision weights with the commands on the main model card. The multi-token prediction head that JetBrains says makes a single response about 1.6x faster is a vLLM feature, still marked “coming soon”, so do not expect it from the GGUF builds.

What settings should you use?

Use temperature 0.6, top_p 0.95 and top_k 20 for chat and one-off coding questions, which is what JetBrains' own examples use. Its non-agentic benchmark scores used greedy decoding, so these are usage settings, not the benchmark settings. The same examples allow up to 81,920 output tokens per reply, which tells you how long the reasoning can run on hard problems. Give it room, or answers get cut off mid-thought.

One detail matters if you run it as an agent. JetBrains' agentic benchmark runs used each model's default sampling, which is temperature 1.0 for Mellum 2.1, not 0.6. If an agent built on it loops or gets stuck, that is one setting to try.

How do you use it in a coding tool?

Point any OpenAI-compatible coding tool at the local server and pick the model. That is http://localhost:11434/v1 for Ollama or the llama-server address with /v1 on the end. Codex CLI, Cline, Zed and most others have a local or OpenAI-compatible option, and llama-server also answers Anthropic's Messages API, which is how tools like Claude Code connect. Our guide to running a local LLM for coding walks through the editor side.

Where this model fits best is as the fast second model, not the main one. Mellum 2.1's pitch is sub-agents: hand it a focused job (read these files, find where this function is called, check this test) while a bigger model plans. In Bodega One Code, you would pull it through the Ollama preset, or add the GGUF in the llama.cpp models view. Bodega runs its own llama-server process for that preset, so the port change above does not apply there. Then make a custom agent that runs on Mellum and let the main agent hand it work as a sub-agent. Bodega checks whether a local sub-agent's model fits in VRAM beside the main one before it starts, so on a single 12 GB card this works best with a cloud main model, or on a machine with room for both. With a local model and air-gap mode on, nothing leaves your machine.

We have not benchmarked Mellum 2.1 in Bodega One Code. This is the same setup any local model uses, and that is all we are claiming. Our docs also note that a small local main model often does the work itself instead of handing it off; a 27B or cloud main model hands off reliably.

What are the honest caveats?

• Every score is JetBrains' own. Same pipeline for every model is better than mixing sources, but nobody has reproduced these numbers independently yet.
• The scores are not from 4-bit builds. JetBrains does not state the precision, presumably the full BF16 weights. The Q4_K_M build matches the full model's top token 88% of the time on JetBrains' test, which used Wikipedia text at a 512-token context, not code. That is normal for 4-bit, and it is not free.
• The speed claim is a server claim. “Almost twice Qwen3.5-9B” was measured under heavy load on an H200. A single request on your GPU will be fast because of the 2.5B active parameters, but no one has published consumer GPU numbers we can cite.
• Thinking costs time. Every answer starts with reasoning tokens. For autocomplete or quick edits, a non-thinking model will feel snappier.
• It is not the best small model at real repo work. On SWE-bench Pro, Qwen3.5-9B scores 38.0 to Mellum's 28.0 on JetBrains' own test.
• The docs are still catching up. JetBrains' launch post and main model card both list GGUF builds as coming soon even though the official GGUF repo is up, and the card's port 8080 is out of date for current llama.cpp.

When is something else the better choice?

For repo-level agent work on 12 GB, Qwen3.5-9B is the stronger pick on JetBrains' own numbers. If you have a 24 GB card, a 27B dense model like Qwen3.8-27B fits in Q4 at about 17 GB and gives you a much larger model to work with. If you have 64 GB of RAM and want Qwen's larger model, our Qwen3.8-Flash-Next with Strata guide covers that route. For the week-by-week picture of what shipped, see What's new in local AI.

Mellum 2.1 earns its place when speed matters more than the last few points: a sub-agent that reads and checks while a bigger model plans, a machine where a dense 9B runs too slowly, including setups that offload to the CPU, since only 2.5B parameters are read per token, or a team serving many requests at once on one GPU. It is small, Apache-licensed and quick, and on competitive coding it punches above its size.

Sources

• Model card, architecture, benchmarks and vLLM commands: huggingface.co/JetBrains/Mellum2.1-12B-A2.5B-Thinking
• GGUF builds, sizes, quality measurements and run commands: huggingface.co/JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF
• JetBrains launch post: blog.jetbrains.com
• Model config used for the context-memory estimate: config.json
• llama.cpp Mellum support (merged June 2, 2026): github.com/ggml-org/llama.cpp/pull/23966
• llama-server default port change to 9931: github.com/ggml-org/llama.cpp/pull/30159
• llama-server flags (--n-cpu-moe, --reasoning-format, Jinja): llama.cpp server README
• Ollama context window default and settings: docs.ollama.com/faq