How to run a local LLM for coding

A local LLM runs on your own computer instead of a company's servers. For coding, that buys you three things: your code never leaves your machine, there is no per-token bill, and it keeps working with the Wi-Fi off. The tradeoff is that you run the model yourself, which sounds harder than it is. On modern hardware it is about a 15-minute setup.

Here is the honest version of the whole process, tool by tool, including the parts nobody tells you. Nothing below is specific to one product. Pick the tools you like at each step.

What you actually need

Every "how do I code with a local model" question is really about one of three things:

• A runtime (the model server) that loads a model and answers requests: Ollama, LM Studio, or llama.cpp.
• A model in GGUF format, sized to fit in your GPU's VRAM.
• A client (an editor, extension, or agent) that sends prompts to the runtime and applies the edits.

Take them in order. Each step below is one of the three.

Step 1: Size a model to your hardware

A model has to fit in GPU memory to run at usable speed. The rule of thumb for a 4-bit (Q4) quantized model is roughly 0.5 to 0.6 GB of VRAM per billion parameters, plus a gigabyte or two of overhead for context. That gives a simple mapping:

No dedicated GPU? A small 3 to 4B model still runs on CPU or integrated graphics fast enough for autocomplete. Apple Silicon is the happy exception: its unified memory runs mid-size models well without a discrete card. To match a specific model to your card, use the VRAM calculator, and the GPU guide maps every tier to real hardware.

Step 2: Install a runtime

Three honest options, in rough order of how little setup they need:

• Ollama is the simplest. One binary, a CLI, and it pulls models with a single command. It serves a local API on http://localhost:11434 that is also OpenAI-compatible at /v1. Best default for most people.
• LM Studio is a GUI. Browse and download models from its Discover tab, watch VRAM usage as you load them, and start an OpenAI-compatible server on http://localhost:1234/v1. Best if you want to see what you are doing.
• llama.cpp is the engine the other two are built on. Run its llama-server against a GGUF file yourself for maximum control and the newest models first. Best if you live in a terminal and want the bleeding edge.

They all serve the same GGUF models over a similar local API, so nothing here locks you in. On macOS and Windows, download the app from ollama.com or lmstudio.ai. On Linux, Ollama installs with one line:

$curl -fsSL https://ollama.com/install.sh | sh

Step 3: Pull a coding model

With Ollama running, one command downloads a model and everything it needs. Start small to confirm the pipeline works, then size up. A 7B coder is a fine first pull on any 8 GB card:

$ollama pull qwen2.5-coder:7b

Then chat with it straight from the terminal to prove it loads:

$ollama run qwen2.5-coder:7b

Rough picks by VRAM, all pullable the same way:

• 6 to 8 GB: qwen2.5-coder:7b or GLM-4.7-Flash. Solid autocomplete and focused edits.
• 10 to 12 GB: qwen2.5-coder:14b. The sweet spot for most single-GPU setups.
• 16 to 24 GB: qwen3-coder (the current flagship open coder) or qwen2.5-coder:32b for multi-file agentic work.

Two things people get wrong here: under 8B, use a Q6 or Q8 quantization rather than Q4, since Q4 costs roughly 8 to 10% accuracy at that size; and do not judge a model by HumanEval, which is saturated. For the full ranking by SWE-bench Verified and VRAM tier, updated monthly, see the best local LLMs for coding page.

Step 4: Connect it to your editor

A running model does nothing until a client talks to it. The connection is always the same idea: give the client the runtime's local URL and pick your model. Any of these work:

• Continue.dev is an open-source extension for VS Code and JetBrains. Add your Ollama or LM Studio model in its config file.
• Cline is an open-source VS Code extension with an autonomous agent. Point its Ollama provider at http://localhost:11434, or use its OpenAI-compatible option.
• Zed has a built-in Ollama provider that auto-discovers the models you have pulled.
• Aider is a terminal pair-programmer that edits files and writes git commits, driven by a local Ollama or OpenAI-compatible endpoint.
• VS Code can now bring your own model, including a local endpoint, with no sign-in for chat and agent mode (inline ghost-text completions still require a GitHub account).
• Bodega One Code ships Ollama, LM Studio, and llama.cpp as built-in presets, so connecting is a dropdown rather than a config file, and the editor, AI chat, and an autonomous agent are all in one app.

Whichever you choose, the URL is the same one the runtime printed on startup: http://localhost:11434 for Ollama, http://localhost:1234/v1 for LM Studio, http://localhost:8080 for llama.cpp.

Step 5: Confirm it is actually local

Two quick checks prove the model is running on your machine and not quietly calling a cloud:

• Pull the plug. Send a prompt, then turn off Wi-Fi and send another. If it still answers, the model is genuinely local.
• Watch the GPU. Your VRAM usage should jump when the model loads, and the GPU should spike while it generates a response.

If you need a hard guarantee that nothing escapes, some tools enforce it rather than trust it. Bodega One Code has an air-gap mode that blocks network egress at the process level, so a local session cannot phone home even by accident. We wrote up how that enforcement works.

Common mistakes, and how to avoid them

• Model too big for your VRAM. It spills into system RAM and generation crawls. Drop one size tier and it will fly.
• Context window too small. 8K tokens is not enough for repo-level work. Aim for a 32K minimum; 64K to 128K is the sweet spot.
• The first response is slow. That is the model loading into VRAM. Every response after is fast until you swap models.
• Expecting frontier-model output on entry hardware. Local models are genuinely good now, but a 7B is not Claude. We wrote an honest assessment of whether local models are good enough for real work.

The all-in-one option

Everything above works with free, open tools, and starting there is the right call. If you would rather not wire a runtime, an extension, and an agent together yourself, that is the gap Bodega One Code fills: a full IDE with the editor, AI chat, and an autonomous coding agent in one app, local providers as built-in presets, an air-gap mode, and bring-your-own-LLM across 10+ providers. Your code stays on your machine. Free for personal use, in open beta now. For the click-by-click version of the local path, see the Ollama setup walkthrough or the LM Studio setup guide.

Sources

• Ollama OpenAI-compatible API and default port 11434: docs.ollama.com
• LM Studio local server (port 1234, OpenAI-compatible): lmstudio.ai/docs
• llama.cpp server (llama-server, GGUF, port 8080): github.com/ggml-org/llama.cpp
• VS Code bring-your-own-key for local models: code.visualstudio.com
• Best local LLMs for coding, ranked by SWE-bench and VRAM tier: bodegaone.ai/local-llms
• VRAM calculator (match a model to your GPU): bodegaone.ai/resources/vram-calculator