Which GPU for local AI? A developer's guide

The question that comes up in every local AI thread: "Can my GPU run this?" Usually someone replies with a number, someone else disputes it, and the thread turns into a benchmark debate. Not super useful if you just want to know whether to bother.

Here's the practical answer. Not a spec sheet. This is a breakdown of what each VRAM tier can actually do, which models work well at each level, and what you'll notice in day-to-day coding use.

Why VRAM is the bottleneck

When you run a local model, the weights load into GPU memory. Not system RAM, not disk. VRAM. A quantized 7B model at Q4 takes roughly 4–5 GB. A 32B model at Q4 takes around 20 GB. If the model doesn't fit, it either falls back to CPU (slow) or fails to load.

System RAM matters for context window overflow and CPU fallback. But if you want GPU inference at useful speeds, VRAM is the number to watch.

Quantization (Q4, Q4_K_M, Q8, etc.) is how the community squeezes large models into smaller VRAM budgets. Q4_K_M is the sweet spot most people land on: good quality loss-to-size ratio, fast enough for real use.

The VRAM tier table

These are the tiers Bodega One Code uses internally to match hardware to model recommendations. If you're using a local provider like Ollama or LM Studio, this maps directly to what you should be pulling.

"Best agent model" here means the largest model that fits comfortably with room for a reasonable context window. You can often run a bigger model if you're careful about context length, but these are the settings that work reliably.

What can each VRAM tier actually run?

Under 4 GB: entry-level GPUs, integrated graphics

You can run something. Qwen3.5-4B Q4 fits in around 3 GB and is the strongest coder at this size. It handles simple tasks, short completions, quick Q&A. For serious coding work (multi-file edits, debugging complex logic, long context), it hits a wall fast. Treat this tier as proof-of-concept, not production workflow.

6–8 GB: RTX 3060, RTX 4060, RX 6600

This is where local AI becomes actually useful. Qwen3.5-9B Q4_K_M fits here and it punches above its size on code tasks. You'll feel the context limit on larger projects, but for focused single-file or small multi-file work it's solid. The RTX 3060 at 12 GB is actually a sweet-spot card, bumping you up to the next tier.

8–12 GB: RTX 3060 12GB, RTX 4060 Ti, Arc A770

Qwen2.5-Coder-14B Q4_K_M lands here. A real jump in reasoning quality. This tier handles most day-to-day coding tasks well: refactors, new component generation, debugging with reasonable context. If you're buying a single card for local AI work and have a $300–400 budget, aim here.

12–16 GB: RTX 3080, RTX 4070, RTX 4080

Devstral Small 2 24B at Q4 lands here (around 14 GB) and is the sweet spot for agentic coding: 68% SWE-bench Verified, Apache 2.0, 256K context. This tier works extremely well, and most people who use Bodega One Code as their primary dev tool are in this range. GLM-4.7-Flash, a 30B-A3B mixture-of-experts model, is another strong option and can even run on 6–8 GB cards: it keeps only ~3B active parameters in VRAM and streams the inactive experts from system RAM, so it punches above its footprint at the cost of speed.

16–24 GB: RTX 3090, RTX 4090, RTX 4080 Super

Qwen3.6-27B at Q4 is the gold standard for local coding models as of 2026. At 77.2% SWE-bench Verified, it's the highest-scoring locally-runnable coding model, ahead of last generation's cloud frontier on that benchmark. At this tier, the main limit is context window, not model quality. The RTX 3090 at 24 GB hits this tier and is the current community favorite for pure local AI work per dollar.

24–48 GB: RTX 3090 Ti, A5000, multi-card setups

KAT-Dev-72B-Exp Q4 fits here at ~40 GB and posts 74.6% SWE-bench Verified, the strongest coding score in the workstation tier. Llama 3.3 70B also fits and remains a strong general-quality pick for research and long-document work, but for coding it trails the purpose-built options. Unless you specifically need a 70B-class model, most developers won't notice the difference over a current 27B like Qwen3.6-27B for typical coding work. The 3090 24 GB is better value for coding unless you have specific needs.

Apple Silicon is different

Apple M-series chips use unified memory, so the GPU and CPU share the same pool. That means a MacBook Pro with 16 GB RAM has 16 GB available for model weights, compared to a 16 GB discrete GPU. The effective headroom is similar but the architecture is different: Apple uses MLX-optimized model formats, not GGUF.

At 16 GB unified: Qwen3.5-9B MLX Q4 runs well and is the recommended starting point. At 32 GB: Qwen2.5-Coder-14B MLX. At 64 GB+: Qwen3.6-27B MLX, the same gold tier as a high-end discrete GPU setup.

The M4 Max at 128 GB can run 70B class models comfortably. Expensive, but it's the only laptop setup that gets you frontier-quality local inference.

Bodega One Code detects Apple Silicon automatically and recommends MLX provider format. If you're on a Mac, set your provider to Ollama with an MLX-tagged model variant for best performance, or use the MLX provider preset directly.

Coding agents need more than chat

Chat performance and agent performance aren't the same thing. A model that handles simple Q&A at 6 GB can struggle when used as an autonomous coding agent, because:

• Agent loops maintain longer context: system prompt, memory, file content, tool history, all at once
• Multi-file tasks accumulate tokens fast
• Instruction-following degradation at the end of long context hits smaller models harder

As a rough rule: bump up one tier from whatever you'd use for chat. If you run 8B for chatting comfortably, a 14B model will serve you better for agentic coding work. If you want serious autonomous agent work (the kind that rewrites files, runs tests, and handles multi-step tasks), 12 GB+ is the practical floor.

Bodega One Code's Quality Enforcement Layer runs verification steps throughout the agent loop, which helps compensate for smaller models' tendency to make hard-to-catch errors. But there's no substitute for raw model quality on complex tasks.

What about CPU-only?

It works for small models. llama.cpp runs on CPU; Ollama falls back to CPU if no GPU is detected. Speeds are typically 1–5 tokens/second depending on your CPU and model size, versus 30–80+ tokens/second on a mid-range GPU.

For interactive chat that's usable. For an agentic coding loop that might run 50+ tool calls per task, 2 tokens/second is painful. CPU-only is a fallback, not a recommendation.

What GPU should you buy for local AI coding?

For pure local AI coding work in 2026, the community consensus:

• Budget pick: RTX 3060 12 GB. Cheap used, gets you into the 8–12 GB tier, runs 14B models well.
• Best value: RTX 3090 24 GB. Still the community pick for 30B-class models, but the used market has heated up: expect roughly $1,200-1,300 in mid-2026 (the GDDR6 squeeze roughly doubled it from a year ago). Fits Qwen2.5-Coder-32B Q4 with room for context. The card most serious local AI developers actually run.
• New card pick: RTX 5060 Ti 16 GB, around $449-549. New warrantied hardware, fits the excellent-tier models. (The RTX 4080 Super, the previous pick here, was discontinued when the 50-series launched and now sells above its old MSRP.)
• Mac: M3 Pro 36 GB or M4 Max 48 GB+ if you're staying in the Apple ecosystem.

VRAM is the spec to optimize for. Everything else (CUDA cores, clock speed, power consumption) matters less for inference workloads than raw memory capacity.

Getting started

If you already have a GPU and want to wire it up to a local AI coding environment, the Bodega One Code quick start guide covers the full setup: provider configuration, model selection, and the first agent task. Bodega One Code auto-detects your hardware on launch and recommends a model tier based on your VRAM.

And if you want to go deeper on what BYOLLM means and why it matters that you can swap models freely.That's the follow-up read.

For a full side-by-side comparison of every model at each tier, with benchmarks, license, Ollama availability, and descriptions, see the Local LLM Guide.