Hardware & System Requirements

Minimum - the app runs, CPU-only inference with small models:

Recommended - comfortable agentic use with mid-size models:

If you have no discrete GPU, Bodega detects CPU-only mode and caps context based on system RAM. You'll see CPU mode - context capped for sustainable inference in the AI panels. 1–4B models are usable; anything larger will be slow.

On every launch, Bodega probes your GPU VRAM using the systeminformation library. The result feeds three things:

• Model recommendations - the Discover tab's Fits My GPU filter and first-run model cards
• Context window ceiling - the max tokens Bodega will send to a local model (written to hardware.local_context_ceiling)
• Cloud Boost escalation - whether your hardware tier triggers automatic escalation on reasoning-heavy tasks

The detection runs once at startup. VRAM capacity is cached for the session. Free VRAM is re-read on demand rather than on a clock: the backend caches it for 3 seconds, so any consumer asking more often than that gets the cached figure. The VRAM timeline strip in the UI asks every 5 seconds, so that is the cadence you actually see. Either way a model already loaded is accounted for.

If the probe hangs (beta.38): the underlying GPU query (WMI on Windows) can occasionally wedge. Each probe now has a hard 10-second deadline - a hung query fails that one reading instead of pinning hardware info, fit checks, and model launches behind it forever. A failed or timed-out probe is never cached as an answer: after a timeout Bodega serves the last good reading (if it has one) for a short backoff window and then retries, so a one-off failure is not remembered as "0 GB VRAM" for the rest of the session.

If a model quietly loads on the CPU: when another program is holding video memory, llama.cpp can fall back to the CPU, answer normally, and report itself ready - many times slower. Bodega reads llama.cpp's own layer-offload report after each load; if the model asked for the GPU and got none of it, the chat status line says so, with the likely cause and what to do (close the other program and reload the model). A deliberate CPU-only run, and the normal partial split for a model slightly larger than the card, stay silent.

Apple Silicon: memory is unified, and Metal grants a fraction of it rather than all of it. Bodega budgets min(sysctl iogpu.wired_limit_mb, total RAM x 0.75) at 64 GB or more, x 0.70 below that. The onboarding badge shows your chip name and memory figure.

Other unified-memory machines (NVIDIA DGX Spark / GB10, AMD Strix Halo): these report 0 VRAM on every controller while having a real, large addressable GPU. Bodega recognises them by model string and budgets total RAM minus an OS reserve (16 GB on Spark-class, 10 GB otherwise). A shared-memory GPU Bodega cannot name (an Intel iGPU, an ordinary APU) deliberately gets no budget rather than a guessed one: over-promising there hands you a model that cannot load.

No GPU: this needs positive evidence - zero controllers, or every controller a known software adapter (Microsoft Basic Display Adapter, llvmpipe, a VM display part). A VRAM reading of 0 on its own is NOT treated as no GPU, because that is exactly what a unified-memory machine reports.

Multi-GPU: llama.cpp splits a model across every visible same-vendor device (--split-mode layer), so Bodega sums the VRAM of the largest same-vendor group for the llama.cpp budget, and pools free VRAM across those same devices when every one of them reports a reading. Cross-vendor sums are never produced - a CUDA build cannot address an AMD card. Ollama is different: it loads a model onto ONE GPU, so its budget stays the largest single card.

A live hardware readout appears at the top of the Hardware section in the Help panel.

1. Open the Help panel: gear icon (top right) → Help
2. Select Hardware in the left rail
3. The Your System card at the top shows: CPU model, total RAM, GPU model, VRAM, and a one-line capability summary

The summary uses these thresholds:

The card fetches from the backend on mount. If the backend isn't running when you open it, the card won't appear - no error is shown.

Bodega uses two separate hardware classification systems. They serve different purposes and have different thresholds - do not confuse them.

Discover tab tiers (used for model recommendations and the Fits My GPU filter):

Routing tiers (used by the agentic loop for model selection and Cloud Boost escalation decisions):

The routing tiers also expose two capability flags: canRunSmartTier (13B+ accessible, requires 10 GB) and canRunCodeTier (code-specialized 7B models, requires 6 GB).

For Q4_K_M quantization - the default in Bodega's managed llama.cpp mode - model weight size in VRAM is roughly:

VRAM (GB) ≈ parameters (B) × 0.55 + KV cache + ~1 GB overhead

Actual sizes from Bodega's catalog (Q4_K_M):

KV cache is the hidden cost. An 8B model's KV cache grows from about 1.2 GB at 8K tokens to ~20 GB at 128K tokens. Context length is where VRAM gets consumed faster than the model weights table suggests. Bodega's context ceilings are set conservatively to prevent thrashing during agentic runs.

After VRAM detection, Bodega computes a context ceiling and writes it to hardware.local_context_ceiling. This caps how many tokens the agentic loop sends to local models.

GPU path (VRAM-based):

CPU-only path (RAM-based, no discrete GPU):

The ceiling calculation uses max(free VRAM, total VRAM − 4 GB) rather than free VRAM alone. This prevents a model that's already loaded from causing the probe to misclassify your card. For example: a 32 GB GPU with a 26B model resident still probes as a 28 GB card for ceiling purposes, not a 4 GB card.

To raise the ceiling manually: Settings → (search) context_window_cap → set llm.context_window_cap to a positive integer. Set it to 0 to return to automatic. The budget bar in each AI panel reflects the active ceiling.

LLM inference is memory-bandwidth-limited, not compute-limited. VRAM capacity and bandwidth matter more than GPU core count.

AMD: The RX 7900 XTX has 24 GB VRAM but ROCm software support on Windows is inconsistent. On Linux, AMD gets the ROCm llama.cpp binary. On Windows, AMD gets the Vulkan build (HIP Radeon support is experimental).

NVIDIA on Linux: Bodega ships a Vulkan llama.cpp binary for Linux NVIDIA - not CUDA. CUDA binaries for Linux are not included in Bodega's managed installation.

On M-series Macs, the GPU and CPU share the same memory pool. Bodega detects Apple Silicon via process.platform === 'darwin' && process.arch === 'arm64'. The effective figure is min(sysctl iogpu.wired_limit_mb, total RAM x 0.75) at 64 GB or more and x 0.70 below that - Metal's recommendedMaxWorkingSetSize grants roughly 70-75% of RAM, not all of it.

Approximate memory bandwidth by chip (from Apple's published specs):

Higher bandwidth means faster token generation - bandwidth scales matter more than raw VRAM here.

When you install llama.cpp through Bodega on macOS, it installs the macos-arm64 binary, which uses Metal acceleration. No configuration required.

If you're using Ollama on Apple Silicon, Ollama handles MLX acceleration automatically. MLX runs roughly 20–30% faster than llama.cpp for inference on Apple Silicon.

Free memory note: macOS manages unified memory dynamically (compression, paging). The free memory reading Bodega sees can fluctuate. The probe uses os.freemem() with a 1 GB kernel reserve as a proxy.

When you install llama.cpp through Bodega (first-run onboarding or Settings → Models → My Models → llama.cpp engine → Install), Bodega picks the right binary for your system automatically:

Bodega reads the CUDA version directly from nvidia-smi output (not inferred from driver version numbers). If nvidia-smi times out or is absent, it falls back to driver major version, then to Vulkan or CPU. The install is never blocked by a missing nvidia-smi.

Choosing it yourself: the automatic pick is a default, not a lock. Settings -> Models -> My Models -> llama.cpp engine has a Build type dropdown listing the builds available for your platform. Your choice is saved to llamacpp.gpu_vendor and is what every later install and update uses. The panel also shows which build is actually on disk (llamacpp.binary_flavor), so you can tell whether your choice has taken effect: if it has not, the button reads Install <build> instead of Up to date. An install that predates that record says so rather than guessing.

Air-gap mode: the binary install endpoint returns 403. Use llamacpp.binary_path_override in Settings to point to a manually downloaded binary.

Each Bodega release pins a specific llama.cpp build. Installing the app does not move an existing installation to that build - updating the engine is a separate, explicit action.

Where: Settings → Models → My Models → llama.cpp engine. It shows two numbers side by side: the build you have installed, and the build this app version bundles. When they differ, an Update button appears.

Some models need a minimum llama.cpp build to load at all. When one does and your build is older, Bodega blocks the download and names the build you need - this panel is where you get it.

Bodega never updates the binary on its own. llama-server is the process your conversations run on; it is not swapped out in the background or at launch. Nothing changes until you click Update.

If the server is running: the update refuses and asks first. A running executable cannot be safely replaced - on Windows the file is locked, and on macOS/Linux the replace appears to succeed while the running process keeps the old binary. Choosing Stop and update stops llama-server (confirmed actually gone, not just asked to stop), then downloads. Your next model load starts the new build. If the stop cannot be confirmed, the update aborts and your binary is left untouched.

Air-gap mode: the update is a network download, so it is unavailable. Install the binary yourself and point llamacpp.binary_path_override at it.

Cloud Boost lets you configure a cloud provider (OpenAI, Anthropic, and others) as a secondary fallback. On low-end hardware, it activates automatically for tasks that need it.

To configure:

1. Go to Settings → Cloud Boost
2. Enable the toggle
3. Choose a cloud provider
4. Enter your API key
5. Set optional daily and monthly budget limits

Auto-escalation triggers - both conditions must be true for trigger B:

• (A) The local model's QEL verification score falls below 50 for two consecutive iterations in the same session, OR
• (B) Hardware tier is minimal (< 6 GB VRAM) or budget (6–10 GB VRAM) AND the current task is classified as reasoning-heavy

Simple conversational messages on minimal hardware do not trigger auto-escalation. The task has to be a planning or reasoning task.

You can also toggle Cloud Boost manually in the chat input area.

Cost tracking: gear icon → Usage Dashboard

Air-gap mode: Cloud Boost is fully blocked when air-gap is enabled (Settings → Privacy & Safety → Air-Gap).