Your hardware and what it can run

Every model, before it can answer anything, has to be loaded into memory. Your computer has more than one kind of memory, and which kind a model lands in changes how fast it runs. The two that matter for running AI locally are RAM (system memory) and VRAM (video memory, on a graphics card).

VRAM is memory built into a GPU, sitting right next to the chip that does the math. It is very fast, which is exactly what the repeated, heavy number-crunching of running a model needs. RAM is the general-purpose memory your whole computer uses for everything: your browser, your operating system, and, if there is no room in VRAM, your model too. RAM is typically slower to read from for this kind of workload, and it is farther from the processor doing the work.

The rule that governs everything else

A model must fit in whichever memory it runs on. If you run it on a GPU, its weights need to fit in that GPU's VRAM. If you run it on the CPU instead, its weights need to fit in system RAM. Fit is not a nice-to-have; a model that does not fit either will not load at all, or will run so slowly it is not usable.

You already know a model's size is measured in parameters, such as 8B for eight billion. What actually determines whether it fits on your machine is how much memory those parameters take up once loaded, and that depends on more than the parameter count alone.

The rough formula

Each parameter is stored using some number of bits. A freshly trained model often stores each one at 16 bits. A quantized model, covered in depth in the GGUF quantization guide, stores each one at fewer bits, commonly 8, 6, 5, 4, 3, or 2, trading some precision for a much smaller file. The approximate memory for the weights alone is:

memory (bytes) ≈ parameters × bits-per-weight ÷ 8

That covers only the weights. On top of that, running a model needs some working memory for the current conversation, called the KV cache, plus overhead from the software actually running the model. The next two lessons unpack both of those. For now, treat the formula above as the floor, not the whole picture.

Where a model actually runs, not just where it is stored, changes how fast it feels to use. There are roughly three setups you will run into.

The three setups

• GPU inference: the model's weights fit in VRAM and a dedicated graphics card does the work. This is generally the fastest option when it fits, because VRAM is fast and the GPU is built for exactly this kind of math.
• CPU inference: the model runs using the processor and system RAM, either because there is no GPU or the model does not fit in VRAM. It works on almost any machine, but token generation is typically noticeably slower than on a capable GPU.
• Unified memory: some machines, most notably Apple Silicon Macs, share a single pool of fast memory between the CPU and GPU instead of giving the GPU its own separate VRAM. That shared pool can be large, which lets these machines run bigger models than their GPU alone would suggest, though not always as fast as a discrete GPU with the equivalent amount of dedicated VRAM.

You now have the pieces: the formula from lesson two, and the idea from the GGUF guide that weights are only the floor. Here is how to put them together before you download anything.

1. 1Pick a model and a quant level, then estimate the weight size: parameters times bits-per-weight, divided by 8. An 8-billion-parameter model at roughly 4.9 bits/weight (a common 4-bit quant) works out to about 8,000,000,000 × 4.89 ÷ 8, which is approximately 4.9 GB.
2. 2Add room for the KV cache, the working memory that grows with your context window (how much conversation and text the model is holding at once) and the number of layers and attention heads in the model. A short chat adds relatively little; a long conversation or a large pasted document can add real gigabytes.
3. 3Add room for runtime overhead: the software actually running the model, plus buffers, plus anything else sharing the same GPU, such as your display. This is not a number you can shave to zero.
4. 4Compare the total, not just the weight size, against the memory you actually have available. As a starting rule of thumb, leave roughly 10 to 20% of your VRAM free above the weight size for short conversations, and budget more if you plan to run a large context window regularly.
5. 5If the total is close to your limit, drop to a smaller model or a lower quant level, or plan to run a shorter context window, and re-check the math.

Worked example: you have a GPU with 8 GB of VRAM. An 8B model at a 4-bit quant needs roughly 5 GB for weights. That leaves about 3 GB for the KV cache and runtime overhead, which is comfortable for a short-to-medium conversation but will get tight if you paste in a very large document or run a long back-and-forth session.

Put everything in this section together and you get a rough starting table. Treat it as a first guess, not a guarantee. Always check the exact file size on the model's download page, and use the math from the last lesson for anything close to your limit.

This table mirrors, in general terms, the hardware guidance in the GGUF quantization guide, which has the fuller version with exact measured sizes for one model family. It is a starting point for exploration, not a hard rule: model architectures differ, mixture-of-experts models follow different math than dense models, and your actual usage (long documents, long conversations) shifts the numbers.