Getting a model running

A model file on its own does nothing. Something has to load it into memory, run the actual math, and hand you a way to send it text and get text back. That something is a runtime. Three names come up constantly, and they are not really three competing choices so much as three layers of the same stack.

The three you will meet

llama.cpp is where most of this ecosystem starts. It is an open-source inference engine written to run GGUF models efficiently on ordinary hardware, CPU included, without needing the heavier machine-learning frameworks a model was originally trained with. Both Ollama and LM Studio lean on the same kind of engine under their own interface.

Ollama wraps that engine in a small CLI and a background service. You type one command to pull a model and another to run it, and it handles finding a GPU, loading the model, and serving a local API without any extra configuration. It is the fastest path from nothing installed to a model answering questions, which is why the Ollama how-to in this Academy uses it as the walkthrough.

LM Studio covers the same ground with a graphical desktop app instead of a terminal: a searchable model catalog, a download manager, a chat window, and a settings panel for things like context length and GPU offload, all without typing a command.

When you pick a model to download, you are not just picking a model, you are picking a quantization level of that model: how much each weight has been compressed to save memory. A quick recap, since the full breakdown lives in its own guide.

A model trained at full precision stores each weight as a 16-bit number. Quantization shrinks that down to fewer bits, commonly 8, 6, 5, 4, 3, or 2, which shrinks both the download and the memory the model needs to run. Fewer bits costs some accuracy, but that loss is gradual, not a cliff, at least down to a point.

• Q8_0 / Q6_K: close to full precision, and the largest of the practical sizes.
• Q5_K_M: a notch of headroom above the everyday default.
• Q4_K_M: the default most people should reach for first. Under 1% quality cost for a real cut in memory.
• Q3_K_M and below: only when a smaller quant is the difference between the model fitting in memory or not.

However you get there, getting a chatbot running locally follows the same basic flow every time. The names of the commands differ between tools, but the shape does not.

With Ollama, each of those steps is a single command. Pulling downloads a model once and stores it on disk; running loads it into memory and drops you into a chat prompt right in the terminal.

ollama pull qwen2.5-coder:7b
ollama run qwen2.5-coder:7b

The first line downloads the model, a one-time cost. The second loads it and opens a chat prompt: type a question, get a streamed reply, type /bye to leave. Run the same command again later and there is no download, it starts straight from the copy already on disk.

Chatting in a terminal or a chat window is only half of what a local runtime gives you. Most of them also run a local API, a network address on your own machine that other programs can send requests to, in the same request-and-response shape covered in the Building with AI course's lesson on calling a model. The difference is where that address points: instead of a cloud provider's servers, it points at your own hardware.

Ollama, for example, serves an OpenAI-compatible API at http://localhost:11434/v1 as soon as it is running. "OpenAI-compatible" means it accepts requests in the same shape as OpenAI's API, so any tool, editor, or SDK that lets you set a custom base URL can point at it instead of a cloud endpoint, with no other code changes.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
reply = client.chat.completions.create(
    model="qwen2.5-coder:7b",
    messages=[{"role": "user", "content": "Say this is a test"}],
)
print(reply.choices[0].message.content)

The api_key field still has to be present because the client library expects one, but a local server does not check it against anything. Nothing about this request leaves your machine: the model, the request, and the response all stay local.

• localhost means "this machine", so the request never touches the network.
• The base URL is the only thing that changes between a cloud setup and a local one; the rest of your code stays the same.
• Other runtimes expose the same idea on their own port, so the pattern carries over even if the exact address differs.

Once a model is running, two questions decide whether the experience is pleasant or painful: how much memory it needs, and how fast it replies. Both have straightforward explanations.

Context costs memory too

The weights are only part of what a running model holds in memory. As a conversation grows, the runtime keeps a running cache of everything in the context window so it does not have to reprocess the whole conversation on every reply. That cache grows with conversation length, so a long chat or a large pasted document uses noticeably more memory than a short question, on top of whatever the weights themselves need.

Speed is measured in tokens per second

How fast a model replies is usually measured in tokens per second: how many tokens it generates for each second of wall-clock time. That number depends mainly on two things: how big the model is, and how capable the hardware running it is. A smaller model on a fast GPU can be many times quicker than a larger model on the same hardware, or the same model on a CPU instead of a GPU.

• If a model that used to run fine suddenly loads onto CPU instead of GPU, another app is probably using the memory it needs.
• A model that barely fits in memory with a short chat can run out of room once the conversation, and its cache, grows.
• When speed matters more than a little extra quality, a smaller model or a lower quantization is usually the first thing to try, not more expensive hardware.