Run your first local model with Ollama

Ollama is the fastest way to get a real language model running entirely on your own machine, no cloud account and no data leaving your computer. This walks through installing it on macOS, Windows, and Linux, pulling a model sized to your hardware, chatting with it, proving to yourself that it is genuinely offline, and then wiring it into your editor over a local API.

1. Install Ollama

macOS and Linux install with the exact same one-line script. Windows uses a PowerShell equivalent. If you would rather click through an installer, download the .dmg or .exe directly instead.¹

curl -fsSL https://ollama.com/install.sh | sh

irm https://ollama.com/install.ps1 | iex

Once installed, Ollama runs quietly in the background and starts automatically on login, on macOS and Windows it registers as a login item, and on Linux it runs as a systemd service.² There is nothing else to start before the next step.

2. Pick a model for your hardware

Ollama runs fine on CPU alone, but it is faster with a supported GPU: Nvidia cards with compute capability 5.0 or newer and a recent driver, AMD Radeon cards through ROCm on Linux and Windows, and Apple Silicon Macs automatically through Metal.³ Without one of those, Ollama quietly falls back to your CPU. It still works, just slower.

As a rough rule, a model needs about as much free RAM or VRAM as its download size, plus headroom for your OS and its context window. This guide uses qwen2.5-coder:7b, a 7-billion-parameter coding model that downloads at roughly 4.7 GB,⁴ so 8 GB of RAM is a safe minimum and 16 GB gives you room to try bigger models later. Smaller and larger sizes of the same model are in the library, and Which GGUF quantization level should you pick? covers exactly what each size level costs you in memory and quality.

In practice, ollama run also downloads a model automatically the first time you use it, so pulling first is not strictly required, it just separates the download from the first chat.⁵

3. Pull the model

Download the model once. After this it lives on disk and starts instantly.

ollama pull qwen2.5-coder:7b

4. Run it and chat

Start a chat right in your terminal.⁵

ollama run qwen2.5-coder:7b

5. Confirm it is actually offline

This is the point of the whole exercise. Turn off Wi-Fi (or unplug ethernet), then send the same chat another prompt. It still answers, because the weights and the computation are entirely on your hardware.

One thing to watch for: some model names carry a :cloud tag, for example gemma4:cloud, and those run on Ollama cloud infrastructure instead of your machine, the exact same way any other cloud API does.⁶ Ollama is explicit that it only processes your prompts when you use a cloud-hosted model; anything run locally never leaves the machine.² Stick to a plain tag, no :cloud suffix, if the goal is staying fully offline. For a fuller look at when local is the right call and when it is not, see the AI Foundations lesson on using AI well.

6. Use it from your editor

Ollama also runs an OpenAI-compatible API on http://localhost:11434/v1. Anything that lets you set a custom base URL and API key, an editor, an agent, or an SDK, can point at your local model instead of a cloud one. The API key is not checked; it just needs to be present.⁷

curl http://localhost:11434/v1/models

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
reply = client.chat.completions.create(
    model="qwen2.5-coder:7b",
    messages=[{"role": "user", "content": "Say this is a test"}],
)
print(reply.choices[0].message.content)

Prefer a native client over the OpenAI-compatible shim? Ollama also ships official Python and JavaScript libraries that talk to its own API directly.⁵ Either route keeps every request, and every response, on your machine.

7. Manage your models

A handful of commands cover everything you need day to day.⁸

Troubleshooting

• Out of memory, or a model will not load: the model needs more RAM or VRAM than is free. Run ollama ps to see whether it loaded onto CPU, GPU, or a split,² close other memory-heavy apps, or switch to a smaller model or a lower quantization; see Which GGUF quantization level should you pick? for the exact memory and quality trade-off.
• Generation is slow: check ollama ps first. A 100% CPU load on a machine with a supported GPU usually means a driver problem, not a software one.² Confirm your GPU meets the driver and compute-capability minimums,³ and check the logs for GPU discovery errors.⁹ A smaller model or a lower quantization also generates faster on the same hardware.
• "model not found," or a pull/run fails: double-check the exact name and tag with ollama ls;⁸ most models need an explicit size tag like :7b, and a bare name defaults to :latest. Confirm the name exists in the model library, then re-run ollama pull.

You now have a model running entirely on your machine, reachable from your terminal and from your editor, with nothing going out over the network. From here, most of the work is picking the right model and the right size for what you have. Which GGUF quantization level should you pick? covers exactly that trade-off, and the free AI Foundations course covers the ideas behind local models, prompting, and agents if you want the fuller picture.