Section 4

Using local AI well

Put it to work, keep it private, and know its limits.

6 lessons25-question quiz

4.1

Local models in your editor and tools

6 min read

The Building with AI course covered calling a model programmatically: sending it a request in code instead of typing into a chat box. That same idea works with a model running on your own machine, and it is one of the more useful things about running local AI at all.

Most local model runtimes expose a request-response API right on your machine, and they commonly speak the same request format as the popular cloud APIs. That means a tool built to call a model, an editor, a coding agent, or a script, usually does not need special local support at all. It just needs a setting for where to send the request.

A typical local-model config, in place of a cloud provider's

base_url = "http://localhost:PORT/v1"
api_key  = "not-checked"
model    = "your-local-model-name"

Why this matters

Bring your own model: the same editor or agent works with whatever you have running locally, in place of a cloud subscription.
Vendor neutral: the tool sends the same shape of request either way. It does not need to know or care whether the model behind that address is local or cloud.
No account to manage: the api_key field is usually a placeholder. Local servers typically do not check it, since there is no cloud account behind the request.

Key idea

Any tool built to call a model over an API can usually be pointed at a local one instead of a cloud one, just by changing the address it talks to. That is the whole trick.

For the fuller idea behind calling a model instead of typing into a chat box, see From chat to calling a model in the Building with AI course.

Key terms

Base URL: The address a tool sends its model requests to.
More
Point it at your local runtime instead of a cloud provider, and a tool built to call a model over an API will use your local model without any other change.

4.2

Air-gapped and fully private

6 min read

Earlier in this course, running a model locally already keeps your prompts off someone else's servers. Air-gapped goes a step further: you cut the network connection entirely, unplugging ethernet, turning off Wi-Fi, or working behind a firewall that blocks all outbound traffic. Nothing can reach the internet from that machine while you work, on purpose.

What genuinely stays on the machine

The model weights already sitting on disk before you disconnect.
Everything you type: prompts, follow-up questions, instructions.
Anything you paste or load in: files, code, documents.
The model's output and your full conversation history.

None of it has anywhere to go. With no network path out, there is nothing on the other end to receive it.

The trade-offs of cutting the cord

No updates while disconnected: the runtime, its libraries, and any installed models are frozen at whatever version was on the machine when you cut the connection.
No new models on demand: getting a different or larger model means reconnecting, downloading, and disconnecting again.
No network-dependent tools: a web search plugin, a live weather lookup, anything that calls out, simply fails or has to be turned off. If it worked, the machine would not really be air-gapped anymore.

An air-gapped machine is only as private as its other network paths are actually closed. Check for background update checks, telemetry, or any add-on tool with its own outbound connection, not just the model itself.

Key idea

Air-gapped is the strongest privacy posture available: cut the network and nothing can leave. In exchange, you give up live updates, new downloads, and any tool that depends on being online.

Key terms

Air-gapped: A machine with no network connection at all, on purpose, so nothing on it can reach the internet or be reached from it.

4.3

Where local models fall short

6 min read

Local models earn their place on privacy, cost, and control. None of that makes them equal to the largest cloud models on every task, and being honest about where the gap still shows up is more useful than pretending it is not there.

	Frontier cloud models	Smaller local models
Hardest reasoning and coding	Generally still ahead	Can miss steps or make more mistakes on hard problems
Context window	Often larger	Often smaller, at the sizes that fit consumer hardware
Speed on modest hardware	Fast, runs on server-grade hardware	Can be noticeably slower, especially without a strong GPU
Cost per use	Ongoing, paid per use on most plans	None once downloaded, though hardware and electricity are real costs

Where the gap still shows up

What this means in practice

For your hardest problems, a frontier cloud model may still get closer to a correct answer.
For everyday tasks, drafting, summarizing, common coding patterns, a well-chosen local model can be genuinely close.
The gap narrows over time as open-weight models improve, but it has not closed.

Key idea

Be honest about the trade: local buys privacy, cost, and control. It does not automatically buy the very best available quality or speed. Match the model to the task, and reach for cloud when a task truly needs the ceiling.

4.4

Choosing which model to run

6 min read

An open-weight model is one whose trained weights are published for anyone to download and run themselves, instead of being reachable only through a company's paid API. Most open-weight models are published on a model hub, most commonly Hugging Face, where you can browse them, read their license, and see notes on how they were trained and evaluated.

Check	Why it matters
Size vs. your hardware	A model that does not fit your RAM or VRAM will not run well, or at all. Match parameter count and quantization level to what you actually have, covered earlier in this course.
License	"Open-weight" is about access, not permission. Read the license before using a model for anything beyond personal use. Some restrict commercial use or redistribution, or require attribution.
Benchmark claims	Leaderboard scores can be optimized for the test, measured differently between labs, or out of date by the time you read them. Treat a benchmark win as a hint, not a guarantee.

Three things to check before you run a model

A quick way to think about it

Start from your hardware limit, not the newest name on a leaderboard.
Skim the license before you commit, especially for anything beyond personal use.
Test it on a task you actually care about. A model's real usefulness to you is what it does on your own work, not its rank on someone else's benchmark chart.

Key idea

An open-weight model earns a place on your machine once it fits your hardware, its license allows what you plan to do, and it holds up on your own tasks, not just on a benchmark chart.

Key terms

Open-weight model: A model whose trained weights are published for anyone to download and run themselves, instead of being reachable only through a company's API.
License (model): The terms attached to a model's weights, covering what you are allowed to do with them, including whether commercial use is allowed.

4.5

Keeping current

5 min read

New open-weight models show up often, sometimes more than one a week, from large labs and smaller independent teams alike. That pace is a big part of why local AI has become genuinely usable on ordinary hardware. It also means you could spend all your time re-downloading and re-testing instead of getting anything done.

A sustainable habit

Pick a small number of places to check occasionally: your runtime's own model library, a leaderboard you trust, or a community that already filters signal from noise.
Check on a schedule that suits you, weekly or monthly, rather than reacting to every announcement.
Only switch models when a new one offers a real improvement for what you actually do, not simply because it is new.
When you do switch, apply the same checks from the last lesson: size, license, and your own test, before trusting it.

Key idea

Keep an eye on the field without chasing it. A model that fits your hardware, respects its license, and does your actual work well is a better daily driver than whatever is newest.

The pace of new releases is a good reason to revisit your choice occasionally, not a reason to rebuild your setup every week.

4.6

Putting it together

6 min read

This course covered why local AI is worth running, how to size a model to your hardware, how to get one actually running, and how to use it well. Put together, it comes down to four steps.

1ChooseAn open-weight model that fits your plans, whose license allows what you intend, tested against your own task rather than a leaderboard alone

2SizeMatch parameter count and quantization level to your RAM or VRAM, so it actually loads and runs at a usable speed

3RunDownload it through a local runtime and get it answering in a terminal or chat window

4UsePoint an editor, agent, or script at its local API instead of a cloud one, or keep it fully air-gapped for the most sensitive work

From picking a model to using it well

Two guides that walk through it hands-on

For the "Run" step done end to end, the how-to Getting a local model running with Ollama walks through installing a runtime, pulling a model, and chatting with it offline. For the "Size" step done right, the guide Which GGUF quantization level should you pick? covers exactly what each file size costs you in memory and quality.

Key idea

Choosing well, sizing correctly, running it, and using it honestly. Four steps, and you have a private, capable, free model doing real work on your own machine.

Section 4 quiz

25 questions. Pass at 75% to master this section. Retakes are unlimited, and the quiz is where the learning sticks.

Section 4 quiz · Using local AI wellQuestion 1 of 25

What do most local model runtimes expose that lets other tools connect to them?

Back to Running Local AI