How to choose an AI model

This is the question every model catalog eventually gets asked: which one do I actually use? It is also the one question a catalog cannot answer well, because the honest answer changes every few weeks. A new release tops a leaderboard, a price drops, a context window doubles, and last month's pick is no longer the obvious one. Naming a single "best" model here would be stale before this page finishes indexing.

What does not go stale is the method. The same handful of questions, asked in the same order, gets you to a sound choice whether the year is 2026 or several years from now. That method, not a ranking, is what this guide teaches.

Start with the job, not the leaderboard

A leaderboard rank is an average across many tasks. Your task is not an average, it is one specific thing: extracting fields from invoices, drafting marketing copy, writing and debugging code, answering questions about a support ticket, running as part of an agent that calls tools. A model that ranks well overall can still be a mediocre fit for your particular job, and a model that ranks lower overall can be the better pick for it. Start from the job and work outward, not the other way around.

The table below is the framework: eight questions, asked in roughly this order, that narrow a huge field down to a short list worth actually testing.

How to read, and distrust, benchmarks and leaderboards

Once the framework has narrowed the field, a benchmark score or leaderboard rank is a reasonable tiebreaker. It is a poor starting point, and it is worth understanding why before you lean on one.

Static benchmarks test a model against a fixed set of questions with known correct answers, then report an accuracy-style score. The problem researchers themselves have pointed to is fragmentation and narrowness: the Stanford CRFM team that built HELM found that before their effort, models on average had been evaluated on just 17.9% of a common core of scenarios, "with some prominent models not sharing a single scenario in common."¹ Two models can each report a strong number on their own preferred slice of tests and still be nearly impossible to compare directly.

Human-preference leaderboards take a different approach: real people compare two anonymous model outputs side by side and vote for the better one, and the votes are turned into a ranking. Chatbot Arena is the best known example, built on "a pairwise comparison approach" with crowdsourced votes converted into a rating with established statistical methods.² This measures something real, which output people tend to prefer, but preference is not the same thing as correctness, and a style people like is not automatically a style that is right.

• A benchmark score measures performance on that benchmark's specific questions, not on yours. Coverage varies enormously between benchmarks, as the HELM findings above show.
• Public benchmark questions can leak into later training data, a problem generally called contamination: a high score can partly reflect a model having seen the answer before, not reasoning its way to it.
• A leaderboard rank is a snapshot. New models are released constantly, so the top of any list reshuffles on a timescale of weeks, not years.
• A preference vote rewards style, tone, and confidence as much as it rewards being right, since the voter usually cannot check the answer in the moment.

Open-weight vs. closed-weight

A closed-weight model is one you can only reach through a provider's API or app. You send a request and get a response; the weights themselves never leave the provider's servers, and you cannot inspect, modify, or run them yourself. An open-weight model publishes the weights, so anyone can download the file and run it on their own hardware or a server they control, the approach covered in the Running Local AI course and the local runtimes and GGUF quantization guides.

"Open weight" describes distribution, not the license terms attached to it, and the two are easy to conflate. Hugging Face's own license list runs from fully permissive terms like apache-2.0 and mit through a long set of custom community licenses built for specific model families, several of which carry usage conditions, such as restrictions above a certain scale of deployment, that a plain permissive license would not.³ Downloadable is not the same as unrestricted. Read the actual license before you build on a specific open-weight model, especially for commercial use.

Neither column is the correct answer in general. Closed-weight models are often the strongest available at a given moment and require no infrastructure of your own. Open-weight models trade some of that ceiling for control: you can run them offline, keep data on your own device, and are not dependent on a provider's uptime or pricing changes. Which one matters more comes straight back to the privacy and cloud-vs-local rows in the framework table above.

Start small and cheap, scale up only if it fails

A durable heuristic, independent of whichever models exist right now: default to the smallest, cheapest model that could plausibly do the job, and only move up a tier when it demonstrably falls short. Reaching straight for the largest, most expensive option out of caution feels safe, but it usually just means paying more, waiting longer, and, for a local model, needing hardware you did not need to buy.

• Try the smallest reasonable candidate first, on real examples of your actual task, not a generic prompt.
• If it fails, look at how it fails. A vague or shallow answer suggests you need more capability. A wrong format or missed instruction sometimes just needs a better prompt, covered in the prompting guide.
• Move up one tier at a time; jumping straight to the largest available model skips useful information about where the real ceiling was.
• Re-check this decision periodically. A smaller or cheaper model released later can sometimes match what only a large one could do before.

Evaluate on your own task, not someone else's benchmark

The most reliable signal is also the most overlooked one: run the actual candidates on a small set of your own real examples and compare the results yourself. This does not require a research setup. Ten to twenty representative cases, the kind of input you will really send it, is enough to see which candidate is genuinely stronger for your job.

1. 1Collect 10 to 20 real or realistic examples of the task, including a few edge cases you know are tricky.
2. 2Write down what a correct or acceptable answer looks like for each one, before you run anything. This stops you from unconsciously grading generously.
3. 3Run the same examples, with the same prompt, through each candidate model.
4. 4Score each output against your own criteria, not against how confident or polished it sounds.
5. 5Re-run this same small set whenever you seriously consider switching models. It also doubles as an early warning if a provider updates a model and its behavior quietly shifts.

Before you even get to running examples, a model's own documentation is worth reading. On Hugging Face, a model card's Direct Use and Out-of-Scope Use sections exist specifically to describe what the model was built for and what it explicitly was not,⁴ which can rule a candidate in or out before you spend any evaluation time on it at all. That same documentation is also where license, training data, and known limitations live, the practical detail behind the framework table's license row.⁵

How to choose, in short

• Work the framework first. Task fit, capability, cloud vs. local, cost, privacy, context window, license, and speed narrow the field before any benchmark enters the picture.
• Treat benchmarks and leaderboards as one weak signal. Coverage is fragmented, scores can reflect contamination, and rank reshuffles on a timescale of weeks.
• Read the actual license on an open-weight model rather than assuming "open" means unrestricted.
• Default small and cheap, scale up only on demonstrated failure against your real task, not out of general caution.
• Build a small eval set on your own task. Ten to twenty real examples, scored against criteria you wrote down first, outperforms trusting any general ranking.