From chat to calling a model

A chat box is a fine way to get one answer to one question from a person sitting at a keyboard. It is a bad way to run the same task a thousand times, connect a model to your own code, or make something happen without a person present to type and click. This lesson covers where that line sits, and why crossing it means calling a model programmatically instead of typing into a browser tab.

What the chat box is actually good at

The chat box interface, the kind you get with a consumer AI product, is built around a single loop: a person types, waits, and reads. That loop suits exploration, one-off writing, quick research, and anything where a human is available to supervise every step and correct course mid-conversation. For a large share of everyday AI use, that is exactly the right tool, and nothing in this course argues otherwise.

Three things a chat box cannot do

• Run without a person present. A chat box needs someone to open it, type, and read the reply. It cannot run at 3 a.m. on a schedule or fire the moment a new support ticket arrives.
• Repeat exactly. Every time you paste a prompt into a chat box, you retype or re-paste it, and small wording drift creeps in. A process that must behave the same way every time needs a fixed, callable input, not a retyped one.
• Live inside other software. A chat box is its own separate application. If you want a model's ability to summarize, classify, or draft text to show up as a feature inside your own app, spreadsheet, or script, the chat box cannot be embedded there.

When the chat box is still the right call

Not every task should become an API call. If you are exploring an idea, need a single answer once, or want a human in the loop reviewing every step, the chat box is simpler and there is no reason to write code for it. Reach for calling the model programmatically when a task needs to run repeatedly, run unattended, or become part of something else you are building.

Underneath every chat product sits an API: a way for a program to send a model some text and get text back. Calling it directly means your code takes over the job the chat box's interface was doing: build the request, send it, and read the response.

The messages array

Almost every provider's API takes the same basic shape: a list of messages, each with a role and content. The role says who "said" that piece of text, and the model reads the whole list, in order, before answering. Three roles cover nearly everything you will write:

• system: standing instructions for the whole conversation, set once (the next lesson covers this in depth)
• user: what the person, or your code standing in for the person, is asking
• assistant: the model's own previous replies, included so a multi-turn conversation keeps its history

A minimal request

Strip a request down to its essentials and it looks something like this. The exact field names differ by provider, but the shape, a model name, an array of role-tagged messages, and a limit on how much to generate, shows up almost everywhere.

POST /v1/chat/completions
{
  "model": "model-name",
  "messages": [
    { "role": "system", "content": "You are a concise assistant." },
    { "role": "user", "content": "Summarize this text in two sentences: ..." }
  ],
  "max_tokens": 300
}

The response comes back in a matching shape: usually an assistant message holding the generated text, plus some accounting data such as how many tokens were used. Your code reads that response the way you would read the model's reply in a chat window, except now a program is doing the reading, and can do anything with the text: save it, pass it to another function, or show it inside your own interface.

API keys and bring-your-own-key

Calling a model over an API almost always requires an API key: a secret string that identifies your account and lets the provider bill your usage and rate-limit your traffic. You get one from the provider, keep it out of any code you share or commit, and send it with every request, usually in a header. Being bring-your-own-key (BYOK) means the software you are using, whether that is your own script or someone else's tool, does not include a subscription baked in. You supply the key for whichever provider you want, and you pay that provider directly for what you use.

Every message you send is a user message, unless you use the other lever available: the system role. A system prompt sets behavior for the whole conversation, once, instead of repeating instructions in every message you send.

What a system prompt actually does

Put text in the system role and the model treats it as standing context: a persona to hold, a set of rules to follow, a tone to keep, a boundary not to cross. It applies to every user message that follows, without you retyping it. Where a chat box's plain conversation only shows user and assistant turns, code that calls a model directly gets a third lever most chat products hide from you entirely.

Persona, standing instructions, and guardrails

• Persona: "You are a patient tutor explaining concepts to a beginner" shapes vocabulary and tone across the whole session, not just one reply.
• Standing instructions: "Always answer in valid JSON matching this structure" or "Never suggest code that requires an internet connection" apply to message one and message fifty alike.
• Guardrails: "Do not discuss unrelated topics" or "If you are unsure, say so instead of guessing" set boundaries the model is meant to hold for the entire conversation.

System prompt vs per-message prompt

A per-message (user) prompt is the specific ask for that one turn: "summarize this," "write a function that does X." A system prompt is the constant backdrop those asks happen against. The practical test: if an instruction should apply to every message in the conversation, it belongs in the system prompt. If it is specific to this one request, it belongs in the user message.

A system prompt is not a lock. Depending on the provider and the model, a conversation can sometimes coax a model away from its system instructions. Treat a system prompt as a strong default, not an airtight guarantee, especially for anything security sensitive.

AI Foundations introduced tokens as the unit a model reads and writes. Calling a model programmatically is where tokens stop being trivia and start being something you manage directly: they set your context limit, they set your bill, and they are the reason a long response can feel slow to arrive unless you handle it right.

A quick recall: what a token is

A token is a chunk of text, often part of a word, that the model processes as one unit. Roughly 100 tokens is about 75 English words. See What AI and models are in AI Foundations for the full explanation.

Cost scales with tokens, in and out

Cloud providers bill per token, and almost always at two separate rates: one for input tokens (everything you send: the system prompt, the conversation history, the user message) and a higher one for output tokens (what the model generates). That means a long system prompt or a long pasted document adds cost on every single call, even before the model writes a word back, and a long generated answer costs more than a short one.

Streaming: responsiveness without waiting for the whole answer

By default, a request can wait until the model finishes generating the entire response before sending anything back. For a long answer, that can mean a real pause with nothing on screen. Streaming asks the provider to send the response back piece by piece, as it is generated, the same way a chat product shows words appearing as the model "types." Streaming does not make the model faster or cheaper; it changes when your code receives the text, which matters a lot for anything a person is watching in real time.

Context-window limits, revisited

The context window from AI Foundations is also a hard ceiling on a request: your system prompt, conversation history, and user message all have to fit inside it, together with room left over for the response. Send too much and the call either gets truncated or rejected outright, depending on the provider. A long-running conversation or a large pasted document is the most common way to hit this limit in practice.