Section 5

Build, evaluate, and ship

Put it together, check it works, and ship it responsibly.

3 lessons25-question quiz

5.1

Building your first small AI feature

7 min read

A single prompt typed into a chat window is not a feature. A feature is that same idea wired into something that runs on its own: paste in a support ticket and get a draft reply, upload a photo and get a caption, paste a log file and get a summary of what broke. The gap between the two is smaller than it looks, and you close it by working in small, testable increments.

The build loop

1Pick one jobOne clear task, not the whole app

2Wire it upOne input in, one output out

3Try itReal inputs, not just the one you imagined

4Check the outputDoes it do the job? Where does it break?

5Improve one thingChange a single variable, then try it again

Five steps, repeated

Start with step one: pick one job. Not "help with customer support," but "draft a first-pass reply to a single ticket." A narrow job is easier to wire up, easier to test, and easier to judge, since you can tell right away whether it worked.

Wiring it up

Wiring it up means connecting the model to that one job: one input goes in, one output comes out. At this stage, resist the urge to add extra steps, extra tools, or a nicer interface. A plain function that takes text and returns text is enough to learn whether the idea works at all.

A minimal wiring, one job only

function draftReply(ticketText) {
  const prompt = "Draft a short, polite reply to this support ticket: " + ticketText;
  return callModel(prompt);
}

Improving one thing at a time

Once you can try it, you will find things to fix: the tone is off, it misses a detail, it runs too long. Change a single variable, then try it again. Change two things at once and a fix you find will not tell you which change actually did the work.

The wording of the prompt itself
Which model answers it
How much surrounding context you give it
The format you ask for back

For the actual wording of that prompt, the prompt patterns lesson from the Prompting course gives you a starting structure instead of a blank page.

Key idea

Build one job at a time: wire it up small, try it on real input, and change a single variable per round so you know what actually worked.

Key terms

Iteration: One pass through the build loop: wire it up, try it, check it, improve one thing.

5.2

Evaluating: is it actually good?

7 min read

After a few tries, a small AI feature starts to feel good. That feeling is not evidence. "Feels good" is a judgment made on whatever inputs you happened to try, in whatever mood you happened to be in, and it evaporates the moment you change the prompt or swap the model. You need something you can check instead of trusting a vibe.

Build a small test set

A test set for this purpose is a short list of real or realistic inputs, each paired with what a good output looks like. It does not need to be large. Five to ten cases that cover a typical input, a couple of edge cases, and one or two known-tricky examples will catch most problems.

What to test	How to check it
A typical, everyday input	Compare the output to what you already know a good answer looks like
An edge case: empty input, odd formatting, a very long input	Check that it fails gracefully instead of crashing or returning garbage
A known-tricky example that has tripped up the model before	Check it directly against the answer you already worked out by hand
A real example pulled from actual use, once you have any	Compare it to what really happened, not a guess

What to test, and how to check it

Running the check

Run every case in the test set through the feature
Mark each one pass or fail against what you expected
Change one thing: the prompt, the model, or a setting
Rerun the whole set, not just the case you were fixing
Compare the new pass count to the old one before you decide the change helped

This habit matters most exactly when you are tempted to skip it: right after a change, when you are eager to believe it worked.

Key idea

Replace "it feels better" with a small test set you can rerun. A change only counts as an improvement if it raises the pass count without breaking cases that used to work.

Key terms

Test set: A short list of inputs paired with what a good output looks like, used to check quality objectively.

5.3

Shipping responsibly

7 min read

Getting a feature working is not the same as being ready to ship it. Once real people, real money, or real data are on the other end, four ordinary concerns become important: cost, privacy, monitoring, and keeping a human in the loop.

Four things to handle before you ship

Concern	What to do about it
Cost	Cap how much each request or user can spend, cache repeated calls, and use a cheaper model where accuracy allows it
Privacy	Know whether you are calling a cloud model or a local one, and treat data accordingly
Monitoring	Log failures and watch for a rising error rate, not just the cases that happen to work
Human in the loop	Keep a person reviewing or approving anything that touches money, real data, or people

Four things to handle before you ship

Privacy, recapped

AI Foundations covered this already: a cloud model sends your data to a provider's servers over the internet, and a local model keeps it on the machine it runs on. Shipping a feature means deciding that on purpose, not by default. If the feature touches anything sensitive, see AI Foundations for the full comparison, and this how-to for what running a model locally actually looks like.

Before you flip it on

Read through the pre-launch checklist: it covers the areas beyond the model itself where real apps break.
Decide in advance what happens when the model is wrong, not after a user hits it.
Set a cost ceiling before launch, not after the first surprising bill.
Write down which parts run without review and which parts wait for a human.

None of this is optional once real users are involved. It is the difference between a feature that works in a demo and one that keeps working.

Key idea

Shipping responsibly means controlling cost, being deliberate about cloud versus local, watching for failures after launch, and keeping a person in the loop wherever a mistake would actually cost something.

Key terms

Human in the loop: A person who reviews or approves an AI system's output before it acts on money, real data, or people.

Section 5 quiz

25 questions. Pass at 75% to master this section. Retakes are unlimited, and the quiz is where the learning sticks.

Section 5 quiz · Build, evaluate, and shipQuestion 1 of 25

What is the first step of the build loop described in the lesson?

Back to Building with AI