RAG and embeddings, explained

Ask a model about something that happened after its training data was collected, or about a document it has never seen, and it does one of two things: it says it doesn't know, or it guesses, fluently and with total confidence, and gets it wrong. Retrieval-augmented generation, RAG for short, is the standard fix. Instead of relying only on what the model memorized during training, a RAG system fetches the specific text the model needs to answer and inserts it into the prompt at the moment it's needed. The model still writes the answer, but it writes it from real material in front of it, not from memory alone. This guide covers the problem RAG solves, what an embedding and a vector search actually do underneath it, how the pipeline fits together end to end, and when RAG is the wrong tool for the job.

The problem RAG solves

A model's knowledge comes from two places: whatever got compressed into its weights during training, called its parametric memory, and whatever you type into the prompt. That split causes four specific problems once you move past small demos.

• Knowledge cutoff: training data has an end date. Anything that happened, was published, or changed after that date, the model has simply never seen.
• Private data: your internal docs, codebase, support tickets, or customer records were never part of any public training set, so the model has no memory of them at all, however good it is in general.
• Context limits: even a model with a large context window has a limit, and pasting your entire knowledge base into every single prompt does not scale in cost, latency, or accuracy.
• Hallucination: when a model doesn't actually know something, it tends to answer anyway rather than admit the gap, and a fluent wrong answer is harder to catch than an obviously broken one.

The original RAG paper, published by Patrick Lewis and coauthors at NeurIPS in 2020, frames this as a limit of parametric memory alone: a model's ability to precisely access and update what it knows is constrained by what got baked into its weights, and that causes measurable gaps on tasks that depend on specific facts.¹ Its proposed fix, pairing a pre-trained model with a retriever that pulls in relevant passages from an outside index, improves factual accuracy and, because the retrieved passages are visible text rather than compressed weights, lets you point to exactly what the answer came from and update the knowledge without retraining the model at all.¹ That combination, a generator plus a retriever pulling from external, inspectable text, is what every modern RAG system still does.

What an embedding actually is

An embedding is a vector, a fixed-length list of floating point numbers, produced by a model that has been trained to place text with similar meaning close together in that number space and text with different meaning farther apart.² Feed it a sentence, a paragraph, or a whole document, and you get back the same shape of vector every time, regardless of how long the input was. Two pieces of text that mean roughly the same thing, even if they use completely different words, land close together in that space. Two pieces of text that use similar words but mean different things do not.

"Close together" is measured with a distance function, and the most common choice for text embeddings is cosine similarity: the angle between two vectors, ignoring their length, scored from -1 (opposite) to 1 (identical direction).² OpenAI's own embeddings documentation recommends cosine similarity specifically, and notes that because its embeddings are already normalized to length 1, cosine similarity reduces to a plain dot product, so it produces the same ranking as Euclidean distance while being cheaper to compute.² A model like text-embedding-3-small produces a 1536-number vector for any input; a larger model like text-embedding-3-large produces 3072 numbers, and some APIs let you shorten that vector deliberately, trading a little accuracy for a smaller footprint.²

Vector search: finding the nearest neighbors

Once every chunk of your data has been turned into a vector, answering a question means embedding the question the same way, then finding which stored vectors sit closest to it. That's nearest-neighbor search. At a small scale, brute force works fine: compare the query vector against every stored vector and sort by distance. Past a few hundred thousand vectors, an exact scan gets too slow to run on every query, so real systems trade a small amount of accuracy for a large speed gain using an approximate nearest neighbor (ANN) index instead.

pgvector, the vector-search extension for Postgres, is a concrete example of what that looks like in practice. It supports several distance operators, including L2 (Euclidean), cosine distance, and inner product, and two ANN index types with a real trade-off between them: HNSW, a graph structure with strong query performance but slower, more memory-hungry builds, and IVFFlat, which clusters vectors into lists for a faster build at the cost of lower recall.³ Whichever database or library you use, the shape of the choice is the same: build time and memory against query speed and how often the index finds the true best matches instead of near misses.

The RAG pipeline

Put the two pieces together, embeddings and vector search, and the pipeline that connects your raw documents to a model's answer runs through five stages.

• Chunk: break each source document into smaller pieces, a paragraph or a section rather than a whole file, so each piece is small enough to embed meaningfully and specific enough to retrieve precisely.
• Embed: run every chunk through an embedding model, producing one vector per chunk that stands in for its meaning.
• Store: save each vector next to the original chunk text in a vector database or a vector-capable extension of a regular database, so a later search can return the readable text, not just the numbers.
• Retrieve: when a question comes in, embed it with the same model, then run a nearest-neighbor search to pull back the handful of chunks closest to it in meaning.
• Augment: insert those retrieved chunks into the prompt, ahead of the actual question, so the model generates its answer with that text in view instead of from memory alone. This is the step that gives retrieval-augmented generation its name.¹

RAG vs long context vs fine-tuning

RAG is not the only way to get outside information or new behavior into a model, and the three common approaches solve different problems. They are not mutually exclusive: many real systems use a large context window for a small set of documents that fit comfortably, and RAG for everything else.

Common pitfalls

A RAG pipeline is simple to sketch and easy to get subtly wrong. Most failures trace back to one of four points in the chain.

• Bad chunking: chunks that are too large drag in irrelevant text alongside the useful part, diluting what the embedding actually captures. Chunks that are too small lose the surrounding context a fact needs to make sense on its own.
• A stale index: source documents change, but if nobody re-chunks and re-embeds them, the stored vectors keep pointing at old text. Retrieval then returns outdated information with the same confidence as current information, and nothing in the pipeline flags the difference.
• Weak retrieval: a mediocre embedding model, or a query phrased very differently from how the source text is written, can surface chunks that are topically related but do not actually answer the question. The model then writes a fluent, confident answer built on the wrong material.
• Over-stuffing the context: cramming in as many retrieved chunks as fit does not reliably improve an answer. More text means more chances for irrelevant material to bury the one chunk that matters, and every added chunk costs latency and money whether or not it helps.

Honest limits

RAG doesn't make a model smarter, it makes it better informed, and only as well informed as the retrieval step managed to be on that particular question. The generation step can still misread a retrieved passage, quote it slightly wrong, or blend it with something from its own training memory without saying so. None of that is a reason to skip RAG for data a model genuinely cannot know on its own. It's a reason to treat retrieval quality, not the model's fluency, as the thing to check first when a RAG-backed answer turns out wrong.