Embeddings and RAG: Memory Beyond the Chat Model

A RAG system holds more than the chat model: embedding weights, a reranker, and a vector index. This page breaks down that separate memory budget and how to size it.

By VRAM Fit editorial Published 2026-09-17 Updated 2026-09-17

A retrieval-augmented generation system is easy to size wrong because the chat model is only one of its memory occupants. The embedding model that turns text into vectors, an optional reranker that reorders results, the vector index that stores those vectors, and the temporary tensors of each query all carry their own budgets. Treating the chat model’s VRAM as the whole story leads to a system that fits the model but runs out of memory the moment it retrieves. This page separates those budgets and shows how a RAG pipeline’s memory is actually composed.

The parts of a RAG memory budget

A typical RAG pipeline has at least four distinct memory occupants. They do not all live in the same place, and they do not all scale with the same variable.

Component	What it holds	Typical location	Scales with
Embedding model weights	The model that encodes text into vectors	VRAM if served on GPU, else system RAM	Model size, quantization
Reranker model weights	A model that scores and reorders candidate passages	VRAM if on GPU, else system RAM	Model size, quantization
Vector index	The stored vectors plus the lookup structure	System RAM or disk; VRAM only for low latency	Number of vectors and dimensionality
Per-query temporary tensors	Activations and buffers for the current request	VRAM or RAM near the compute	Batch size and concurrency
Concurrent retrieval peak	All of the above, multiplied by live queries	Same as components above	Number of simultaneous requests

The crucial point is that the index is a stored structure, not a compute graph. It wants memory and fast access, not GPU throughput, which is why it usually sits in system RAM or on disk rather than VRAM.

The chat model is not always the main occupant

In a RAG system the chat model is often assumed to be the dominant memory user, but that assumption breaks as soon as the document set grows. The vector index scales with the corpus, and a large knowledge base can hold far more memory in its index than the chat model holds in weights. The embedding service is a model in its own right, loaded independently of the chat model, with its own weights and its own working memory.

The two budgets do not share capacity assumptions. The chat model’s VRAM estimate says nothing about how much room the index needs, and the index size says nothing about the model. Planning must account for both as separate line items. The VRAM calculator helps with the model side; the index side is a function of corpus size and embedding dimension, covered below.

Where the vector index belongs

The default home for a vector index is system RAM, with disk as the backing store for very large corpora. The index is queried by distance search over stored vectors, which is memory-bandwidth and latency sensitive but not floating-point heavy in the way model inference is. GPU VRAM is the scarce, expensive resource; spending it on a lookup structure is only justified when query latency is the binding constraint.

When the index does move to VRAM, it is usually for a latency-critical, high-throughput service where the saved milliseconds per query pay for the occupied memory. For most document-question-answering and agent-retrieval setups, system RAM is sufficient and leaves the GPU free for the models. The local hardware guide frames capacity decisions around how much memory is needed, which applies to RAM as much as to VRAM.

How the index grows with the corpus

The index footprint tracks the number of stored embeddings and their dimensionality, not the chat model or the context length. More documents mean more chunks, each producing one or more vectors; a higher embedding dimension means each vector is larger. The relationship is direct: grow the corpus, grow the index.

Document scale	Index footprint direction	What drives it
A few hundred documents	Small	Low vector count
Thousands of documents	Moderate, grows with count	More chunks, more vectors
Tens of thousands of documents	Large	Vector count dominates
Very large corpora	Largest tier; disk-backed common	Count and dimensionality both matter

The exact gigabytes are not stated here because they depend on the embedding dimension, the chunking strategy, and the index format. The direction is what matters for planning: the index rises with corpus size and embedding dimension, independently of the chat model’s requirements.

Embedding and generation on the same card

Putting the embedding model and the chat model on one GPU is tempting because it is one machine. The risk is that their memory adds. If the embedding model’s weights, the chat model’s weights, the KV cache, and the reranker all load together, the total is the sum of each, measured against a single capacity. A card that comfortably holds the chat model alone can overflow once the embedding service and reranker join it.

The failure mode is subtle because it depends on timing. If embedding and generation run at different moments, the peaks may not overlap and the card survives. If they run together — as a live RAG service does — the budgets collide. Options that avoid the clash: serve the embedding model on CPU or a separate machine, load the reranker only when needed, or keep the index out of VRAM so the GPU is reserved for the models.

Estimating the two budgets separately

The clean way to size a RAG system is to build two independent estimates and add them only at the points where they truly overlap in time.

Model budget. Estimate the chat model weights from the parameter count and quantization using the standard weight formula, add the KV cache for the expected context, and add engine headroom. The quantization comparison and the KV cache size page support this step.
Retrieval budget. Estimate the embedding model and reranker weights separately, and estimate the index size from corpus scale and embedding dimension. Keep the index in system RAM or on disk unless latency forces VRAM.
Overlap check. Decide whether the model and retrieval run at the same time. If they do, their VRAM or RAM sums must fit the shared pool. If they are staged, plan each peak separately.

This separation is the difference between a system that fits and a system that loads the model but falls over on the first retrieval.

Concurrency is the hidden peak

A single retrieval query uses one set of temporary tensors and touches the index once. A service answering several users at once multiplies that: each live query holds its own temporary buffers, and the index may be queried in parallel. The peak memory is therefore higher than a single query implies, and it rises with concurrent load.

Plan for the busiest moment, not the average, because the memory is allocated per live request. The batch size and throughput page covers how concurrency multiplies memory in the inference path, and the same logic applies to the retrieval path. A RAG system’s worst case is many simultaneous users asking questions at once, with the embedding service, reranker, chat model, and index all active together.

Measuring the real footprint

Because the index and embedding service are independent of the chat model, the only reliable total comes from measuring the running system. Watch both VRAM and system RAM, since the index may live in RAM while the models live in VRAM, and the two limits are independent. The measure VRAM usage page describes how to read actual memory across systems, and the Ollama VRAM requirements page shows what common models consume if the embedding or chat model runs through Ollama.

What this page does not tell you

This page does not give specific gigabyte figures for any index size, embedding model, or reranker, because those depend on the embedding dimension, the chunking strategy, the index format, and the model chosen, none of which are fixed. It does not recommend specific embedding or reranker models, and it does not benchmark query latency for any configuration. Disk-versus-RAM tradeoffs for very large corpora, distributed vector stores, and multi-GPU splits for the model side are covered elsewhere. The claim that the index often dominates is a directional observation about structure, not a measured share of memory on a specific deployment.

Asked before trying it

Is the chat model the biggest memory user in a RAG system?

Not always. The chat model is one budget, but the embedding model, an optional reranker, and the vector index are a separate budget that does not share the chat model's capacity. On many RAG setups the index and embedding service dominate, especially when the document set is large.

Where should the vector index live?

Usually in system RAM or on disk. Only when low query latency is required does it make sense to keep the index in VRAM. The index is mostly a lookup structure, not compute, so it does not need the GPU the way the models do.

Do the embedding model and the chat model share VRAM?

They can be on the same card, but then their weights and working memory add up against one capacity. If both are loaded at once, the total is the sum, not the larger of the two. Splitting them across machines or loading one at a time avoids the clash.

How does document count affect the index size?

The index grows with the number of stored vectors and their dimensionality, not with the chat model at all. More documents means more embeddings stored, so the index footprint rises with corpus size and embedding dimension.

What is the peak memory event to plan for?

Concurrent retrieval. Each in-flight query pulls temporary tensors and may touch the index in parallel, so the peak is higher than a single query would suggest. Plan for the busiest moment, not the average.

Can the reranker be skipped to save memory?

It can be omitted, and many pipelines do skip it. A reranker improves relevance by scoring candidate passages but adds a model that must be loaded. The trade is relevance versus a second model's memory and compute budget.