A multimodal model that accepts images looks like a language model with an extra input. Under the hood it is exactly that: a vision encoder turns the picture into a sequence of tokens, a projection layer maps those tokens into the language model’s space, and then the language model proceeds as usual. The interesting memory fact is that the image does not stay separate. It becomes tokens, those tokens join the context, and the image’s cost therefore lands on the same KV cache that text tokens use. This page shows where the VRAM goes and why the story ends at the KV cache, which is also the bridge back to the site’s text-model material.
The weight portion has two extra blocks
A plain language model’s weights are one block. A multimodal model adds at least two more:
- The vision encoder is a network, often a transformer or a convolutional backbone, that reads the image and produces patch representations. Its weights must be loaded to process any image.
- The projection layer maps the encoder’s output into the language model’s embedding space so the tokens can be fed in as if they were text. It is small relative to the encoder, but it is an added resident block.
On top of these sits the language model itself, with its usual weight cost. So the weight floor of a multimodal model is the language-model weights plus the encoder plus the projection. The encoder is typically far smaller than the language model, which is why the weight increase is modest compared with the jump from a text-only model to a much larger text model.
Images become visual tokens
The crucial step is what the encoder’s output becomes. The image is divided into patches, a grid of small regions. Each patch is turned into one visual token. The sequence of visual tokens is then prepended to the text tokens and fed to the language model together.
This is the moment the image stops being “an image” for memory purposes and becomes “a run of tokens in context.” From here, the visual tokens behave like text tokens: they occupy positions in the sequence, and the model keeps key and value vectors for each of them.
The number of visual tokens is not fixed. It depends on the image resolution and on how the encoder tiles the image into patches. A higher-resolution image yields more patches, hence more visual tokens. Because the patches come from a two-dimensional grid, raising the resolution on both sides increases the token count along both axes, so the visual token count grows with image area.
Resolution raises visual tokens, which raise the KV cache
Here is the chain that matters for memory:
- A higher-resolution image is split into more patches.
- More patches become more visual tokens.
- More visual tokens enter the language model’s context.
- Each token in context gets a key and value vector in the KV cache.
- The KV cache therefore grows with the number of visual tokens, exactly as it grows with text tokens.
So the image’s memory cost is not a special “image block.” It is KV cache, sized by the visual token count. The KV cache size explained page derives the cache formula from layer count, head count, and sequence length; the visual tokens are simply part of that sequence length.
This is the bridge between this page and the text-model pages. A multimodal model does not invent a new memory category for images. It converts the image into tokens and lets the existing KV cache absorb the cost. Anyone who understands context-length cost for text already understands image cost for a multimodal model; the only new step is counting visual tokens instead of text tokens.
The multimodal VRAM breakdown
The table separates the memory components of a vision-capable multimodal model. As elsewhere, the figures are qualitative: exact bytes depend on the language model size, the encoder, the patch strategy, the resolution, the quant, and the engine. The structure is the point.
| Component | What it is | Scales with | Often missed? |
|---|---|---|---|
| Language model weights | The text backbone | Model size and quant bits | No |
| Vision encoder weights | Image-to-patch network | Encoder size and quant bits | Yes — added on top of the LLM |
| Projection layer weights | Maps visual tokens to text space | Small, fixed by design | Yes — easy to forget |
| Visual tokens (encoder output) | Patches turned into tokens | Image area (more patches at higher res) | Yes — the image’s real cost |
| KV cache for visual tokens | Key/value per visual token | Visual token count, then context | Yes — lands on the same cache as text |
| KV cache for text | Key/value per text token | Text length | No |
| Encoder residency | Kept loaded across images | Per image processed | Yes — matters when many images arrive |
The rows marked “often missed” are the ones that make a multimodal model use more than a naive “it is just the LLM” estimate. The encoder and projection are added weights, and the image’s true cost is the KV cache for its visual tokens, which competes with the text for the same context budget.
One image equals how many text tokens
A frequent question is how many text tokens a single image “costs.” The honest answer is that it varies, and the right way to think about it is by patch count rather than by a fixed ratio.
The encoder tiles the image into a grid. If the grid is, for example, a certain number of patches on each side, the visual token count is the product of those sides. Raising the resolution adds patches on both axes, so the token count scales with image area, not with a single linear dimension. A modest resolution bump can therefore add a surprising number of visual tokens, each of which takes a slot in the KV cache.
There is no universal “one image equals N text tokens” because the patch size and the tiling strategy differ between models. The useful habit is to treat the image as a block of context of size equal to its patch count, and to check whether that block plus the planned reply fits the context setting. The context length cost page shows how context length translates into KV cache; substitute the visual token count for part of that length and the arithmetic is the same.
Why high-resolution images can blow the budget
Because visual tokens sit in the same context as text, a high-resolution image can consume a large share of the context by itself. If the context setting is fixed, the visual tokens leave less room for the reply and for any long document the model is also reading. The KV cache for the whole sequence grows with the combined token count, so a big image plus a long answer can push memory well past what the text-only case would suggest.
This is the practical trap of multimodal use. A model that handles short text comfortably may struggle once an image is added, not because the weights grew, but because the image injected a block of visual tokens that enlarged the KV cache. Lowering the image resolution, or cropping to the region of interest, reduces the patch count and therefore the cache, often enough to fit.
Batching and many images
Serving several images, or several users each with an image, multiplies the working set the way concurrency multiplies cost in text serving. Each image becomes its own run of visual tokens in its own context, and each context carries its own KV cache. The language model weights are shared, but the caches are not. The batch size and throughput page describes this linear multiplier for any batch, and it applies directly when many images are processed at once.
The encoder adds another wrinkle: if the encoder is kept resident to handle a stream of images, its weights stay in memory throughout, which is fine but is a cost that disappears from view once attention moves to the language model. Counting only the LLM understates the resident footprint of a busy multimodal service.
Sizing a multimodal model
The sizing order mirrors the text-model order with two extra steps:
- Take the language model weight size from its parameter count and quant, as on the VRAM calculator.
- Add the vision encoder and projection weights.
- Estimate the visual token count from the image resolution and patch strategy; treat it as context length.
- Add the KV cache for visual tokens plus the planned text, using the formula on the KV cache size explained page.
- Add engine headroom and, for serving, multiply the cache by expected concurrent sessions.
- Compare the sum to the card’s usable memory, noting the caveat in the GPU VRAM reference that nominal capacity is not all available.
A model that fits as text-only can fail once images are added, and the failure is almost always the KV cache for visual tokens rather than the weights.
Connecting multimodal models to the rest of the site
The vision encoder and projection follow the same weight arithmetic as any network, covered on the VRAM math cheatsheet and the model file size lookup. The visual tokens reuse the language model’s context machinery, so the context length cost page and the KV cache size explained page are the natural companions here: the image’s cost is just context cost with a different source. The quantization formats compared page sets the weight floor for each block, and the local LLM hardware guide gives the judgment criteria for whether the whole stack fits, without naming specific cards.
What this page does not tell you
This page does not give a per-model VRAM number for any specific multimodal model, because the figures depend on the language model size, the encoder, the patch strategy, the resolution, the quant, and the engine, and a single quoted number would mislead at other settings. It does not cover training or fine-tuning, which add gradients and optimizer state on top of the weights and caches described here. It does not benchmark speed, which depends on the runtime and hardware rather than on the memory layout. It does not recommend specific graphics cards; the GPU VRAM reference lists nominal capacities only, and usable memory is always below nominal. It also does not address video understanding, which introduces a time dimension and a different token-counting scheme from single images.