A diffusion image model does not fit on a card the way a language model does. The weights are only part of the budget. A large and easily underestimated share of memory goes to the intermediate activations produced while the network denoises a picture, and those activations scale with the image’s pixel area rather than with a token count. This page breaks the VRAM bill of an image model into its component parts, marks the items that are most often missed, and explains why the resolution term behaves nothing like the linear context growth seen in text models.
What stays resident: the weight blocks
The first portion of VRAM is the model weights, loaded once and held for the whole generation. An image pipeline is not a single network but a stack of them, and each one carries its own weights.
- The denoising backbone (commonly a UNet or a transformer-style backbone) is the largest weight block. It runs every single denoising step, so it must be fully resident.
- The text encoder(s) turn the prompt into conditioning vectors. Some pipelines load one encoder; others load two or more, and each additional encoder adds its own weight footprint. This is the first easily missed item, because a reader counting only the backbone undercounts the true resident memory.
- The VAE is the autoencoder that compresses images into a latent space and decodes them back. Its encoder is used up front, and its decoder is used at the end. Whether the VAE weights are kept resident or reloaded depends on the engine, but in many setups they are held in memory alongside the backbone.
- Optional condition modules such as ControlNet add still more encoder-weight blocks that participate in the denoising loop.
The weight portion is mostly fixed for a given model and quantization. Unlike a language model, where loading once and sharing across users is the norm, an image pipeline still pays the per-block weight cost every time it renders.
The activation term: why resolution dominates
The part that surprises people is the activation memory. During each denoising step the backbone computes feature maps at every layer, and the size of those maps is tied to the latent resolution, which in turn is tied to the output image’s pixel dimensions.
The key relationship is that activation memory grows with pixel area, not with a linear count. If the output is made twice as tall and twice as wide, the number of spatial positions goes up by roughly four. The activations at each layer scale with that area, so the activation cost multiplies by about four for a 2x size increase. This is a quadratic relationship in the two spatial dimensions.
That stands in sharp contrast to a language model. In a text model, the activation and KV cache grow linearly with context length: twice the tokens means roughly twice the cache. An image model’s activation growth is quadratic in resolution, so the two curves separate quickly. A rule of thumb that works well for a chat model at longer context — “a bit more memory per extra token” — badly under-predicts an image model at higher resolution, where “a bit more per extra pixel” compounds into “four times as much per doubled side.”
This is the central reason image-model VRAM intuition cannot be borrowed from language-model intuition. The backbone weights are comparable in spirit, but the activation term follows a different law.
The image-generation VRAM breakdown
The table below lists where the VRAM goes for a typical text-to-image diffusion pipeline. It is qualitative by design: the actual bytes depend on the specific model family, its parameter count, the quantization, and the engine. The point is the shape of the budget and which rows are easy to overlook.
| Component | What it is | Scales with | Often missed? |
|---|---|---|---|
| Denoising backbone weights | The main network run every step | Model size and quant bits | No |
| Text encoder weights | Prompt conditioning network(s) | Number and size of encoders | Yes — a second encoder is common |
| Text encoder residency | Kept loaded across steps | Same as above, if resident | Yes — only counted when reloaded |
| VAE encoder/decoder | Latent compress/decompress | Model size; full-res feature map at decode | Yes — the decode burst is forgotten |
| Activations per step | Feature maps during denoising | Pixel area (quadratic in resolution) | Yes — the dominant variable cost |
| Batch size / concurrency | Several images at once | Roughly linearly with count | Partly |
| ControlNet / extra condition modules | Added encoder blocks | Module size and resolution | Yes — silent weight + activation add |
The rows marked “often missed” are the ones that make a real-world render use far more than a naive “backbone size plus a little” estimate. Anyone sizing a card should add the encoders, the VAE decode burst, and the resolution-driven activation term before concluding that the model fits.
Batching and concurrency
Running more than one image in a batch multiplies activations. Each image in the batch carries its own feature maps through every layer, so a batch of N images at a fixed resolution raises the activation memory by roughly a factor of N on top of the shared weights. This mirrors the concurrency story in text serving, where each live sequence gets its own cache, but here the multiplier is spatial rather than sequential.
A practical consequence: a model that renders a single small image comfortably can overflow the card the moment several images are requested together, not because the weights grew, but because the activation term scaled with the batch. Lowering the batch size is the cheapest fix when an image pipeline hits an out-of-memory error mid-generation.
Conditioning modules add both kinds of cost
ControlNet and similar modules are loaded as additional encoder-weight blocks that feed into the backbone at each step. They therefore add in two places at once: a resident weight block, and extra per-step activations because their feature maps flow through the denoising loop. At higher resolution both contributions grow, since the conditioning features are themselves spatial.
The effect is that enabling a control module is not a small toggle. It shifts the model from “backbone plus prompt” to “backbone plus prompt plus a second network that runs every step.” Anyone who has fit a base model but cannot fit the same model with a control module active has hit exactly this dual addition.
Reading a memory number for an image model
When a memory figure is quoted for an image pipeline, the questions to ask are:
- At what resolution was it measured? Because activation is quadratic, the same model at two resolutions can show very different totals.
- Were both text encoders resident, or was one reloaded? That changes the constant overhead.
- Was the VAE kept in memory, and was the decode burst included? Short benchmarks often miss the tail.
- What batch size was used? A batch of one versus a batch of four is not the same model.
Treat any single quoted number as specific to those conditions. A figure taken at a low resolution says little about a high-resolution render, and a figure for a base pipeline says little about the same pipeline with a control module attached.
Connecting image models to the rest of the site
The weight portion of an image model follows the same arithmetic as any other neural network: parameter count times bits per parameter, divided by eight, as covered on the VRAM math cheatsheet. The difference is the activation term, which has no direct equivalent in a language model’s fixed-weight-plus-KV picture.
For readers coming from text models, the context length cost page shows the linear growth that image models do not follow. The model file size lookup explains how weight size is derived from quant bits, which applies to image backbones as much as to language models. The quantization formats compared page describes the bits-per-parameter trade that sets the weight floor for any model family. And for hardware that must hold all of these blocks at once, the local LLM hardware guide gives judgment criteria rather than purchase advice, the same criteria an image pipeline must meet.
What this page does not tell you
This page does not give a per-model VRAM number for any specific image model, because the figures depend on the exact architecture, quant, engine, and resolution, and quoting one would mislead at other settings. It does not cover training or fine-tuning memory, which adds gradients and optimizer state on top of everything here. It does not benchmark speed, which depends on the compiler, driver, and offload setup rather than on the memory layout alone. It does not recommend particular graphics cards; the GPU VRAM reference lists nominal capacities, and the usable memory is always less than the nominal figure. It also does not address video or 3D generation pipelines, whose activation patterns differ again from single-image diffusion.