Running a large language model on your own hardware comes down to one question: does the model’s memory fit in the memory you have? This page is the entry point for the whole site. It lays out the path from choosing a model to getting it to load, names the four numbers that decide success, and points to every other page that works out the details. Nothing here is a product recommendation; the only judgment offered is about memory.
A model that does not fit will not load, and a model that fits with no headroom will fall over the moment the context grows. The math is smaller than it looks: a handful of inputs, a couple of multiplications, and a comparison against your card. The rest of the site exists to make each input exact.
From zero to running: the decision path
The path has five steps. Each step answers one question and points to the page that answers it precisely. Do them in order; a later step is wasted if an earlier one says the model cannot fit.
- Pick a model. This step fixes the parameter count, which sets the floor on weight memory before any other choice is made. A 70B model and a 7B model are not the same planning problem. Start here: run a 70B model locally.
- Compute the VRAM. This step decides whether the model fits at all, by combining weights, the KV cache, and engine overhead into one number. If the sum clears your usable memory, continue; if not, revisit the model or the quantization. Do it here: VRAM calculator.
- Choose an engine. This step decides how the memory is laid out and whether you can serve more than one user. The engine changes startup behavior and concurrency cost, not the weight size. Compare them here: LLM engine comparison.
- Choose quantization. This step decides the trade between weight size and output quality. A lower tier fits a smaller card; a higher tier keeps more of the original model. Weigh the tiers here: quantization formats compared.
- Troubleshoot. This step decides what to change when the model fails to load or drops mid-session. The cause is almost always context, quant tier, or offload headroom. Start here: Ollama out of memory.
The same path, as a reference table:
| Step | What this step decides | Where to go |
|---|---|---|
| 1. Pick a model | The parameter count, which sets the minimum weight memory | run a 70B model locally |
| 2. Compute VRAM | Whether the model fits, combining weights + KV cache + overhead | VRAM calculator |
| 3. Choose an engine | Memory layout and whether multiple users are supported | LLM engine comparison |
| 4. Choose quantization | Weight size versus quality trade-off | quantization formats compared |
| 5. Troubleshoot | What to change when loading or running fails | Ollama out of memory |
The four numbers that decide everything
Four inputs settle nearly every local-LLM memory question. Learn what each one is, why it matters, and which step consumes it.
1. Parameter count
The parameter count is the size of the model in billions of weights. It matters because it sets the floor: no amount of clever engineering removes the weights from memory. A 7B model and a 70B model differ by an order of magnitude in weight memory before quantization is even considered. This number enters at step 1 (picking the model) and step 2 (computing VRAM), where it is multiplied by the bits-per-weight. The exact per-model figures live in the model file size lookup.
2. Quantization bits
The bits-per-weight is how many bits store each parameter on disk and in memory. It matters because it is the main lever you control after picking a model: dropping from 16-bit to roughly 4.5-bit (the Q4_K_M tier) cuts the weight memory by more than three times. As a rule of thumb, Q4_K_M is about 0.6 GB per billion parameters, with a small margin. A 70B at Q4_K_M is approximately 39.4 GB in weights; a 7B at the same tier is approximately 3.9 GB. Treat these as estimates: real GGUF files often run 5 to 10 percent higher because token embedding and output tensors stay at higher precision, so the actual file size is the figure to trust. This number is chosen at step 4 and consumed at step 2. The trade against quality is covered by quantization quality loss.
3. Context length
The context length is how many tokens the model holds at once. It matters because it drives the KV cache, the part of memory that scales with use rather than with the model itself. The weights are fixed the moment the file is built; the cache is not. Doubling the context doubles the cache. On a 70B-class shape the cache is approximately 10 GB at 32K context and approximately 40 GB at 128K context, figures derived from the model’s public config.json (check the specific model’s config for exact layer and head counts). A model that loads comfortably at 4K can become impossible at 128K purely from the cache. This number is set during step 3 or step 5 and sized exactly by the KV cache calculator and the context length cost page.
4. Available VRAM
The available VRAM is the memory your model can actually use. It matters because it is the budget everything else spends against. Note that nominal VRAM is not the same as usable VRAM: the display output and driver reserve part of it on Windows and Linux, and macOS reserves part of unified memory too. On Apple Silicon the pool is shared and can reach 192 GB on high-end machines, which changes what fits, as Apple Silicon unified memory explains. Nominal figures for common cards, such as 24 GB for an RTX 4090 or 80 GB for an A100, are manufacturer labels and should be checked against the vendor spec page. This number is the comparison target at step 2. The GPU VRAM reference lists nominal values across cards.
How to read the rest of the site
The pages below are the rest of the site, grouped by what they help you decide. Each row names the page and the one problem it solves. Use the decision path above to know which group to open next.
Tools (calculators)
| Page | What it solves |
|---|---|
| VRAM calculator | Computes total VRAM as weights + KV cache + overhead in one step. |
| KV cache calculator | Sizes the context-scaling KV cache for a model’s attention shape. |
| Context length calculator | Turns a context length into KV cache size and leftover headroom. |
| Token cost calculator | Compares local electricity amortization against per-API pricing at your token volume. |
| Model file size lookup | Looks up the on-disk size of a model at each quantization tier. |
| Open model catalog | A crawled list of open-weight families with reported file sizes, context windows and the class of card each tag needs. |
Decide before you download
| Page | What it solves |
|---|---|
| Run a 70B model locally | Walks the memory math for getting a 70B model to fit and run on real hardware. |
| Local LLM hardware guide | Gives the judgment standard for what card fits your model, not a buy list. |
| Apple Silicon unified memory | Explains how Apple Silicon’s shared pool changes what fits. |
| GPU VRAM reference | Lists nominal VRAM across common cards so you can match model to hardware. |
| Ollama VRAM requirements | Shows the real Ollama memory footprint for common models and quant tiers. |
| LLM engine comparison | Compares llama.cpp, Ollama, and vLLM on memory overhead, startup, and concurrency. |
How memory actually works
| Page | What it solves |
|---|---|
| KV cache size, explained | Explains the formula behind why context length, not weights, fills memory. |
| VRAM math cheatsheet | Collects the core VRAM formulas and shortcuts in one reference sheet. |
| Mixture of experts memory | Shows how MoE models store more weights than they run per token. |
| Quantization formats compared | Compares GGUF quant tiers by bits-per-weight and file size. |
| GGUF vs safetensors | Contrasts the two formats on disk and in-memory behavior. |
| Quantization quality loss | Summarizes how much quality each quant tier tends to give up. |
| Context length cost | Connects chat length to KV cache and how batching multiplies it. |
Make it fit and run faster
| Page | What it solves |
|---|---|
| llama.cpp offload layers | Explains splitting a model across GPU and CPU by layer when it will not fit. |
| Multi-GPU VRAM | Covers how tensor and pipeline parallelism split memory across cards. |
| CPU offload performance | Maps how much VRAM each offload step saves against the speed it costs. |
| Batch size and throughput VRAM | Shows how concurrent sequences grow memory linearly on top of weights. |
| Measure VRAM usage | Lists ways to read the actual memory a running model uses. |
| Embeddings and RAG memory | Estimates the vector-store and embedding memory a RAG setup needs. |
Other model families
| Page | What it solves |
|---|---|
| Stable Diffusion VRAM | Sizes the VRAM for image generation models. |
| Whisper and audio VRAM | Sizes the VRAM for speech-to-text and audio models. |
| Vision model VRAM | Sizes the VRAM for multimodal vision models. |
| Fine-tuning VRAM | Shows the memory multiples each fine-tuning method adds over inference. |
Running costs
| Page | What it solves |
|---|---|
| LLM API cost comparison | Compares API provider pricing per token, with update timestamps. |
| ChatGPT cost per month | Compares regional subscription tiers and what they include. |
| AI Excel formula | Points to AI helpers for spreadsheet formula work. |
| Ollama out of memory | Diagnoses common OOM causes when an Ollama model fails to load. |
A note on estimates
Every number on this site that is derived rather than measured is an estimate, and each depends on inputs you supply. The weight sizes depend on the exact GGUF file, because real files carry higher-precision embedding and output tensors. The KV cache depends on the model’s layer, KV-head, and head-dimension counts from its config.json. The usable memory depends on your operating system and what it reserves. When a page gives a figure, treat it as a planning estimate and confirm against the calculator and your own hardware before committing a download of tens of gigabytes.
What this page does not tell you
- It does not recommend a specific graphics card to buy, and it does not link to vendors. The judgment offered is always about memory needed versus memory available, never about a purchase.
- It does not give tokens-per-second figures. Speed depends on the build, drivers, batch size, and offload ratio, and those vary too much to state a number.
- It does not measure your machine. The available-VRAM figures are nominal manufacturer labels; read your actual usage with the measure VRAM usage page.
- It does not cover training from scratch or distributed cluster setups beyond the memory multiples in fine-tuning VRAM.
- It does not price hardware or electricity; the token cost calculator and the LLM API cost comparison page handle cost comparisons with their own update notes.
- It does not substitute for the model’s own documentation, the engine’s README, or the vendor spec page, which remain the authoritative sources for exact numbers.