Why an LLM Fits in VRAM but Still Runs Out of Memory

A local language model can load successfully, answer a short question and then fail when you paste in a document. That does not necessarily mean the installation is broken. Loading the model and running your intended workload put different demands on the graphics card.

Before buying a GPU or renting a larger one, separate three things: the space occupied by model weights, the memory needed for the conversation, and the working memory used by the software. A useful hardware test exercises all three.

Start with the weights

The parameter count gives you a rough starting point. If every parameter of an eight-billion-parameter model used two bytes, its weights would occupy about 16 GB. At four bits per parameter, the simple calculation gives 4 GB.

Those numbers are lower-bound calculations. Quantized files also contain metadata, and some tensors may remain at higher precision. The inference engine needs additional allocations when it loads and runs the model. A file that fits on disk is therefore not a reliable specification for the GPU you need.

Keep the units consistent, too. Decimal GB and binary GiB are different: 16 billion bytes is about 14.9 GiB. Check the units reported by your monitoring tool before comparing them with a spreadsheet.

Record the exact model version and quantization format. “An 8B model” is too vague for a comparison that someone else needs to reproduce.

Budget for the conversation

Many transformer models store attention information in a key-value cache, usually called the KV cache. Reusing it avoids recalculating the entire conversation at every generated token. That saved computation comes with a memory cost.

A longer prompt and its generated response need more cached tokens. Several independent conversations can need separate cache space. The model weights can stay unchanged while the total working set grows substantially.

For a simplified full-attention cache, estimate:

KV bytes = 2 × layers × KV heads × head dimension
           × cached tokens × bytes per cache element

The factor of two accounts for keys and values. Use the number of KV heads, which can differ from the number of query heads.

Consider a hypothetical architecture with 40 layers, eight KV heads, a head dimension of 128 and two-byte cache elements:

These are cache-only calculations, before weights and runtime overhead. They assume unshared caches and full attention; sliding windows, other architectures and cache formats can change the result.

This explains why testing one short chat can give an overly optimistic impression of a card. The difficult workload might be a long document or several users arriving together.

Set limits that match the task

Decide what the application actually needs before selecting the largest supported context window. A short classification task and a coding assistant reading several files have different requirements.

Write down a representative prompt length, an allowance for the generated answer and the number of simultaneous requests you expect. Include the system prompt, conversation history and retrieved material in the input budget.

When comparing cards, AtmosCompute’s guide to estimating GPU memory for an LLM provides a useful starting calculation. Use that estimate to narrow the options, then measure the exact model and workload.

Runtime controls matter here. In vLLM, max_model_len and max_num_seqs let you bound context and batch size. In Ollama, check the context configured for the running model instead of assuming a default will suit your workload.

Reducing these limits may make a configuration usable, but it also changes what the application can handle. Keep the limits alongside every result so a faster or cheaper setup does not quietly mean a shorter conversation.

Measure a realistic request

For an NVIDIA GPU, begin with:

nvidia-smi --query-gpu=name,memory.total,memory.used,memory.free --format=csv

Check the card before loading the model, after loading it and while running the workload. A single idle reading cannot reveal every temporary allocation, so use repeated observations alongside the engine’s logs.

For Ollama, ollama ps also shows whether the model is running fully on the GPU or partly on the CPU. A successful response with CPU offloading can have very different latency from an entirely GPU-resident run.

Use a small, repeatable test sequence:

1. Run one short request to check that the setup works.
2. Submit a representative long prompt and generate the intended answer length.
3. Repeat with the number of simultaneous requests you plan to allow.
4. Record failures, response time and the memory observations for each case.

Keep the model, precision, prompt and output allowance fixed while comparing GPUs. Save the software versions and configuration with the results.

Choose the change that addresses the failure

If the model fails during loading, investigate the weights, format support and baseline memory allocation. A smaller model, a compatible quantized checkpoint or more VRAM may be appropriate.

If loading succeeds but a long request fails, inspect the context budget and cache requirements. If one request works but several fail together, examine concurrency and scheduling. Some serving engines reserve a cache pool at startup, so a high memory reading alone is not evidence that the GPU is overloaded.

Do not assume two smaller cards automatically work like one larger card. The engine must support distributing the chosen model, and communication between cards can affect performance.

Before committing to hardware, keep one reproducible configuration that survives your longest expected request at the intended concurrency. That result gives you a much better purchasing basis than the model’s download size or a successful “hello” prompt.