Your Quantized LLM Loads, Then Runs Out of Memory. What Changed?

A quantized model loading successfully does not prove that its generation workload fits in GPU memory. Weight storage, attention state for processed tokens, temporary computation and runtime reservations are different demands. First identify whether the failure occurs during loading, warmup, a long prompt, later generation or concurrent requests. Those observations lead to different next checks.

This guide concerns text generation on GPU with decoder-style models. The diagnostic sequence below is our synthesis of official documentation, not a tested recipe for a particular card. Multimodal encoders, training and distributed placement need additional accounting.

Why did four-bit weights not solve the problem?

Weight quantization changes how model parameters are represented. In the Transformers bitsandbytes integration, quantized linear layers replace ordinary linear layers; other modules can retain a different dtype. A checkpoint’s advertised precision therefore does not describe every runtime allocation. Transformers bitsandbytes documentation.

During autoregressive generation, attention also retains keys and values from processed tokens. This KV cache is separate from the weights and is maintained per layer. Its tensor dimensions include sequence length and batch size. More retained tokens or simultaneous sequences can therefore increase its payload without changing the model’s weight file. Transformers caching explanation.

Do not infer that a four-bit weight format gives you a four-bit KV cache. Check the actual cache representation and implementation separately.

Original diagnostic diagram. The arrows show a diagnostic test order, not a strict runtime sequence. These are observation checkpoints, not measured memory proportions or proof that a particular component caused the failure.

At which stage does memory run out?

Start a small record before changing configuration. Record the model revision, runtime version, device, weight format, KV dtype and cache implementation. Keep all settings fixed except the variable under investigation.

First failure observed	Question to investigate	Controlled comparison
Weight loading	Can the selected representation and loading path fit at all?	A supported smaller model or weight representation, with the same device placement
Warmup or graph capture	Does runtime setup reserve additional memory?	The runtime’s documented eager mode versus graph capture
Processing a long prompt	Does the input create a peak beyond the short-prompt case?	Short versus long tokenized inputs, one active sequence
Later token generation	Does retained state grow beyond available capacity?	Same input and concurrency, different output caps
Concurrent requests	Is capacity sufficient for multiple active sequences?	Same input/output bounds, one versus several active sequences

This table proposes investigations, not diagnoses. For example, a failure late in generation can involve retained tensors or another process, rather than KV growth alone. Record the failing operation and peak memory, not just the final error string. Keep synthetic inputs free of private data when preparing a reproducible report.

In vLLM, the current conserving-memory guide documents max_model_len and max_num_seqs as memory levers. It also says CUDA graph capture consumes extra GPU memory and documents smaller capture sets or enforce_eager as alternatives. These are speed/capacity trade-offs, not interchangeable fixes. Match options to your installed version and compare performance before retaining a change. vLLM conserving memory.

How much can KV state add? A fictional ledger

Consider a deliberately simplified, single-GPU, full-attention decoder with:

• 32 layers, 8 KV heads per layer and head dimension 128;
• two bytes per cached scalar;
• no beam expansion, shared prefixes, sliding windows or quantized cache.

Each stored token needs keys and values across these layers:

2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB

At 8,192 stored tokens, that is 1 GiB per sequence. Four such active sequences require 4 GiB of raw KV payload. Doubling each sequence to 16,384 stored tokens doubles that payload to 8 GiB. This is dimensional arithmetic based on per-layer key/value storage, not a benchmark or a real model specification. Use KV heads, not the larger query-head count in a grouped-query model. See the separate num_key_value_heads and num_attention_heads definitions in LlamaConfig. Cache tensor structure.

For a fictional budget of 12 GiB, suppose resident weights take 5 GiB and a separate non-KV allowance takes 2 GiB:

Scenario	Weights	KV payload	Other allowance	Total
Four sequences, 8,192 stored tokens each	5 GiB	4 GiB	2 GiB	11 GiB
Four sequences, 16,384 stored tokens each	5 GiB	8 GiB	2 GiB	15 GiB

The second scenario exceeds the assumed budget. The first is not certified to fit. The allowance is invented, not measured; temporary peaks, allocation granularity and other processes can still invalidate it. For unequal sequence lengths, use the sum of stored tokens. Count processed input and generated-token state, rather than only the requested output cap.

Real implementations complicate the ledger. Static caches reserve capacity in advance, while sliding-window or chunked layers can stop growing at their limits. Paged allocation and prefix sharing also require implementation-specific accounting. The calculation explains a failure mechanism; it is not a universal VRAM calculator. Transformers cache strategies.

Why does clearing the cache not free the memory I expected?

“Cache” names different things. The KV cache holds model state; PyTorch’s caching allocator manages memory blocks. Its memory_allocated() counter concerns tensor storage, while memory_reserved() concerns allocator-managed memory, including allocations. Do not add those two counters as independent costs.

PyTorch documents that unused reserved blocks can still appear occupied in nvidia-smi. empty_cache() releases unused allocator blocks, not live tensor storage. It cannot make an oversized live weights-plus-state requirement disappear. A high reserved number alone is not evidence of a leak. PyTorch 2.15 memory management.

Which trade-off should you try next?

Our recommended order is to change one constraint, then measure the same workload again:

1. Bound the workload. Set useful input/output limits and an explicit active-sequence limit. Reject or queue excess work rather than assuming a successful short request establishes serving capacity.
2. Inspect the cache choice. Transformers documents offloading KV state to CPU and separate KV quantization. Transfers can reduce throughput; quantized caches can add latency when memory was already sufficient. Test both capacity and task quality, not only whether the process survives. Cache strategies and trade-offs.
3. Inspect runtime overhead at the failing stage. Compare supported graph-capture settings if warmup is the problem. Do not change every tuning flag at once.
4. Recheck the actual service envelope. Include representative long inputs, output bounds and concurrency, plus latency and throughput. If those cases still fail, reduce the supported workload or change the hardware/model plan rather than advertising the larger capacity.

A shorter output cap can also produce incomplete application responses. If that happens, distinguish an intentional generation limit from a memory failure using the response-completion diagnostic. Provider prompt-cache billing is another separate question; a billing cache hit does not establish local GPU capacity.

Scope and source dates

Sources were checked on November 5, 2026 JST. Transformers pages are current unversioned documentation; their exact revision dates were not exposed. The vLLM stable guide displays October 9, 2026 but did not expose an exact release identifier in the retrieved page. The PyTorch source is explicitly version 2.15. Check your installed version before applying an option. No GPU workload, model, throughput result or memory profiler was run for this article.