Estimate the GPU memory your LLM can require.
Calculate theoretical model-weight memory, optionally add architecture-aware KV cache, then apply an explicit runtime safety margin. Unknown architecture details stay unknown.
- Primary use
- Inference planning
- Weight precision
- 2–16 bit
- KV cache
- Optional exact formula
- Performance claim
- None
Define the inference load
Start with theoretical weight memory; add exact architecture fields when available.
Capacity breakdown
GiB uses binary units. The recommended class rounds up from included inputs.
Memory capacity is only one deployment constraint.
Weight memory is parameter count × bits per weight ÷ eight, converted to GiB. When every architecture field is supplied, KV cache uses two tensors (key and value) across layers, context tokens, concurrent sequences, KV heads, head dimension, and cache precision. The safety margin is then applied to the included subtotal.
Real runtimes also allocate activations, quantization metadata, kernels, framework buffers, CUDA context, and fragmented memory. Tensor parallelism and offloading add communication and placement constraints. The result does not predict tokens per second, latency, software compatibility, or whether a particular GPU is economical.