LLM Inference: VRAM & Performance Calculator
Pick a model, a quantization and a device — see whether it fits and how fast it runs.
Precision of the model weights. Lower uses less VRAM but costs quality.
KV cache precision. Dominates VRAM at long context.
Hardware
One machine, or GPUs spread over several nodes you lay out yourself.
Select your GPU or configure a custom device.
Tensor-parallel replicas. Comms overhead is included.
Physical chassis, and the GPUs installed in each.
TP splits every layer across GPUs, PP splits the layer stack into stages. Whatever is left over replicates. Auto picks the layout that fits.
Link between GPUs in one machine. More bandwidth scales tensor parallelism better.
Lets the model exceed VRAM — at host-bandwidth speed.
Workload
Sequences processed per step. Raises throughput, costs KV cache.
Tokens per sequence (prompt + generation). Drives KV cache.
Simultaneous requests. Multiplies KV cache, splits per-user speed.
Inference simulation
Press play to watch this configuration generate at its estimated rate.
0 GB
of 0 GB usable
Assumes the device runs continuously at the estimated utilization.