Quantization on a budget: what actually fits in 12 GB

The question I keep getting asked is simple: what can I run on a mid-range card with 12 GB of memory? The honest answer is "more than you think, less than the benchmarks suggest".

The rule of thumb

Weights at 4-bit take roughly half a byte per parameter, plus some overhead. A 14B model lands around 8–9 GB, which leaves room for a reasonable context window. Push to 8-bit and the same model no longer fits at all.

Where it falls apart

Long contexts. The key-value cache grows with every token, and on a 12 GB card it is the context, not the weights, that runs out first. Quantizing the cache helps more than squeezing the weights further.

weights (4-bit)   ~ 0.55 GB per 1B params
kv cache          grows with context length
headroom          keep ~1 GB free for the driver

For daily use I settled on a 4-bit 14B model with an 8k context. It is not the smartest setup, but it answers fast and never swaps.