What is layer offload?

Running some transformer layers on CPU/system RAM so the GPU need not hold the full model.

PCIe or memory traffic for offloaded layers is far slower than on-GPU matmuls.

Experiments, one-off jobs, or proving a prompt before buying hardware — not production chat latency.

Does VRAMwise simulate exact tok/s?

No. The offload tool is a transparent penalty sketch.

CPU offloading: how much speed you lose

Offload is a VRAM relief valve, not a free performance feature.

The mechanism

If the full model does not fit, runtimes can keep some layers on the host. Every token may pay a tax moving activations across a slow link.

How to think about the penalty

Our sketch uses effective ≈ tps × (1 − frac + frac × penalty_factor). Penalty factor ≪ 1 models slow host paths. Calibrate against a real run on your machine when possible.

Better alternatives when possible

Smaller quant, shorter context, or a GPU with more VRAM usually beats aggressive offload for interactive use.

General information, not personalized advice.

Models in band

Llama 3.1 8B Instruct · Llama 3.3 70B Instruct · Qwen3 8B · Qwen3 14B · Qwen3 32B

Apple contrast

M1 · M1 Pro · M1 Max

Runtime tools

VRAM fit · Context VRAM · Quant compare

Catalog exits

Guides · Tools

Methodology paths

Offload sketch · Fit checker

CPU offloading: how much speed you lose

The mechanism

How to think about the penalty

Better alternatives when possible

Frequently asked questions