Measured tokens/sec: GPUs and Apple Silicon running local LLMs

Discrete GPUs: Llama 3 8B and 70B at Q4_K_M

All figures below are measured text-generation throughput (average over 1,024 generated tokens) from a single community benchmark suite run on identical model files, so rows are directly comparable. OOM = the model does not fit in that card's VRAM. Machine-readable copy: benchmarks.json.

Source & date: XiongjieDai, GPU-Benchmarks-on-LLM-Inference — llama.cpp, Llama 3 8B/70B Q4_K_M, average of 1,024 generated tokens. Retrieved 2026-07-13.

RTX 5090: newer models at Q4, 4k context

The RTX 5090 postdates the suite above; the figures below come from a separate llama.cpp test on Qwen3-family models at Q4_K_XL with a 4k context, so compare them against each other rather than row-for-row with the first table.

Source & date: Hardware Corner, RTX 5090 LLM benchmarks — llama.cpp, Q4_K_XL quantization, 4k context. Retrieved 2026-07-13.

Apple Silicon: 7B Q4_0 generation across the whole lineup

The canonical Apple-to-Apple comparison is the llama.cpp maintainers' benchmark thread: identical LLaMA 7B v2 Q4_0 model, 128 generated tokens, all layers on GPU. Ranges reflect multiple submissions per chip.

Source & date: llama.cpp discussion #4167, "Performance of llama.cpp on Apple Silicon M-series" — LLaMA 7B v2 Q4_0, 128-token generation, ngl=99. Retrieved 2026-07-13. M5-family chips had no comparable entries in the thread at retrieval time; bandwidth figures for them are in the Apple Silicon memory table.

What the measurements show

Generation speed tracks memory bandwidth, not marketing TOPS. Sort the Apple table by the bandwidth column and it is already sorted by tokens/sec, across four chip generations; the same holds for discrete cards once architecture differences are accounted for. That is the empirical basis for the estimator model documented on our methodology page: tokens/sec ≈ bandwidth × efficiency ÷ bytes per token, with measured efficiency clustering near 50–70% on CUDA GPUs and 35–55% on Apple Metal. It also explains apparent anomalies — the 12 GB RTX 4070 Ti (504 GB/s) losing to the 10 GB RTX 3080 (760 GB/s) is exactly what the bandwidth column predicts.

The 70B column is a capacity story first and a speed story second. Every single consumer card shows OOM: 24 GB simply cannot hold ≈46 GB of weights-plus-cache, as the model VRAM table breaks down. The setups that do run it — dual 24 GB cards, 48 GB workstation boards, high-memory Macs — land in the 4–25 t/s range, with dual-GPU rigs paying a visible tax: two RTX 4090s generate an 8B model slower (122.6 t/s) than one (127.7 t/s), because layer-split inference round-trips activations over PCIe every token.

Finally, architecture increasingly beats raw size. The MoE rows are the clearest case: Qwen3 30B-A3B holds a 32B-class file but generates at 234 t/s on an RTX 5090 — faster than the dense 8B — because only ≈3.3B parameters are read per token. If your hardware sits in the 16–24 GB band, a Q4 MoE model can deliver dense-70B-class quality signals at 8B-class speed; the minimum GPU per model tier table folds these tradeoffs into a quick lookup. Treat every figure here as llama.cpp-specific: vLLM, MLX, and TensorRT-LLM each shift throughput, usually upward on batch workloads.