Research · Inference
LLM inference benchmarks and serving research
Measured results for serving large language models on the GPUs people can buy, from the RTX 4090 to the B200, with the setup and the limits of each number stated.
The benchmark articles compare the RTX 4090, RTX 5090 and RTX PRO 6000 on vLLM with quantized Llama and Qwen models, put the PRO 6000 against the H100, H200 and L40S, and take the B200 through 16K-token long-context inference next to the H200, H100 and PRO 6000. Each one reports throughput and cost for the exact configuration we ran, so you can tell which results carry over to your own setup.
The optimization articles show how far a single card can be pushed. Qwen3 Coder went from 277 to 1,207 tokens per second on a PRO 6000 and from 556 to 1,157 on an RTX 5090, with the recipes to reproduce it. On Gemma 4 12B, kernels generated by Emmy, our compiler, reach up to 1.6× cuBLAS on the RTX 5090 and ship as a vLLM plugin that beats stock vLLM and llama.cpp. A network-attached KV cache extends an RTX card’s effective memory for long-context work, and the two FlashAttention articles derive the kernel from first principles and then build it to FlashAttention-2 parity on the 5090.
The track also carries a July 2026 report on open-weight coding models: what agentic developers run, how far the gap to closed models has closed, and the hardware the best open models now need.
AUG 2026 · BENCHMARKS · LLM
Outperforming vLLM and Llama.cpp on Gemma4-12B
Emmy, an AI-driven compiler, generates CUDA kernels automatically instead of by hand. On Gemma 4 12B in FP16 its kernels reach up to 1.6× over cuBLAS on the RTX 5090, achieved via TMA transport, hybrid FP16/FP32 accumulation, and FlashAttention-2 parity, shipped as a drop-in vLLM plugin.
JUL 2026 · LLM · OPEN SOURCE
The State of Open Coding AI Models in July 2026
A data report on open-weight coding models in July 2026: what agentic developers actually run, how far the open-to-closed gap has closed, and why the best open coders have outgrown the hardware most people own.
JUL 2026 · TUTORIALS · GPU
Learning FlashAttention the Hard Way — Part 2
Part 2 takes Part 1's twisted monoid down to the metal: a generic compiler moveset walks one attention kernel from 25959µs to 206µs on an RTX 5090 — on par with FlashAttention-2 — with a measured number for every rung, the optimizations measurement refused, and the reasons FA-3/4 cannot land on consumer silicon.
JUL 2026 · TUTORIALS · ALGEBRA
Learning FlashAttention the Hard Way — Part 1
FlashAttention looks like a bespoke GPU kernel, but it is not much different from a regular parallel reduction. Safe softmax becomes an associative reduction via denominator trick, Welford's variance, and flash attention is the same concept. This part is the complete theoretical foundation, with the numerical-stability analysis and a test for when any loop is secretly associative.
MAR 2026 · BENCHMARKS · LLM
Optimizing Qwen3 Coder for RTX 5090 and PRO 6000
I got Qwen3 Coder from 277 tok/s to 1,207 tok/s on a PRO 6000, and from 556 to 1,157 tok/s on an RTX 5090. Here's exactly how, with reproducible recipes.
JAN 2026 · BENCHMARKS · LLM
Blackwell Dominates. Benchmarking LLM Inference on NVIDIA B200, H200, H100, and RTX PRO 6000
We benchmarked NVIDIA B200, H200, H100, and RTX PRO 6000 for long-context LLM inference using 8K input + 8K output (16K total). B200 delivers up to 4.9× the throughput of RTX PRO 6000 and is now the cost efficiency leader across all models.
NOV 2025 · BENCHMARKS · LLM
RTX PRO 6000 vs H100, H200, and L40S: LLM Inference
RTX PRO 6000 beats H100 SXM on single-GPU LLM inference at 28% lower cost per token. H100 and H200 NVLink pull 3-4x ahead on 8-way tensor parallel.
OCT 2025 · BENCHMARKS · LLM
RTX 4090 vs 5090 vs PRO 6000: LLM Inference Benchmark
RTX 4090, 5090, and PRO 6000 benchmarked on vLLM with three quantized models (24GB, 48GB, 96GB). PRO 6000 hits 3.7x the 4090 throughput on single GPU.
SEP 2025 · BENCHMARKS · LLM
Benchmarking LLM Inference on RTX 4090, RTX 5090, and RTX PRO 6000
We ran a series of benchmarks across multiple GPU cloud servers to evaluate their performance for LLM workloads, specifically serving LLaMA and Qwen models and on RTX 4090, RTX 5090, and RTX PRO 6000 GPUs.
AUG 2025 · AI TOOLS & WORKFLOWS · LLM
How to Give Your RTX GPU Nearly Infinite Memory for LLM Inference
Network-Attached KV Cache for Long-Context, Multi-Turn Workloads. Let's be honest — we can't afford an H100. Learn how to extend your RTX GPU's effective memory using innovative KV cache offloading techniques.