Google's DiffusionGemma hits 1,288 tokens per second — but quality lags behind standard Gemma 4

A new open-source model applies image-diffusion logic to text generation, unlocking parallel decoding and self-correction. The speed gains are real. So is the quality trade-off.

Written by Lena Armitage · Bureau Tech · Jun 14, 2026

The surprising part isn't the speed — it's what Google admitted

Google's DiffusionGemma reaches 1,288 tokens per second on a single Nvidia H200. That's the headline number, and it's striking. But the more notable moment in the launch was what Google said next: the model's overall output quality is lower than standard Gemma 4, and teams that need maximum quality should use standard Gemma 4 instead.

That kind of candor is worth noting. It also tells you exactly what this model is and isn't.

What diffusion means for text

Diffusion, in the image-generation sense, means starting with random noise and iteratively refining the entire output in parallel until it converges. Stable Diffusion works this way. Standard language models do not — they generate one token at a time, left to right, with no ability to revise a committed output.

DiffusionGemma applies the diffusion approach to text. It begins with a block of 256 random placeholder tokens — effectively a blank canvas — and runs multiple refinement passes over the entire block simultaneously. On each pass, it locks in the positions it's most confident about. Low-confidence positions get randomized and reconsidered on the next pass, informed by what was resolved in the previous round.

Two architectural properties follow from this. First, **self-correction**: unlike an autoregressive model, which is stuck with a wrong token once it's committed, DiffusionGemma can identify uncertain positions and re-evaluate them. Second, **bidirectional context**: every token position attends to every other position in the block at once, including tokens that appear later in the sequence.

Where the speed numbers actually apply

At batch size 1 on a single H100, vLLM's published benchmarks put the FP8 model at roughly five times a standard autoregressive baseline. On H200, roughly six times. These are peak figures under optimal conditions: single user, dedicated hardware, FP8 quantization.

The gains are real in **local inference and low-concurrency deployments**, where the GPU has spare compute and memory bandwidth is the bottleneck. DiffusionGemma's parallel block generation fills that gap.

In **high-throughput cloud serving**, the picture changes. When a server is batching hundreds of concurrent requests, autoregressive models already saturate available compute, and parallel decoding provides diminishing returns.

This distinction matters for anyone evaluating the benchmarks. The 6x figure is not a general-purpose throughput improvement.

How it differs from speculative decoding

Speculative decoding — a common inference optimization — uses a smaller draft model to guess several tokens ahead, then has the target model verify them in one pass. The output distribution stays identical to the target model. The architecture is unchanged.

DiffusionGemma is not a decoding trick applied to a standard model. It creates a noisy 256-token canvas and repeatedly denoises the whole block in parallel. The generation paradigm itself is different.

Where the architecture has a structural edge

Google demonstrated bidirectional context with a fine-tuned Sudoku solver. The base model solved zero puzzles. After fine-tuning on a Sudoku dataset, it reached an 80% success rate and converged in 12 denoising steps rather than 48 — an efficiency gain that came directly from self-correction and early stopping.

The same structural advantage applies to **code infilling, structured data generation, and any task where correct output depends on context not yet generated**. These are the workloads worth evaluating DiffusionGemma against.

For open-ended generation, standard Gemma 4 remains the stronger option. Google says so directly.

What enterprises need to know

DiffusionGemma serves via a standard vLLM OpenAI-compatible endpoint. No diffusion-specific pipeline changes are required. The model is available under the Apache 2.0 license.

The vLLM integration required new engineering: DiffusionGemma alternates between causal and bidirectional attention as it cycles through prompt reading, canvas refinement, and block commit. The team built per-request attention switching into both the Triton and FlashAttention 4 backends. The new ModelState interface is designed to support additional diffusion models as they emerge.

For teams running local or low-concurrency inference on dedicated GPU hardware, this is a third path that didn't exist before: comparable parameter footprint to a large model, consumer-hardware compatibility, and same-day vLLM support. The quality trade-off is real. Whether it matters depends entirely on the task.

Key takeaways

DiffusionGemma generates 256 tokens simultaneously by starting with a noisy placeholder block and refining it across multiple passes — a fundamentally different generation paradigm from standard autoregressive models.
On a single Nvidia H200 at batch size 1, the FP8 version reaches 1,288 tokens per second — roughly six times a standard autoregressive baseline, per vLLM benchmark results.
Google explicitly states that DiffusionGemma's overall output quality is lower than standard Gemma 4 and recommends standard Gemma 4 for applications requiring maximum quality.
The speed advantage is conditional: it applies most clearly to local inference and low-concurrency deployments, not high-throughput cloud serving where autoregressive models already saturate available compute.
The model is a 26B Mixture of Experts architecture that activates only 3.8B parameters during inference and fits within 18GB VRAM, making it compatible with consumer hardware including the RTX 4090.

FAQ

What is a diffusion language model?

A diffusion language model generates text by starting with a block of random placeholder tokens and iteratively refining the entire block in parallel across multiple passes, rather than generating one token at a time from left to right. The approach is analogous to how image diffusion models like Stable Diffusion work.

Is DiffusionGemma better than standard Gemma 4?

Not overall. Google explicitly states that DiffusionGemma's output quality is lower than standard Gemma 4 and recommends standard Gemma 4 for applications requiring maximum quality. DiffusionGemma has a structural advantage on constrained generation tasks — code infilling, structured data, problems requiring bidirectional context — but on open-ended generation, standard Gemma 4 is the stronger choice.

What hardware does DiffusionGemma require?

The quantized model fits within 18GB VRAM, making it compatible with consumer hardware including the Nvidia RTX 4090 and 5090. Google and Nvidia also optimized it for enterprise Hopper and Blackwell servers using NVFP4 kernels.

Does the 6x speed improvement apply to all deployments?

No. The peak figures — roughly six times a standard autoregressive baseline on H200 — reflect batch size 1, single-user, FP8 quantization conditions. The speed advantage is most meaningful for local inference and low-concurrency deployments. In high-throughput cloud serving with many concurrent requests, autoregressive models already saturate available compute and the parallel decoding advantage diminishes.

How does DiffusionGemma differ from speculative decoding?

Speculative decoding is an optimization applied to a standard autoregressive model: a smaller draft model guesses tokens ahead, and the target model verifies them. The output distribution is unchanged. DiffusionGemma is a different generation paradigm entirely — it creates a noisy 256-token canvas and denoises the whole block in parallel, rather than applying a decoding trick to a standard architecture.

Citations

Google's DiffusionGemma generates 256 tokens in parallel and self-corrects as it goesDiffusionGemma generates a 256-token block in parallel, reaches 1,288 tokens per second on H200, and Google acknowledges its overall output quality is lower than standard Gemma 4.
DiffusionGemma — Google Gemma (vLLM benchmark results)At batch size 1 on a single H100, the FP8 version reaches approximately 1,008 tokens per second — roughly five times a standard autoregressive baseline; on H200, roughly six times, per vLLM benchmark results.
DiffusionGemma Sudoku fine-tuning resultAfter fine-tuning on a Sudoku dataset, DiffusionGemma reached an 80% success rate and converged in 12 denoising steps rather than 48, compared to 0% for the base model.
DiffusionGemma architecture: 26B MoE, 3.8B active parameters, 18GB VRAMDiffusionGemma runs as a 26B Mixture of Experts model that activates only 3.8B parameters during inference and fits within 18GB VRAM on consumer hardware.