NVIDIA Groq 3 LPX Hits 3,431 Tokens/Sec on Gemma 4

NVIDIA has demonstrated the capabilities of its new Groq 3 LPX inference accelerator, paired with the Vera Rubin NVL72 platform. In a third-party evaluation by Artificial Analysis, the setup clocked a median speed of 3,431 output tokens per second on a 100K context benchmark using the Gemma 4 31B dense model. When tested at a shorter 10K context length, the system maintained a similar median speed of 3,382 tokens per second, showing minimal latency variation. On the coding-specific SPEED-Bench benchmark, the system reached a median speed of 4,767 tokens per second, with a P80 of 5,520 tokens per second.

To achieve these speeds without losing precision or model quality, the system relies on a deterministic execution model. The Groq 3 LPX architecture features 256 LP30 local processing units (LPUs), 128 GB of total SRAM-based memory, and 96 chip-to-chip links per chip operating at 112 Gbps. A specialized compiler maps out data transfers down to the clock cycle before execution, eliminating real-time routing conflicts. It also overlaps computation and communication at the level of 320-byte vectors, allowing LPUs to transmit matrix multiplication slices immediately rather than waiting for entire operations to finish.

For developers building multi-turn agentic systems, this architecture solves the high latency typically caused by tensor parallelism at small batch sizes. Instead of waiting 50 seconds to generate 5,000 tokens at a standard rate of 100 tokens per second, practitioners can receive the same output in roughly 1.5 seconds. The platform supports co-execution configurations with the Vera Rubin NVL72, including prefill-decode disaggregation, attention-FFN disaggregation, and speculative external-drafter decoding. These hybrid setups allow the hardware to scale to multi-trillion parameter models, such as a projected 2-trillion-parameter GPT-OSS model.