Skip to content
NewsProduct launch

NVIDIA Groq 3 LPX enters full production with record 3,400 tokens per second for agentic AI inference

· by Pondero Newsdesk

The short version

NVIDIA's Groq 3 LPX inference accelerator hit a benchmark record of 3,400 output tokens per second on Gemma 4 31B with a 100K-token context. Nebius is first to deploy; Groq inference cloud follows.

NVIDIA Groq 3 LPX enters full production with record 3,400 tokens per second for agentic AI inference

The record-breaking number is 3,400 output tokens per second on Gemma 4 31B running a 100,000-token context. That figure, from Artificial Analysis benchmarking, sits at the center of NVIDIA's August 24 announcement that its Groq 3 LPX inference accelerator entered full production. The harder story is why output-token speed matters more for agents than for human-facing chat, and what reaching this benchmark at full-context length changes for teams building multi-step pipelines.

What

NVIDIA announced on August 24, at the Hot Chips conference, that the Groq 3 LPX is now in full production, per the NVIDIA Newsroom press release. The chip extends the Vera Rubin platform, which already powers NVIDIA's NVL72 rack-scale systems for training and general inference. The LPX variant concentrates its architecture on a single phase of inference: the decode step, which determines how fast tokens are generated for an individual user or agent loop.

In Artificial Analysis benchmarking, the Groq 3 LPX delivered 3,400 output tokens per second running Gemma 4 31B at a 100,000-token context. NVIDIA described that result as the fastest performance ever recorded for the model. Per NVIDIA, the chip delivers 4x faster responsiveness for agents and latency-sensitive workloads than the nearest alternative platform. NVIDIA did not identify the competing platform or cite a third-party source for the 4x figure.

Nebius is the first AI cloud to commit to deploying Groq 3 LPX in production, through its Token Factory platform. The inference cloud Groq, which runs separate infrastructure at groq.com, said it plans to be among the earliest adopters, per a Groq blog post linked from the NVIDIA announcement.

Why it matters

Token generation speed is largely irrelevant for human-in-the-loop chat. Humans reading responses cannot process 3,400 tokens per second; 50 is plenty. The math shifts entirely for agentic systems.

A coding agent completing a real task, locating a bug in a large codebase, drafting a fix, running tests, and iterating, can require hundreds of sequential inference steps. Each step generates output that becomes input for the next. For a task requiring 200 steps at 500 output tokens each, the generation phase alone produces 100,000 tokens. At 3,400 t/s compared to a 500 t/s baseline, the time spent on that generation phase drops by roughly 85%. Wall-clock completion time depends on many factors beyond generation speed, but generation is typically the dominant component in extended agent loops.

The 100,000-token context length in the benchmark adds a second relevant data point. Agentic systems frequently carry long context windows to maintain state across steps. A chip that sustains high output speed at 100K tokens of context rather than degrading at longer windows is more practically useful for production agent deployments than a chip optimized only for short-context tasks.

For operators running agentic pipelines today, Groq 3 LPX's entry into production means the performance ceiling on the decode phase just moved. Access comes through Nebius Token Factory initially and through the Groq inference cloud once that deployment lands.

Context: Vera Rubin and the decode bottleneck

NVIDIA's Vera Rubin platform was positioned as a modular AI factory architecture from launch. The NVL72 configuration handles both training and general inference. The LPX is an additive extension for workloads where output generation speed, not training throughput or prefill speed, is the binding constraint.

The naming creates a potential source of confusion. "NVIDIA Groq 3 LPX" uses the "Groq" brand from NVIDIA's acquisition of Groq Inc., the company that originally developed the LPU (Language Processing Unit) tensor streaming processor. The inference cloud at groq.com is a separate entity that built its product on early Groq LPU hardware and is now a customer and early adopter of the NVIDIA Groq 3 LPX, not the same company.

Jensen Huang, NVIDIA's founder and CEO, said in the announcement: "Vera Rubin extends that vision with workload-optimized AI factory configurations designed for the era of agentic AI, advancing the performance frontier with LPX for ultrafast token generation."

Danila Shtan, Nebius's chief technology officer, said generation is "the phase of inference that determines how responsive an AI system actually is," and described the Nebius deployment as delivering instant-feeling agent loops through the same API developers already use, with no migration to a new stack.

Cloud adoption and access

Nebius and the Groq inference cloud are the two confirmed early access points. NVIDIA has not published a broader general-availability timeline or third-party access pricing for the Groq 3 LPX. Both providers run managed inference APIs, so developers would access the chip's throughput through existing SDK or REST calls rather than direct hardware procurement.

The Token Factory designation is specific to Nebius's production inference product. The Groq inference cloud has announced intent but has not published a date for Groq 3 LPX availability on its platform.

What to watch next

Whether CoreWeave, Together AI, Lambda, and other cloud inference providers announce Groq 3 LPX integration timelines is the next concrete signal. Broad adoption across the cloud layer would set a new practical floor for decode performance across the industry, making the 3,400 t/s result a commodity rather than a differentiated feature. The published benchmark covers Gemma 4 31B; performance figures on larger models such as Llama 4 Maverick would give enterprise operators running bigger-context tasks a clearer picture of real-world throughput at the scales they actually deploy.

Sources