Groq

Overview

Groq is the fastest LLM inference platform on the public market in 2026 — the developer-facing product built on top of the company's custom LPU (Language Processing Unit) hardware, delivering open-weight models like Llama 4, Qwen, and DeepSeek at throughput numbers that closed-provider APIs on GPUs cannot match. If you have ever wondered what a chat app feels like when tokens stream faster than a human can read, Groq is where you find out.

Founded in 2016 by Jonathan Ross (one of the original Google TPU architects), Groq spent six years building a deterministic inference chip before pivoting into public API access in 2024 and becoming the de facto latency benchmark the rest of the industry gets measured against. The LPU is not a GPU repurposed for inference — it is a purpose-built ASIC with a compiler-scheduled memory hierarchy, which is why the throughput numbers on 70B-parameter models routinely exceed 400 tokens per second and often clear 800 on smaller ones.

The one-line positioning: Groq is the pick when latency is the product — when the difference between a 200 ms and a 2-second time-to-first-token changes what users can build. It does not host GPT-5 or Claude — the platform is open-weight-only by design — but for the open-weight tier, it is the fastest option any developer can access with a credit card.

The product surface is deliberately minimal: an OpenAI-compatible chat completions endpoint, a small catalog of hosted open-weight models, JSON mode, function calling, and Whisper for speech-to-text. Point your existing OpenAI SDK at https://api.groq.com/openai/v1, change the model name and the API key, and you are running. That drop-in compatibility is intentional — Groq's bet is that low switching cost plus dramatic speed advantage will win a slice of the workload no other provider can serve.

Key Features

Groq's feature set is narrow on purpose. The parts that matter:

• LPU-backed inference at 400-1000+ tokens per second. Llama 3.3 70B streams at roughly 400 tokens/sec, Llama 3.1 8B at 750+, and Llama Guard and smaller specialist models frequently break 1000 tokens/sec. Compare against 60-90 tokens/sec typical on frontier closed APIs and the interactive-UX implications are obvious.
• Sub-second time-to-first-token in most regions. TTFT on Groq typically lands between 150 and 300 ms — fast enough that streaming visibly starts before a user finishes reading the previous line. Artificial Analysis and independent benchmarks consistently rank Groq at the top of latency leaderboards.
• Open-weight model catalog. Llama 4 8B and 70B, Llama 3.3 70B and Llama 3.1 8B, DeepSeek R1 distillations, Qwen 2.5, Mixtral 8x7B, and a rotating catalog of smaller specialists. When Meta or DeepSeek ships a new open model, Groq is typically among the first hosts to serve it in production.
• OpenAI-compatible API surface. Chat completions endpoint that accepts the same request shape as OpenAI. Existing SDK code migrates by changing three values (base URL, API key, model). For teams already on openai, this is the cheapest migration in the market.
• Function calling and JSON mode. Native tool-use with the same request/response shape as OpenAI. Not as sophisticated as Anthropic's tool use, but sufficient for most agentic workloads.
• Whisper large-v3 for transcription. Whisper hosted on LPUs runs 100-200x realtime — a one-hour audio file transcribes in about 30 seconds. This alone makes Groq the pick for high-throughput STT workloads, independent of the LLM story.
• Free tier with generous rate limits. Every account starts on a rate-limited free tier that is genuinely usable for hobby and prototype work — no credit card required to get real experience with the platform.
• Deterministic inference. Because of the LPU's compiler-scheduled architecture, inference is bit-reproducible for a given seed and prompt. For evaluation harnesses and regression testing, that determinism is a real quality-of-life feature.

Pricing

Groq is freemium with a paid tier that scales linearly with usage. Approximate per-million-token prices in mid-2026 for the flagship open-weight models:

Model	Input ($/M)	Output ($/M)	Speed (tok/s)
Llama 4 70B	$0.59	$0.79	~350
Llama 4 8B	$0.05	$0.08	~800
Llama 3.3 70B	$0.59	$0.79	~400
Llama 3.1 8B	$0.05	$0.08	~750
DeepSeek R1 distill Llama 70B	$0.75	$0.99	~275
Qwen 2.5 32B	$0.29	$0.39	~400
Mixtral 8x7B	$0.24	$0.24	~500
Whisper large-v3	$0.02/audio-hour	—	~200x realtime

Free tier: rate-limited to roughly 30 requests per minute and a moderate daily token budget, sufficient for evaluation and hobby projects. Paid tier: usage-based per the table above, with dedicated capacity available for enterprise customers who need reserved throughput.

The pricing that actually matters: Llama 3.3 70B at $0.59/$0.79 with 400 tok/sec streaming is roughly one-fifth the price of GPT-5 mini and 5x the throughput. For workloads where Llama-3.3-70B-tier quality is enough — most classification, extraction, moderation, and light chat — Groq is the cheapest and fastest option available in production. The quality gap versus GPT-5 flagship is real and matters for some workloads, and irrelevant for many others.

Pros and Cons

Pros

• Fastest LLM inference on the public market — 5-10x the throughput of GPU-hosted equivalents
• Sub-second time-to-first-token changes what interactive UX can feel like
• OpenAI-compatible API means near-zero migration cost from existing SDK code
• Prices on open-weight models undercut closed-API-hosted equivalents by 3-10x
• Generous free tier lets you evaluate before paying — a rarity in this market
• Whisper large-v3 at 100-200x realtime is genuinely differentiated for STT workloads
• Deterministic inference is a real testing-and-evaluation advantage

Cons

• No frontier closed models — no GPT-5, no Claude, no Gemini. If you need frontier quality, Groq is not the platform
• Model catalog is narrower than Together AI or Hugging Face — Groq curates rather than hosts everything
• Rate limits on the paid tier are conservative for new accounts and require capacity requests to unlock
• No fine-tuning — you use the base open-weight models as shipped
• No native embeddings model — pair with OpenAI, Voyage, or a self-hosted embeddings service
• Context windows on hosted models track the underlying open-weight defaults (typically 128K on Llama 3.3-4), smaller than Claude Sonnet's 1M
• Latency advantage matters less if your bottleneck is your database, retrieval layer, or tool calls

Best Use Cases

• Interactive UX where streaming latency matters. Chat interfaces, coding assistants, voice agents, live translation. When 400 tok/sec versus 60 tok/sec is the difference between a fluent user experience and a laggy one, Groq wins the comparison.
• High-volume classification and extraction. Route the top of your ticket queue to Llama 4 8B at $0.05/M input and process a million tickets a day for under $50. This is the workload where Groq's price-performance dominates.
• Voice agents and real-time transcription pipelines. Whisper at 200x realtime combined with a Llama-3.3-70B response at 400 tok/sec gets you from audio-in to audio-out in under a second — the latency envelope needed for phone-quality voice interfaces.
• Agentic loops with many small calls. Agents that do 10-20 LLM calls per user interaction benefit disproportionately from per-call latency reduction. Groq's throughput compounds across the loop.
• Prototyping and hackathon builds. The free tier plus OpenAI compatibility means you can ship a working prototype in an evening without spending anything. Once traffic grows, the paid tier scales without a rewrite.
• Cost-sensitive B2B tools where Llama quality is enough. Internal tools, batch jobs, and customer-facing features where GPT-5-flagship-tier quality is not the point.

Alternatives

Groq competes on two axes — latency and open-weight price — and different competitors win each:

• Together AI — broader open-weight catalog, dedicated endpoints, fine-tuning, and per-token prices in the same neighborhood as Groq. Slower on throughput but wins on model breadth and customization.
• Hugging Face — inference endpoints for any of the 1M+ hosted models. Pick this if you need a specific model Groq or Together does not host.
• OpenAI API — GPT-5 family. Pick this if you need frontier quality and can accept lower throughput.
• Anthropic API — Claude Sonnet 4.5 and Opus 4.7. Pick this for writing quality, coding depth, or long-context work.
• Cerebras Inference — the closest direct competitor on the "custom silicon for inference" thesis, with comparable throughput on smaller model tiers. Groq's ecosystem and pricing tend to be more mature in 2026.
• Fireworks AI — another open-weight fast-inference host, competitive on price and speed. The delta versus Groq is small and workload-dependent — worth benchmarking on your own traffic.

For consumer-side comparisons of the frontier models Groq does not host, see ChatGPT, Claude, and Gemini.

Getting Started

1. Sign up at console.groq.com and generate an API key. No credit card required for the free tier — you get real rate-limited access to every hosted model immediately.
2. Point your OpenAI SDK at Groq. For the Python openai SDK: client = OpenAI(base_url="https://api.groq.com/openai/v1", api_key=os.environ["GROQ_API_KEY"]). Change the model name to llama-3.3-70b-versatile or similar and your existing code works.
3. Benchmark your actual workload. Compare tokens/sec, time-to-first-token, and end-to-end cost against your current provider on real traffic. The latency delta is easy to measure and the cost delta usually surprises people in a good way.

If you are moving production traffic, request a rate limit increase early — the default tier is conservative and enterprise capacity is available on request. For voice pipelines, benchmark Whisper large-v3 against your current STT provider first; the price-performance advantage is often the biggest single win.

FAQ

Does Groq host GPT-5 or Claude? No. Groq is open-weight-only by design. If you need GPT-5, use the OpenAI API; for Claude, use the Anthropic API.

How does Groq get such high throughput? Custom LPU hardware — an ASIC designed specifically for LLM inference with a compiler-scheduled memory hierarchy that avoids the memory-bandwidth bottleneck GPUs hit at long sequence lengths. This is a hardware advantage, not a software trick.

Is Groq the same as Grok (X.ai)? No — completely different companies. Groq is the inference platform founded by Jonathan Ross. Grok is Elon Musk's chatbot at X.ai. The name collision is unfortunate for both.

What's the largest context window on Groq? Track the underlying open-weight model. Llama 4 supports 128K on Groq as of mid-2026; Llama 3.3 supports 128K; DeepSeek distills typically expose 32-64K. Larger context windows exist on other platforms (Claude Sonnet at 1M, Gemini at 2M).

Can I fine-tune models on Groq? No. Groq serves the base open-weight models as released. For fine-tuning open-weight models, use Together AI, Hugging Face Inference Endpoints, or run your own fine-tuning pipeline.

Verdict

Groq is the pick when latency is the product. For any workload where sub-second time-to-first-token or 400+ tokens/sec streaming changes the user experience, Groq is the cheapest and fastest way to get there in 2026. The combination of custom LPU silicon, aggressive open-weight pricing, and OpenAI-compatible API surface makes it the easiest experiment any team can run: five minutes to migrate, immediate throughput gains, often a 3-5x cost reduction on the same workload.

Where Groq stops being the obvious choice is on the two things the platform deliberately doesn't do: frontier quality (no GPT-5, no Claude, no Gemini) and model customization (no fine-tuning, curated catalog only). Teams whose products depend on the sharpness of the frontier models cannot substitute Llama 4 70B for GPT-5 flagship and expect the same output. Teams that need a specific fine-tuned model will look at Together AI or Hugging Face instead.

The honest recommendation for most teams in 2026: treat Groq as the fast tier of a multi-provider stack. Route latency-sensitive interactive calls, high-volume classification, voice pipelines, and simple agent loops to Groq — where the open-weight tier is good enough and the speed matters. Keep frontier providers on the calls where quality is the point. Almost every serious AI product with real-time UX in 2026 uses Groq for at least one part of its stack, and the ones that don't usually discover the missing latency budget after their first user-experience review.

Explore more in the Developer API category, or read The Economics of AI Inference at Scale for the broader token-pricing landscape.