Skip to content
WildRun AI home Get Your Free Audit
Tools & APIs · June 26, 2026 · Updated November 8, 2026 · 12 min read · WildRun AI Engineering

Best LLM for Voice Agents in 2026: A Developer's Comparison

Compare GPT-4.1, Claude Sonnet 4.6, Gemini 2.5 Flash, and Groq for voice AI: response speed, function calling, cost, and TypeScript you can adapt.

Retro-style illustration features a green megaphone with telephone handset emitting sound waves beside an open-engine unit.

The STT and TTS layers in a voice pipeline are largely commoditized — Deepgram Nova-3 and ElevenLabs Flash v2.5 are common defaults in 2026. The variable that actually separates a voice agent that feels natural from one that feels laggy is the LLM in the middle. Pick wrong and every turn can feel slow, tool calls can misfire, or your bill can grow fast at scale.

This post compares the five models worth considering for production voice agents today: GPT-4.1, Claude Sonnet 4.6, Gemini 2.5 Flash, Groq + Llama 3.3 70B, and Claude Haiku 4.5. Rough TTFT numbers, function-calling notes, cost at scale, and the TypeScript to wire each one into a Vapi pipeline.

The constraint set that makes voice different

Voice agents operate inside a latency budget that chat interfaces never face. From the moment STT emits its final transcript, your LLM has roughly 200–700ms before the TTS pipeline can produce audio. Anything past 700ms TTFT can sound like an unnatural pause to callers, and delays over 1 second tend to frustrate them. All latency and speed figures in this post are rough planning numbers, not WildRun benchmarks. They vary by region, time of day and prompt, so measure your own.

That budget breaks down roughly like this (typical figures; measure your own):

  • STT finalization: ~100ms (Deepgram Nova-3)
  • LLM time to first token: 150–500ms — the variable you control
  • TTS first audio chunk: ~80ms (ElevenLabs Flash v2.5)
  • Telephony and network round-trip: ~60ms

Two traps reliably kill TTFT in production. First: reasoning mode. Every frontier model in 2026 ships a reasoning or thinking toggle. That toggle is off for voice — without exception. Reasoning mode can add many seconds before the first token on complex prompts. That is not a slow turn; it is a dropped call. Second: unbounded context growth. Voice transcripts accumulate fast. Feeding thousands of tokens of conversation history into every LLM call adds delay compared to a trimmed 1,500-token window. Keep turn-by-turn context lean and summarize aggressively at turn 10.

Vapi times out a turn that takes too long to respond. Aim for a median TTFT well under 1 second and a slow-case (95th percentile, or P95) TTFT under 2 seconds. That leaves headroom for slow networks.

The five models worth evaluating

GPT-4.1 — the production default

TTFT: ~450ms median, ~800ms P95. Throughput: ~125 tok/s. Cost: $2.00 input / $8.00 output per million tokens.

GPT-4.1 is a common choice for production Vapi deployments, and the reasons are mundane: it works reliably. In our experience it follows instructions well on noisy speech-to-text transcripts: filler words, stumbles and mid-sentence restarts. Function calling usually fires correctly on the first attempt. Tool call failures in a voice agent mean dead air while the agent retries, so a low failure rate matters. The 1M-token context window is overkill for single calls but useful if you inject lengthy service catalogs or knowledge bases into your system prompt.

The weakness is cost. At $8.00/M output tokens, a 10-minute call generating around 4,000 output tokens per minute runs to roughly $0.32/call on the LLM alone. That is before STT, TTS, Vapi's platform fee (listed by third-party reviews at about $0.05/min; check current rates), and telephony. Those costs add up quickly at volume.

Claude Sonnet 4.6 — best tail-latency predictability

TTFT: ~500ms median, ~900ms P95. Throughput: ~90 tok/s. Cost: $3.00 input / $15.00 output per million tokens.

Sonnet 4.6's median TTFT is slightly slower than GPT-4.1 in our estimates, but we expect its slow cases to stay closer to the median. In practice that means fewer calls where the agent suddenly pauses for 2+ seconds. Verify this in your own tests. For voice, tail latency (the slowest turns) matters more than median latency: one bad turn can end the call. Sonnet 4.6 also handles complex multi-step reasoning inside a single turn well — dynamic intake questions that require branching on caller responses, or instructions that require reading between the lines of ambiguous speech.

Output cost is the limiting factor: $15.00/M output tokens is nearly double GPT-4.1. For a typical intake agent generating 300 output tokens per turn × 20 turns per call, that is ~$0.09/call on LLM output alone. Fine at low volume; significant at 10,000 calls/month.

Gemini 2.5 Flash — the cost-performance option

TTFT: ~600ms median (endpoint-dependent). Throughput: ~204 tok/s — fastest of any tested provider. Cost: $0.30 input / $2.50 output per million tokens (Google's pricing page; check current rates).

Gemini 2.5 Flash changes the unit economics significantly. At $2.50/M output tokens, it costs roughly one-third of GPT-4.1 and one-sixth of Claude Sonnet 4.6 on output. For high-volume commodity tasks — appointment confirmation, FAQ deflection, basic lead triage — the quality difference between Gemini Flash and GPT-4.1 may not justify the cost gap. Its fast output speed also means shorter waits on longer replies.

The caveats matter. Slow-case TTFT can get much worse on the Google AI Studio public endpoint during busy US business hours. Using Vertex AI with a regional endpoint (us-west1 is the closest to Central Oregon) should tighten that. Function calling accuracy on complex nested tool schemas also lags behind GPT-4.1. Test your specific tool definitions before committing at production scale.

Groq + Llama 3.3 70B — the speed outlier

TTFT: ~180ms median (8B model), ~250ms median (70B). Throughput: 500+ tok/s. Cost: $0.59 input / $0.79 output per million tokens (Llama 3.3 70B).

Groq's LPU (Language Processing Unit) hardware produces inference speeds that GPU-based providers cannot match. At 180ms median TTFT on the 8B model, Groq creates genuine headroom in your latency budget — you can add a CRM lookup before the LLM call and still hit under 700ms total end-to-end. The 70B Llama 3.3 model follows instructions well enough for structured workflows like appointment booking and FAQ routing.

The production constraints are real. Groq's free tier allows about 30 requests per minute (RPM) on the 70B model. Paid tiers raise this, but limits still cap how many calls you can run at once. Complex parallel tool calls are also less consistent on open-weight models than on GPT-4.1. Use Groq for simple, single-tool workflows or as a speed-optimized fallback path — not as the primary model for complex intake flows.

Claude Haiku 4.5 — the economical Anthropic option

TTFT: Sub-600ms median. Cost: $1.00 input / $5.00 output per million tokens (Anthropic).

Haiku 4.5 sits between Sonnet 4.6 and Gemini Flash in both capability and cost. It follows instructions well on speech-to-text transcripts. It is the right tier when you need Anthropic's API and a HIPAA BAA (Business Associate Agreement, a contract healthcare data rules require) but cannot absorb Sonnet 4.6 pricing at scale. Confirm with Anthropic that your model and setup are covered. Function calling is reliable on simple single-tool schemas; less consistent on deeply nested multi-tool flows where Sonnet 4.6 is the safer choice.

Pipeline architecture

The most cost-efficient production deployments do not pick a single model for every call — they route by turn complexity. A caller asking to reschedule an appointment is structurally different from one who needs you to look up their insurance details, validate the policy number, and branch on three possible eligibility outcomes.

textVoice pipeline with LLM routing
Caller speech
     │
     ▼
[Deepgram Nova-3 STT]  ~100ms
     │ transcript + metadata
     ▼
[Complexity Classifier]  <20ms (regex/keyword — no LLM)
  ├─ simple FAQ / booking   ──► Gemini 2.5 Flash    ~600ms TTFT
  ├─ multi-step / tool-heavy ─► GPT-4.1            ~450ms TTFT
  └─ HIPAA intake / complex ──► Claude Sonnet 4.6  ~500ms TTFT
     │
     ▼
[LLM response + tool calls]
     │
     ▼
[ElevenLabs Flash v2.5 TTS]  ~80ms
     │
     ▼
Caller hears audio
Total target: <750ms end-to-end

The classifier itself must be cheap and fast — a regex or keyword pass over the transcript keeps classification under 20ms. Do not route the routing decision through a second LLM call.

Connecting models in Vapi

Vapi's assistant configuration accepts a model object that lets you swap providers with a single field change. Here is the base pattern covering all three primary models. A routing function picks one based on call complexity:

TypeScriptVapi assistant model configs
import Vapi from "@vapi-ai/server-sdk";

const vapi = new Vapi({ token: process.env.VAPI_API_KEY! });

const SYSTEM_PROMPT = `You are a helpful scheduling assistant.
Be concise — this is a phone call, not a chat.
Speak in complete sentences. Never read back JSON or code.`;

// GPT-4.1 — production default for complex flows
const gpt41Config = {
  provider: "openai" as const,
  model: "gpt-4.1",
  temperature: 0.3,
  maxTokens: 400,
  messages: [{ role: "system" as const, content: SYSTEM_PROMPT }],
};

// Gemini 2.5 Flash — high-volume, cost-sensitive paths
const geminiFlashConfig = {
  provider: "google" as const,
  model: "gemini-2.5-flash",
  temperature: 0.2,
  maxTokens: 300,
  messages: [{ role: "system" as const, content: SYSTEM_PROMPT }],
};

// Claude Sonnet 4.6 — HIPAA intake, complex branching logic
const sonnet46Config = {
  provider: "anthropic" as const,
  model: "claude-sonnet-4-6",
  temperature: 0.2,
  maxTokens: 500,
  messages: [{ role: "system" as const, content: SYSTEM_PROMPT }],
};

type Complexity = "simple" | "complex" | "sensitive";

function selectModel(complexity: Complexity) {
  if (complexity === "sensitive") return sonnet46Config;
  if (complexity === "complex") return gpt41Config;
  return geminiFlashConfig;
}

export async function createVoiceAssistant(complexity: Complexity) {
  return vapi.assistants.create({
    name: `intake-agent-${complexity}`,
    model: selectModel(complexity),
    voice: {
      provider: "elevenlabs" as const,
      voiceId: "21m00Tcm4TlvDq8ikWAM",
      model: "eleven_flash_v2_5",
      stability: 0.5,
      similarityBoost: 0.75,
    },
    transcriber: {
      provider: "deepgram" as const,
      model: "nova-3",
      language: "en-US",
    },
    firstMessageMode: "assistant-speaks-first",
    firstMessage: "Thanks for calling. How can I help you today?",
  });
}

Function calling: where models diverge most

Latency benchmarks get most of the attention, but function calling reliability is the real failure mode in production. A voice agent that says "One moment while I check that" and then silently fails to call the tool — because the LLM emitted malformed JSON or picked the wrong function — produces a dead-air moment that callers interpret as the line dropping. That failure is invisible in your latency metrics.

The tool definition schema you write matters as much as the model. Keep schemas flat. Name parameters with full English words, not abbreviations. Include an example value in every description field. In our experience, all three top models do better with those practices in place.

TypeScriptTool schema that works reliably across all three providers
// This schema shape produces consistent tool calls on GPT-4.1, Sonnet 4.6, and Gemini 2.5 Flash
const checkAvailabilityTool = {
  type: "function" as const,
  function: {
    name: "check_appointment_availability",
    description:
      "Check whether a specific appointment slot is open. " +
      "Call this whenever the caller asks about scheduling, rescheduling, or availability. " +
      "If the caller mentions a day without a specific time, ask for the time before calling.",
    parameters: {
      type: "object" as const,
      properties: {
        requested_date: {
          type: "string",
          description:
            "The date the caller wants, in YYYY-MM-DD format. Example: '2026-07-15'",
        },
        requested_time: {
          type: "string",
          description:
            "The time slot in HH:MM 24-hour format. Example: '14:00'",
        },
        service_type: {
          type: "string",
          enum: ["cleaning", "exam", "emergency", "consultation"],
          description:
            "Type of appointment. Infer from context if not explicitly stated.",
        },
      },
      required: ["requested_date", "requested_time", "service_type"],
    },
  },
};

export async function createAgentWithTools(complexity: Complexity) {
  return vapi.assistants.create({
    name: `intake-agent-tools-${complexity}`,
    model: {
      ...selectModel(complexity),
      tools: [checkAvailabilityTool],
      toolChoice: "auto",
    },
    voice: {
      provider: "elevenlabs" as const,
      voiceId: "21m00Tcm4TlvDq8ikWAM",
      model: "eleven_flash_v2_5",
    },
    transcriber: {
      provider: "deepgram" as const,
      model: "nova-3",
      language: "en-US",
    },
    firstMessageMode: "assistant-speaks-first",
    firstMessage: "Thanks for calling. How can I help you today?",
  });
}

Dynamic routing via Vapi webhook

Rather than hardcoding one model per assistant, you can route at the start of each call using Vapi's assistant-request webhook event. When this event fires, you receive the caller's number and any metadata attached to the call — enough to pick a model tier before the first LLM call executes.

TypeScriptVapi webhook — dynamic model routing
import { Hono } from "hono";

interface VapiWebhookPayload {
  message: {
    type: string;
    call: {
      customer?: { number?: string };
      metadata?: Record<string, string>;
    };
  };
}

const app = new Hono();

// Classify complexity in <5ms — no LLM needed here
function classifyComplexity(
  callerNumber: string,
  metadata: Record<string, string>
): Complexity {
  // Known patients flagged in your PMS metadata → HIPAA-eligible model
  if (metadata.patientStatus === "existing") return "sensitive";
  // After-hours calls in Central Oregon tend toward urgent or non-routine
  // PDT is UTC-7; business hours roughly map to 15:00–01:00 UTC
  const utcHour = new Date().getUTCHours();
  const isAfterHours = utcHour < 15 || utcHour >= 1;
  if (isAfterHours) return "complex";
  return "simple";
}

app.post("/vapi/webhook", async (c) => {
  const payload = await c.req.json<VapiWebhookPayload>();

  if (payload.message.type === "assistant-request") {
    const { call } = payload.message;
    const complexity = classifyComplexity(
      call.customer?.number ?? "",
      call.metadata ?? {}
    );
    const assistant = await createVoiceAssistant(complexity);
    return c.json({ assistantId: assistant.id });
  }

  return c.json({ received: true });
});

export default app;

Production gotchas

Reasoning mode silently wrecks latency. OpenAI exposes a reasoning_effort parameter; Anthropic uses a thinking block in the request body. If you copy config from a coding or analysis project, verify neither is set. Even a low reasoning setting can add seconds of TTFT. Log your actual TTFT per turn in production — do not rely on local testing where reasoning latency is masked by IDE response caching.

Gemini regional variance is significant. Gemini 2.5 Flash on the public AI Studio endpoint can be much slower in the slow cases during US business hours than Vertex AI us-west1. If you benchmark on AI Studio and deploy to Vertex — or vice versa — your production latency profile will not match your test results. Pick one surface and stay on it.

Streaming mode changes your effective latency. When streaming is enabled, Vapi can begin the TTS pipeline as soon as the first non-tool-call tokens arrive from the LLM. If streaming is off, Vapi waits for the complete LLM response. Check your Vapi plan and explicitly set stream: true in any direct API calls outside Vapi's SDK. This single setting can cut the delay callers notice on each turn.

Context window creep is a hidden latency tax. A long call piles up thousands of tokens of transcript. At turn 20, you are paying to re-process the entire call history on every LLM invocation. Implement rolling summarization at turn 10: compress turns 1–8 into a single summary message and drop the raw turn objects. This alone can cut TTFT on long calls with any of these models.

Anthropic's BAA is not automatic compliance. Signing a Business Associate Agreement with Anthropic can make your Claude-based deployment HIPAA-eligible. It does not make it HIPAA compliant. Your call transcript logging, storage, retention, and access controls must also be configured correctly. Vapi stores call transcripts on its own infrastructure by default; review Vapi's data handling agreements separately from your LLM provider's BAA.

Groq rate limits are set per model. The free tier for Llama 3.3 70B sits around 30 RPM, and paid tiers are higher but still capped. At 50 concurrent voice calls, you can hit the ceiling and see throttling errors. In a voice pipeline those show up as long delays and timeouts. Build a fallback to GPT-4.1 mini on Groq throttle errors, or negotiate higher limits before launch.

When NOT to build this yourself

Multi-model LLM routing adds real operational complexity. You now maintain three provider SDKs, three sets of API keys, three billing dashboards, and a routing layer that can fail silently. Before committing to this architecture, answer these questions honestly:

Is your call volume under 3,000 calls/month? At that volume, the dollar gap between GPT-4.1 and Gemini Flash is usually small next to the engineering time to build and maintain a routing layer. Pick one model, tune your prompt, and ship.

Are your tool schemas simple? If your agent books appointments and does nothing else, GPT-4.1's function calling accuracy is high enough that the reliability advantage over cheaper models does not translate to meaningful caller experience improvement.

Do you have monitoring in place? Multi-model routing fails silently in ways that single-model deployments do not. If your classifier is sending 15% of calls to the wrong tier and you cannot see that in your dashboards, the routing layer is actively degrading quality. Build routing only after you have per-turn TTFT logging and tool-call success rate tracking already running.

For most small businesses — medical practices, law firms, home services contractors across Central Oregon — a single well-configured GPT-4.1 agent on Vapi with a tested system prompt outperforms a poorly instrumented multi-model setup. See the complete voice agent latency optimization guide for the full pipeline tuning approach before adding LLM routing complexity.

For new production deployments, as of November 2026 (check current models and prices first):

  • STT: Deepgram Nova-3 via Vapi built-in
  • LLM default: GPT-4.1 — reliable function calling, predictable latency, 1M context window
  • LLM at scale (>3,000 calls/month): Gemini 2.5 Flash on Vertex AI us-west1, after validating your tool schemas
  • LLM for HIPAA intake: Claude Sonnet 4.6 — predictable slow-case latency plus Anthropic BAA availability (confirm your model is covered)
  • LLM for speed-critical simple paths: Groq + Llama 3.3 70B with a GPT-4.1 fallback on rate-limit errors
  • TTS: ElevenLabs Flash v2.5 — built for low latency (check ElevenLabs' own figures and test in your pipeline)
  • Context: Rolling summarization at turn 10, maximum 1,500 tokens of active turn history

If you are building a voice pipeline for a client rather than your own business, a team that has already worked through the production edge cases can handle the multi-model routing for you. WildRun AI can build voice pipelines like this for Central Oregon businesses, scoped and priced in writing per job. See our AI automation services or contact us to talk it through.

Architecture
Voice Call Pipeline — LLM Routing

Caller speech
     |
     v
[Deepgram Nova-3 STT]  ~100ms
     | transcript + metadata
     v
[Complexity Classifier]  <20ms (regex/keyword, no LLM)
  +- simple FAQ / booking   --> Gemini 2.5 Flash    ~600ms TTFT
  +- multi-step / tool-heavy --> GPT-4.1            ~450ms TTFT
  +- HIPAA intake / complex --> Claude Sonnet 4.6   ~500ms TTFT
     |
     v
[LLM response + tool calls]
     |
     v
[ElevenLabs Flash v2.5 TTS]  ~80ms
     |
     v
Caller hears audio
Total target: <750ms end-to-end

Frequently asked questions

What is the best LLM for voice agents in 2026?

GPT-4.1 is a safe default for most production voice agents: reliable function calling, roughly 450ms median time to first token (TTFT, the delay before the first word of a reply), and a 1M-token context window. For high-volume, cost-sensitive deployments, Gemini 2.5 Flash on Vertex AI costs about one-third as much on output tokens and may be enough for simple conversations. Test both on your own calls.

How much latency budget does the LLM have in a voice pipeline?

Roughly 200 to 700ms from when speech-to-text finalizes the transcript to when text-to-speech can start producing audio. That window includes your LLM's TTFT. Above 700ms, callers tend to hear an unnatural pause, and above 1 second they tend to get frustrated.

Can I use reasoning models for voice agents?

No. Reasoning and thinking modes can add many seconds before the first token, which makes them unusable for real-time voice. Turn reasoning off on any model you use for voice, and check that it is off. Some API client library defaults vary by version.

Is Groq production-ready for voice agents?

For simple workflows with a single tool call, yes. Groq is known for very fast inference, and a Llama model on Groq can leave extra room in your latency budget. The limits are tighter rate limits on the free tier and less consistent multi-tool function calling on open-weight models.

Which LLM should I use for HIPAA-eligible healthcare voice agents?

Claude Sonnet 4.6 is one reasonable choice for HIPAA-adjacent deployments. Anthropic signs Business Associate Agreements (BAAs) for eligible API use, so confirm your model and setup are covered. Signing the BAA can make your deployment HIPAA-eligible, but your logging, storage, and access controls must also be set up correctly.

When does multi-model LLM routing make sense for voice agents?

Once your call volume is high enough that the cost gap matters, and you have per-turn latency and tool-call-success monitoring in place. Below roughly 3,000 calls per month, the savings are unlikely to justify the added work of maintaining multiple provider integrations and a routing layer that can fail silently.

Is your business showing up in AI answers?

Request a free review of your Google, Maps, website and AI-search visibility from WildRun, an SEO, AEO and GEO studio in Bend, Oregon.

Get Your Free Visibility Audit