How to Build an AI Voice Agent with Vapi in TypeScript

Vapi is a voice orchestration platform that wires together speech-to-text, a language model, and text-to-speech into a single real-time pipeline. Instead of managing three separate API contracts and stitching audio streams yourself, you configure an assistant object and Vapi handles the telephony, codec negotiation, latency management, and interruption detection. This guide walks through a complete TypeScript implementation — from creating your first assistant to handling tool calls and surviving production.

The end result is a phone agent that answers inbound calls, understands natural language, calls your server when it needs to take action (book an appointment, look up an account), and responds in a human-sounding voice. On a well-tuned stack, end-to-end latency lands between 500ms and 900ms — close enough to human response time that most callers will not notice the difference.

What you are building

The architecture has three distinct layers: Vapi's managed orchestration, your assistant configuration, and a webhook server that handles tool calls. Understanding the boundary between these layers is the most important concept in this guide.

Caller (PSTN / WebRTC)
        |
        v
  Vapi Orchestration Layer
  +--------------------------------------+
  |  STT: Deepgram Nova-2    (~100ms)    |
  |  LLM: gpt-4o-mini        (~300ms)    |
  |  TTS: ElevenLabs Flash   (~75ms)     |
  +--------------------------------------+
        | POST /webhook (tool calls only)
        v
  Your Server (Express / Workers)
  +-- checkAvailability(date)
  +-- bookAppointment(name, phone, slot)
  +-- lookupAccount(callerId)

Vapi holds the phone call and audio stream at all times. Your webhook server receives structured JSON only when the LLM invokes a tool — it never touches raw audio. This separation is what keeps the implementation manageable as complexity grows.

Prerequisites

• A Vapi account — the free tier covers development and testing without needing a phone number
• Node.js 20+ with TypeScript 5+
• A publicly reachable HTTPS endpoint for webhooks — ngrok works for local development
• An inbound phone number provisioned inside the Vapi dashboard under Phone Numbers

npm install @vapi-ai/server-sdk express zod
npm install -D typescript @types/express @types/node tsx

Creating your assistant

An assistant is a persistent configuration object that defines the full voice pipeline: which STT, LLM, and TTS providers to use, the system prompt, available tools, and telephony behavior. Create it once and reference it by ID on every call — or create it dynamically per call to inject per-customer context before the conversation begins.

import Vapi from "@vapi-ai/server-sdk";

const vapi = new Vapi({ token: process.env.VAPI_API_KEY! });

async function createReceptionistAssistant(): Promise<string> {
  const assistant = await vapi.assistants.create({
    name: "Front Desk",
    model: {
      provider: "openai",
      model: "gpt-4o-mini",
      temperature: 0.3,
      systemPrompt: `You are the front desk for Cascade Physical Therapy in Bend, Oregon.
Your job: greet callers, answer questions about hours and location, and book appointments.
Keep every response under two sentences. Never say "certainly" or "absolutely".
When booking: confirm first name, callback number, and preferred day before calling bookAppointment.
If a tool call fails, say: I am having trouble with that right now — can I take a message?`,
      tools: [
        {
          type: "function",
          function: {
            name: "checkAvailability",
            description: "Returns open appointment slots for a given date",
            parameters: {
              type: "object",
              properties: {
                date: {
                  type: "string",
                  description: "ISO 8601 date string, e.g. 2026-06-15",
                },
              },
              required: ["date"],
            },
          },
        },
        {
          type: "function",
          function: {
            name: "bookAppointment",
            description: "Books a confirmed appointment slot for a caller",
            parameters: {
              type: "object",
              properties: {
                firstName: { type: "string" },
                phone:     { type: "string", description: "E.164 format, e.g. +15415551234" },
                date:      { type: "string", description: "ISO 8601 date" },
                time:      { type: "string", description: "HH:MM 24-hour format" },
              },
              required: ["firstName", "phone", "date", "time"],
            },
          },
        },
      ],
    },
    voice: {
      provider: "elevenlabs",
      voiceId: process.env.ELEVENLABS_VOICE_ID!, // store in env — IDs can be deprecated
      model: "eleven_flash_v2_5",
      stability: 0.5,
      similarityBoost: 0.75,
    },
    transcriber: {
      provider: "deepgram",
      model: "nova-2",
      language: "en-US",
    },
    firstMessage: "Thanks for calling Cascade PT — this is the front desk. How can I help you today?",
    serverUrl: process.env.WEBHOOK_URL!,
    silenceTimeoutSeconds: 10,
    maxDurationSeconds: 600,
  });

  console.log("Assistant created:", assistant.id);
  return assistant.id;
}

Provider selection drives most of the latency outcome. Deepgram Nova-2 delivers ~100ms for STT consistently. gpt-4o-mini at temperature 0.3 handles short structured workflows faster than a full-size model; Groq-hosted Llama 3.1 70B can cut LLM time to 150–250ms at the cost of slightly less reliable tool-call formatting. ElevenLabs eleven_flash_v2_5 reaches first audio at ~75ms — switching to eleven_turbo_v2_5 adds roughly 200ms but improves prosody on longer sentences.

Note the voice ID is pulled from an environment variable, not hard-coded. ElevenLabs has removed voice IDs from its library without advance notice; a hard-coded ID fails silently in production. Store it where you can swap it without a deploy.

Handling the webhook

Vapi POST-calls your serverUrl for function-call events (LLM invoking a tool), assistant-request events (dynamic assistant assignment at call start), and end-of-call-report (full transcript and metadata). Vapi allows up to 10 seconds before timing out a tool call, but anything over 100ms produces audible silence. Treat 80ms as your real budget.

import express, { Request, Response } from "express";

const app = express();
app.use(express.json());

type BookParams = {
  firstName: string;
  phone: string;
  date: string;
  time: string;
};

type VapiMessage = {
  type: string;
  functionCall?: { name: string; parameters: Record<string, unknown> };
  call?: { customer?: { number?: string } };
  [key: string]: unknown;
};

app.post("/webhook", async (req: Request, res: Response) => {
  const { message } = req.body as { message: VapiMessage };

  switch (message.type) {
    case "assistant-request": {
      const hour = new Date().getUTCHours();
      const assistantId = isBusinessHours(hour)
        ? process.env.DAYTIME_ASSISTANT_ID!
        : process.env.AFTERHOURS_ASSISTANT_ID!;
      res.json({ assistantId });
      return;
    }

    case "function-call": {
      const { name, parameters } = message.functionCall!;
      let result: string;
      try {
        result = await routeToolCall(name, parameters);
      } catch {
        // Always return a string — silence on the call is worse than a graceful failure
        result = "I am having trouble with that right now. Let me take a message instead.";
      }
      res.json({ result });
      return;
    }

    case "end-of-call-report": {
      // Fire-and-forget — never await here, never block the response
      persistCallRecord(message).catch((err: unknown) =>
        console.error("Failed to persist call record:", err)
      );
      res.sendStatus(200);
      return;
    }

    default:
      res.sendStatus(200);
  }
});

async function routeToolCall(
  name: string,
  parameters: Record<string, unknown>
): Promise<string> {
  switch (name) {
    case "checkAvailability": {
      const { date } = parameters as { date: string };
      // Replace with your calendar API (Google Calendar, Calendly, etc.)
      const slots = await calendarService.getOpenSlots(date);
      return slots.length
        ? `Available times on ${date}: ${slots.slice(0, 3).join(", ")}.`
        : `No openings on ${date}. Next available: ${await calendarService.nextOpenDay()}.`;
    }
    case "bookAppointment": {
      const p = parameters as BookParams;
      const conf = await calendarService.book(p);
      return `Done — ${p.firstName} is booked for ${p.time} on ${p.date}. Confirmation: ${conf.id}.`;
    }
    default:
      throw new Error(`Unknown tool: ${name}`);
  }
}

function isBusinessHours(utcHour: number): boolean {
  // Pacific time offset: UTC-7 summer, UTC-8 winter — adjust as needed
  const ptHour = (utcHour - 7 + 24) % 24;
  return ptHour >= 8 && ptHour < 17;
}

// Stub — implement against your database
async function persistCallRecord(_msg: VapiMessage): Promise<void> {
  // write transcript, duration, outcome to your DB
}

// Stub — implement against your calendar system
const calendarService = {
  getOpenSlots: async (_date: string): Promise<string[]> => [],
  nextOpenDay: async (): Promise<string> => "",
  book: async (_p: BookParams): Promise<{ id: string }> => ({ id: "" }),
};

app.listen(3000, () => console.log("Webhook listening on :3000"));

Testing locally before touching production

Use the Vapi dashboard's Talk to Assistant button for development. It opens a browser-based WebRTC call that runs through your full pipeline — including webhooks — without telephony charges. Point the assistant's serverUrl at your ngrok tunnel while developing.

For automated regression testing, Vapi's REST API includes a call simulation endpoint that replays a transcript against your webhook server without initiating a real call. Wire this into your CI pipeline with a fixture set of common caller intents. Silent regressions in system-prompt behavior — where the agent starts answering correctly only half the time — are the hardest class of bugs to catch in voice agents. Automated transcript replay against a known fixture set catches them before callers do.

Latency optimization

End-to-end latency across different production configurations lands between 500ms and 900ms. Three levers account for most of that range:

1. Provider selection compounds

Deepgram Nova-2 adds ~100ms for STT. gpt-4o-mini via OpenAI adds 200–400ms for short completions — Groq-hosted Llama 3.1 70B can reduce this to 150–250ms but produces less reliable tool-call JSON formatting, which causes its own latency when the model has to self-correct. ElevenLabs Flash v2.5 at ~75ms is the fastest TTS on Vapi's provider list. Every component swap propagates — the total reflects every choice.

2. Response length is the highest-leverage prompt variable

Vapi's default chunk plan accumulates tokens before dispatching batches to ElevenLabs for better prosody. Longer LLM responses trigger more accumulation cycles, each adding 100–300ms of perceived latency. A system prompt that enforces two-sentence maximum responses is worth more than any infrastructure change. Measure with your specific model and prompt before drawing conclusions — the difference between a verbose and concise system prompt can be 400ms of caller-perceived latency.

3. Webhook I/O is the most common bottleneck

Every millisecond your tool handler spends on I/O is dead silence on the call. Pre-warm database connection pools, use read replicas for availability lookups, and never run synchronous HTTP calls to slow third-party APIs in the webhook hot path. If a downstream service reliably takes 300ms+, return a brief holding phrase from the assistant ("Let me check that for you") and complete the lookup in a second pass. Vapi supports mid-stream tool responses for exactly this pattern.

Production gotchas

These come from real deployments, not from documentation examples.

Webhook timeouts fail silently and dangerously

If your handler does not respond within Vapi's 10-second budget, Vapi continues the call without the tool result. The LLM fills the gap — in the best case it says "I am having trouble with that"; in the worst case it hallucinates an appointment confirmation that never gets written to your calendar. Always wrap outbound calls inside your handler with an explicit 8-second race timeout and return a graceful fallback string before the deadline, never let the request time out cold.

VAD false triggers in noisy environments

Vapi's default voice activity detection is tuned for quiet office conditions. In a Central Oregon HVAC shop, auto shop, or restaurant kitchen, background noise and HVAC hum will interrupt the agent mid-sentence. Increase silenceTimeoutSeconds to 1.5–2.0 and test with recorded background noise from your actual deployment environment before going live. This single setting has a larger effect on perceived call quality than any provider swap.

First-message cold start adds invisible latency

The TTS connection initializes on the first message, adding 200–400ms of one-time overhead not present on subsequent turns. Make firstMessage a short, static string with no dynamic data. The warmup is invisible when the greeting starts immediately; it becomes a noticeable pause before the greeting if you attempt to inject live data (current time, caller name lookup) into the first message.

SIP trunk codec negotiation breaks audio quality

If you route calls through an existing RingCentral or Dialpad SIP trunk before handing off to Vapi, verify the trunk negotiates G.711 µ-law (PCMU), not G.729. G.729 compression clips the higher-frequency detail that ElevenLabs voices depend on and produces a tinny, artificial quality that callers notice even on short calls.

Cloudflare Workers: 30-second CPU limit is a hard wall

If you host the webhook on Cloudflare Workers, the 30-second CPU time limit per request terminates mid-call with no recoverable error — it is not a soft timeout you can catch. Workers is well-suited for stateless lookups against D1 or KV; keep stateful, connection-holding operations on a traditional server process.

ElevenLabs voice IDs deprecate without advance notice

ElevenLabs has removed voice IDs from its library without warning. A hard-coded ID causes Vapi to fall back to a default voice or error the call depending on your configuration — neither outcome is obvious in logs unless you are monitoring TTS-specific error rates. Store voice IDs in environment variables and add a periodic health check that verifies the ID still resolves.

When NOT to build this yourself

Vapi is the right choice when you need tight control over a specific call flow, when your business logic is too custom for a configurable managed service, or when you are building voice AI as a product. It is not the right choice in every scenario.

Skip the build if your use case is standard reception. Holiday schedules, voicemail fallback, HIPAA call logging, Salesforce or HubSpot CRM sync — managed services have already solved these edge cases. Rebuilding them from Vapi primitives typically takes three to six months and requires ongoing maintenance. If you are deploying for a healthcare client, read the HIPAA-compliant AI voice agent guide first — the compliance requirements alone add significant scope to any custom build.

Skip it if you do not have ongoing engineering capacity. Voice agents drift. Prompts need retuning as new call patterns emerge. Provider APIs deprecate. The firstMessage that worked in June sounds wrong in December when business hours change. A production voice agent is a running service that needs an owner, not a project that gets shipped and forgotten. If the client does not have a developer who will maintain it, a managed deployment is more honest than shipping a Vapi build. Talk to us about what a managed deployment looks like for your situation.

Skip it if compliance review has not happened yet. HIPAA, state bar rules around AI-assisted legal intake, and FINRA regulations around recorded financial conversations vary by state and are actively evolving in 2026. Vapi does not provide compliance guarantees — BAA agreements, call recording consent flows, data residency, and audit trail requirements are your responsibility. That legal review belongs at the start of the project, before a single line of code is written.

Do build it when you need full control over the call flow, when integration with a proprietary system (a custom Dentrix module, an internal scheduling engine) requires direct API access a managed service cannot provide, or when you are building a voice product where the call behavior is itself the differentiator. This guide gives you a working foundation — the real work starts when you tune the system prompt against real caller transcripts from your specific environment.

Caller (PSTN / WebRTC)
        |
        v
  Vapi Orchestration Layer
  +--------------------------------------+
  |  STT: Deepgram Nova-2    (~100ms)    |
  |  LLM: gpt-4o-mini        (~300ms)    |
  |  TTS: ElevenLabs Flash   (~75ms)     |
  +--------------------------------------+
        | POST /webhook (tool calls only)
        v
  Your Server (Express / Workers)
  +-- checkAvailability(date)
  +-- bookAppointment(name, phone, slot)
  +-- lookupAccount(callerId)

Frequently asked questions