How to Build Voice AI Agents

A voice agent listens, understands, decides, and speaks back in real time. It can answer calls, book appointments, take orders, or answer questions, and pass the caller to a person when needed.

Pipeline

Speech-to-text, then a language model, then text-to-speech; flexible and easy to swap parts

Speech-to-speech

One model hears and speaks directly, such as OpenAI Realtime; very natural, less flexible

A good voice agent sounds human, but it never pretends to be one.

• Booking and rescheduling appointments
• Answering common questions after hours
• Qualifying leads before a sales call
• Taking simple orders or messages

One narrow job done well beats a general agent that fails half the time.

ElevenLabs Agents

Known for very natural voices

Vapi

Developer-first, bring your own speech, model, and voice providers

Retell

Managed phone agents you can ship quickly

OpenAI Realtime

Native speech-to-speech in one API

Test two platforms on your own calls before choosing. Published comparisons are often written by vendors or partners.

You are the booking assistant for [business]. You help callers book, move, or cancel appointments. Speak in short, friendly sentences. Confirm the date, time, and name before booking. If the caller asks about anything else, or sounds upset, offer to transfer them to a person. Never give medical, legal, or financial advice.

• Write for the ear: short sentences, no lists or symbols
• Tell it what to do when it does not know
• Set a clear greeting and goodbye

• Connect your calendar or booking system
• Add a knowledge base with prices, hours, and policies
• Let it send a confirmation text or email
• Log every call to your CRM

• Keep response delay low; under a second feels like conversation
• Let callers interrupt the agent mid-sentence
• Choose a clear voice that fits your brand
• Handle silence, background noise, and accents

Most platforms let you buy a number or connect your existing phone system. Start with a separate test number before you route real customers to it.

• Transfer when the caller asks for a person
• Transfer when the agent fails twice
• Pass a short summary so the caller does not repeat themselves
• Offer a callback outside opening hours

• Call it yourself 20 times with different requests
• Ask friends to try confusing or rude calls
• Listen to recordings and read transcripts
• Fix the prompt, then test again

Most platforms charge per minute, and the total includes speech recognition, the language model, the voice, and telephony. Track cost per resolved call, not just cost per minute.

• Tell callers they are speaking with an AI
• Get consent before recording where the law requires it
• Follow calling and marketing rules in your country
• Never clone a real person’s voice without their permission

• Review a sample of calls every week
• Track resolution, transfers, and complaints
• Add missing answers to the knowledge base