An AI voice agent is software that answers a phone call, listens to what the caller says, works out what they want and replies out loud, often while checking a calendar or writing to a CRM. Each turn of the conversation runs through four steps: hearing, thinking, speaking and acting. It only feels natural when those steps happen quickly enough that the caller never notices them.
This guide follows one example call through those four steps, then covers the two things that decide whether a voice agent is pleasant or painful to talk to: delay and interruptions.
What an AI voice agent does on a single call
Picture a physiotherapy clinic that closes at 6pm. A caller rings at 6.40pm.
Agent: Thanks for calling Harbour Physio. You’ve reached the clinic’s automated assistant. How can I help?
Caller: Hi, I’ve done my back and I’d like to see someone tomorrow if I can.
Agent: Sorry to hear that. Have you been to us before?
Caller: No, first time.
Agent: No problem. I have 8.30am or 4.15pm tomorrow with one of our physios. Which suits you better?
Caller: 8.30.
Agent: Booked for 8.30am. Can I get your name and a mobile number so I can text you the details?
That is a two minute call with no one at the front desk. Behind it, four separate pieces of technology took turns. The sections below take them one at a time.
Step one: hearing
The agent has to turn the caller’s voice into text, live, while they are still talking. This is speech recognition. Phone audio is compressed and often noisy, so it is a harder job than dictating into a laptop.
Two things go wrong most often. The first is names: surnames, suburbs and street names are exactly the words a recogniser has the least reason to expect. The second is knowing when the caller has finished. People pause in the middle of a thought (“I need to book… actually, it’s for my son”), and an agent that jumps in at the first gap sounds rude.
A well-built agent repeats back anything that matters, such as a phone number or an address, and asks the caller to confirm it. You do the same thing on the phone yourself.
Step two: thinking
The text goes to a language model, which is the part people usually mean by “the AI”. It reads the conversation so far, plus a set of instructions written for your business, and decides what to do next: ask a question, answer one, look something up or hand the call to a person.
The instructions matter more than the model. They hold your opening hours, your services, what you will and will not discuss, and what counts as an emergency. A language model can produce a confident answer that is wrong, so a sensible set-up limits it to information you have supplied. When the answer is not there, the agent should say it will pass the question on instead of guessing. The guide on writing an AI receptionist script shows what those instructions look like.
Step three: speaking
The reply, written as text, is turned into a spoken voice. You choose the voice, and you can usually adjust pace and tone. Accent matters to callers more than most owners expect, and so does pronunciation of local place names. We cover that separately in getting an AI voice agent to sound Australian.
Good spoken replies are short. A paragraph that reads well on a web page is tiring to listen to. Callers want one fact or one question at a time: “I have 8.30am or 4.15pm. Which suits you?” is better than a list of six times. This is the main difference from the phone menus most callers know, compared in IVR vs conversational AI.
Step four: acting
This is the step that separates a voice agent from a talking answering machine. During the call the agent can use tools, which are connections to your other systems:
- Check your real calendar for free times, and write the booking into it.
- Create or update the contact in your CRM, with notes from the call.
- Send the caller a text confirmation.
- Transfer the call to a staff member, or notify them with a summary.
Without those connections the agent can only take a message. With them, the booking in our example exists by the time the caller hangs up. How much this matters depends on your business: a clinic with a busy online calendar needs it, while a one-person trade that only wants messages taken may not. The guide to AI appointment setters goes further into booking.
Why delay matters on a phone call
Every step above takes a moment, and the moments add up. The system must detect that the caller has stopped, finish the transcription, let the language model decide, generate the voice and start playing it. If one stage is slow, the caller hears silence.
Silence on a phone is uncomfortable in a way it is not in a chat window. A few seconds of nothing makes people say “hello?” or hang up. Builders reduce the gap in practical ways:
- Keep replies short, so the voice can start speaking before the whole answer is ready.
- Use a brief acknowledgement before a slow step, such as “One moment while I check the diary.”
- Keep the instructions focused, since a very long set of instructions gives the model more to work through.
- Only call outside systems when the caller has actually asked for something that needs them.
You do not need a figure from a brochure to judge this. Call the test line, say something, and notice how long you wait. If you would be annoyed as a customer, so will your customers.
Why interruptions matter
People talk over each other on the phone constantly. A caller hears the agent start to explain parking and says, “No, I just want a time.” If the agent keeps talking through that, the call is already going badly.
A good voice agent stops speaking the moment the caller starts, listens and then responds to what was actually said. This is often called barge-in. The opposite problem also exists: the agent cutting off a caller who has only paused. Background noise adds a third problem, because a barking dog or a passing truck can be mistaken for the caller and make the agent stop mid-sentence.
Test it deliberately when you try one. Talk over it. Say “sorry, what was that?” Cough. Give an address slowly with pauses. Change your mind halfway through a booking. How it copes with all of that tells you more than a polished demo.
When it should hand over to a person
An AI voice agent should know its limits and pass the call on cleanly. Reasonable reasons to hand over:
- The caller asks for a person.
- The caller sounds distressed or the matter sounds urgent or unsafe.
- The question is outside the approved information, such as a clinical question or a complaint.
- The agent has misheard the same thing twice.
During opening hours that can mean a transfer to a staff member’s phone. After hours it more often means taking a message and sending the team a summary, with the caller told exactly what will happen next and when. Many businesses also choose to tell callers up front that they are speaking with an automated assistant. Rules on this and on recording calls differ by country and state, so check what applies to you.
Questions to ask before you trust one with your phone
- Can I ring a test line and try to break it?
- What does it do when it does not know an answer?
- Does it book into my real calendar, or does it only take details?
- Where do the call notes and recordings go, and who can see them?
- Who fixes it when something changes, such as new opening hours or a new service?
For costs, see our guide to what an AI receptionist costs. If you want a system built around your own calendar and CRM, CentriWeb builds and looks after AI phone receptionists, and the first step is a short call about where your phone calls are currently being lost.