What the job involves day to day
Most production voice agents run as a pipeline. Speech to text transcribes the caller, one or more language model calls work out the reply, and text to speech speaks it. Andrew Ng, writing for DeepLearning.AI, prefers this design when he needs tight control over output, because the reasoning happens in text where you can add guardrails. His example: before an agent tells a caller a refund is coming, the system checks that the refund fits policy and that the refund API will actually be called.
The other route is a speech to speech model that takes audio in and produces audio out. Ng calls these good for quick prototypes and low-stakes conversations, but harder to control. Much of the engineer's job is picking between the two, then building the parts around the model: turn taking, interruptions, calls to business systems, testing and monitoring. Prototyping is quick. Ng reports connecting a Twilio phone number to a speech to text, LLM and text to speech pipeline in a few hours.
Why latency takes up so much of the work
People will wait several seconds for a text chatbot. On a phone call, a pause that long feels broken. Ng puts face to face conversational latency at around 0.3 to 1 second. His team's first agentic voice workflow took 5 to 9 seconds per reply. They brought it down to about 0.5 to 1 second, partly by speaking a short pre-response while the full answer was still being generated.
Salesforce's team behind Agentforce Voice describes the same pressure. If replies took 1.5 to 2 seconds, callers lost the thread or assumed the system had failed. Every service in the chain had to get faster, and the team aimed to cut 100 milliseconds from each one. They also built semantic end-pointing, which judges from the words so far whether the caller has finished or is only pausing, for example partway through reading out a card number.
Handoffs, contact center platforms and measurement
Voice agents usually live inside a contact center platform. In Amazon Connect, the Get customer input flow block takes keypad (DTMF) input or hands the caller to an Amazon Lex bot, and it supports barge-in, so callers can talk over a prompt. Dialogflow CX ends a session and passes the conversation to a human agent through an END_FLOW_WITH_HUMAN_ESCALATION transition. For direct phone connections, OpenAI's SIP documentation leaves webhooks, session setup, call decisions and business logic to your application.
Measurement is part of the job too. Google's contact center data dictionary defines virtual agent resolution rate as the share of virtual agent interactions the bot resolved, which is close to what many teams call containment. Google's platform also collects customer satisfaction ratings on calls. Salesforce made time to first audio its main metric and separated planned human transfers from escalations caused by agent failures, so that a deliberate handoff is not counted as a failure.
Moving in from chatbot or support engineering
Chatbot developers already know intents, dialog flows, fallbacks and escalation logic. What they usually need to add is audio: streaming speech to text and text to speech, turn detection, interruptions and latency budgets. Support and contact center engineers come in from the other side. They know routing, queues, telephony and the metrics operations teams watch, and they need to learn the language model and evaluation side.
Either way, the quickest proof is a working phone agent that you have measured. Build one end to end, record time to first audio for every turn, and log every handoff with its reason. Then test it with scripted synthetic callers. Salesforce generated varied test utterances with text to speech so that latency and accuracy results could be repeated. A project like that shows a hiring manager more than a certificate does.
Pay
USD a year, before tax. US figures from BLS OEWS May 2025, moved to 2026 with the Employment Cost Index. No official series covers voice AI work: software developers is the nearest series, and it covers the whole occupation, not voice or AI specialists alone. Refer to the Conversational AI & Chatbot Developers salary guide for figures from live adverts.
What you need
- Voice pipeline skillsYou need to know how speech to text, language models and text to speech fit together, and where each one adds delay. You also need to judge when a speech to speech model is the better fit.
- Latency measurementYou should be able to measure time to first audio and end-to-end latency per turn, and know the difference. Salesforce once saw 70-second readings because its tests timed the whole spoken reply instead of the first audio.
- Turn taking and interruptionsAgents have to detect when a caller has finished speaking and stop talking when the caller cuts in. Amazon Connect calls this barge-in.
- Contact center integrationExpect to work with platform flows, bot services and human escalation paths, such as Amazon Connect with Amazon Lex or Dialogflow CX handoffs, plus telephony links like SIP.
- Guardrails and business logicThe agent's words have to match what the backend actually does. Check that refunds, bookings and account changes are allowed and actually executed before the agent confirms them.
- Evaluation and metricsYou need to track bot resolution, customer satisfaction and escalations, with planned transfers kept apart from failures. Repeatable synthetic test callers make the numbers reliable.
How to get there
- StartBuild a prototype phone agentConnect a phone number to a speech to text, language model and text to speech pipeline, and get one complete call working.
- NextMeasure and cut latencyLog time to first audio for every turn. Try streaming and a short pre-response to bring replies close to the 0.3 to 1 second range of human conversation.
- NextHandle interruptions and turn endsAdd barge-in and end-of-turn detection that copes with natural pauses, such as a caller reading out a number.
- ThenAdd human handoff and backend actionsRoute to a human with context passed along, record a reason for every escalation, and check that actions the agent promises really run.
- ThenIntegrate with a contact center platformRebuild the agent inside a platform flow, such as Amazon Connect with Amazon Lex or Dialogflow CX, so you learn how real routing and escalation work.
- OngoingReport outcomesTrack resolution rate, customer satisfaction and escalation types with repeatable synthetic test callers, and use those numbers as your portfolio evidence.
Open now on AutomationRoles.ai
Questions people ask
Is a voice AI engineer different from a chatbot developer?
The two roles overlap on dialog design, fallbacks and escalation. Voice adds audio streaming, speech recognition, synthesis, turn detection and strict latency limits. A text user will wait a few seconds for a reply, while a caller hearing a 1.5 to 2 second pause may think the system has failed.
How fast does a voice agent need to respond?
Andrew Ng puts human face to face conversational latency at about 0.3 to 1 second, and his team got an agentic voice system down to about 0.5 to 1 second. Salesforce engineers found that even 300 milliseconds of extra work could spoil the experience.
What is containment in a voice agent project?
Containment is the share of conversations the bot handles to the end without passing them to a person. Google's contact center platform tracks a close equivalent called virtual agent resolution rate: resolved virtual agent interactions divided by all virtual agent interactions.
Should I use a speech to speech model or a pipeline?
It depends on how much control you need. Andrew Ng finds speech to speech models good for prototypes and low-stakes conversations. He prefers a speech to text, language model, text to speech pipeline when output has to follow business policy, because guardrails are easier to build in text.
How much does a voice AI engineer earn?
No official US series covers voice AI work, and we found no reliable source for this specialty on its own. The nearest proxy is the BLS software developers series, moved to 2026 values: $108,500 at the 25th percentile, $140,200 at the median and $177,400 at the 75th. That series covers all software developers, not only voice AI specialists.
Is there a license for voice AI engineers?
No. No license exists for AI work and no regulator covers the job itself. Platform vendors offer certifications, but employers mostly judge you on working systems with measured results.
Sources
- Salesforce Engineering: sub-second latency for Agentforce Voice
- DeepLearning.AI The Batch: What I've learned building voice applications
- DeepLearning.AI: Building AI Voice Agents for Production
- AWS: Amazon Connect Get customer input flow block
- Google Cloud: Dialogflow CX handoff (Agent Assist)
- Google Cloud: Contact center data dictionary
- OpenAI: Telephony and SIP guide
- BLS Occupational Employment and Wage Statistics: software developers
