What broke when I put an AI voice agent on real phone lines
I build AI phone agents for home service companies. A voice agent (Retell) answers the call, and n8n workflows do the work behind it: checking a calendar, booking, writing to the CRM. These are the failures that taught me the most, and what fixed each one. 1. The model was bad at "are we open right now?" The prompt had the business hours and the current time. The model still got the open or…
Building AI phone agents for home service companies led to several failures, which shaped the author's approach to creating more effective agents. The first issue was the model's difficulty in determining whether a business was open or closed. The prompt contained the business hours and current time, but the model still got the answer wrong frequently.
The solution was to stop relying on the model entirely and have an n8n step compute the answer and pass it as a variable instead. The author now follows the rule that if a rule misfires twice, it should leave the prompt and compute the answer upstream.
Another significant failure was when the agent incorrectly stated that a caller was booked when no appointment existed. This was a huge problem because callers would think they had an appointment and then be inconvenienced when it didn't materialize. The author resolved this by adding a structural rule that requires the booking tool to go through an n8n proxy first. The proxy reads the transcript and ensures that the agent only says "booked" when the tool returns success.
Dead air during tool calls was another issue. When the calendar was slow, callers sat through up to 27 seconds of silence. To combat this, the author reduced the timeout for the booking tool to 25 seconds and allowed the agent to talk while the tool ran in the background.
The fifth failure came when publishing a new version of the agent succeeded, but callers kept hearing the old version. To fix this, the author now checks which version the phone number is actually bound to after a publish call and ensures that a real call is placed on the new version.
The sixth and final challenge the author faced was when the same caller made two bookings an hour apart. To prevent this, the booking step now looks for an existing booking from the same number and moves it if necessary. The author advises anyone starting with this process to read the actual tool call data, not just the transcript, as the transcript is what the model said happened, and anything that has a single right answer (time, hours, dates) should be coded.
Additionally, a publish should only be considered successful when a real call lands on the new version.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.