AI Hiring in a Live Company: The Agent Filled the Funnel. The AI Interview's Labels Did Not Line Up With Who Is Still Working

In the second part of the series on AI agents performing management tasks in a live US telemedicine company, the focus shifts to the hiring process for a phone-based role. Over five weeks, an AI agent processed 1,973 job applications and scheduled interviews without the need for a recruiter. However, by day 36, only 2 out of 7 hires and 6 out of 12 candidates labeled as "maybe" were still working, indicating the labels did not accurately reflect which candidates were still employed.

The AI agent responsible for the hiring process was named Claude. It read the applications, selected candidates based on predefined rules, entered them into the CRM, and booked interview slots. The AI interview was introduced as a replacement for the human first-round interview, but its results showed no clear distinction between candidates who were still working and those who were not.

During the period, the AI interview invited 41 candidates, with 22 completing the interview, amounting to a 54% completion rate. The remaining 19 candidates either did not complete the interview or were not labeled as hires. The AI interview process involved a personal link, a 15-minute video interview with questions about past sales experience, a typical workday, commitments, full-time availability, and a start date.

The interview was scored automatically using a separate model, resulting in "HIRE," "MAYBE," or "NO" labels.

Despite the AI interview's introduction, the hiring process still faced challenges such as a high no-show rate (over 30% in the first two weeks) and a delay in making hiring decisions for many candidates. The AI supervisor from the previous part of the series also noted that a significant number of candidates were waiting without receiving a hiring decision, highlighting the inefficiencies in the hiring funnel.