I Built an AI Pipeline That Reads Support Emails and Drafts Replies (Here's What Actually Broke, and the Math on Whether It's Worth It)

A small one-person development company faced the common problem of support agents spending excessive time answering repetitive customer inquiries. To address this, the developer built a pipeline that uses Gmail, Claude's API for classification, and Slack notifications to automate the process. The pipeline pulls unread messages from Gmail, extracts the message body while handling various formats, classifies the email with structured output from Claude, routes the email to a human based on the category and confidence score, generates a draft reply using the same API call, posts a Slack card with the information, marks the email as read, and stores the result in SQLite for future runs.

The developer faced several unexpected issues while implementing the pipeline. The first bug was related to OAuth scopes. Initially, the developer used both gmail.readonly and gmail.send scopes, but adding the mark as read step required write access. Google silently dropped the redundant scope, causing the token refresh to fail without any clear indication. To avoid this issue, it is essential to request the narrowest single scope that covers all necessary actions.

The second bug occurred while extracting the message body from Gmail messages, which are frequently multipart/alternative structures containing both plain text and HTML parts. The developer's initial approach recursively searched for plain text and fell back to HTML if it wasn't found. However, if the HTML part appeared before the plain text part in the tree structure, the function incorrectly returned the HTML version.

To resolve this issue, the developer implemented two separate passes searching for plain text first, and only if that fails, proceeding to search for HTML. This approach ensures that the preferred format is found, and the fallback is only triggered if the preferred format is absent throughout the entire structure.

The third bug was related to the routing process based on category and confidence score. Initially, the developer assumed a simple lookup table for assigning emails to specific human handlers based on the category. However, a test case revealed an exception to this rule: if the model's confidence score was below a certain threshold, the email would be routed to a broader category handler, regardless of its category.

To address this issue, the developer modified the routing logic to account for the confidence threshold. Emails with lower confidence scores are assigned to a person with broader context, while those with higher confidence scores follow the category-based routing table.

In summary, the developer's pipeline successfully automates the process of answering customer support emails by utilizing Gmail, Claude's API for classification, and Slack notifications. However, the developer encountered three unexpected bugs during implementation: an OAuth scope trap, issues with handling multipart/alternative message bodies, and a routing error based on confidence thresholds.

By understanding and addressing these issues, the developer was able to create an efficient and reliable system for managing customer support inquiries.