Five Ways to See LLM Chatbot Token Costs: Tenant-Aware Sales CRM Operations

To determine the most cost-effective LLM chatbot solution for a tenant, it is crucial to analyze the total cost of each candidate by measuring the entire work unit, not just the token usage. A work unit encompasses the entire sales call process from input to CRM action, including retries and any necessary follow-up actions.

When comparing chatbot production costs, consider the input and output tokens separately, as different providers may excel in one aspect while underperforming in another. Including the number of retry attempts in the calculation provides a more accurate representation of the true production cost.

To conduct a fair comparison, replay the same sanitized call set, CRM action schema, and acceptance rules for each candidate. Record a tenant's journey from input to CRM action, tracking metrics such as the number of received calls, accepted actions, input and output tokens, retries, and estimated cost. Calculate the cost per accepted CRM action, as it better reflects the operational efficiency of the chatbot solution.

Retrieval steps, such as searching prior calls or account notes, should also be considered part of the work unit, as they belong to the numerical representations used for search and related tasks. It is essential to keep the tenant ID on every observability signal, such as metrics, logs, and traces, to trace any issues back to a specific tenant easily.

In summary, to find the most cost-effective LLM chatbot solution, compare the total cost of each candidate based on the complete work unit, including input and output tokens, retries, and the number of accepted CRM actions. By following this approach, a tenant can identify the candidate that not only has the lowest advertised token rate but also aligns with their quality and latency constraints.