Your Customer Health Score Is Probably Astrology: Here's How to Test It

Many SaaS companies use a customer health score, which is a numeric rating based on metrics such as logins, seats, Net Promoter Score, and support tickets. However, few of these scores have been rigorously tested to determine if they accurately predict churn.

To test a health score, create a table called "health_snapshots" with columns for account ID, snapshot date, score, usage change, tickets, and churned_on status. This table should contain one row per account per week or month, capturing the score and relevant inputs at that point in time.

First, label each snapshot with what happened next - did the account churn within the following 90 days? Create a "labeled" table that includes this information.

Next, analyze whether red, yellow, or green bands (based on the score) churn more frequently than the others. In a simulated dataset of 1,500 accounts over 18 months, red bands churned at 1.5 times the rate of green bands, indicating that focusing on the red list may not be as effective as focusing on accounts with falling usage.

Finally, calculate an overall accuracy score (AUC) to determine how well the health score predicts churn compared to random chance. In the simulation, the AUC was 0.556, indicating that the score only performed slightly better than random guessing. However, using a specific rule such as "usage fell 20% or more compared to three months ago" had a higher AUC of 0.643, suggesting that this metric outperformed the health score.

Key takeaways:

1. Test your health score by creating a labeled "health_snapshots" table.

2. Focus on inputs that significantly improve the accuracy score (AUC) compared to random chance.

3. Drop inputs that don't perform well (i.e., those with an AUC of 0.5 or below).

4. Calibrate the health score bands to meaningful churn rates and publish the backtest alongside the score.