Stop guessing auto scaling targets for TTS on Amazon SageMaker AI
Teams deploying custom text-to-speech models on Amazon SageMaker AI often guess at autoscaling thresholds, and a wrong guess either wastes GPU spend or breaches a latency SLO. This post deploys Kokoro-82M as a custom container and derives a measured autoscaling policy from two live-endpoint measurements, Q_max and T_total, instead of guesswork.
Auto scaling for text-to-speech endpoints on Amazon SageMaker AI often fails due to incorrect targets. This is because auto scaling policies are based on guesses rather than accurate measurements. If the target value is too low, the fleet uses GPU instances that sit mostly idle, wasting resources. If the target value is too high, instances risk carrying a risk nobody sees.
The issue lies in the fact that traditional auto scaling metrics don't work for TTS synthesis, which involves holding connections open for several seconds without token generation to attach a metric to. Kokoro-82M, an 82-million-parameter model, is deployed as a custom container on an Amazon SageMaker AI real-time endpoint. To replace guesswork with accurate scaling, the article measures two key metrics: Q_max, the highest concurrency one instance sustains before breaching the SLO, and T_total, how long a scale-out takes end to end.
These metrics are derived from actual deployment and measurement runs. By using these two measurements, the article calculates the scale-out threshold, scale-in threshold, and cooldowns. The deployment code uses AWS CDK stacks to only attach auto scaling resources once a real, measured plan exists for the model, preventing a guessed configuration from reaching production silently.
Written by urgent.news from AWS Machine Learning's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
