Adopting an Autoscaling Policy You Didn't Write: ECS, ASG, VMSS and MIG Without Silent Overwrites
Every team eventually inherits autoscaling configuration it didn't write. The author left, the Terraform was half-applied, someone tuned a threshold in the console during an incident two years ago, and now the scaling behavior of production is defined by an unknowable merge of code, clicks, and defaults. Everyone is afraid to touch it, which is exactly how it stays wrong. The technical trap that…
Many teams encounter inherited autoscaling configurations they did not create themselves. These configurations often originated from a combination of code changes, console edits, and default settings over time. The issue is exacerbated by silent overwrites, where multiple writers make changes to the same autoscaling policy simultaneously without noticing the conflict.
This silent overwrite phenomenon occurs across various platforms such as AWS ASGs, ECS services, Azure VMSS, and GCP MIGs. The technical challenge with inherited autoscaling is that scaling policies exist as separate API objects with last-writer-wins semantics, meaning code-defined policies can silently override console changes, and vice versa.
This results in the system's behavior changing without any explicit deployment, pull request, or alert. To safely adopt and manage inherited autoscaling configurations, follow these steps:
1. Inventory every autoscaling policy currently in use. Rather than relying solely on the code definitions, query the API to see what policies are actually being applied in production. For AWS, use the command `aws autoscaling describe-policies`; for ECS, run `aws application-autoscaling describe-scaling-policies`; in Azure, use `az monitor autoscale list`; and for GCP, execute `gcloud compute instance-groups managed list`.
2. Freeze the configuration before attempting to fix it. Establishing a single source of truth is crucial to avoid further conflicts. Export the live state of the scaling policies into your infrastructure as code (IaC) tools, including all the quirks and inconsistencies. Import these policies back into your IaC tools using the appropriate commands (e.g., `terraform import` for Terraform, or the Azure/Bicep/gcloud equivalents).
Perform a diff to ensure that the plan shows zero changes, indicating that the live configuration matches your code. This step ensures that the incident-era tweaks are documented in code, eliminating them as hidden landmines.
3. Audit the newly visible configuration. With one definitive source of truth in place, it's easier to identify common issues in inherited policies. Look out for settings like cooldown periods shorter than 120 seconds, which can cause oscillations between scaling up and down, leading to unnecessary costs and availability issues.
Targets set too high (above 90%) might trigger scaling too late, while targets set too low (below 30%) can result in over-provisioning. Additionally, policies with identical min and max values do not perform any autoscaling and should be clarified explicitly. Furthermore, ensure that all groups expected to have policies actually have them configured, as failing to do so can lead to unintended behavior.
4. Enforce single ownership and alert on unauthorized changes. After successfully adopting the policies, it's essential to prevent out-of-band changes from occurring again. Utilize CloudTrail logs for AWS, Azure activity logs, and GCP audit logs to track any policy writes that are not originating from your defined deployment pipeline.
Set up alerts on these policy-write events to ensure that any unauthorized modifications are caught promptly. This approach allows legitimate incident-time tuning to occur, but it should be accompanied by a follow-up task to codify the changes or revert them, rather than leaving them unmanaged.
5. Only then proceed with tuning. Once the inherited autoscaling configuration is stable and owned by a single team, you can begin making changes. However, it's advisable to make changes incrementally, one variable at a time, through your deployment pipeline. Monitor the effects of each change across a full traffic cycle to avoid the pitfalls of batch tuning, which often leads to unpredictable behavior due to the complex interactions between different autoscaling settings.
By enforcing a single owner for each scaling policy, ideally the team responsible for the service that the policy scales, you can prevent inherited configuration decay into an unmaintainable state.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
