An $11,400 AWS bill, and CloudWatch Logs was $2,100 of it

On a Tuesday morning, a founder shared credentials with a reporter, hinting at potential overpayment. The monthly spend totaled $11,400, which seemed excessive for the product's 800 active users. Intrigued, the reporter spent the first 15 minutes merely observing the account, mapping the structure and reasoning behind the costs.

A glaring red flag emerged: CloudWatch Logs accounted for $2,100. This figure was particularly concerning given the size of the account. The reporter delved into the log groups, finding 43 of them without any retention policy, causing logs to accumulate indefinitely. Several of these groups were receiving data from services that no longer existed, further complicating the situation.

One log group stood out: it was ingesting a staggering 180GB of data each month from a service that was operating in debug mode during production. This meant every function call, variable state, and internal operation was being shipped to CloudWatch and stored at full price. This single log group was responsible for $900 of the monthly CloudWatch bill.

The service responsible for this excessive logging had been deployed eight months prior, and debug logging had been left on during the initial development stage, never being turned off. The reporter noted that nobody had taken the time to review the CloudWatch bill specifically, as it was just another component of the gradually increasing total that went unquestioned.

After setting retention policies on all 43 log groups and turning off debug logging on the offending service, the reporter deleted the log groups receiving data from services that were no longer operational. The following month, the CloudWatch bill had dropped to a more reasonable $340, resulting in a savings of $1,760. All this was achieved with just one afternoon of work, and they hadn't even touched the rest of the account yet.