I Spent a Month on Architecture The Migration Took Two Days

A month-long effort went into migrating a production backend service from one cloud provider to another. The primary motivation was cheaper auto-scaling and flexible compute that scaled with demand, issues with the previous provider, and permission problems. The migration itself took just two days, while designing the architecture took the entire month. The author initially believed migration was the hardest part, but in reality, it was the design that required the most effort.

The old system had a setup process where each machine needed to establish its own identity before it could perform any work. The author assumed this approach would work on the new provider, but it didn't. After multiple failed attempts, the author learned that the original setup should not have been copied from one machine to another. Instead, each machine should have been installed fresh, logged in through a plain configuration file, and set up from scratch. This approach worked immediately.

The author ran a load test to understand the actual performance of the new machine. Surprisingly, the CPU usage did not scale linearly with increasing load. The new machine had the same CPU as the old one, but with more memory. At light load, the CPU usage was around 22%, and it jumped to 92% at moderate load. The author lowered the safe capacity limit based on this real data, realizing that the real limit was CPU, not memory.

During the migration, the author faced a trade-off. When users returned to the system, they could be routed to a different machine than before. Building a more complex routing layer to ensure returning users landed on the same machine was not deemed necessary. Instead, the system was designed to tolerate the problem by starting fresh wherever the user landed. The author also chose to route traffic through a simple passthrough instead of a heavier encryption option, as the service already handled encryption itself.

The main reason for the migration was to take advantage of cheaper auto-scaling and flexible compute. The setup could scale a small always-on baseline with cheaper burst capacity added on top when needed. However, the author did not implement a fully automatic scaling policy at that time. They wanted real usage data before deciding on an automatic policy, as an automatic policy built on a guess would either react too late or too early, and the author would not know which until it was live.

Lastly, the author focused on designing access properly. They reviewed every broad permission in the new setup, kept only those that genuinely couldn't be scoped any narrower, and added explicit restrictions on sensitive actions. This review caught a real gap before it caused a problem. The principle was to design access narrowly, ensuring the system was secure and efficient.