A cloud engineer walked into a bar and – no joke – ended up having to migrate a datacenter
The budget was minimal, FedEx was the bottleneck, and DNS was of course the source of trouble
Dave Bradley, a senior cloud engineer at Vizient, found himself in an unusual situation after a night out with colleagues from the tech team. After enjoying some drinks, Bradley decided to stop by a second bar when he noticed his boss conversing with other senior executives there. When asked if it was possible to move the Chicago datacenter in two weeks, Bradley's initial reaction was skepticism. However, with the help of a colleague from the networking team, he began brainstorming the project.
The migration, which was initially scheduled for a two-week timeframe, was later extended to six to seven months. Vizient used Pure Storage arrays and had 10Gbps links between the two datacenters. The main challenge was the different IP addressing schemes and insufficient hardware to support all workloads during the downtime. Bradley and his team solved this issue by consolidating workloads onto a subset of servers and storage arrays in Chicago, freeing up hardware that was then shipped to Dallas.
Bradley's team planned a three-night migration push, shifting different parts of the company's infrastructure on each night. The first night saw them move the development environment, consisting of about 70 VMs and three VLANs. They created orchestrations that shut down workloads in Chicago and spun up new VLANs in Dallas ready to run the apps once they arrived.
Bradley's team also had to handle the DNS migration, learning that SRM can glitch when asked to move a server with both DNS and Active Directory, resulting in some workloads needing to remain in Chicago.
The final push to move the production environment took three-and-a-half hours, with minimal downtime of about two hours. Bradley credits his approach to tech work and the combined efforts of his team as the key to the project's success. He offers three takeaways from the experience: transport can be a bottleneck, DNS is always a challenge, and proper planning and communication are crucial for successful migrations.
Written by urgent.news from The Register Science's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
