They say timing is everything. This is especially true when you plan a server deployment. When crucial equipment arrives at the wrong time, it can ruin even the best plans. Even so, good planning and a good infrastructure can make up for even the worst timing. For a good example of this I only need to recall our last corporate retreat.
A corporate retreat is a good time to get to know your coworkers in a relaxed setting. While some companies head to the beach and others to the forest, we headed to the hills; specifically, to a ski lodge for a weekend of both team building and relaxation. The surroundings were scenic and remote with plenty of comforts (such as in-room wifi to keep the geeks in the group happy).
For the entire week leading up to our retreat, we had been waiting on some critical servers to arrive in our United Kingdom data center. This extra capacity was important to a client and we had a team of people ready to deploy software to the servers once they were installed. As we left for the retreat on Friday, we had basically resigned ourselves to a missed deadline--the servers still were not delivered, so there was no way we would have them ready by the beginning of the following week.
Timing is everything. Shortly after the end of the opening ceremonies for our retreat, we found out (surprise!) that the servers, which we had never actually seen, mind you, had arrived. After some quick discussions among the management, they decided that this deadline was critical enough that it trumped the retreat activities. Now with our traditional infrastructure in place, this would mean that I would need to fly back home, grab my passport, and then take the next flight to the UK. Once there, I would rack the machines and pop in the latest OS install CD to set them up.
I have written previously about our team's goal of "administering from the Bahamas." This wasn't just a slogan we kept to ourselves, we had been actively promoting to management this id
To continue reading for free, register below or login
To read more you must become a member of SearchDataCenter.com
ea that no matter where we were, we could set up and repair servers with this new remote management infrastructure. In theory we should be able to set up servers from anywhere, even this ski lodge.
Well, here was our chance to prove this theory. Over the rest of the evening we planned the chain of events. First we had to work out how to get a remote console on these blank machines. Luckily, these servers use DHCP by default to get an IP address for their remote lights out management. We arranged for the "smart hands" to rack and connect the servers and that evening while on top of a peak accessible only by cable car, we configured our remote DHCP server with our EVDO-connected laptops (you get surprisingly good cell reception on the top of a mountain).
The next morning we got word that the machines were racked and immediately started on the install. We had instructed the smart hands to email back the asset tag information, which included the default remote management password. We had four servers with four different passwords, but since we could check which IPs had been handed out, process of elimination could pair each IP with its respective password. As we sat around a table in our hotel room the first breakthrough came: we successfully logged in to all four machines and could access their remote power and remote console.
With remote control over the machines, the big test was next--we had to install an operating system on a server from thousands of miles away. In preparation for this setup, we had configured a PXE server in the UK data center and copied over the latest packages from our operating system install. When we told these machines to boot off of the network, the PXE server would present us with a menu containing all of our OS choices and allow us to set extra settings like the hostname for the machine. The PXE server also doubled as a kickstart server, so instead of installing the OS from a CD, these machines could pull all of the files they needed directly over the network.
Finally, we powered up the first machine and booted it over the network. There was our friendly PXE menu. A few keystrokes later we watched as it partitioned its drive, installed its OS, and finally rebooted to the familiar Linux login prompt. About that time a new email came in--it was the server sending me information about its new OS. Success. All that was left was a few tweaks, a few kickstarts, a few high fives, and we sent out the notice that the machines were up and ready to go--all in time for dinner.
Not only were the servers ready on time, we were still able to enjoy the rest of the weekend. Sure, a corporate retreat is a great place for team building. But if you have the right remote management in place, it turns out it's not such a bad place for server building either.
ABOUT THE AUTHOR: Kyle Rankin is a systems administrator in the San Francisco Bay Area and the author of a number of books including Knoppix Hacks and Ubuntu Hacks for O'Reilly Media.
