Who turned out the lights?

We had one of our racks at the hosting facility lose power last night. I had thought everything was setup so that we had redundant power supplies and this would not be a problem, apparently that was not the case. Do to our switchover from some old hardware to new hardware the rack that went down also happened to house our newer 1U servers (instead of the old 3Us) which meant when 1 of 5 racks went out it took out more then half our active servers. The hosting site we are in is closing down and I really hope when we move to a new one this summer we can address all these problem and do more to ensure that something silly like that doesn’t happen again. I’d gotten there and started pulling plugs on things and moving the down servers power over to other racks so we could at least get them back online, as I had no idea how long it was going to take for the facility guys to get out there and get power back to that rack. Mike (the Sysadmin) ended up coming out there even though he’d called in sick that day which I was very thankful for, as I had no contact numbers for Switch and Data or the people that were on call for the facility nor any sort of relationship with them. It was much easier for him to get ahold of everyone and direct traffic, and it was nice having him out there as one of the old servers blew a CPU fan and he could work on that while I was in back doing all the replugging. Had there have been some sort of network issues or one of the other boxes that I don’t have a root login to shown issues I would have been in a world of hurt without him. In the end repairs happened quicker than I thought they would have, so my unplugging of things probably created more work than was needed as I had to reroute everything back to that rack. Had the knowledagable person who had previously been running the facility not just flown back into town I doubt that it would have been as quick a job to get the power back. So Friday ended up being an awefully long day at the office and hopefully there are no gremlins this weekend and I get a little bit of relaxation.

I did lean a useful tidbit, as I hadn’t worked with the 1Us much, the new ones have a little light button that you can push on them that turns on an led in the front and the back to make it easier to tell which server is which once you walk around the cages. I always had a problem with trying to remember it was the 6th server down from the top, and then by the time I got around to the back I’m second guessing myself as to which server it was. A very small thing simple thing, but so very helpful.