gah server death
This space is where instructions for Ubuntu install of the Backup Exec agent would go if the server had not died on me yet again. Time to use one of the G4 chassis (grumble grumble)
Thou shalt not fail
Still doing battle with the damnable Backup Server. Backup Exec is up and running now and it’s pulling and getting things to tape, but the drives in the server still like to flake on me. I think using one of the older G3s for this process might not have been the best thing and eventually I’ll need to talk about upgrading it to a new(er) server. I’ll post install instructions, based on what I do at least, for getting the covert, undercover agents working within the realm of Ubuntu. It’s not all that bad in actuality but it seems pretty silly to use a license for a backup agent on a free OS. Once I get more into Xen virtualization, have a decent load balancer and have multiple instances running of all major appliances I can then do backups of those at the virtualization server level and create snapshots using LVM.
Over coffee today there was a lot of discussion about frameworks, of which I wasn’t heavily involved since I’m the padawan learner in this area. I’m also a bit put out by frameworks in that in our small shop the problem will be that there will be a new framework we’d like to work in every two years or so.  What there needs to be is framework for frameworks so they can easily talk to each other or some sort of mystic translator so you can take your Struts setup to Maypole for a year and then press the magic switch and have it shoot over Rails, maybe that’s not as far off as I think.
On another note I need to learn to swing the golf clubs at some point. The family golfs and I have an itchy feeling that there really is no other sport out there that’s better for “networking” than becoming a golfer. I suppose I could go the route of the horrible golfer and comic relief but my competitive spirit wouldn’t allow for that and I’d end spending far too much time and money that I don’t have on getting a little bit better so as not to be the rear of a foursome.
Side note: Macs are awesome except when it comes to vim behavior from server to server and the whole one button thing. Give me two buttons and boot Mac-Roman for Latin-1.
joost
I don’t suppose anyone out there is sitting on a Joost invite they just don’t know what to do with and would be willing to get me in.
back back up exec
Backup Exec server is running once again, all HP drives this time around and I moved the Seagate drives to another server chassis in which I built another Redhat server on so I can test and see if they will blow up on me. I am getting pretty efficient at installing Win 2k3 server at this point, which is not something you want to brag about when you are a sysadmin who’s focus is supposed to be Linux. I must admit though it is nice having one Windows desktop in the mix of servers.
Email annoys me, spammers annoy me and the people that block spam annoy me. In order to block spam there are things which are actually legit via RFCs and email standards but because it possibly could be exploited you can be put on a spam list. Big companies out there will decide to flag you with the evil moniker of spammer based on their own perception of their company wants the rules to be. There was actually a company at one point that marked us as a spammer and the only way to get off that list was to contact the list maintainer and pay them a $1000/hr consultancy fee to figure out why they had decided we were spammers. Eventually this list maintainer fell to the wayside and people stopped using their list, but it was a commonly referenced blacklist for a long time. Email is something we really probably need to allocate a person to specifically handle or stop delivering them entirely, I suppose there are people out there that would take it over for us for some sort of revenue sharing opportunity. I cringe every time I hear revenue share though, I think this really translates to two companies not wanting to fully invest in something and spread their risk, if it’s successful it then falls to disrepair is both side feel cheated about not getting the full bite or if it’s unsuccessful nobody wants to pull the plug on it. Of course that’s not always the case but it does seem to work out that way often.
Sounds like grub is on, time to refuel.
no style
I was thinking that I’d be able to write these entries in a certain style and I’m realizing I have enough problems updating the way it is that throwing the extra burden of trying to stylize them would make it that much harder so I think I’ll skip that for now. I am down at Big Iron today and first thing out of the gate is SpamCop is reporting us as a spammer. Not that were really doing anything wrong mind you, it’s just that Qmail accepts messages by default and then sends bounce backs if it figures out there is not valid mailbox. This is valid mail behavior but the problem is this sets up a situation where a spammer can say their mail is from the person they want to spam to an invalid mailbox on our domain. A body of X comes with that email which then normally gets forwarded back with the bounce notification that things failed. I’ve taken the qmail servers out of the loop for now and I’ll need to apply a local check during the SMTP session to verify valid recipients before it accepts mail in the first place, there’s a few patches out there that do such things.
I need to contact Extreme also, I can go copper from the Pix to the 8806 but I can’t go from fiber directly in and currently I’m going through a small switch. I’ll bet it’s something entirely simple I’m over looking and I’ll feel like an ass once I figure out what it is. I don’t have time to go through the 1000+ pages of command reference to look for what I need at this point though.
Build, build, build the backup server… and this time you will keep running damn you. I think the Seagate and HP drive mix was freaking the array controller out and I really hope it’s not the controller itself.
how long can it last
It’s been a good few years now that I’ve been working for the company oft referred to as The Syndicate. I didn’t start at the beginning of the war but I’ve been there through some pretty ugly spells and some of the brighter days. A few comrades have come and gone in my time and there’s still a few old timers around to bang around with when things get bad or the situation calls for it. Mostly now though my role has me in a black ops mode only surfacing from time to time, lots of solo operations and independent action. This is what has caused me to think it’s time to start a journal, in case something happens and I don’t come back, in case the fight gets to big and someone has to come picking up the pieces. Today for instance we got some news from Corporate M1 through a subsidiary that there was a potential for security protocols to beak down and cause them a potential leak. Personally I’m not a huge fan of M1 but their not the kind of people you want to piss off if you value your life. I had been spending most the day packing up pieces from the recent campaign anyway, so getting a quick job like this brought me out of my haze. Half the task in these matters is figuring out the intel and comms, but once that was in hand it didn’t take long to root out the trouble. Two small caliber rounds and the offenders were neutralized for the time being, I left any real cleanup to the rest of the crew and painted the area bright enough that even the greenies should be able to spot trouble should it come knocking again. No response from M1 after debriefing but that’s often the protocol.
Tomorrow I need to head down to Big Iron, that’s where the brunt of my work seems to take me these days. I envision a future where I’ll have it all tidied up enough that only small incursions are required and I can spend most days running with the crew in the Paper Jungle, but that’s neither here nor there right now. The task at hand is getting our underground network back in place, the B branch that I’d setup for archiving has been taken out by what appears to be a small team of locals multiple times. My first go was a hasty attempt and I hadn’t planned on any resistance. The second attempt was a little better thought out but didn’t prove successful. This next round though I’ll be bringing a little larger ammo and I have a few new structural fortifications they shouldn’t be expecting. I don’t think it will be an in and out operation, but I should be able to lessen the load by making use of the existing agent network which has gone to silent running since the take down of B branch. If possible I need to start mapping Big Iron, it’s a massive area but should I manage to get a good plot of everything my potential for control will be amazing.
Best get some shut eye, it’s tough to be accurate between yawns.
New Facility
Going to the new facility we cut down to four racks of space which was made very much possible by moving from our old 3U 1850 and DL380s we purchased in around 2001, to new DL360 G5s which are 1U. I can run about 4 virtual servers of similar memory and processor on the newer low end DL360 in 1U space as 4 3Us of 1850s took up, not to mention the faster speed of all the components and the fact that it’s a beast to try to get an recent OS release running on hardware that old. My work isn’t done at the new facility but here’s how it’s shaping up right now.
I’ve managed to keep the backs of the racks fairly clean and not have tons of cord wads back there. I removed all of the 6 foot power cords in preference to 3 foot ones as the longer ones ended up leaving a huge mass of cord bundles blocking air flow at the rear.
The fronts of the racks look better, mainly due to the fact of the switch to more G5 servers which are on the bottom. The G1 and 1850 servers located on top need to go soon and there are plans to get all the applications that remain on them off, we just need the time to get the portage done.
The network gear rack still has the most physical work to be done. I wish I would have gotten the cable management in earlier but as I wasn’t sure what I wanted and hadn’t done this before it’s getting done later rather than earlier. The Extreme switch should get hooked up tonight and one of the three PIXs and a small switch should get removed while we then switch over to the other two. Standard at the new facility is CAT6 wiring, this may be a bit overblown for things such as the console cables to the network gear, they could run on lesser wiring but I think in the interest of keeping it clear it will be better to create of a standard and stick to it so everything is the same. There is a set of patch panels which run one on the network rack to one on each server rack so that there will be no cable running from rack to rack to plug things in. There’s a trade off by having more cable to cable connections from the switch to the server but I think signal degradation should be minimized by having CAT6 in there. The amount hassle saved by just plugging to a patch panel in both the network and server cabinet will justify this and the potential for creating problem by tracing cords back and forth will be eliminated in this scenario. Also in many instances the network gear ends up facing in the opposite direction of your servers and thus the idea of hot and cold isles runs into problems when you cable servers and switches directly together, so this should keep both pieces of hardware breathing cold air. This is the network rack in it’s early stages.
So that’s where I’m at now, by the middle of July I hope to have it cleaned up more, maybe I’ll install some neon underglow in the cabinets so they are rock star cool 😉
Clean it up, clean it up
It’s been a crazy last few weeks but the server move has gone off fairly well. I managed to move all of the uclick servers in one weekend from our old facility to the new one. In the mix I managed to change internal IPs of all of our servers which at current count sits at about 50. I had a lot of help from Samuel and Doug in the recoding department and from what I count in subversion we’ve got a bit over 200,000 lines of code in perl modules and a little more than 40,000 in command line programs, my best guess is that’s about half the code we actually have on the servers. To get all that fixed up in less than a week is an accomplishment I think we should be proud of and I’m happy with myself as I’ve been the “official” Sysadmin for a couple of months now. The week before Memorial Day, Memorial Day weekend and a couple days after I didn’t really exist for anything other than work so it was extremely nice of Kate to put up with my work induced absence during those days. I’ve got a couple of photos of the old facility, the first is from the days I went down and pulled all the boxes of items that had been stashed inside of cabinets. I’d say about ninety percent of it was trash that went directly into the dumpster.
There’s various reasons it got to the point that it is and my goal though is to keep the new facility in showable condition. In order to do that I can’t have the network closet look like this
You can’t see it but that’s an Extreme 8806 behind the network wiring and it’s a shame to have it that way, it’s a very nice piece of hardware. These racks were crap really and there wasn’t room mount things to the side, not much additional depth beyond what was needed to get the server into the rack, the mounting posts were fixed and C channels just to add an extra piece of complexity. Our new server racks aren’t the ultimate design but they are definitely a step up from what we had before. Next week I’ll get some pictures of the new racks and the layout I’ve got going there. My networking cable isn’t exactly where I want it to be at in that process yet but I’ve got some items I’ll be getting from Panduit to help resolve that situation. I’ve learned more about Pix firewalls and VPNs in the last week than I will probably ever use in my life again and while I had a fairly good set of cisco configs to work from I had to figure out how two rework two new switches, upgrade and downgrade the OS on a Pix 525 as well as learning how to establish and troubleshoot a VPN tunnel on a 515. Pretty good for a weekends work. Now we’ll be in four racks instead of six but by going from DL380 G1 (3Us) to DL360 G5s the amount of space being saved is tremendous. The new facility still has a few of those old G1 (and 1850) beasts running but they should be on the short timers list and migrated over to newer hardware soon. I would say the new facility is overbuilt, but it’s cheaper to over build a little bit now and take the hit to buy products that allow for expansion at a slightly higher cost than have to worry about how to do it later at higher cost and the possible addition of downtime when we start to notice problems cropping up, that’s my theory anyway.