“Tubes” Senator Indicited for Lying to Feds
Today, Alaska Senator Ted “Series of Tubes” Stevens was indicited for lying to federal investagors. Apparently he falsified some information on finacial records.
Stories like this reaffirm my belief in karma.
Senator Stevens became famous for his explanation of how the internet works by comparing it to a series of tubes, but also explaining that the internet isn’t a big truck. Yes, it sounds just as retarded listening to him explaining it as I did… but I posted a youtube video of him trying to explain.
Now remember, he’s trying to convince people on why Net Neutrality shouldn’t be enforced. Because we all know it’s okay for ISPs to filter data off the internet we’re using… like Charter nuking Vonage connections, Comcast dropping Peer to Peer connections, and the like. Good riddance Ted.
Listen in here:
Shocking… well, not really.
The last few months at my new job I’ve been squishing small and medium bugs to get systems up to par. Service packs, patching, firmware updates, software upgrades, or just organized to make life easier for everyone involved.
Our server room has been one of those infested areas… I’ve been squashing the easy bugs but the room is frankly a disaster waiting to happen. It’s not large by data center measurements, frankly it’s just a small class room with three ceiling mounted cooling units and seven racks of equipment. Three two post network racks and four cabinets for servers. Problem is almost nothing is labled. Power cables, random colors of ethernet, and dayglo orange fiber cables are intertwined in a quilt of choas behind the cabinets. The cable ladder above the racks is about 12 inches too far away and has a large power bus bar below it. But that’s not difficult to fix. Yes, it’s a time consuming job – but working for a college has advantages.
The biggest problem is: I don’t know how much power I have to work with. 30 circuits of power and I haven’t a clue what goes where or how much I’m using.
Today was the big day I was waiting for. An electrician arrived and performed a detailed analysis and audit of our power usage. He started from the UPS inputs and worked through the distribution panel and finally labled and measured the outlets in the server room. This is where my worst fears were realized…
we were this close to a massive cascading power failure. Three circuits have been identified as being over 75% utilized, one is at 96%…
Bad news: Nine servers are connected to this circuit.
Worse news: Three servers totally reliant on it, both power supplies are connected to this.
Even worse news: Two of those servers are part of a three node ESX cluster with twenty two virtual machines hosted on them.
Worse bad news: If that circuit trips, it’ll force the other six servers to pull power from another circuit almost as loaded, which will most likely put it over the top and trip that second breaker.
And, to top it all off: Our UPS load is really unbalanced, but not in a way we can fix with medication. You see, this room is fed with three feeds of electricity called “phases” or “legs”. Equipment like large appliances or electric motors run more efficiently using more than one phase. In this case, the UPS (our battery backup device for the servers) pulls electricity equally from all three phases, conditions it, charges its batteries, and then feeds it to a breaker box. In this breaker box are thirty 20A circuits. Each is connected to one of those phases. Our core switches are large units, so they get two circuits (and two phases) for each of their power connections. It’s a bit complicated, but the simple rule is – load the boat evenly and it won’t capsize.
Right now, phase one is running 3% over, phase two is 33% under, and phase three is 24% over average. So the devation between L2 and L3 is 58%! It’s no wonder the UPSs have only been living for two or three years. When a UPS has to supply power to a system, it performs better when the load across all of its connections are close to the same. Deviations up or down simply chew up UPS components and spit them out. Oh, and there is no UPS maintenance by-pass switch so if the UPS dies – the room dies. If we want to replace the UPS we have to kill the room until the hardwire connection is bypassed by an electrician.
But all is not lost.
Now that I have a detailed map of our power usage and outlets that are labeled, I’m throwing together an emergency change plan to migrate servers onto other circuits to reduce the load on the heavily loaded circuits AND to balance the load across phases.
In August we plan on installing new three phase power distribution units from APC with onboard monitoring and access to all three phases on the PDU. This will make balancing and loading a lot easier. Until then, I’m juggling power cables to anonymous power strips… but at least NOW they’re labeled.
Knowing is half the battle.
Life update.
It’s been a while since I’ve updated this blog, the real life has been keeping me busy but I try to get to the forums every day or two.
My new career has kept me hopping. The college is busy and turn over has lead to an open position that I’m going to try to fill. Unfortunately it was the position that performed similar duties, so the team’s current work load has increased.
I have a few large projects under my wing. I’m working hard to meet our budget restraints but keep service levels high.
We’re looking at new UPS installations in two campus buildings, one a small 6-10KVA and the other is for our core server room weighing in at 60-80KVA. We lack a generator and no budget to buy one so we’ll need to run on battery long enough to shutdown gracefully in the even of a power outage.
Ramp up our ESX cluster and virtualize as much as possible. Many of our HP servers are out of warranty and still run critical apps. I’ll be starting with some of the easier redundant servers and working toward more critical servers as we balance the load and ensure our core switch fabric can handle the increased iSCSI load the ESX and SAN environments will be placing on them.
In my spare time, I’m troubleshooting our TrendMicro AV solution along with Exchange email issues that creep up. Personally, I think Trend is bucket of chum with a pretty bow on it – but now isn’t the time to forklift in a new solution, it’s time to wring as much support out of them as our contact gives us… so far I’m less that impressed at their customer service. It’s frustrating when you want to give your clients the best service possible, but a vendor prevents you from meeting those goals.
On the home front, life has been equally busy. Sam crawled for the first time. He’s getting more mobile every day. Anna scored three ribbons in her last swim competition… I could see her smile from a block away when she brought them home.
Nap time is over for Sam… time to save and log off. Peace!
First day.
First day of my new job went pretty well. Half of the day was supposed to be spent going over benefits and legal stuff but they shifted that to the afternoon. So my morning was spent getting a more formal tour of the central campus.
The college is a distance learning center, they have a lot of nifty audio video technology including polycom systems in a theater classroom for teaching large classes remotely or two classes in different locations. They also are planning to do media streaming, pod casting, and other cool stuff over their intranet.
The campus has a nice cafeteria onsite – which will be great in the winter time. Even though I’m ten minutes from home, when old man winter is waiting for you in the parking lot it’s best to hunker down at work.
I also got to see the loading dock / storage garage / staging area that they queued up their new Dell workstations. All of them have been imaged and preconfigured with our specific image directly from Dell, even the BIOS is locked down with our password straight from the factory.
I pulled in the driveway at 5:16pm this evening and started in on my list of to-do’s.
