Shocking… well, not really.
The last few months at my new job I’ve been squishing small and medium bugs to get systems up to par. Service packs, patching, firmware updates, software upgrades, or just organized to make life easier for everyone involved.
Our server room has been one of those infested areas… I’ve been squashing the easy bugs but the room is frankly a disaster waiting to happen. It’s not large by data center measurements, frankly it’s just a small class room with three ceiling mounted cooling units and seven racks of equipment. Three two post network racks and four cabinets for servers. Problem is almost nothing is labled. Power cables, random colors of ethernet, and dayglo orange fiber cables are intertwined in a quilt of choas behind the cabinets. The cable ladder above the racks is about 12 inches too far away and has a large power bus bar below it. But that’s not difficult to fix. Yes, it’s a time consuming job – but working for a college has advantages.
The biggest problem is: I don’t know how much power I have to work with. 30 circuits of power and I haven’t a clue what goes where or how much I’m using.
Today was the big day I was waiting for. An electrician arrived and performed a detailed analysis and audit of our power usage. He started from the UPS inputs and worked through the distribution panel and finally labled and measured the outlets in the server room. This is where my worst fears were realized…
we were this close to a massive cascading power failure. Three circuits have been identified as being over 75% utilized, one is at 96%…
Bad news: Nine servers are connected to this circuit.
Worse news: Three servers totally reliant on it, both power supplies are connected to this.
Even worse news: Two of those servers are part of a three node ESX cluster with twenty two virtual machines hosted on them.
Worse bad news: If that circuit trips, it’ll force the other six servers to pull power from another circuit almost as loaded, which will most likely put it over the top and trip that second breaker.
And, to top it all off: Our UPS load is really unbalanced, but not in a way we can fix with medication. You see, this room is fed with three feeds of electricity called “phases” or “legs”. Equipment like large appliances or electric motors run more efficiently using more than one phase. In this case, the UPS (our battery backup device for the servers) pulls electricity equally from all three phases, conditions it, charges its batteries, and then feeds it to a breaker box. In this breaker box are thirty 20A circuits. Each is connected to one of those phases. Our core switches are large units, so they get two circuits (and two phases) for each of their power connections. It’s a bit complicated, but the simple rule is – load the boat evenly and it won’t capsize.
Right now, phase one is running 3% over, phase two is 33% under, and phase three is 24% over average. So the devation between L2 and L3 is 58%! It’s no wonder the UPSs have only been living for two or three years. When a UPS has to supply power to a system, it performs better when the load across all of its connections are close to the same. Deviations up or down simply chew up UPS components and spit them out. Oh, and there is no UPS maintenance by-pass switch so if the UPS dies – the room dies. If we want to replace the UPS we have to kill the room until the hardwire connection is bypassed by an electrician.
But all is not lost.
Now that I have a detailed map of our power usage and outlets that are labeled, I’m throwing together an emergency change plan to migrate servers onto other circuits to reduce the load on the heavily loaded circuits AND to balance the load across phases.
In August we plan on installing new three phase power distribution units from APC with onboard monitoring and access to all three phases on the PDU. This will make balancing and loading a lot easier. Until then, I’m juggling power cables to anonymous power strips… but at least NOW they’re labeled.
Knowing is half the battle.
TrendMicro Appliance Randomly Blocking
I’ve been fighting an odd issue and finally found a resolution with the assistance of TrendMicro’s support.
A few users (six out of 22k) reported that they weren’t getting email from anyone outside of the network. A few test messages from my web mail accounts (Gmail, Hotmail, and my own domain) revealed an interesting issue.
These few accounts were getting this error:
Hotmail
Reporting-MTA: dns;blu0-omc1-s38.blu0.hotmail.com
Received-From-MTA: dns;BLU119-W30
Arrival-Date: Thu, 3 Jul 2008 06:02:56 -0700Final-Recipient: rfc822;[deleted@for.security]
Action: failed
Status: 5.7.1
Diagnostic-Code: smtp;554 5.7.1: Recipient address rejected: Access denied
GMail
This is an automatically generated Delivery Status Notification
Delivery to the following recipient failed permanently:
[deleted@for.security]
Technical details of permanent failure:
PERM_FAILURE: Google tried to deliver your message, but it was rejected by the recipient domain. We recommend contacting the other email provider for further information about the cause of this error. The error that the other server returned was: 554 554 5.7.1 < [deleted@for.security]>: Recipient address rejected: Access denied (state 14).
I tested SMTP connectivity to the Exchange server by telnetting to the device from outside and inside the network to attempt to narrow down the block. Our Exchange server is protected by TrendMicro ScanMail, and we utilize a TrendMicro Interscan Messaging Security Appliance on our DMZ to provide more spam and virus protection.
I narrowed it down to the IMSA appliance but couldn’t locate the problem in the logs. The MTA logs simply stated Access Denied… not very helpful. So after a short wait on hold, TrendMicro support asked me to deactivate the Network Reputation Services, a learning adaptive IP filtering system that blocks spam senders before they finish connecting.
I later found that the NRS is configured on the appliance AND on TrendMicro’s Email Reputation Service website. lets you create an account using your IMSA’s activation code. Then you can log in and configure the “aggressiveness” of the NRS filters.
If you’ve already laid out the cash for the IMSA, get your email servers registered on this site to make sure they don’t get blocked or at least you’ll have a higher rating with other Trend users on the internet.
It was a frustrating problem that I hope nobody else has, but if they do I hope you find this helpful. If this doesn’t fix it, give Trend a call. Enterprise wait time was less than a minute and had me up and running in less than 10 minutes.
Guitar Hero Robots… rock on, geeks!
A few hardcore geeks have solved the latest problem holding back their community – how to beat some of these insane Guitar Hero songs on expert or Dragon Force’s Through the Fire and Flames on any difficulty.
Well a few ideas are looking promising.
Kevin Herron developed Tom Hannu, a Guitar Hero bot and a YouTube user he uses to post videos of the bot slaying songs on expert with 100% accuracy. The songs are preprogrammed into three applications which then spews out the strums and notes out to a breadboard wired into a dissected Guitar Hero controller. Very cool and very accurate.
Demo Video:
Second bot:
Jeremy Blum, went with more of a hardware attack on this problem.
This slick bot that actually watches the live video using optical sensors to pickup incoming notes and then relaying the strums and notes to a dissected Guitar Hero controller. It’s not infallible, but pretty darn good for not knowing what the next note is going to be until it shows up on screen. Some of the effects in the game mess with the optical sensors – but a few tweaks and I think this will be almost perfect.
Here’s a sample:
Digital Cuba
Circumventing government censorship is nothing new to people in Cuba. The New York 
Times posted a story today about how students in Cuba are challenging the status quo. It’s interesting to see how people get around the limitations other people put on them. For example Havana currently has only one Internet cafe, it’s owned by the government, it costs five U.S. dollars of use. This may not seem like a lot of money but according to the New York Times that’s about 1/3 the average Cuban monthly salary.
People in Cuba have gone to great lengths to get the basic Internet access that we take for granted. The the most popular way to get Internet content is through the use of memory sticks. A lot of different software has been developed over the last few years that allows for offline browsing. The software allows one user to select content and have a download from the Internet and stored on the thumb drive for later viewing. When you think about it anything with a memory stick could actually be a mule for Internet Data. Digital cameras, iPods, watches, in just about anything else with writable memory.
The story also mentions that some industrious people have smuggled in satellite dishes to use live satellite based Internet connections. Most if not all of these connections would be paid for by family members living in the United States earning a much higher wage.
Hotels that cater to tourists are expected to provide free wireless access to their guests. I don’t think I’ve stayed in the hotel and the last five years that hasn’t had this feature. Anyone with a laptop and a wireless card would be able to use this including the local residents. And it appears the locals don’t 
keep the Internet connection for themselves, they tend to download everything they can and share it with others who don’t have access.
Wii Wireless Woes and Fix
Problems have been plaguing my Wii to the point of being useless.
Symptoms include: intermittent connectivity; no connectivity; random error codes ranging from unable to connect to your router to you have no internet connection; all while being able to easily see the wireless access point and connect to it.
It all started when I upgraded my wireless access point to a Linksys WRT54GS router. My old D-Link would spontaneously reboot itself when more than one wireless device was connected and I was using processor intensive WPA wireless encryption. Now that we have two laptops, a Wii, and a few other devices that can use WPA2 – it seemed the thing to use.
I set the WRT54GS to use WPA2 wireless encryption, changed the admin password, and changed the name of the SSID.
All the laptops connected fine. My smartphone connected fine. The Wii could see the AP (with a green, three bar signal icon) and displayed the proper encryption method… but would intermittently fail connection tests. If it passed a test, it would fail almost every time I launched an online channel.
After further troubleshooting I reset the wireless to:
G only. No change.
B only. No change.
Mixed. No change.
Tried every channel from 1 – 12, no difference. All the while my two laptops with Intel 802.11 a/b/g wireless are working flawlessly.
So I powered up an old 802.11b access point that only has WEP, and I left that disabled. The Wii connected fine, test passed, and I could browse with the internet channel. I wasn’t going to leave an open AP on in my neighborhood.
So now I’ve confirmed something is amiss with the Wii.
Solution:
I took my 12 character long alphanumeric password and shortened it to 8 characters. Ding! WPA2 and WPA will work just fine.
I have yet to test > 12 or < 8 character passwords, but right now I’m comfortable with WPA2 rotating the encryption keys every 3600 seconds with an 8 character non-dictionary word password.
If you’re having the same problem, try an 8 character password using random letters and numbers. Even though it’ll be shorter than I’m usually comfortable using, making it hard to guess will keep you safe.
