Sophos ES4000 Active Directory Fun
The college recently purchased a new Sophos Email Security appliance model. It was very easy to setup and I’m looking forward to having PureMessage filtering our spam and crapmail attacks, it’ll be a good thing.
The Active Directory integration is not a polished as their Web Security appliances’ are. We have two WS1000 appliances, also from Sophos. Both hooked right into AD and pulled down both students and staff accounts without issue. Even indicated what sub-domains it found during the process. Top notch, no brainer installation.
The problem I’m writing about is the ES4000 appliance’s inability to detect our second domain in the same forest as the domain our service account is in. First off, it couldn’t even automatically detect settings using the same service account using the “Detect Settings…” feature. An undocumented bug was documented on experts-exchange.com with the workaround being you have to use an account with Schema Admin privileges in the domain’s original Users OU. Once detected, you could move the user and modify the DN used to authenticate.
Okay, that one was fixed. But I still couldn’t sync both staff and students – even if I pointed the Base DN to the top domain or left it blank.
I opened a case with Sophos and went through first level support. After 48 hours (plus a weekend) of remote support they kicked me to second tier.
Second tier connected remotely and continue the troubleshooting. After an hour or so they found a workaround and had me test it. Success.
Fix: Replace the Base DN for users/groups with a single space. Done and now it works. I’m not much of an LDAP junkie, but I would consider that a bug.
Anyway, it works for me and I hope it helps someone else out there scratching their head wondering why the eff their ES4000 is not working.
Side note: All in all, Sophos support is pretty good I just wish they would read my entire email before firing back the first canned response that essentially was exactly what I had already done. For anyone absolutely buried with this product, I can highly recommend leveraging their consulting services. Well worth the small price to get it done right the first time.
Rotten BlackBerries Make Good Whine.
Now to be perfectly fair, I’m not going to blame all of my issues on BlackBerry or their Enterprise Server. I will however, consider their method of message relay and integration with our environment quite a hack and their support staff a challenge to work with, especially late at night.
That being said… here’s my problem and what I’ve found that fixes it – I hope it may help you.
Read more
Journalspace – overwritten
Popular blogging site journalspace.com has been wiped off the Internet by a suspected disgruntled engineer. The engineer was fired months ago for stealing from the company but may have planted a logic bomb or delayed virus that initiated a full server wipe.
The server in question held the database for journalspace.com, which is the contents of every blog on the site. It was confirmed today that DriveSavers was unable to recover the data on the mirrored hard drives. Journalspaces did not Use any other form of backup so they are unable to recover their site or any used content. They, however, recommend checking Google’s cache to attempt to save published content.
There is no excuse for not having backups on tape or even an external drive. This disaster could have easily been recovered from with minimal data loss using free software and a hundred dollar hard drive.
Vendor Hell
A vendor that shall remain anonymous (for now) has almost guarenteed they won’t be getting future business for me any longer. The sad thing is that it wasn’t directly related to any mistake they made, but how they follow up on mistakes made by their suppliers.
Frankly, I don’t care who makes the mistake – they sold us the product. And on three seperate occasions couldn’t tell us when it was going to be delivered or where it was in transit. Another incident happened yesterday where we received the completly wrong product, even though the packaging slip shows the correct one. Sure, nothing we can blame our vendor directly – they put the order through correctly. But when I call and get no response on when the right one will be arriving, I begin to discover why these schmucks guys were the lowest bidder on our government contract.
Unfortunately this little snafu has thrown a huge monkey wrench into our schedule. A typically lenient college schedule only affords us a few weeks of network and service instability before everything must be running again. A luxury in most companies, I’ll admit that. But then again, we’re working with a smaller budget. Now everything is on hold and nobody can give me an answer.
I’ve unleashed the dogs and hopefully we’ll get some answers today, but I have to say a certain company has lost a lot of respect in our department.
Charley Delta Whisky… folks, it can’t get much clearer than that.
Shocking… well, not really.
The last few months at my new job I’ve been squishing small and medium bugs to get systems up to par. Service packs, patching, firmware updates, software upgrades, or just organized to make life easier for everyone involved.
Our server room has been one of those infested areas… I’ve been squashing the easy bugs but the room is frankly a disaster waiting to happen. It’s not large by data center measurements, frankly it’s just a small class room with three ceiling mounted cooling units and seven racks of equipment. Three two post network racks and four cabinets for servers. Problem is almost nothing is labled. Power cables, random colors of ethernet, and dayglo orange fiber cables are intertwined in a quilt of choas behind the cabinets. The cable ladder above the racks is about 12 inches too far away and has a large power bus bar below it. But that’s not difficult to fix. Yes, it’s a time consuming job – but working for a college has advantages.
The biggest problem is: I don’t know how much power I have to work with. 30 circuits of power and I haven’t a clue what goes where or how much I’m using.
Today was the big day I was waiting for. An electrician arrived and performed a detailed analysis and audit of our power usage. He started from the UPS inputs and worked through the distribution panel and finally labled and measured the outlets in the server room. This is where my worst fears were realized…
we were this close to a massive cascading power failure. Three circuits have been identified as being over 75% utilized, one is at 96%…
Bad news: Nine servers are connected to this circuit.
Worse news: Three servers totally reliant on it, both power supplies are connected to this.
Even worse news: Two of those servers are part of a three node ESX cluster with twenty two virtual machines hosted on them.
Worse bad news: If that circuit trips, it’ll force the other six servers to pull power from another circuit almost as loaded, which will most likely put it over the top and trip that second breaker.
And, to top it all off: Our UPS load is really unbalanced, but not in a way we can fix with medication. You see, this room is fed with three feeds of electricity called “phases” or “legs”. Equipment like large appliances or electric motors run more efficiently using more than one phase. In this case, the UPS (our battery backup device for the servers) pulls electricity equally from all three phases, conditions it, charges its batteries, and then feeds it to a breaker box. In this breaker box are thirty 20A circuits. Each is connected to one of those phases. Our core switches are large units, so they get two circuits (and two phases) for each of their power connections. It’s a bit complicated, but the simple rule is – load the boat evenly and it won’t capsize.
Right now, phase one is running 3% over, phase two is 33% under, and phase three is 24% over average. So the devation between L2 and L3 is 58%! It’s no wonder the UPSs have only been living for two or three years. When a UPS has to supply power to a system, it performs better when the load across all of its connections are close to the same. Deviations up or down simply chew up UPS components and spit them out. Oh, and there is no UPS maintenance by-pass switch so if the UPS dies – the room dies. If we want to replace the UPS we have to kill the room until the hardwire connection is bypassed by an electrician.
But all is not lost.
Now that I have a detailed map of our power usage and outlets that are labeled, I’m throwing together an emergency change plan to migrate servers onto other circuits to reduce the load on the heavily loaded circuits AND to balance the load across phases.
In August we plan on installing new three phase power distribution units from APC with onboard monitoring and access to all three phases on the PDU. This will make balancing and loading a lot easier. Until then, I’m juggling power cables to anonymous power strips… but at least NOW they’re labeled.
Knowing is half the battle.
