Binaryspiral

My original post included a quick and dirty test on raw hard drive performance using HDTach to give me an idea of what I was working with with my new DroboPro. Of course true to any benchmarking test – there are so many metrics that can be tested that may be able to give me a clearer picture, but I didn’t have the luxury of time to do so.

After publishing my testing, I felt that it was worth going back and getting more data. It’s obvious from the hit count I’ve seen – that this is an interesting topic that a few people actually find interesting. I’m in a unique position as a consumer to be able to test this unit in a great development environment against some really good equipment that others can identify with.

So, without further ado… here’s what I did.

New testing utility – the venerable IOmeter to test the same virtual machine on a vSphere environment.

This time we hit the same three storage devices:

1. Direct attached storage: two 72GB 2.5” SAS disks, spinning at 10k rpm, and attached to an HP P400 raid control card with BBU and 512MB cache – configured in RAID 1 (mirrored).
2. HP/LeftHand NSM2120 storage modules (2 units clustered for availability). Each unit has 12 x 500GB 7200 RPM SATA disks, each six disks are configured in a RAID 6 configuration.
3. DroboPro – with 8 x 500GB 7200 RPM SATA disks, configured in single disk failure protection. Firmware 1.4.1.

with four tests: 4k read test, 4k write test, 32k read test, and a 32k write test.

On each test, we give the test a 5 second ramp up time (especially useful on throughput and response numbers so that cache on all the devices doesn’t skew the numbers). Each test runs for 15 seconds.

Test 1 – Megabytes per second

This test gathered during each of the four tests – shows the total throughput each device was able to push over the 15 second test window.

As you can see the DAS was over the top on reading data – but actually the raid controller hurt the write performance, even below that of the NSM and DroboPro. In this test the DroboPro held its own against the NSM for throughput on all four tests. I was impressed on this, because up until the end of last year – these NSM units were the primary storage for 60 VM servers in our production virtual infrastructure. No, they weren’t fast – and performance was always an issue – but it ran without fail. So I’m comfortable a unit that can match these units would be worthy of a development environment or even a small production requirement.

Results 2 – IOPS (Input / Output Operations Per Second)

This test again, gathers the total number of iops over the entire window of the test. This metric is especially interesting to database administrators or exchange administrators who have users that complain of slow access. Anything transaction based will perform based on these numbers. The DAS wipes the board with its read performance then again drops to below the two units with write performance. The DroboPro achieved an average of 63% of the NSM’s IOPS performance. Ranging from 78% as fast on the 32K reads, to only 34% of the 4k read.

Results 3 – Response Times

Response times are an average during the entire test for each device for each test.

The DroboPro was consistently slower – but not by much – than the NSM unit. Which is impressive considering the cost differences (and annual support renewal costs).

The maximum response graph is also included to show you the peak response times, but these are the single highest access times at any time during the tests – the average times are a better representation of the overall performance.

Final Thoughts

I have users who have local storage requirements of over 2TB and this unit will fit perfectly on their desk. It’ll also take a load off our SAN and the network segments between their desk and our storage networks. It’ll also provide them a considerable amount of space that they can manage themselves.

If When a drive dies, I send them a new one – any size will work. It won’t have to be an HP 80GB or a Dell supplied 120GB, just whatever is on sale or sitting in my cabinet of spare parts. If they need more room – I send them something bigger. I can still pull backups of this unit over the network using our Veritas / Symantec Backup Exec DLO so the data is still safe.

VMWare Answer

Performance is much less of an issue now that I’ve seen a more detailed battery of tests. The LeftHand units I tested the Drobo against were capable of hosting 60 virtual machines for an ESX 3.5 environment of 3 ESX hosts. The numbers I’ve seen today have been impressive now that I’ve received these new numbers from testing.

The DroboPro is not the VMWare answer Data Robotics has any longer – even their website has stopped pushing the Pro for VMWare users and are now courting customers with the DroboElite.

Unfortunately the costs of the Elite price it quite higher than the Pro (2x) for the same storage capacity. Although I’d be glad to test it, if Data Robotics would like to provide me a unit to beat on for a week or two to see if the additional costs are worth it. However as of this writing, I can’t get one from our supplier (CDW-G) without hard drives.

Another hitch, remote monitoring.

Until Data Robotics provides me a way to receive failure notifications or monitor the DroboPro remotely (with SNMP or the like) without running the Drobo Dashboard application on a workstation – I can not recommend it for production environments.

As one user in a forum suggested – they could put a web cam in front of the unit to watch the lights on the drive to alert them of issues. I know this is in jest, but seriously – why hasn’t Data Robotics thought of this as part of their rack mount kit? They’d like us to place this unit somewhere other than our desks – so who wants to play roulette with 3 TB worth of data sitting in a rack somewhere?

-Update-

I’m testing DroboPro’s dashboard service/application in our development environment that may provide me with remote monitoring capabilities. Data Robotics also has a DroboElite en route to me for testing. I’ll post the raw data and details as soon as I get the unit and test it.

Recently I was able to purchase and test two Drobo storage appliances.

A DroboS with five disks and a DroboPro with eight disks. All the disks are the same size, speed, and manufacturer. The specs are listed further below for your consumption.

The DroboS is attached via USB 2.0 to my desktop workstation – a Dell OptiPlex 745 with 4GB RAM and running Windows 7 (64bit). I have an eSATA card on order. I’ll be posting more details on this specific setup at a later date.

The DroboPro is the device I’m most interested in. Depending on the test results that I get – it may live in a remote campus building attached to two small ESX hosts with six virtual servers on them.

Currently our environment is hosted on three MPC DataFrame 120’s. These units are no longer covered by manufacturer’s warranty due to MPC’s crash and burn bankruptcy. However, I have two HP/LeftHand 2120 storage modules that could replace them, but it may be overkill. The HP units each have 12 disks and use much more power than our existing MPC units or the Drobo. Two domain controllers, two file servers, a print server, and an application server.

Read on for the gory details.

Test Environment

VMWare Host:

· HP DL360 G6 (Sixth Generation)

CPU: 1 x Intel Xeon E5530 Quad Core 2.4 GHz with Hyper threading enabled (8 total cores available)
Memory: 3 x 4GB PC3-10600R RDIMMs DDR3
Network: Six total 1Gbps Ethernet ports (2 on-board, 4 port PCIe NIC)

· Local Storage:

HP P400 Raid Controller with 512MB Cache and Battery Backup Unit (for full read/write caching)
4 x 72GB 10k RPM SAS (2 x RAID1)

· NAS Storage

2 x LeftHand (HP) NSM 2120 Storage Modules (mirrored but not load balanced)
2 x RAID 6 arrays and combined into one storage pool using SAN iQ

· VMFS volumes:
3 x 1TB (vmware default configurations)
1 x 250GB (vmware default configurations)

DroboPro

1 x 1Gbps Ethernet connection
Latest firmware (1.4.1)
8 x 500GB 7200RPM SATA
Single drive failure protection (dual disk disabled for testing)

· VMFS volumes:
2 x 2TB (8MB blocks, per best practice documentation from Drobo)

· Network:

1 x HP 3500 YL gigabit Ethernet switch (24 port – 101Gpps backplane)

· Testing Software:

Host OS: vSphere 4.0
VM OS: Windows XP SP3 (1GB RAM, 9GB HDD)
Notes: VMware tools and latest virtual hardware upgrades applied.
HD Tach 3.0.4.0

Notes:

HD Tach uses base 10 for conversion of bytes to megabytes and gigabytes. I use base 2 to reflect the OS and application measurements.

First Test:

Test VM on LeftHand 1TB VMFS volume.

We start off slow but maintain a high throughput with minimal latency, averaging 8.4ms on the random access tests. Average read speeds are 51.5 MB/s with a burst above 61MB throughout the test. The test bandwidth ranges from 28.6 MB to over 66 MB, but mostly staying above 42 MB during the entire test.

Second Test:

Test VM on Drobo 2TB VMFS volume.

After migrating the same test machine over to the Drobo, I ran the exact same test. We see much less top end and very low valleys on the ranges of throughput. The test ranges from 10MB on the lowest to a hair over 45MB/s on the bursts. The throughput averaging 27.56 MB/s with a latency of over 450ms. I reran the test at a later time and had a better result.

Third Test:

Test VM on Drobo 2TB VMFS volume (#2)

I reran the HD Tach test on the Drobo to test, this time using 32MB blocks. The random access latency was much better at 14.4ms. The throughput remained the same – averaging 29.85 MB/s, the difference easily attributed to the larger block size.

Fourth Test:

Test VM on Drobo 2TB VMFS volume during a storage vmotion:

I started a storage VM Motion of a powered off clone of the test VM. This copy was being svmotioned to the second VMFS volume on the DroboPro, and finished during the first 3^rd of the test. As you can see, the disk access for the running machine was crushed by the svmotion process of the vSphere host.

Fifth Test:

Text VM on HP local storage.

I also wanted a benchmark in this test to prove the virtual machine and the host are not the bottlenecks in this test. I’ll let the graph speak for itself.

The sustained throughput and burst speeds are more than enough to prove that vsphere host and virtual hardware have opened up the floodgates to high I/O applications.

Final Thoughts:

I’m expecting too much from this SMB storage device. While I point out the performance issues I found, it really isn’t fair – I’m comparing apples to oranges in this environment.

The Pro is really a good unit and has a lot of potential for SMBs or workgroups. It’s brain dead simple to use and manage. The Drobo line of products are devices that “just works” out of the box. Plug it in, slap a pair of hard drives in, and turn it on.

I don’t think I can recommend this unit for the original purpose it was purchased for: Shared storage for two ESX hosts and 4-6 virtual servers in a remote campus. Even though it is certified by VMWare, it’s apparent by the tests that it would not support more than a few virtual machines – and certainly not any file servers that require a decent storage subsystem to keep users from complaining about slow file access or delays.

This is not to say “don’t buy it” because I really do think it’s a worthy product if you need a lot of storage that can be upgraded over time with very few technical skills. What other device out there can you yank a 500GB drive out and slap in a 1TB and start using it right away? That’s the power of these units.

I can think of a dozen situations here at work that this unit would be a perfect fit. Just not this one.

The lack of monitoring (remote or self) also removes it from our production environment. I brought this question up with Drobo support. They were very prompt in their turnaround, as I received a reply in less than an hour with a follow up question and then another 45 minutes with a recommendation.

They recommend I install the Drobo dashboard software on a virtual machine and setup email alerts from within the software. The catch is that the dashboard is an application – not a service. I would have to remain logged in for the dashboard to be running. This is not an option.

Alternatives:

The DroboPro is not the biggest dog in Drobo’s kennel. They recently released the Drobo Elite which offers a faster processor, a second gigabit nic, and some additional file system features to allow for multiple computer access from the network. The cost is about double what the Pro was.

However, I was hoping for more performance from an 8 drive unit than I got. It’s just not fast enough for my environment. I will post additional test information from my DroboS when I get the eSATA card installed. I’m looking forward to that!

-Update-
I reviewed my test data and methods and wanted a clearer picture of where the DroboPro was on my chart of storage. So I retested the DroboPro with additional software and tests for additional data. Read on to DroboPro Testing, Part 2.

VMWorld 2009  and many other VMWarevirtualisation topics are still rolling around in my head. So many ideas, so many paths… so many vendors calling me after the conference.

Some interesting tidbits of discussions included baremetal VMWare View clients. In response to portable thinclients like the WyseMobile Thinclient, VMWare wants to install its hypervisoron your hardware then install the VM workstation on top of that – checking in the deltas to a central server for management, backup, and business continuity.

VMWare View workstations with support for VoIP, presumably leveraging the PCoIP protocol learned from Teradici.

And this week after a 64GB memory upgrade on our virtual desktop environment, the talk of thinclients has risen to a roar. It looks like we’re going to be spending upwards of a half a million on workstation refresh this year – do we plunk down a core duo micro tower twenty five times a room or invest in a virtual desktop environment that’s infinitely easier to manage and much more mobile.

We’re also implementing a new NetAppSAN. The gear just arrived this afternoon – five pallets worth. It’s a thing of beauty, and it’s not even out of the box yet. Dual FAS3140 controllers, 28 x 450GB 15,000 RPM FiberChannel drives and 14 x 1TB 7200 RPM SATA drives.

We’ve already moved the LeftHand/MPC equipment to a temp rack  until we can migrate the data off of them.

It’s going to be a busy week.

I’ve started running with Kathy a few weeks ago – she’s already kicking my arse, but supportive of my efforts to run. Nike+ and my iPod have made running almost fun. Feel free to add me as a friend if you’re out there… just don’t point and laugh – I’m a noobie in this endeavor.

I love deliveries. Especially this time of year. UPS, FedEx, AirTrans, carrier pigeon… I don’t care – it’s usually something expensive and always something that is going to make my job easier.

Today, FedEx delivered a pallet of new HP servers and parts for our new campus. The pallet also contained a few parts for servers we had just got last week.

So I’ve got five new HP DL360 servers, sixth generation. HP just released their new line last month with the new Intel 5500 Xeon processors. I like to compare them like the pro version of the i7 consumer chips. Four hyperthreading cores with onboard memory controller per chip.  Yeah, and even though it matches clock speed with our existing G5 servers – it’s smoking fast.

Opening the little 1U server chassis, shows a lot of room for expansion – given the amount of gear this unit has already. It has an onboard raid control card that can address up to eight 2.5” SAS or SATA drives. It also has an IDE controller for optical media. It also includes a USB port and SD Card slot on the motherboard… great for those moronic copy protection dongles or emergency boot drives or utilities.

I’m not going sit here and try and sell you a server by just spewing specs… what HP really did to impress me is cut the noise and power usage so drastically I seriously thought there was something wrong with it.

These servers are usually so loud I can’t build them at my desk – I had to take them and bench build them in our staging room. Not anymore.

I actually had this DL360 G6 installing windows 2008 64bit from DVD on a bench next to a Dell OptiPlex 755 sitting idle. When I placed my head between the two to check if the fans were actually spinning on the HP – the Dell was louder. I have never actually heard an SAS drive until today… amazing.

After diving into the onboard monitoring systems, I found out how they are able to keep the fans spinning at 19% while keeping cool – 28 onboard temp sensors watching everything in the box… if a section gets warmer – only the fans dedicated to that area increase their speed and only as much needed to move more air to cool it.

With a single quad core processor, three 2GB memory cards, four 10,000 rpm hard drives, and a four port gig nic PCI-x card – this server only pulled 130 watts of power out of both power supplies at its peak. When it was idling it sat at 93 watts. The only time I ever heard the fans is when I started the server after that near silence.

Yes, I’m that impressed with this new line – I’m looking forward to the next year when we upgrade our ESX environment to G6 host servers… Maybe I won’t be able to hear the server room from down the hall.