TechHead would like to welcome James Pearce as a contributor to the site*.   James is a Kent based qualified accountant, currently working in information security and technical architecture with  most of his  time “being spent on virtualisation and business continuity at the moment”.

James has written the following good article on his experiences installing and running a Dell PERC 5i controller inside his HP Proliant ML115 G5 VMware ESXi home lab server.  I have personally only used an HP e200 array controller (as well as the onboard) in my ML115 or ML110’s so far and have found it good though the Dell PERC 5i does, in my opinion, offer advantages such as cost and the ability to present the status of the controller and attached disks back to ESXi.  After reading James’ article I may have to go and buy myself a PERC 5i for my own home lab.

Thanks James for such an informative article and I hope you find time to write some more. 

I’ve recently gotten ‘into’ ESXi thanks to the excellent TechHeads article on getting the quad-core ML115 G5 running with ESXi 3.5 U4. But I wanted a hardware RAID solution and could only find a load of sketchy information on the options for this machine, besides very expensive “supported” options from HP of course. So what we have here is a quick guide, backed up with performance and power consumption data and integration hurdles, on getting a Dell PERC 5i RAID controller working with the ML115 G5.

Base Machine

• HP Proliant ML115 G5, Quad-core Opteron 1352 CPU (2.1GHz)
• 8GB ECC RAM (4x 2GB)
• 3x Western Digital RE2 500GB SATA disks
• 1x Western Digital Caviar 1TB Green Power SATA disk (used to backup the VMs)
• 8GB USB drive (internally mounted) booting ESXi 3.5 U4

Why Hardware RAID

Two reasons – performance and reliability. When a disk goes bad, I won’t lose all my VMs and my server will still work. The health of the disks attached to the PERC 5i are also reported in the ESXi management console (unlike the onboard controller).

Why the Dell PERC 5i

I’d read several references to using the Dell PERC 5i in the ML115 and so I took the plunge and ordered one from eBay, brand-new for under £100 complete with a four-way SATA cable and 512MB of cache installed. I’ve no idea why these are so cheap on eBay, but this seemed to be at least half the price of anything else and, since it serves well in some of the most popular enterprise servers, it seemed good enough for me.

Of course nothing is ever simple and two problems were immediately present,

• There is insufficient ventilation in the ML115 expansion slots for the card as-is; it would quickly overheat, and
• The card was delivered without the battery-backup, which is absolutely critical for ESXi with RAID.

The card I got was shipped for Dell systems with a custom carrier, but fortunately the seller included a standard PCI bracket with it. They are also available even from Dell themselves with a standard PCI bracket.

Cooling the Card

The PERC 5i has two processors, an Intel IOP333 and an ARM 1068. In a Dell server it would be directly in the air flow path, so is fitted with only a small heat sink on the Intel chip despite its 11W TDP. Although a tiny fan could be mounted on the heat sink, and would probably be sufficient (the IOP333 has a maximum working temperature of 95?C), I was nervous about relying on an unmonitored fan as if it failed, the card would quickly overheat and break.

To solve this, I decided to fit a large North Bridge heat sink and reuse the supplied heat sink for the ARM processor. I also added a fan (see below) but even if the fan does fail, the chip temperatures won’t exceed their absolute maximums – hopefully!

The supplied heat sink is mounted to the IOP333 with a simple spring-clip which can be re-used to mount the new heat sink, but it does take a bit of persuasion to separate it. The supplied heat sink can be mounted to the ARM processor using a couple of spots of SuperGlue, gently pressing it on with a circular motion to spread it as thinly as possible for the best possible thermal contact.

Battery Backed Write Cache (BBWC)

Without the BBWC, write performance will be dire – much less than 1MB/s, due to the RAID-5 write penalty being experienced on every sector written by ESXi (two reads and two writes).

There are three parts to the battery backup – the cable, the holder, and of course the battery. Dell sells these together as service kit XJ547, which I picked up a brand-new from eBay for £15. There is obviously nowhere for the holder to be mounted in the HP server, but the battery fits in a gap just below the rear system fan. I used a cable tie through some convenient openings to hold it (see picture of the case below).

Note that it is extremely important that the battery is never disconnected from the PERC card, even when the machine has been gracefully shutdown. ESXi does not fully flush the hardware cache when shutting down, so if the battery is disconnected then the data on the volume will be corrupted, as I discovered. Fortunately I still had all my VMs on a backup disk.

Additional Case Ventilation

I’ve always felt the chipset of the ML115 runs much too hot, and since the PERC is so close to it I felt the whole area could benefit from a bit of circulation. I mounted a standard case fan, simply with a double-sided foam sticky pad, at the base of the case angled slightly to blow across the PERC towards the chipset. I selected that spot since I didn’t want to disrupt (too much) the overall front-to-back cooling by drawing more air into the case low down, thereby reducing the flow over the disks.

Obviously I’ve not done any scientific testing but with it in operation, the hard disk temperatures seem the same and there is still a good inflow of air across the front vents. The PERC and the chipset are also very much cooler. Talking of which…

Power Consumption

The power consumption of the server was increased by about 19W with the PERC 5i installed, which includes the PSU overhead (the units are stated as 65% efficiency in the ML115). Overall this server, with its four drives, runs at about 137W idle, rising to 170W with workload and possibly more. It’s not particularly efficient, then again running four VMs continuously that’s only 35W per machine I suppose.

Performance

OK the important bit, how does it stack up?

I tested straight sequential throughput and multi-threaded random IO in various configurations from an XP guest with Passmark Advanced Disk Test. With the three drives operating in RAID-5, it’s consistently twice as fast as one drive connected to the onboard controller. It seems likely therefore that a further 50% improvement would be there if using a four-drive array. In sequential throughput I got 80MB/s (read or write).

The 40MB/s achieved on the nVidia SATA controller seems to be a controller limit, since copying files between data stores on different disks also peaked at 40MB/s total (i.e. each disk halved to 20MB/s). This was also the peak performance attainable copying large files between guests when running from different disks.

Similar tests with the PERC fared less well, turning in around 16MB/s, presumably because more physical disks are needed to spread the IO loads since both VMs are providing contention to the same disk resources – on the RAID array the disks must physically move between read and write locations, whilst in the test between two disks connected to the onboard SATA controller all IO will be broadly sequential.

* If anyone else out there would like to submit an article to TechHead please get in touch – I would love to hear from you.

Related posts:

1. Can you use the HP Proliant ML115 G5 Onboard RAID with VMware ESXi 3.5 Update 4 (U4)? Can you use the HP Proliant ML115 G5 onboard...
2. Updating the HP E200 StorageWorks Array Controller Firmware on your HP Proliant ML110 or ML115 Server. There are many of you out there that after...
3. VMware ESXi Purple Screen of Death on HP Proliant ML110 or ML115 A common occurrence discovered by HP Proliant ML110 and...
4. Installing and Running VMware vSphere (ESX 4.0) on your lab HP Proliant ML110 or ML115. With the release of VMware’s vSphere and ESX 4.0...
5. VMware ESX – Prevented Migration due to “Device SCSI Controller is a SCSI controller engaged in bus-sharing” error When going to vMotion a server either manually, via...