Posts Tagged ‘# VMware’

Dell EqualLogic PS4000: Hands-on Review Part 4

# VMware
About the Author

Author: James PearceJames is regular guest contributor to TechHead and is a Kent based qualified accountant, currently working in information security and technical architecture with  most of his  time “being spent on virtualisation and business continuity at the moment”. Check out his new virtualisation and storage blog here for more interesting and informative posts.

_________________________________________________

 Dell EquallogicThe Dell EqualLogic PS4000 series of iSCSI storage arrays are positioned above the MD3000i series and targeted at the SME sector, especially for vmware virtualisation, as well as “branch office” use for larger companies.  The model reviewed here is the XV model, with sixteen 300GB 15k SAS drives, and dual controllers and dual power supplies.

This is quite a complicated product, so I’ve split this into four sections:

Part 1 – The PS4000

Part 2 – EqualLogic Networking with Force10

Part 3 – System Management and Monitoring

Part 4 – Performance

 

Part 4 – Performance

Performance

OK the important bit – the numbers that EqualLogic don’t seem to want to print.  I’ve tested this array in Raid-50, 5 and 6 using a Windows Server 2003 VM running IOMeter on a Dell R610 (with 1 vCPU and 4GB RAM allocated) using the methods described on my blog, here.  For comparison, I’ve also run the tests on a few different storage systems, including a single SATA drive and a few local RAID arrays (note that the IOPS graphs have a logarithmic scale).

Sequential Read

iSCSI with Gigabit Ethernet (GbE) just doesn’t provide great sequential throughput as GbE is good ‘only’ for about 110MB/s.  With some command-line configuration, as described in part 2, a pair of GbE’s can be used load balanced, providing total bandwidth between ESX and the EqualLogic of about 220MB/s.

Running in Raid-50, the PS4000 gets very close to saturating this network connectivity, providing 210MB/s with 64 outstanding IOs.  Raid-5 and 6 fair slightly less well, at about 170MB/s.  Still, compare this to the old Dell Clarion based AX150i iSCSI chassis, providing only 40MB/s:

Of course the throughput is dwarfed by the local Perc-6i with 5 SAS drives, which puts in an impressive 330MB/s.

Sequential Write

The first configuration tested was Raid-6, which provides about 120MB/s on writes.  Given the dual-parity nature of Raid-6, this seems like reasonable performance.

But then there is a surprise – Raid-50 and Raid-5 provide similar performance at 130 and 140MB/s respectively.  Again for comparison, the local Perc-6i sustains over 330MB/s in the same test using Raid-5.

Still, sequential throughput isn’t a deal breaker and is relatively unimportant for virtualised workloads expect perhaps for volume based backups, and even then these figures are likely to be easily sufficient for perhaps 90% of real-life workloads.

Random Throughput

Random IO is what this array is all about – serving multiple contending workloads without allowing latency to rise enough to impact application response times.  This is where the PS4000 really scores over local storage, primarily due to its 16 drive bays.  With 3.4ms average seek and 2ms latency, each 15k6 should give at least 185 IOPS.  In practice,  command queuing and the fact the test file is very much smaller than the overall capacity, in an effort to simulate real life workloads, mean numbers considerably higher than this can be expected in these tests.  EqualLogic pre-sales presentations claim around 270 IOPS per drive in RAID-5 – but does it deliver?

The random workload used is 8K blocks, 100% random IO, 70% read, against a 30GB test volume.  The Raid-6 performance seems amazingly strong, just 10% down on Raid-5 although this is essentially due to the 70:30 weighting of the testing.

Achieving these IOPS rates needs a good queue depth – performance levelling out at about 64 outstanding IOs.  At this level, and even up to 192 outstanding IOs, latency remains comfortably within vmware’s recommended 50ms.

Recovery Mode

After a disk failure, the array immediately starts a rebuild using the spare drive.  As a result of this, performance is impacted – the IOMeter results roughly halved whilst this was ongoing.  Sequential write was hit hard, being reduced to around 30MB/s.

 

In Summary

The EqualLogic PS4000 is an accessible storage system for the SME market built to the usual Dell standards which performs well, scales now up to fourteen units, and should prove reliable thanks to top-quality components throughout.

Failure of disks, power supplies, interconnecting switches and controllers seems to work well and is gladly demonstrated by the commissioning engineer.  Dual-controllers also enable firmware updates to be made without downtime (although a maintenance window should still be used).

Limitations

The system isn’t however without it’s limitations.

In testing, an old Oracle 8i import job proved a corner-case where the array performed badly.  It should be noted that this is a pretty ancient piece of software that works with a low queue depth and is unlikely to see much use today – the problem essentially being the underlying command latency, which will always be higher for iSCSI compared to local storage.

Another limitation is present in preparing for a remote replication deployment.  The system works with block change tracking utilising (internally) 64MB blocks – hence a database changing a single sector will trigger a 64MB transfer to the remote site.  It would be helpful if the system could be used with the block change tracking enabled without a replication partner, in order to collect the statistics needed to assess the viability of SAN layer replication prior to committing a good wedge of cash to it.

Another issue is with the thin provisioning, however this is another iSCSI limitation.  Industry seems to be demanding iSCSI over NFS despite the protocols being very similar in terms of efficiency.  The issue with iSCSI though is that there is no delete command, so a run-away log file in a VM running on a thin-provisioned volume, or a carelessly thick-provisioned VM, will wipe out those gigabytes on the EqualLogic pretty much forever.  The only way to retrieve the space from the LUN is to move the VMs to a new LUN and delete the old.  In contrast, NFS storage simply doesn’t suffer part of this issue since ESX is communicating with it at the file system level (zeroing free space and a storage vMotion would still be needed to shrink thin-provisioned VMDKs).

Finally the sequential write performance seems disappointing, but it should be noted that the interoperable PS6000 series have higher speed processors and 10-GbE in some models, and would probably be better suited to such workloads.  In random performance – much more of interest in virtualisation projects and in particular database workloads – the EqualLogic is of course way ahead of a typical 6-drive server, running as it does at over 4,000 8K IOPS in Raid-50 with latency of under 16ms with 64 outstanding IOs.

Tuning

Getting the network layer configuration right is the key to getting the best from these arrays.  The mostly browser or Java based management interfaces offer excellent performance monitoring capabilities, but there is almost nothing by way of tuning.  This is really a good thing though since the system is self managing and will automatically move data between arrays based on their performance profiles in use.  And less settings to adjust really means less to get wrong.

vStorage

As the long awaited vStorage API seems finally due to make an appearance, the EqualLogic product will get even better since it will be one of the first able to provide hardware off-load capabilities to ESX, such as snapshots.

And Finally

After the release of the first part of this series several months ago, Dell were quick to get in touch to offer any assistance needed including direct access to a high-level engineer and access to an invitation-only user group.  The testing methodology has been reviewed by them and the performance metrics provided here were confirmed to be in the range expected.

The few niggles aside, the EqualLogic PS4000 is an excellent product with excellent management tools, and its support backing and simple configuration (networking aside) are real strengths in this product.

VN:F [1.9.3_1094]
Rating: 0.0/5 (0 votes cast)
VN:F [1.9.3_1094]
Rating: 0 (from 0 votes)

VMware ESX “I moved it” or “I copied it” – What’s the difference?

# VMware

When you copy or move the data store location of an existing VM you will be presented with a message box (as seen below) in the vCenter Client asking if your VM has either been ‘moved’ or ‘copied’.  As you can see the message box also mentions “msg.uuid.altered: This virtual machine may have been moved or copied”, but what does this actually mean?

VMware ESX msd.uuid.altered

Figure 1. Has the VM been moved or copied?

What is a VM’s UUID?

Firstly, it is important to have an understanding of what a ‘UUID’ (universally unique identifier) is.  As the name suggests the UUID is a ‘identifier’ (128 bit integer) which is ‘unique’ to that VM, and effectively gives it a digital fingerprint to differentiate it from other VMs.

The UUID is automatically generated when a VM is first powered on or moved, with the UUID value being based on the physical host’s identifier and also the path to the VM’s configuration (vmx) file.  Within this configuration file the UUID value is stored in two places:

  • uuid.bios
  • uuid.location (hash based on the current path of the VM)

For example: uuid.bios = "56 4d 5e 58 66 f5 2d 04-03 31 0a bd 6f a7 19 88"

The UUID is also stored in the SMBIOS system information (ie: the BIOS of the VM) descriptor.  When the VM is started or moved the location UUID (ie: uuid.location) which is hashed from the VM’s data store path is compared to the UUID location hash which already exists in the configuration file.  At this point if the new and existing location UUID value differs then ESX knows that the VM is now running from a different data store location and will present the ‘Virtual Machine Message’ in figure 1 above.

But why do we care if the VM has the same or a new UUID? 

We saw in the message above provided by ESX informing that the UUID has in someway been altered but why does this really matter?  The answer to this you’ll be pleased to know is quite The vmx file contains important VMware ESX UUID informationsimple.  A VM’s unique UUID is used to generate other unique values used by the VM such as the unique MAC (media access control) address of the network card(s).  For example if you had multiple copies of the same VM/Guest OS running in your vSphere environment all with the same (ie: non-unique) network MAC address you will likely receive duplicate MAC address error messages within the guest OS which can cause a number of issues.

Another potential point to be mindful of is that some software licensing can be linked to a MAC address of a guest OS’s network card.  This includes software such as Microsoft Windows where changing the MAC address and some other key hardware components (eg: moving from an Intel based ESX host to a AMD based ESX host) can mean you have to re-activate the software again.  The changing of a VM’s MAC address will occur when you select “I copied it”, the next couple of sections will go into more detail on what exactly is altered.

 

Should I Select “I Moved It” or “I Copied It”?

So what is the difference between selecting “I_moved it” or “I_copied it”?  The easiest way to demonstrate the differences is by viewing the configuration file (vmx) for the VM before and after the two different options have been selected.

 

“I Moved It”

By indicating that you had moved the VM (instead of copying it) the only UUID change that is made to the configuration file is to the ‘uuid.location’ setting, which as you’d expect indicates a change of location for the VM. The ‘uuid.bios’ and the existing generated network MAC address remains that same.

You will also notice that the CPUID settings have also changed which is also the case for when you indicate that the VM was copied.

The “I Moved It” option should be used when ‘moving’ the location of where a VM resides and a copy of the VM has not been made.

VMware ESX I Moved It

 

“I Copied It”

When you select that the VM has been copied then there a few more changes that are made to the VM’s configuration file when compared to just moving it.  These changes are to the ‘uuid.bios’, ‘uuid.location’ and as a result of these changes a newly generated network MAC address (ethernet.generatedaddress).

The “I Copied It” option should be used when you’ve made, and intend to run, more than one copy of the VM in your vSphere environment.

VMware ESX I Copied It

 

To summarise, here is a table which outlines the changes that are made when either the “I Moved It” or “I Copied It” are selected

  “I Moved It” (change?) “I Copied It” (change?)
uuid.bios uuid.bios uuid.bios
uuid.location uuid.location uuid.location
ethernet.generatedaddress ethernet.generatedaddress ethernet.generatedaddress
guestCPUID.x guestCPUID guestCPUID
hostCPUID.x hostCPUID hostCPUID
userCPUID.x userCPUID userCPUID

 

As you can see it is worth spending the time to understand the changes which will be made when presented with the “I moved it” or “I copied it” options as it can impact (eg: software re-activation) the guest OS of the VM.

I hope this helps clarify this small aspect of vSphere administration which can sometimes be an area of confusion.

 

VN:F [1.9.3_1094]
Rating: 0.0/5 (0 votes cast)
VN:F [1.9.3_1094]
Rating: +7 (from 7 votes)

DroboPro – Thin Provisioning with VMware ESX and using RDMs

# VMware

This guest post is by well known storage expert, Chris M Evans who writes his own blog on storage and virtualisation at www.thestoragearchitect.com. His blog is an excellent source of storage, virtualization and enterprise information – well worth adding to your browser favourites & RSS feed.

 

One of the unique selling points of the Drobo series of devices is that they are capable of detecting and understanding common O/S file systems.  For a DroboPro connected to a single host, the device is able to track file deletions and immediately recover released space.  However if you’re using the DroboPro in a VMware environment, exactly how can you make use of this unique feature?  The answer is to use RDM devices.

 

Background

Firstly, here’s a little extra background on the Drobo.  The original Drobo "classic" model offered up to 4 physical drives in a single host connection using USB.  The host sees a single (or multiple, depending on the drive capacity) large 2TB LUN.  Version 2 supported FireWire and the DroboPro model introduced single-user iSCSI.  We’ve since seen the release (last November) of the Drobo-S and Elite models which provide additional capacity, connectivity (eSATA) and multi-host iSCSI support.

image

 

Using A DroboPro With VMware ESX

When the DroboPro is connected to a PC or physical server, formatting is based on the standard file system of the host. Even using iSCSI, the devices are directly mapped to the host, enabling the DroboPro to see and understand the file system.  However, when using VMware, disk devices are presented as vmdk devices to the host on a VMFS file partition.  Drobo devices don’t currently support VMFS and so the benefit of deleted space reclamation is lost.

 

Use RDM Devices

There is a workaround.  If the entire LUN can be presented as an RDM (raw device mapping) LUN in VMware, then the DroboPro is able to see and understand the file system, reclaiming deleted space automatically.  Here’s how I tested this configuration on my existing Drobo environment.

 

Configuration

My Drobo "lab" setup consists of a dual 4-core Intel CPU server, 16GB of memory and various storage devices.  I’ve most data on an Iomega ix4-200d, internal SAS disks and a DroboPro.  The ‘Pro is configured with 16 (sixteen) 2TB thin provisioned LUNs and 6.4TB of physical storage, however for this test I’ve removed all but two of the drives, leaving 2.4TB of raw storage available.  The first screenshot from my vSphere client shows the drives, some of which I’ve specifically named as part of this test.

 VMware and Drobo Pro RDM Devices

 

In order to obtain a fair comparison I’ll be using two LUNs (LUN10 and LUN11) from the DroboPro.  LUN10 will be presented to my test Windows 2008 host as an RDM device; LUN11 will be formatted with vmfs as a datastore.  The next two screenshots highlight this.

 

VMware ESX and Drobo Pro RDM Devices

VMware ESX and Drobo Pro RDM Devices

 

The formatted LUN11 with VMFS initially occupies 600MB of space.

So, I’ve presented my iSCSI LUN to the DroboPro using the vSphere client.  The test host is running Windows 2008 Server and the Drobo Dashboard so we can see what’s happening as files are created.

Here’s my DroboPro dashboard.  The device has around 153MB allocated to the formatted iSCSI LUN, labelled "X:".

 

Drobo Dashboard

 

The next step is to create some data on this test LUN.  For that I’ll be using the fsutil command, which lets me create a file of any size.  There’s no real data in the file, but the Windows MFT entry will reflect that a file of the specified size has been created on the volume.  If the DroboPro is watching the MFT, it should detect a file has been created and change the capacity figures in the dashboard accordingly.  This can only be the case if it understands the file system layout, as we’re creating no real data with this test.

The following video shows the creation of multiple 100GB files on the test volume and the change in the DroboPro Dashboard as this process occurs.  This test is performed on the iSCSI RDM LUN and clearly shows that the DroboPro is identifying the space usage via the file system.

 

 

 

Using VMFS

So, it seems the DroboPro can detect the creation and deletion of files through an RDM device.  However just to be certain, I repeated the task on the second LUN connected to the host as a LUN on the vmfs formatted data store.  As expected the Drobo wasn’t able to detect the create/delete process, however worse than that, the device wasn’t able to detect the amount of configured space in use until it resumed from standby.

I contacted Data Robotics and they provided this response:

Yes this is a known behaviour. The ‘cleverness engine’ works for a list of known file system types. Currently the list includes NTFS, EXT3, HFS+, and FAT32.  There is a level of effort in supporting VMFS
and it is something we intend to do. However, we do not have a date for support.

In the mean time we recommend using VMware based tools to track the utilization of the Smart Volume(s) on which datastores have been created.

Since we do not report utilization properly, we recommend that the actual available space in a DroboElite be the same or larger than the provisioned space. For instance, if you have 10TB usable in a DroboElite, the sum of all datastores on Smart Volumes should not be greater than 10TB.

What this means is the thin provisioning benefits of the Drobo are lost in a VMware environment as the device can’t file system changes.  So perhaps RDM devices are a good solution, but they’re only good if you are using them exclusively as the Drobo can’t keep track of a mixture of both supported and non-supported file systems.  I’m keen to see how VMFS will be supported, however in the meantime, keep an eye open for my next test – DroboPro on Hyper-V.

 

This guest post is by Chris M Evans who writes his own blog on storage and virtualisation at http://www.thestoragearchitect.com.

VN:F [1.9.3_1094]
Rating: 0.0/5 (0 votes cast)
VN:F [1.9.3_1094]
Rating: 0 (from 0 votes)

Call for Information! Compiling details on VMware ESX compatibility with HP Proliant ML110 and Ml115′s models.

# VMware

Hi all,

I am looking at compiling information on, and presenting via the site, peoples experiences with running VMware ESX (versions 3.0-v3.5, including v3i) on HP Proliant ML110 and Ml115 servers.  I get quite a few queries regarding what is compatible and what isn’t so thought it be a good idea to compile this information, including workarounds, gotchas and hot deals in an easy to follow format.

These are great little, cost effective servers that make an ideal ESX test environment.

HP_ML110_G5

If you can let me know what you have found to work for you, were any tweaks/workarounds required along with any other information you think may be useful to others.

I can confirm that the ML110 onboard storage controller and onboard NIC’s work just fine under v3.5.  See my previous blog article here for more information.

Here’s a rough outline of information to give a basic example.  This will be a work in progress and a fresh blog article created to house this information.

UPDATE: Check out my blog article here on installing VMware ESX 3i 3.5 on an ML110 G5.

Make/Model: Disk Controller: NIC: Work Around?
ML110 G4 SC44Ge NC320i N/A
ML110 G5   NC105i  
ML115   NC320i  
ML115 G5   NC105i  

Useful Additional Information:

VMWare ESX v3.5 I/O Compatibility Guide

List of unsupported servers that work with ESX v3.5 or v3i

 

Technorati Tags: ,,,,,,

VN:F [1.9.3_1094]
Rating: 0.0/5 (0 votes cast)
VN:F [1.9.3_1094]
Rating: 0 (from 0 votes)

VMware ESX Hyperthreading is Inactive.

# VMware

An area of confusion that seems to arise quite often is that around that of the Hyperthreading (HT) feature within ESX Virtual Infrastructure Client (VIC) being displayed as being ‘Inactive’.

VMware ESX Hyperthreading is disabled

Here is a link to Wikipedia that outlines what Hyperthreading is about and how it works.  This is another article with easy to follow information on the different CPU architectures.

If you see Hyperthreading within the VIC set to ‘Inactive’ you’re probably thinking “Hey, what’s going on here?  I’ve got a modern processor that can do this sorta stuff.”

Well, the answer is.. it depends.

VMware ESX Hyperthreading is disabledIf you have a single core processor and it is a Pentium 4 or early’ish (vague technical term :) ) Xeon then if it likely that you have Hyperthreading on the CPU’s die.  Sometimes it must be enabled in the BIOS first.  This article tells you what to look for.  Also, this list from Intel outlines which Intel processors and chipsets support Hyperthreading technology.

Hyperthreading disappeared from the Intel range of CPU’s with the introduction of the Merom, Conroe, and Woodcrest range of multi-core processors.

So if you are using a semi-recent PC/Server to run ESX then chances are you don’t have Hyperthreading technology on your CPU(s) and hence it is showing up as ‘Inactive’ within the Virtual Infrastructure Client.

VN:F [1.9.3_1094]
Rating: 0.0/5 (0 votes cast)
VN:F [1.9.3_1094]
Rating: 0 (from 0 votes)
TrainSignal - vSphere Pro
PHD Virtual - esXpress
VizionCore
Veeam - Backup & Replication