How to measure IOPS for VMware

http://blog.synology.com/blog/?p=2225

Executive Summary
This article, intended towards IT Professionals, is a follow up towards my previous article, How to measure IOPS for Windows will be covering how to measure IOPS consumption of Guest VMs from VMware which is commonly asked when using our Synology products. This information can be used to aid in diagnostics of the storage performance on the VMware Hypervisor, or can be used to determine storage performance capacity of the storage array or a new deployment of a storage array. Using this process does require familiarity with determining IOPS from Windows. This article contains a proof-of-concept method of gathering data from a VMware Hypervisor to determine the IOPS consumption of the Guest VMs hosted by the Hypervisor.

Problem: IT Professionals needs to determine the amount of IOPS on Guest VMs in VMware
Just as there were questions on how to identify IOPS consumption on Windows, there are plenty of requests made to me to help determine and analyze the IOPS consumption of guest VMs within VMware. This was originally covered in the IOPS – Performance Capacity Planning Explained article; however, there was no method of recording and analyzing the data. The procedure below provides the full details on how to record IOPS data from VMware, and then using techniques from the measuring IOPS from Windowsarticle, will illustrate how to analyze the data from guest VMs.

Procedure – Collecting data for IOPS measurement

1. Have the Guest VMs operating
2. Run this command in the esxcli
   1. For complete background information about this command, please refer to VMware KB 1006797
   2. esxtop –b –a –d 2 –n 43300|gzip -9c> outputFileName.csv.gz
   3. “–b” Run in Batch mode
   4. “–a” Record all parameters, this is useful for future reference
   5. “–d” Delay Time between samplings, shortest delay is two seconds
   6. “–n 43300″ Number of iterations
   7. “|gzip -9c>” gzip the output datafile
   8. “outputFileName.csv.gz” is the file name
3. After the data collection is completed, move the file to “/vmfs/volumes/datastore1″
4. Use the vSphere Client to download the file from the datastore to a Windows Desktop
5. Use perfmon to iosolate the data to the specific Guest VM that requires analysis
   1. For further background information about using perfmon to isolate data, please refer to Microsoft TechNet Article
6. Open perfmon from a Windows Computer
7. Remove all the current counters
8. Right click on the white field, and click Properties
9. Click on the Source Tab
10. Select the decompressed csv file
11. Click Apply
12. Observe the date range become active
    1. Adjust the date range if it is needed
13. Click Ok
14. Right Click, and click Add Counters
15. Add the counters
    1. For Background Information about the VMware Counters, please refer to Interpreting esxtop Statistics article
    2. \Physical CPU\
       1. % Util Time\Total
    3. \Memory\Free MBytes
    4. Virtual Disk
    5. \Avg ms/Read
    6. \Avg ms/write
    7. Cmds/sec
    8. MB Read/Sec
    9. MB Write/Sec
    10. Reads/Sec
    11. Writes/Sec
        1. VMs Mount Points. iSCSI or NFS is displayed here. The data isolation can be set to a specific Guest VM, or a specific storage entity.
16. Right Click on the white field, and click Save Data As
    1. Save the data as a CSV
17. Open the isolated data file in Excel and begin sorting columns of data for easier analysis.
18. The following values were calculated
    1. Write/Read IOP Cluster
    2. Average IOP Cluster
    3. Concurrent Throughput
19. Generate the graphs as before with the Windows IOPS Collection procedure

Description of Counters

1. \Physical CPU\% Util time\Total
   • Total CPU Utilization
2. \Memory\FreeMBytes
   • Amount of Free Memory on the Hypervisor
   • VirtualDisk
     1. \Avgms/Read
        1. Average latency time in milliseconds for each read IOP
     2. \Avgms/Write
        1. Average latency time in milliseconds for each write IOP
     3. Cmds/sec
        1. Number of IOPS
     4. MBRead/Sec
        1. Megabytes Read Per Second
     5. MBWrite/Sec
        1. Megabytes Written Per Second
     6. Reads/Sec
        1. Number of Read IOPS
     7. Writes/Sec
        1. Number of Write IOPS

Procedure – Processing the data
In my example, I used a VMware ESXi 4.1 Host to gather the data for about two hours for a proof-of-concept. I did test the commands against a new deployment of ESXi 5.1 – and it produced the same results, with respects of collecting data. The data that I collected isn’t accurate as it was Win7 install procedure, mostly peak behavior, and not production use data. The ESXi 4.1 machine I was using is part of my lab use – which is in semi-production use on a day-by-day basis, and is not part of a permanent VDI solution.
To download an example of this VMware IOPS data, please download the measuring VMware IOPS sample file.. The file is about 2MB in size when decompressed.

Procedure – Analyzing the data

• VMware Guest IOPS over time
  • Comprised of Write/Read IOPS over time with percentile analysis based on the highest values out of read, write or concurrent of the individual Guest VM.
  • This data, for the Guest VM (or storage entity) will provide the basis of analysis for determining the IOPS consumption of the array.
• VMware Guest IOP Block Size over time
  • Comprised of Write/Read IOP Block Size over time, displayed in KB.
  • This data will show the average IOP block size that is experienced by the storage array. This data can help gauge what kind of storage array performance is needed.
• VMware Guest Throughput over time
  • Comprised of Write/Read Bytes/Sec over time, displayed in MB/Sec. This data will determine the basic infrastructure needed to support the storage array.
• Vmware Guest Latency over time
  • Comprised of Write/Read Latency over time, displayed in milliseconds.
  • This data can be used to identify any latency errors.
• VMware Host CPU Utilization over Time
  • This graph is used in conjunction with all other graphs in scale, to determine the processor utilization with respect to other aspects of the storage array.
• VMware Host Memory Utilization over time
  • This graph is used in conjunction with all other graphs in scale, to determine the memory utilization with respect to other aspects of the storage array.
• Summary Analysis
  • Considering that this is a proof-of-concept test – the data being generated isn’t 100% accurate. However, if I had to make a decision of what kind of storage array that is needed to support a VM that is utilizing around 700 IOPS at the 95th percentile, I would strongly consider using a Synology XS+ Series RackStation for my storage.

Summary
With this procedure IT Professionals are able to collect the IOPS data being consumed from the Guest VM itself. This information is useful for determining the current storage performance utilization of the current array, or can be used to aid in the deployment of a new storage array, such as a Synology XS+ Series RackStation.

References

1. Synology Blog – How to measure IOPS for Windows
2. Synology Blog – IOPS: Performance Capacity Planning Explained
3. VMware KB – 1006797 – Using performance collection tools to gather data for fault analysis
4. Microsoft TechNet – Open Log Files in Performance Monitor
5. VMware Community – Interpreting esxtop Statistics
6. Measuring VMware IOPS Sample File Note: 1.6MB File