CPU Masking

Several months ago, I ended up in an unfortunate situation. We invested a sizeable amount of money on a powerhouse server to add to our ESX cluster. While the server was the same model as the rest of the machines in our cluster, the vendor had added SSSE3 support to this latest revision, something I was not aware of at the time of purchase. The end result of adding these new instructions to the cluster was vMotion incompatibility between this server and our other ESX hosts.

While not supported by VMWare, CPU masking is a feature available in Virtual Center that will allow you to overcome situations like this. While in my case, SSE3 servers to an SSSE3 server, compatibility certainly cannot be guaranteed in these situations and I must emphasize that you proceed with caution. In addition, if the differences in my CPU's were more profound (say AMD vs. Intel), I would never attempt this, especially in a production environment.

In some situations, Enhanced vMotion will allow you to overcome minor CPU differences and I would certainly recommend that as a first attempt. You will need to be running at least Virtual Center 2.5 Update 2 and ESX 3.5 Update 2 to enable this. You will also need to recreate your cluster. You can find information on EVC as well as supported processors in KB 1003212.

VMWare outlines the process of CPU masking in KB 1993. In my case, I wanted to mask only the needed instruction sets and it took a little while to figure out exactly which combination of masks to use between each of the registers leveraged. The end result was to navigate to C:\Documents and Settings\All Users\Application Data\VMware\VMware VirtualCenter and edit the vpxd.cfg XML configuration file. The section of text below was all that was needed to accomodate the differences between these processors. Sorry for the screenshot, but formatting XML for web presentation is a pain...

Adding Raw Device Mappings for Maximum Performance

We know that adding an RDM to your SQL and Exchange hosts can have an enormous impact on IO performance. In addition, under Windows Server guests prior to Server 2008, the partition must be properly aligned when accessing a RAID volume in order to achieve maximum performance. This is due to the fact that with a physical disk that maintains 64 sectors per track, Windows always creates the partition starting at the sixty-forth sector, therefore misaligning it with the underlying physical disk. In some cases, aligning a partition can boost IO performance by up to 50 percent! Here is a step by step guide to creating, attaching and aligning an RDM partition in ESX 3.5 on Server 2003.

Attaching a Raw Device Mapping

1. Log on to your SAN and create your LUN. Make this LUN available to all ESX hosts in your cluster. The steps needed to do this will vary by SAN and Fibre switch, so consult your vendor’s documentation for more info.
2. Log in to Virtual Infrastructure Client and connect to your Virtual Center instance.
3. Click on an ESX host and choose the Configuration tab.
4. Click on Storage adapters and rescan for new storage devices. Your new LUNs should show up. You do NOT want to create a VMFS here.
5. Repeat this procedure for each ESX host in your cluster.
6. Right click on the guest OS that you will be attaching the RDM to and click on Edit Settings.
7. Click Add in the hardware tab, choose Hard Disk and then Raw Device mapping.

Adding your new disk to the guest OS

1. Log on to your guest OS and launch Device Manager.
2. Scan for hardware changes.
3. Open Disk Management and initialize the new disk. Do not create a partition at this time.

Adding a properly aligned partition to the RDM

1. Log on to the server.
2. Type diskpart at the command line to launch the diskpart utility.
3. Type list disk to see a list of disks present. For my example, I will be creating a partition on Disk 2 and it will be the only partition on this disk.
4. Type select . In my example, you would type SELECT Disk 2.
5. Type create partition primary align=64 to create a primary partition that takes up the entire disk. You can use the size keyword if you are creating more than one partition on the disk.
6. After you have finished here, you will need to go in to Disk Management, format the partition and assign it a drive letter as you would normally.

Running SQL Server on ESX

There is currently a lot of debate going on about whether or not you should be running SQL server or Exchange on ESX. Rather than jump into the middle of this debate, I'll offer the best way that you could possibly run a high performance SQL server on virtual hardware.



I/O

I/O is by and large the biggest concern when running a database on an ESX cluster. Your main concern here is going to be ensuring that your database files are not impacted by I/O operations on your other VMs. We can ensure this through Raw Device Mappings.

Generally, for a high-performance database, you'll want your data files, log files, tempdb and program/OS/backup files to all live on separate spindles. To accomplish this, we do the following:

Data Files - RAID5 or RAID10 (RAID5 is more economical but you will suffer a hit on write performance).

Log Files - RAID10

TempDB - RAID10

OS/Program Files/Backup/File Share - This can be run from a standard VMDK.


Memory

In addition to I/O, memory is very important. If a database server has sufficient memory, it will not need to go to disk for its data as often, vastly improving performance. To be absolutely sure that your server receives sufficient memory, you can set your memory reservation equal to the amount of memory given to your VM. This will ensure that ESX never swaps out active memory from your database and that the SQL query optimizer can make accurate predictions about hardware performance.


CPU

On my key instances, CPU has never been much of a bottleneck, so I generally treat this as I would with any other VM.

ESX Partitioning

While most of the defaults in an ESX installation will be fine, I always take the time to edit the partition scheme. Since most new servers will come with drives no smaller than 60GB and I'll be storing all of my VMDKs on shared storage, there's no reason to not allocate extra space to places that will use it. Here is my usual breakdown:

/boot - 100MB - ext3 - The default is fine here.

/ - 10GB - ext3 - If you ever want to update your Service Console, it's nice to have some extra space available.

(none) - 1600MB - swap - The service console can access a maximum of 800MB of RAM. Your swap file should always be at least twice the size of memory being used. Since this partition cannot be resized without doing a reinstall, I always set it to the max in case I need to allocate more memory to the SC down the road a bit.

/var/log - 2GB - ext3 - Having a separate partition for your logs prevents them from filling up your root partition in the case of system issues.

(none) - 100MB - vmkcore - While this is optional, it holds the kernel dump if you have a Purple Screen of Death. When you call VMWare support in such a case, they will want to look at the contents held here.

/home - ? - ext3 - If you plan on storing scripts and other such files on your ESX server, you may want to carve out an extra home partition.

/vmfs/volumes/xyz - ? - vmfs-3 - Any leftover space can be set aside as a spare VMFS volume.

VMWare Network Communications Diagram

If you have a segmented network, as you probably should, where your ESX/Virtual Center/vMotion/Storage traffic has to pass through a firewall, you've had to comb through various documents to track down which ports need to be opened to allow various types of communication to be passed around. Jason Boche has posted an easy to read PDF that ties it all together for you here:

http://www.boche.net/blog/?p=323

Thanks to VM/ETC for bringing this to my attention.

How Virtual Center Calculates HA Failover Capacity

From the Resource Management Guide:

"HA plans for a worst‐case failure scenario. When computing required failover capacity, HA calculates the maximum memory and CPU reservations needed for any currently powered on virtual machine and calls this a slot.... HA determines how many slots can “fit” into each host based on the host’s CPU and memory capacity. HA then determines how many hosts could fail with the cluster still having at least as many slots as powered on virtual machines. This number is the current failover level."

What this means is, if I have one really powerful VM on my cluster (say 4vCPU and 4GB RAM reservation), VC will calculate ALL of my VMs as being this powerful when determining failover capacity, even if all of the other VMs are 1vCPU and have a low memory reservation. Since the calculated failover capacity will obviously be much lower than what I can actually run on, I opted for a workaround.

1) Set Admission Control to allow VMs to power on even if they violate availability constraints.

2) Create a non-expandable resource pool with unlimited access to resources. Set the reservations for CPU and memory to be the sum of all of your remaining hosts should your highest resource machine die. Take the following example:

ESX1 - 2 x 2.4GHz, 24GB RAM
ESX2 - 2 x 2.4GHz, 32GB RAM
ESX3 - 4 x 2.4GHz, 24GB RAM

My resource pool reservation would be set to 4 x 2.4GHz (my two lowest CPU producers) and 48GB RAM (my two lowest memory producers). Make sure to adjust these numbers down to allow overhead for the host.

3) Make sure to add all VM's to this resource pool and keep your reservations up to date.

Adding AD Integration to ESX 3.x

While most administration on ESX servers is done through Virtual Center, there are some tasks that still must be run directly from the service console. Furthermore, if your environment isn't large enough to justify running Virtual Center, sometimes the ESX host is your only option for administration. These instructions will allow you to authenticate through your Active Directory infrastructure so that you don't have to manage a parallel set of usernames and passwords.

run the following command for each user you're adding:
useradd

edit /etc/pam.d/vmware-authd and insert at the beginning of the file:
auth sufficient pam_krb5.so use_first_pass

run the following command:

esxcfg-auth --enablead --addc=.com --addomain=.com

run more /etc/krb5.conf to verify the results of your work.

Make sure that the date is set correctly on the ESX server or you will not be able to log in. Type date to check.

when changing the date use the following command specifying military time:
date MMDDTTTTYYYY

Installing the VMWare Infrastructure Remote CLI for sVmotion

When my Virtual Infrastructure finally grew to a state in which I had a few different VMFSes spread across several SAN LUNs, it became time to get my feet wet with Storage vMotion. Here’s how to download, install, and utilize the VMWare Infrastructure Remote CLI virtual appliance which has everything needed in order to utilize this feature from one centralized location.

Download
- Go to http://www.vmware.com/download/vi/drivers_tools.html and click on the Download link under Vmware Infrastructure Remote CLI.
- You must have an account on the VMWare website to proceed.
- Agree to the license agreement and download the zip file for the Virtual Appliance. At the time of download, this file was 258MB. After unpacked and imported, it will take up 4GB on your VMFS volume.
- Unzip the file to a location on your desktop.

Installation
- Using your VI client, connect to your Virtual Center.
- In the inventory pane, click on the host where you’d like to keep the virtual appliance.
- From the File menu, select Virtual Appliance – Import.
- Choose Import from file and browse to the OVF file that you decompressed.
- Choose your VM name, location and datastore to proceed.

Running the RCLI VM
- Select the Virtual Machine in your inventory and power it on.
- At first boot, you will be prompted to agree to the Perl toolkit license agreement, set a root password and set up networking.
- You’re ready to go! To download the Remote Command-Line Interface Installation and Reference Guide, go to http://www.vmware.com/download/vi/drivers_tools.html and click on Documentation.

ESXKILL Shutdown Script

A little while back, I was looking into getting Power Chute set up for our ESX hosts. The unfortunate news is that APC does not yet support ESX 3.5 U2. While researching this, I came across a script to shut down all guest OSes and then an ESX host, though, which I have slightly modified and deployed to all of my ESX hosts. The original can be found at http://www.tooms.dk/ By copying the script to a location that can be accessed in my PATH, I can quickly log on to each server in the case of disaster and type 'esxkill". This will usually shut everything down gracefully in 5-10 minutes. Here's the syntax:


#####################################################################
#!/bin/sh
#
# UPS shutdown script for VMware ESX 3.5 U1
#
# 20060911 First version by tooms@tooms.dk
# 20081006 Revised by ian.reasor@gmail.com
#####################################################################
# Set PATH variable to the location of ESX utilities
PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin"
#####################################################################
# Attempt a graceful shutdown through VMWare Tools
count_vm_on=0
for vm in `vmware-cmd -l` ; do
#echo "VM: " $vm
for VMstate in `vmware-cmd "$vm" getstate` ; do
#echo $VMstate
# If the VM is power ON
if [ $VMstate = "on" ] ; then
echo " "
echo "VM: " $vm
echo "State: is on and will now shut down"
echo "Shutting down: " $vm
vmware-cmd "$vm" stop trysoft
vmwarecmd_exitcode=$(expr $?)
if [ $vmwarecmd_exitcode -ne 0 ] ; then
echo "exitcode: $vmwarecmd_exitcode so will now power off"
vmware-cmd "$vm" stop hard
fi
count_vm_on=$count_vm_on+1
sleep 2
# if the VM is power OFF
elif [ $VMstate = "off" ] ; then
echo " "
echo "VM: " $vm
echo "State: is already powered off"
# if the VM is power suspended
elif [ $VMstate = "suspended" ] ; then
echo " "
echo "VM: " $vm
echo "State: is already suspended"
# if state is getstate or =
else
printf ""
#echo "unknown state: " $VMstate
fi
done
done
########################################################################
# wait for up to 5 min for VMs to shut down
#
if [ $count_vm_on = 0 ] ; then
echo " "
echo "All VMs are powered off or suspended"
else
echo " "
vm_time_out=300
count_vm_on=0
echo "Waiting for VMware virtual machines."
for (( second=0; second<$vm_time_out; second=second+5 )); do sleep 5 printf "." count_vm_on=0 for vm in `vmware-cmd -l` ; do for VMstate in `vmware-cmd "$vm" getstate` ; do if [ $VMstate = "on" ] ; then count_vm_on=$(expr $count_vm_on + 1) fi done done if [ $count_vm_on = 0 ] ; then #echo "exit for" break fi done #echo $VMstate fi #echo $count_vm_on ##################################################################### # Check to see if all VMs are off and if not then power them down for vm in `vmware-cmd -l` ; do #echo "VM: " $vm for VMstate in `vmware-cmd "$vm" getstate` ; do # If the VM is power ON if [ $VMstate = "on" ] ; then echo " " echo "VM: " $vm echo "it is still on but will now be powered off" vmware-cmd "$vm" stop hard sleep 2 fi done done ##################################################################### # Shutdown ESX Host echo " " echo "All VM's have been successfully shut down, halting ESX Host" echo " " shutdown -h now

Virtual Center Performance Reporting

While Virtual Center provides a great performance monitoring interface to gain some insight into VM resource consumption, I wanted a way to automatically get a run down of my VM's memory and CPU utilization e-mailed to me on a weekly basis. This is especially useful when setting resource pool reservations. If you would like to do the same for your VC database, you'll need to set up database mail on your SQL server and create a SQL Server Agent job to run each of the following scripts. Set up a schedule and you're on your way!

Script 1: Memory Utilization

EXEC msdb.dbo.sp_send_dbmail@recipients = 'yourname@yourdomain.com', @subject = 'Virtual Center Memory Utilization Report',@body_format = 'HTML',@body = 'Here are some memory utilization metrics for the last month.',@query = ' USE VirtualCenter select vms.NAME as ''VIRTUAL_MACHINE'' , (AVG(STAT_VALUE)/10000) as ''MEMORY_PERCENT''
into #STAT_REPORT_TEMPfrom dbo.VPXV_HIST_STAT_YEARLY
JOIN VPXV_VMS vmsON SUBSTRING([ENTITY] ,4 , 4) = vms.VMID
where stat_group = ''mem''and entity like ''vm%''and sample_time > (getdate() - 30)
group by vms.NAME
go
select VIRTUAL_MACHINE as ''Virtual Machine'' , MEMORY_PERCENT * vm.MEM_SIZE_MB as ''Memory Usage in MB''
from #STAT_REPORT_TEMP stat
JOIN VPXV_VMS vmsON VIRTUAL_MACHINE = vms.NAME
JOIN VPX_VM vmON vms.VMID = vm.ID
ORDER BY VIRTUAL_MACHINE
go
drop table #STAT_REPORT_TEMP',@attach_query_result_as_file = 1,@query_attachment_filename = 'Virtual Machine Memory Stats.xls'

Script 2: CPU Utilization

EXEC msdb.dbo.sp_send_dbmail@recipients = 'yourname@yourdomain.com', @subject = 'Virtual Center CPU Utilization Report',@body_format = 'HTML',@body = 'Here are some CPU utilization metrics for the last month.',@query = ' USE VirtualCenter select vms.NAME as ''VIRTUAL_MACHINE'' , (AVG(STAT_VALUE)) as ''CPU_USAGE''
from dbo.VPXV_HIST_STAT_YEARLY
JOIN VPXV_VMS vmsON SUBSTRING([ENTITY] ,4 , 4) = vms.VMID
where STAT_NAME = ''usagemhz''and entity like ''vm%''and sample_time > (getdate() - 30)
group by vms.NAMEorder by vms.NAME',@attach_query_result_as_file = 1,@query_attachment_filename = 'Virtual Machine CPU Stats.xls'

Each of these scripts takes an average for each metric over the last month but if you prefer, you could edit these to use a different time frame.

Blog Launch

As I have learned about virtualization over the last year, a bunch of blogs out there helped to point me in the right direction. Now that I have gotten my feet wet and am starting to learn some tricks of my own, I decided to launch this blog as a space to share them. I hope that you find the content helpful and would love to hear your feedback.