ESX Partitioning

While most of the defaults in an ESX installation will be fine, I always take the time to edit the partition scheme. Since most new servers will come with drives no smaller than 60GB and I'll be storing all of my VMDKs on shared storage, there's no reason to not allocate extra space to places that will use it. Here is my usual breakdown:

/boot - 100MB - ext3 - The default is fine here.

/ - 10GB - ext3 - If you ever want to update your Service Console, it's nice to have some extra space available.

(none) - 1600MB - swap - The service console can access a maximum of 800MB of RAM. Your swap file should always be at least twice the size of memory being used. Since this partition cannot be resized without doing a reinstall, I always set it to the max in case I need to allocate more memory to the SC down the road a bit.

/var/log - 2GB - ext3 - Having a separate partition for your logs prevents them from filling up your root partition in the case of system issues.

(none) - 100MB - vmkcore - While this is optional, it holds the kernel dump if you have a Purple Screen of Death. When you call VMWare support in such a case, they will want to look at the contents held here.

/home - ? - ext3 - If you plan on storing scripts and other such files on your ESX server, you may want to carve out an extra home partition.

/vmfs/volumes/xyz - ? - vmfs-3 - Any leftover space can be set aside as a spare VMFS volume.

VMWare Network Communications Diagram

If you have a segmented network, as you probably should, where your ESX/Virtual Center/vMotion/Storage traffic has to pass through a firewall, you've had to comb through various documents to track down which ports need to be opened to allow various types of communication to be passed around. Jason Boche has posted an easy to read PDF that ties it all together for you here:

http://www.boche.net/blog/?p=323

Thanks to VM/ETC for bringing this to my attention.

How Virtual Center Calculates HA Failover Capacity

From the Resource Management Guide:

"HA plans for a worst‐case failure scenario. When computing required failover capacity, HA calculates the maximum memory and CPU reservations needed for any currently powered on virtual machine and calls this a slot.... HA determines how many slots can “fit” into each host based on the host’s CPU and memory capacity. HA then determines how many hosts could fail with the cluster still having at least as many slots as powered on virtual machines. This number is the current failover level."

What this means is, if I have one really powerful VM on my cluster (say 4vCPU and 4GB RAM reservation), VC will calculate ALL of my VMs as being this powerful when determining failover capacity, even if all of the other VMs are 1vCPU and have a low memory reservation. Since the calculated failover capacity will obviously be much lower than what I can actually run on, I opted for a workaround.

1) Set Admission Control to allow VMs to power on even if they violate availability constraints.

2) Create a non-expandable resource pool with unlimited access to resources. Set the reservations for CPU and memory to be the sum of all of your remaining hosts should your highest resource machine die. Take the following example:

ESX1 - 2 x 2.4GHz, 24GB RAM
ESX2 - 2 x 2.4GHz, 32GB RAM
ESX3 - 4 x 2.4GHz, 24GB RAM

My resource pool reservation would be set to 4 x 2.4GHz (my two lowest CPU producers) and 48GB RAM (my two lowest memory producers). Make sure to adjust these numbers down to allow overhead for the host.

3) Make sure to add all VM's to this resource pool and keep your reservations up to date.

Adding AD Integration to ESX 3.x

While most administration on ESX servers is done through Virtual Center, there are some tasks that still must be run directly from the service console. Furthermore, if your environment isn't large enough to justify running Virtual Center, sometimes the ESX host is your only option for administration. These instructions will allow you to authenticate through your Active Directory infrastructure so that you don't have to manage a parallel set of usernames and passwords.

run the following command for each user you're adding:
useradd

edit /etc/pam.d/vmware-authd and insert at the beginning of the file:
auth sufficient pam_krb5.so use_first_pass

run the following command:

esxcfg-auth --enablead --addc=.com --addomain=.com

run more /etc/krb5.conf to verify the results of your work.

Make sure that the date is set correctly on the ESX server or you will not be able to log in. Type date to check.

when changing the date use the following command specifying military time:
date MMDDTTTTYYYY

Installing the VMWare Infrastructure Remote CLI for sVmotion

When my Virtual Infrastructure finally grew to a state in which I had a few different VMFSes spread across several SAN LUNs, it became time to get my feet wet with Storage vMotion. Here’s how to download, install, and utilize the VMWare Infrastructure Remote CLI virtual appliance which has everything needed in order to utilize this feature from one centralized location.

Download
- Go to http://www.vmware.com/download/vi/drivers_tools.html and click on the Download link under Vmware Infrastructure Remote CLI.
- You must have an account on the VMWare website to proceed.
- Agree to the license agreement and download the zip file for the Virtual Appliance. At the time of download, this file was 258MB. After unpacked and imported, it will take up 4GB on your VMFS volume.
- Unzip the file to a location on your desktop.

Installation
- Using your VI client, connect to your Virtual Center.
- In the inventory pane, click on the host where you’d like to keep the virtual appliance.
- From the File menu, select Virtual Appliance – Import.
- Choose Import from file and browse to the OVF file that you decompressed.
- Choose your VM name, location and datastore to proceed.

Running the RCLI VM
- Select the Virtual Machine in your inventory and power it on.
- At first boot, you will be prompted to agree to the Perl toolkit license agreement, set a root password and set up networking.
- You’re ready to go! To download the Remote Command-Line Interface Installation and Reference Guide, go to http://www.vmware.com/download/vi/drivers_tools.html and click on Documentation.

ESXKILL Shutdown Script

A little while back, I was looking into getting Power Chute set up for our ESX hosts. The unfortunate news is that APC does not yet support ESX 3.5 U2. While researching this, I came across a script to shut down all guest OSes and then an ESX host, though, which I have slightly modified and deployed to all of my ESX hosts. The original can be found at http://www.tooms.dk/ By copying the script to a location that can be accessed in my PATH, I can quickly log on to each server in the case of disaster and type 'esxkill". This will usually shut everything down gracefully in 5-10 minutes. Here's the syntax:


#####################################################################
#!/bin/sh
#
# UPS shutdown script for VMware ESX 3.5 U1
#
# 20060911 First version by tooms@tooms.dk
# 20081006 Revised by ian.reasor@gmail.com
#####################################################################
# Set PATH variable to the location of ESX utilities
PATH="/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin"
#####################################################################
# Attempt a graceful shutdown through VMWare Tools
count_vm_on=0
for vm in `vmware-cmd -l` ; do
#echo "VM: " $vm
for VMstate in `vmware-cmd "$vm" getstate` ; do
#echo $VMstate
# If the VM is power ON
if [ $VMstate = "on" ] ; then
echo " "
echo "VM: " $vm
echo "State: is on and will now shut down"
echo "Shutting down: " $vm
vmware-cmd "$vm" stop trysoft
vmwarecmd_exitcode=$(expr $?)
if [ $vmwarecmd_exitcode -ne 0 ] ; then
echo "exitcode: $vmwarecmd_exitcode so will now power off"
vmware-cmd "$vm" stop hard
fi
count_vm_on=$count_vm_on+1
sleep 2
# if the VM is power OFF
elif [ $VMstate = "off" ] ; then
echo " "
echo "VM: " $vm
echo "State: is already powered off"
# if the VM is power suspended
elif [ $VMstate = "suspended" ] ; then
echo " "
echo "VM: " $vm
echo "State: is already suspended"
# if state is getstate or =
else
printf ""
#echo "unknown state: " $VMstate
fi
done
done
########################################################################
# wait for up to 5 min for VMs to shut down
#
if [ $count_vm_on = 0 ] ; then
echo " "
echo "All VMs are powered off or suspended"
else
echo " "
vm_time_out=300
count_vm_on=0
echo "Waiting for VMware virtual machines."
for (( second=0; second<$vm_time_out; second=second+5 )); do sleep 5 printf "." count_vm_on=0 for vm in `vmware-cmd -l` ; do for VMstate in `vmware-cmd "$vm" getstate` ; do if [ $VMstate = "on" ] ; then count_vm_on=$(expr $count_vm_on + 1) fi done done if [ $count_vm_on = 0 ] ; then #echo "exit for" break fi done #echo $VMstate fi #echo $count_vm_on ##################################################################### # Check to see if all VMs are off and if not then power them down for vm in `vmware-cmd -l` ; do #echo "VM: " $vm for VMstate in `vmware-cmd "$vm" getstate` ; do # If the VM is power ON if [ $VMstate = "on" ] ; then echo " " echo "VM: " $vm echo "it is still on but will now be powered off" vmware-cmd "$vm" stop hard sleep 2 fi done done ##################################################################### # Shutdown ESX Host echo " " echo "All VM's have been successfully shut down, halting ESX Host" echo " " shutdown -h now

Virtual Center Performance Reporting

While Virtual Center provides a great performance monitoring interface to gain some insight into VM resource consumption, I wanted a way to automatically get a run down of my VM's memory and CPU utilization e-mailed to me on a weekly basis. This is especially useful when setting resource pool reservations. If you would like to do the same for your VC database, you'll need to set up database mail on your SQL server and create a SQL Server Agent job to run each of the following scripts. Set up a schedule and you're on your way!

Script 1: Memory Utilization

EXEC msdb.dbo.sp_send_dbmail@recipients = 'yourname@yourdomain.com', @subject = 'Virtual Center Memory Utilization Report',@body_format = 'HTML',@body = 'Here are some memory utilization metrics for the last month.',@query = ' USE VirtualCenter select vms.NAME as ''VIRTUAL_MACHINE'' , (AVG(STAT_VALUE)/10000) as ''MEMORY_PERCENT''
into #STAT_REPORT_TEMPfrom dbo.VPXV_HIST_STAT_YEARLY
JOIN VPXV_VMS vmsON SUBSTRING([ENTITY] ,4 , 4) = vms.VMID
where stat_group = ''mem''and entity like ''vm%''and sample_time > (getdate() - 30)
group by vms.NAME
go
select VIRTUAL_MACHINE as ''Virtual Machine'' , MEMORY_PERCENT * vm.MEM_SIZE_MB as ''Memory Usage in MB''
from #STAT_REPORT_TEMP stat
JOIN VPXV_VMS vmsON VIRTUAL_MACHINE = vms.NAME
JOIN VPX_VM vmON vms.VMID = vm.ID
ORDER BY VIRTUAL_MACHINE
go
drop table #STAT_REPORT_TEMP',@attach_query_result_as_file = 1,@query_attachment_filename = 'Virtual Machine Memory Stats.xls'

Script 2: CPU Utilization

EXEC msdb.dbo.sp_send_dbmail@recipients = 'yourname@yourdomain.com', @subject = 'Virtual Center CPU Utilization Report',@body_format = 'HTML',@body = 'Here are some CPU utilization metrics for the last month.',@query = ' USE VirtualCenter select vms.NAME as ''VIRTUAL_MACHINE'' , (AVG(STAT_VALUE)) as ''CPU_USAGE''
from dbo.VPXV_HIST_STAT_YEARLY
JOIN VPXV_VMS vmsON SUBSTRING([ENTITY] ,4 , 4) = vms.VMID
where STAT_NAME = ''usagemhz''and entity like ''vm%''and sample_time > (getdate() - 30)
group by vms.NAMEorder by vms.NAME',@attach_query_result_as_file = 1,@query_attachment_filename = 'Virtual Machine CPU Stats.xls'

Each of these scripts takes an average for each metric over the last month but if you prefer, you could edit these to use a different time frame.

Blog Launch

As I have learned about virtualization over the last year, a bunch of blogs out there helped to point me in the right direction. Now that I have gotten my feet wet and am starting to learn some tricks of my own, I decided to launch this blog as a space to share them. I hope that you find the content helpful and would love to hear your feedback.