Why Your Virtual Infrastructure Network Should Be Segmented

When architecting a Virtual Infrastructure, networking is one of your most important concerns. Dropping 50 servers into 5 buckets creates new IO and security concerns. Here is how we have implemented this and had good results:

1) Service Console vLAN - Having a separate vLAN for your service consoles not only brings up the level of security around them, but ensures that your ESX to ESX pings for high availability cannot be bogged down by other network traffic.

2) vMotion Physically Segmented - The obvious reason for this is to improve performance. Running a vMotion or an svMotion sends an enormous amount of data over the network. So as to not impact the performance of other applications on your network and to get the best throughput possible in your migrations, having this physically separate is key.

The less obvious reason for this is security. When a vMotion is happening, the raw contents of a VM's memory are sent across the network. Should a malicious party be listening on this vLAN, they would be able to read the entire memory contents of your server. If you have any type of sensitive data being processed on that server, this creates an additional point of exposure.

3) Virtual Machine Network - In our case, we use the same vLAN for our VMs that we use for our physical servers.

4) Storage Network - If you will be using iSCSI or NFS on the back end, it makes sense to segment this out at least virtually if not physically to be able to ensure proper performance of your storage devices.

ESX Partitioning

While most of the defaults in an ESX installation will be fine, I always take the time to edit the partition scheme. Since most new servers will come with drives no smaller than 60GB and I'll be storing all of my VMDKs on shared storage, there's no reason to not allocate extra space to places that will use it. Here is my usual breakdown:

/boot - 100MB - ext3 - The default is fine here.

/ - 10GB - ext3 - If you ever want to update your Service Console, it's nice to have some extra space available.

(none) - 1600MB - swap - The service console can access a maximum of 800MB of RAM. Your swap file should always be at least twice the size of memory being used. Since this partition cannot be resized without doing a reinstall, I always set it to the max in case I need to allocate more memory to the SC down the road a bit.

/var/log - 2GB - ext3 - Having a separate partition for your logs prevents them from filling up your root partition in the case of system issues.

(none) - 100MB - vmkcore - While this is optional, it holds the kernel dump if you have a Purple Screen of Death. When you call VMWare support in such a case, they will want to look at the contents held here.

/home - ? - ext3 - If you plan on storing scripts and other such files on your ESX server, you may want to carve out an extra home partition.

/vmfs/volumes/xyz - ? - vmfs-3 - Any leftover space can be set aside as a spare VMFS volume.