Nested ESXi Auto-Deploy

A lot of my work these days revolves around creating labs for various vendors and departments, including VMware ironically. I needed a way to quickly deploy and create Nested ESXi hypervisors via vCloud Director that had a consistent configuration. While just cloning an ESXi installation would work, it often had several catches such as the following:

  1. Storage Devices had the same names
  2. Mac Addresses and UUID’s were the same

This led to quite a bit of re-work once a clone was taken. Taking a queue from this blog: how-to-provision-nested-esxi-hosts I was able to create a local.sh shell script that can be run during the first bootup of a new ESXi hypervisor. One bit that took a bit was guaranteeing that the storage device was unique between all hosts. This code accomplishes it:

In my environment the VMDK was identified by the following: ‘/vmfs/devices/disks/mpx.vmhba1\:C0\:T0\:L0\:3’ So I needed to remove the partition name so the below code would create a unique partition name for each host based on the ComputerName.

	vmkfstools -C vmfs5 -b 1m -S $vCloud_computerName /vmfs/devices/disks/mpx.vmhba1\:C0\:T0\:L0\:3

To prep a host we need to do the following. This will reset the Mac Address of the VMkernel interfaces, reset the System UUID and backup the configuration before rebooting.

esxcli system settings advanced set -o /Net/FollowHardwareMac -i 1
sed -i "/\/system\/uuid/d" /etc/vmware/esx.conf
/sbin/auto-backup.sh

 

 

The result is on each deployment of the ESXi server boots with it’s own variables as passed from vCloud. The below script initializes interfaces, DNS servers, datastore names reboots the system. To prevent a never ending loop one of the last bits I added was to delete the script and reboot, may be a bit blunt but it gets the job done.

#!/bin/sh</code>

# local configuration options

# Note: modify at your own risk! If you do/use anything in this
# script that is not part of a stable API (relying on files to be in
# specific places, specific tools, specific output, etc) there is a
# possibility you will end up with a broken system after patching or
# upgrading. Changes are not supported unless under direction of
# VMware support.
getprops_from_ovfxml()
{

/bin/python - &lt;/tmp/ovf.xml 2&gt;/dev/null
[ -s /tmp/ovf.xml ] || {
log "ERROR: Cannot get OVF parameters through VMware Tools. Exiting ..."
exit 1
}
o=`hostname`
eval `getprops_from_ovfxml /tmp/ovf.xml`
[ "${o%%.*}" == "$vCloud_computerName" ] &amp;&amp;
{
log "Hostname is unchanged. Skipping OVF customizations ..."
sed -i '60,73d' /etc/rc.local.d/local.sh
/sbin/auto-backup.sh
exit 0
}
grep "fqdn.fqdn" /etc/dhclient-vmk0.conf &gt;/dev/null &amp;&amp;
{
log "OVF customization already done. Exiting ..."
exit 0
}
log "Applying OVF customizations ..."
rm -f /tmp/ovf.xml
esxcli system hostname set --host=$vCloud_computerName
esxcli network ip interface ipv4 set -i vmk0 -I $vCloud_ip_0 -N $vCloud_netmask_0 -t static
esxcli network ip route ipv4 add -g $vCloud_gateway_0 -n default
esxcli network ip dns server add --server=$vCloud_dns1_1
esxcli network ip dns server add --server=$vCloud_dns2_1
esxcli network ip interface ipv4 set -i vmk1 -I $vCloud_ip_1 -N $vCloud_netmask_1 -t static
esxcli system hostname set --domain=$vCloud_suffix_1
esxcfg-route -a 10.252.0.0/16 gw $vCloud_gateway_1
esxcfg-route -a 10.12.42.0/24 gw $vCloud_gateway_1
esxcli system maintenanceMode set --enable true --timeout=1
vmkfstools -C vmfs5 -b 1m -S $vCloud_computerName /vmfs/devices/disks/mpx.vmhba1\:C0\:T0\:L0\:3
/sbin/auto-backup.sh
esxcli system shutdown reboot --reason="Updated System UUID"
esxcfg-init --set-boot-progress-text "OVF Settings applied...."
exit 0

vCloud Next Free IP

I’ve been writing a bit of automation these last few weeks and I came across a need to determine the next block of currently unassigned IP’s within vCloud Director. Those of you who use vCD will know that when you deploy an Edge vCD will automatically grab the next available IP, but I found there was no easy way to get this same information from vCD itself. Put together a little powercli script that takes as it’s input variable the name of the external network we want to query and returns an array of unassigned IP’s.

What makes this task a little more difficult is the fact that vCloud has both Assigned IP’s and Gateway IP’s. IP’s can be assigned to a gateway but not be in use (suballocated) so when determining what IP is free next we have to query both assignments and compile a single list.


function Get-FreeExtIPAddress([String]$extnetName){
$extnet = Get-ExternalNetwork -name $extnetName
$ExtNetView = $Extnet | Get-CIView

$allocatedGatewayIPs = $extnetView.Configuration.IpScopes.IpScope[0].SubAllocations.SubAllocation.IpRanges.IpRange

[int]$ThirdStartingIP = [System.Convert]::ToInt32($extnet.StaticIPPool[0].FirstAddress.IPAddressToString.Split(".")[2],10)
[int]$ThirdEndingIP = [System.Convert]::ToInt32($extnet.StaticIPPool[0].LastAddress.IPAddressToString.Split(".")[2],10)
[int]$FourthStartingIP = [System.Convert]::ToInt32($extnet.StaticIPPool[0].FirstAddress.IPAddressToString.Split(".")[3],10)
[int]$FourthEndingIP = [System.Convert]::ToInt32($extnet.StaticIPPool[0].LastAddress.IPAddressToString.Split(".")[3],10)
$octet = $extnet.StaticIPPool[0].FirstAddress.IPAddressToString.split(".")
$3Octet = ($octet[0]+"."+$octet[1]+"."+$octet[2])
$2Octet = ($octet[0]+"."+$octet[1])
$ips = @()

if ($ThirdStartingIP -eq $ThirdEndingIP) {

$ips = $FourthStartingIP..$FourthEndingIP | % {$3Octet+'.'+$_}

} else {

do {

for ($i=$FourthStartingIP; $i -le 255; $i++) {
$ips += ($2Octet + "." + $ThirdStartingIP + "." + $i)
}
$ThirdStartingIP=$ThirdStartingIP + 1

} while ($ThirdEndingIP -ne $ThirdStartingIP)
for ($i=0;$i -le $FourthEndingIP; $i++) {
$ips += ($2Octet + "." + $ThirdStartingIP + "." + $i)

}
}

$allocatedIPs = $ExtNetView.Configuration.IpScopes.IpScope[0].AllocatedIpAddresses.IpAddress

for ($i=0;$i -le $ips.count; $i++) {
for ($j=0; $j -lt $allocatedGatewayIPs.count; $j++) {
if ($ips[$i] -eq $allocatedGatewayIPs[$j].StartAddress) {
$ips = $ips | Where-Object { $_ -ne $ips[$i] }
$i--
}
}
for($z=0;$z -lt $allocatedIPs.count;$z++) {
if ($ips[$i] -eq $allocatedIPs[$z]) {
$ips = $ips | Where-Object { $_ -ne $ips[$i] }
$i--
}
}
}
return $Ips
}

Getting Started

After months and years of swearing I would never put together a blog I finally caved…..In my field it’s inevitable I think that everyone eventually develops a blog if for nothing more than to show what they do on a daily basis. The purpose of this blog is to give back to the VMware community by sharing my day to day experiences working as a Product and Operations Engineer at the largest VMware service provider. Here goes nothing……