darrylcauldwell.com On a journey around the datacenter and public cloud.

vSphere 6, VSAN & NSX 6.2 Homelab In The Public Cloud

Over the last few years the suite of products which VMware offer has grown its made it less practical to obtain and run enough hardware for a home learning lab. At the same time the cloud promises access to resources on demand and paid for as consumed. I came across Ravello who offer a unique proposition, namely running nested ESXi on public cloud.  They are able to achieve this as they have written an abstraction layer called HPX which offers nested virtualisation engine, a software defined network and a storage overlay. The solution is pay as you use, and so seems potentially the ideal way to run my homelab cloudlab.

Windows Management Host

The first thing to build is a server to host all management tools and build out the environment. I chose Windows Server 2012 R2 as this will also host Active Directory. The high level steps I followed,

  • Upload Windows Server 2012 R2 and ESXi 6.0 Update 2 ISO files to Ravello Library -> Disk Images
  • Create new Ravello Application called 'vSphere 6'
  • Change to Canvas tab within application and add VM named 'Empty' from library
  • Select the 'Empty' VM and its specification will be displayed in right pane and change following
    • General -> Name to 'ad'
    • General  -> Description to 'Active Directory'
    • General  -> Hostnames to 'ad'
    • System -> CPUs to '2'
    • System -> Mem Size to 4GB
    • Disks -> Disk 1 Sized to 60GB
    • Disks -> CD to the uploaded Windows 2012 R2 ISO
    • Networks - Network 1 to have IP Configuration
      • Static IP 172.16.20.5
      • Netmask 255.255.255.0
      • Gateway 172.16.20.254
      • DNS 172.16.20.5
    • Services Remove ssh
    • Services Add rdp Port 3389
  • Power on the VM and open the VNC console
  • Unbind IPv6
  • Configure IPv4 address 172.16.20.5, net mask 255.255.255.0, gateway 172.16.20.254 DNS server 172.16.20.5
  • Give computer name as 'ad'
  • Disable IE Enhanced Security
  • Enable Remote Desktop connection,  find public IP from VM Summary tab and connect using mstsc
  • Install all Windows Updates
  • Add Active Directory, DNS, DHCP and .net Framework 3.5 using Roles and Features
  • Install Google Chrome
  • Install Google Chrome PostMAN Rest Client App
  • Install putty and WinSCP
  • Create a folder C:\ISOs
  • Download vCenter appliance 6.0 Update 2 ISO file
  • Create a DNS Forward lookup zone for darrylcauldwell.home
  • Create a DNS Reverse lookup zone for 172.16.20.0
  • Create A & PTR record in DNS for 'ad' with IP '172.16.20.5'
  • Add a new Active Directory forest named darrylcauldwell.home
  • Install License Key and Activate Windows

ESXi Hosts

From ravello web application open the vSphere 6 application and change to canvas view. Add three new VMs from the ‘Empty ESX’ library object.  For each esx VM change the following.

  • General -> Name to 'esx1, esx2 OR esx3'
  • General  -> Description to 'esx1, esx2 OR esx3'
  • General  -> Hostnames to 'esx1, esx2 OR esx3'
  • System -> CPUs to '4'
  • System -> Mem Size to 16GB
  • Disks -> Disk 1 Sized to 5GB (Make bootable)
  • Disks -> Disk 2 Sized to 4GB
  • Disks -> Disk 3 Sized to 2048GB
  • Disks -> CD to the uploaded ESXi ISO
  • Networks - Network 1 named mgmt to have IP Configuration
    • Static IP 172.16.20.11, 172.16.20.12 OR 172.16.20.13
    • Netmask 255.255.255.0
    • Gateway 172.16.20.254
    • DNS 172.16.20.5
  • Networks - Network 2 named vmotion to have IP Configuration
    • Static IP 172.16.21.11, 172.16.21.12 OR 172.16.21.13
    • Netmask 255.255.255.0
  • Networks - Network 3 named vsan to have IP Configuration
    • Static IP 172.16.22.11, 172.16.22.12 OR 172.16.22.13
    • Netmask 255.255.255.0
  • Networks - Network 4 named nsx to have IP Configuration
    • Static IP 172.16.23.11, 172.16.23.12 OR 172.16.23.13
    • Netmask 255.255.255.0

Power on all three ESX VMs and perform installation of ESX to the 5GB disk.  Using console configure IPv4 address, DNS to 172.16.20.5, hostname, disable IPv6, enable SSH and Shell.

Configure ESXi Hosts

All disk we have is hdd but in order to form VSAN we need to  have some SSD,  we should mark our 4GB disk with the enable_ssd option using the method described in  kb2013188. For how my ESX detected devices the command syntax was

 
esxcli storage nmp satp rule add --satp=VMW_SATP_LOCAL --device mpx.vmhba2:C0:T0:L0 --option "enable_ssd"

In order to run VMs on these we need to add the ‘vmx.allowNested’ flag on each ESX host.  Run

 
vi /etc/vmware/config 

and add the line

vmx.allowNested = "TRUE"

Run

/sbin/auto-backup.sh

Ignore any warnings.

VSAN Bootstrap

In order to deploy vCenter we require some storage,  as we require vCenter to form VSAN we need to form a VSAN bootstrap with one host so we can deploy VCSA then form VSAN between all hosts.  The method is described in this  William Lam guide.

vCenter Appliance

Mount the vCenter Appliance ISO file to AD server and from this install \vcsa\VMware-ClientIntegrationPlugin-6.0.0.exe then open vcsa-setup.html in Chrome and allow it to access the client integration plugin.

  • point installer to the ESXi host you created the bootstrap VSAN on
  • name appliance 'vcenter'
  • use embdeed psc
  • create new sso domain called vsphere.local
  • ensure Tiny size is selected
  • select vsan datastore
  • use embdedded PostgresSQL DB
  • network address 172.16.20.10
  • system name vcenter.darrylcauldwell.home
  • netmask 255.255.255.0
  • gateway 172.16.20.254
  • dns server 172.16.20.5
  • ntp server 172.16.20.5
  • enable ssh

During the deployment it powers on the VM,  due to a limitation in how Ravello presents hardware to ESX as vCenter boots it presents a warning and doesn’t boot automatically.  As its deploying connect to C# client console and press any key to continue past warning,  if you don’t do this the installation process fails due to timeout.

Configure vCenter Cluster

Create a Datacenter and a Cluster then add the three ESXi hosts using FQDN.

Update the networking on each ESXi host and add a vStandard Switch,  linked to vmnic1,  with IP address (172.16.21.11, 172.16.21.12 OR 172.16.21.13) and enabled for vMotion.

Update the networking on each ESXi host and add a vStandard Switch,  linked to vmnic2,  with IP address (172.16.22.11, 172.16.22.12 OR 172.16.22.13) and enabled for VSAN.

Enable VSAN, DRS and HA for the cluster.

NSX

Create a distributed switch, portgroup and add all three hosts using vmnic3 for uplink. Deploy NSX Manager OVA give name nsx and IP address 172.16.20.8. Do not check box to power on after deployment.

As our ESX hosts only have 16GB we need to reduce the configured size from 16GB to 8GB.  First we need to remove the memory reservation of VM object and as this is a very small environment reduce the assigned memory to 8GB.

The NSX Manager virtual machine deploys a VMXNet3 NIC this doesn’t play nicely with Ravello so remove this NIC and add a e1000 NIC.  If you do this prior to powering on then the installation scripts use this.

Now you can power on NSX Manager.

Proxy

The Ravello networking allows internet incoming and outgoing traffic to VMs running in Ravello via NAT rules. This means VMs nested into vSphere don’t get internet access,  as such its useful to deploy a Squid Proxy.

Download CentOS ISO and upload to Ravello library,  deploy a Empty VM and attach the ISO, configure Static Networking  for IP 172.16.20.250.  During CentOS setup wizard,  enable network and configure IP address as in ravello console. Once installed open console and run

yum -y update
yum -y install squid

The default configuration works with no changes needed and offers working proxy to your nested VMs (vCenter and NSX) on 172.16.20.250:3128.

Be social and share this post!