Automating VM Builds in My Home Lab

Content Library Templates

Within the last two years I have had unfortunate circumstances where power issues or something else has caused hardware failure that resulted in some sort of corrupted data in my home lab. I swore the next time this happens I’d build a script that would help redeploy and configure my setup, mainly my Horizon environment.

Well.. I found that time again when I lost two drives in my vSAN three node NUCs. Not the vSAN’s fault that I bought cheap spinning disks, or that I waited too long to replace the first failed drive. But I’ve always treated my NUCs as throwaway environments that I can spin up and down anytime. My main Dell T710 hosts my vCenter, one of my domain controllers (plus DNS), my main Win10 jump box and a small Ubuntu docker host to play with other things. My NUCs are truly where I spun up vROps, vRLI and my VDI environments.

My first step was to create a new Content Library for the NUCs and create three VM Templates for 1) Windows 10 2) Server 2019 Standard and 3) Windows 2019 Datacenter needs. After installing off of the ISOs and doing the needed patching I did a Google search on how to use Powershell to update these VMs remotely so that I can automate this monthly. The below script is what I found from https://4sysops.com/archives/powershell-remoting-over-https-with-a-self-signed-ssl-certificate/ because Microsoft would prefer the remote and client hosts be on the same domain. In my case I’m going with best practices and leaving my Templates out of the domain so I don’t have to deal with Trust issues later. You have to start off with building that Self Signed Cert and the Windows Remote Management services to leverage HTTPS as the transport mechanism. The commented lines are for the Powershell host where you need to import the newly created Self Signed Certs into the cert store.

# https://4sysops.com/archives/powershell-remoting-over-https-with-a-self-signed-ssl-certificate/

# Below on Remote/Template VMs

$Cert = New-SelfSignedCertificate -CertstoreLocation Cert:\LocalMachine\My -DnsName $env:COMPUTERNAME
mkdir c:\temp
Export-Certificate -Cert $Cert -FilePath C:\temp\$env:COMPUTERNAME
Enable-PSRemoting -SkipNetworkProfileCheck -Force
dir wsman:\localhost\listener
Get-ChildItem WSMan:\Localhost\listener | Where -Property Keys -eq "Transport=HTTP" | Remove-Item -Recurse
Remove-Item -Path WSMan:\Localhost\listener\listener* -Recurse
New-Item -Path WSMan:\LocalHost\Listener -Transport HTTPS -Address * -CertificateThumbPrint $Cert.Thumbprint –Force
New-NetFirewallRule -DisplayName "Windows Remote Management (HTTPS-In)" -Name "Windows Remote Management (HTTPS-In)" -Profile Any -LocalPort 5986 -Protocol TCP
Set-Item WSMan:\localhost\Service\EnableCompatibilityHttpsListener -Value true
Set-NetConnectionProfile -NetworkCategory Private
Disable-NetFirewallRule -DisplayName "Windows Remote Management (HTTP-In)"

# Below on Client/Host Management Powershell
# Import-Certificate -Filepath "S:\certs\DESKTOP-KPQJPMS" -CertStoreLocation "Cert:\LocalMachine\Root"
# Import-Certificate -Filepath "S:\certs\WIN-1DBMFPA2ND2" -CertStoreLocation "Cert:\LocalMachine\Root"
# Import-Certificate -Filepath "S:\certs\WIN-93OI9TU9IFP" -CertStoreLocation "Cert:\LocalMachine\Root"

Now what I did was create the Windows Task Scheduler on my Powershell host that you will later use the invoke-command to trigger the task to start the Windows Updates to check for, update and reboot. I then exported that task so I wouldn’t have to set it up manually again and can use the top of the script to automatically create by importing for me (which is the first commented out line.) Then I create my variables for the three VMs that we will use as our Templates in the Content Library that we will actually delete every month and upload the new VMs instead. But by defining the same name each time I can create other scripts, like automating the deployment of my Horizon Connection Servers. I do always include how I create my credential (.cred) files but comment them out so that if I end up rebuilding my Powershell host I can easily redo those tasks.

# This script requires a Task Scheduler instance setup to do the download and install of Windows patches. This is due to security policies
# in place that you cannot download anything while in a remote session.  So for each new Master Image be sure to create this which can be done 
# once then export it to a file share and run this command so it's the same in each VM.
#
# schtasks /create /xml "\\192.168.1.95\share\Scripts\Download Windows Updates.xml" /tn "Windows Download and Install Updates" /ru "Administrator" /rp "yourpassword"
# 
# For Windows 10 make sure Windows Remote Management is running as a service, set start automatically in each Master Image VM.
#
# Also change the default web operation timeout from the 300 seconds to -1 to be infinite or to another larger timeout. In my case my home lab NUCs are slow disks so 
# the default 5 mins wasn't long enough and would throw an error but the task would complete within vCenter.  
# Set-PowerCLIConfiguration -WebOperationTimeoutSeconds -1 and then restart the powershell window.
#

#region Variables
$vc = "jw-vcenter.iamware.net"
$Server2019DC = "Server2019-DC"  # Content Library Item Name
$Server2019DCVM = "Server2019-DataCenter" # VM Name
$Server2019STD = "Server2019-STD"  # Content Library Item Name
$Server2019STDVM = "Server2019-Standard" # VM Name
$Win10GI = "Win10-Gold-Image"  # Content Library Item Name
$Win10GIVM = "Win10-IC-Parent" # VM Name
$contentlibrary = "NUC-ContentLibrary"
$cluster = "vSAN-Cluster"
$vmhost = "jw-esx-01.iamware.net"
$vmfolder = "Template-Masters"


# I leave these processes in my scripts so I can create the credential file when needed on new PS host but keep the lines commented out except for import line that is needed.
#$localcred = get-credential # For non-domain user needs
#$localcred | Export-Clixml -path S:\scripts\localcred.cred
$localcred = Import-Clixml -path S:\scripts\localcred.cred
# Save vCenter credentials - Only needs to be ran once to create .cred file.
# $credential = Get-Credential # Can be a service account or domain user
# $credential | Export-Clixml -path S:\scripts\jw-vcenter.cred
# Import cred file
$credential = import-clixml -path S:\scripts\jw-vcenter.cred
#endregion

#region Let's get started!
# Connect to vCenter with saved creds
Write-Host "Connecting to $vc"
connect-viserver -Server $vc -Credential $credential

# Get list of VMs based upon folders in vCenter
$vmservers=get-vm -location (Get-Folder -Name Template-Masters)
$vmservers | select Name | export-csv s:\scripts\templates-masters.csv -NoTypeInformation
$servers = import-csv S:\scripts\templates-masters.csv | Select -ExpandProperty name
Write-Host "Starting $servers on $vc"
Start-VM -VM $servers

function Start-Sleep($seconds) {
    $doneDT = (Get-Date).AddSeconds($seconds)
    while($doneDT -gt (Get-Date)) {
        $secondsLeft = $doneDT.Subtract((Get-Date)).TotalSeconds
        $percent = ($seconds - $secondsLeft) / $seconds * 100
        Write-Progress -Activity "Sleeping.." -Status "Powering on VMs.." -SecondsRemaining $secondsLeft -PercentComplete $percent
        [System.Threading.Thread]::Sleep(500)
    }
    Write-Progress -Activity "Waiting.." -Status "Letting the OS boot.." -SecondsRemaining 0 -Completed
}

# Sleep for 5 minutes to give the OS time to customize.
Start-Sleep 300

Get-VM -location (Get-Folder -Name Template-Masters) | Select Name, @{N="IP Address";E={@($_.guest.IPAddress[0])}} | Format-Table -HideTableHeaders

# IPs for Master Images used for templates - you may want to reserve the IP and MAC so doesn't change in future.
$IP1 = (Get-VM -Name $Server2019DCVM | Select @{N="IP";E={@($_.guest.IPAddress[0])}} | Format-Table -HideTableHeaders | Out-String).Trim()
$HostName1 = (Get-VM -Name $Server2019DCVM | Select @{N='FQDN';E={$_.ExtensionData.Guest.IPStack[0].DnsConfig.HostName}} | Format-Table -HideTableHeaders | Out-String).Trim()
$IP2 = (Get-VM -Name $Win10GIVM | Select @{N="IP";E={@($_.guest.IPAddress[0])}} | Format-Table -HideTableHeaders | Out-String).Trim()
$HostName2 = (Get-VM -Name $Win10GIVM | Select @{N='FQDN';E={$_.ExtensionData.Guest.IPStack[0].DnsConfig.HostName}} | Format-Table -HideTableHeaders | Out-String).Trim()
$IP3 = (Get-VM -Name $Server2019STDVM | Select @{N="IP";E={@($_.guest.IPAddress[0])}} | Format-Table -HideTableHeaders | Out-String).Trim()
$HostName3 = (Get-VM -Name $Server2019STDVM | Select @{N='FQDN';E={$_.ExtensionData.Guest.IPStack[0].DnsConfig.HostName}} | Format-Table -HideTableHeaders| Out-String).Trim()

Add-HostFileEntry -hostname $Hostname1 -ipaddress $IP1
Add-HostFileEntry -hostname $Hostname2 -ipaddress $IP2
Add-HostFileEntry -hostname $Hostname3 -ipaddress $IP3

#endregion This is just the beginning..

# For testing purposes to make sure I can still connect to the 3 OS instances and verify there are Updates to install.
invoke-command -ComputerName $Hostname1 -scriptblock {get-windowsupdate} -UseSSL -Credential $localcred
invoke-command -ComputerName $Hostname2 -scriptblock {get-windowsupdate} -UseSSL -Credential $localcred
invoke-command -ComputerName $Hostname3 -scriptblock {get-windowsupdate} -UseSSL -Credential $localcred

#region Let's update the first master image..
Write-Host "Connecting to $Server2019DCVM and running Task Scheduler to download and install recent updates.."
Invoke-Command -ComputerName $Hostname1 -ScriptBlock {
schtasks /Query /TN "Windows Download and Install Updates"
Install-Module -Name PSWindowsUpdate -Confirm:$false -Force
Get-WindowsUpdate

Start-ScheduledTask -TaskName "Windows Download and Install Updates"

function Start-Sleep($seconds) {
    $doneDT = (Get-Date).AddSeconds($seconds)
    while($doneDT -gt (Get-Date)) {
        $secondsLeft = $doneDT.Subtract((Get-Date)).TotalSeconds
        $percent = ($seconds - $secondsLeft) / $seconds * 100
        Write-Progress -Activity "Sleeping.." -Status "Waiting for Windows Updates to install.." -SecondsRemaining $secondsLeft -PercentComplete $percent
        [System.Threading.Thread]::Sleep(500)
    }
    Write-Progress -Activity "Sleeping.." -Status "Applying new patches and updates.." -SecondsRemaining 0 -Completed
}


Start-Sleep 600
} -UseSSL -Credential $localcred
#endregion

#region Here's the second master image..
Write-Host "Connecting to $Win10GIVM and running Task Scheduler to download and install recent updates.."
Invoke-Command -ComputerName $Hostname2 -ScriptBlock {
Start-Sleep -seconds 5
schtasks /Query /TN "Windows Download and Install Updates"
Install-Module -Name PSWindowsUpdate -Confirm:$false -Force
Get-WindowsUpdate
Start-ScheduledTask -TaskName "Windows Download and Install Updates"

function Start-Sleep($seconds) {
    $doneDT = (Get-Date).AddSeconds($seconds)
    while($doneDT -gt (Get-Date)) {
        $secondsLeft = $doneDT.Subtract((Get-Date)).TotalSeconds
        $percent = ($seconds - $secondsLeft) / $seconds * 100
        Write-Progress -Activity "Sleeping.." -Status "Waiting for Windows Updates to install.." -SecondsRemaining $secondsLeft -PercentComplete $percent
        [System.Threading.Thread]::Sleep(500)
    }
    Write-Progress -Activity "Sleeping.." -Status "Applying new patches and updates.." -SecondsRemaining 0 -Completed
}


Start-Sleep 600
} -UseSSL -Credential $localcred
#endregion

#region Last master image
Write-Host "Connecting to $Server2019STDVM and running Task Scheduler to download and install recent updates.."
Invoke-Command -ComputerName $Hostname3 -ScriptBlock {
Start-Sleep -seconds 5
schtasks /Query /TN "Windows Download and Install Updates"
Install-Module -Name PSWindowsUpdate -Confirm:$false -Force
Get-WindowsUpdate
Start-ScheduledTask -TaskName "Windows Download and Install Updates"

function Start-Sleep($seconds) {
    $doneDT = (Get-Date).AddSeconds($seconds)
    while($doneDT -gt (Get-Date)) {
        $secondsLeft = $doneDT.Subtract((Get-Date)).TotalSeconds
        $percent = ($seconds - $secondsLeft) / $seconds * 100
        Write-Progress -Activity "Sleeping.." -Status "Waiting for Windows Updates to install.." -SecondsRemaining $secondsLeft -PercentComplete $percent
        [System.Threading.Thread]::Sleep(500)
    }
    Write-Progress -Activity "Sleeping.." -Status "Applying new patches and updates.." -SecondsRemaining 0 -Completed
}

Start-Sleep 600
} -UseSSL -Credential $localcred
#endregion

#region Now to restart the VMs..
# Restart VMs to get a clean system
Write-Host "Rebooting $servers to clear up any last update installs.."
Restart-VMGuest -VM $servers
Start-Sleep 600
#endregion

#region Now to shutdown the VMs..
# Shutdown VMs
Write-Host "Shutting down $servers on $vc.."
Shutdown-VMGuest -VM $servers -Confirm:$false

function Start-Sleep($seconds) {
    $doneDT = (Get-Date).AddSeconds($seconds)
    while($doneDT -gt (Get-Date)) {
        $secondsLeft = $doneDT.Subtract((Get-Date)).TotalSeconds
        $percent = ($seconds - $secondsLeft) / $seconds * 100
        Write-Progress -Activity "Sleeping.." -Status "Cleanly doing an OS shutdown.." -SecondsRemaining $secondsLeft -PercentComplete $percent
        [System.Threading.Thread]::Sleep(500)
    }
    Write-Progress -Activity "Sleeping.." -Status "Waiting for VMs to safely complete the shutdown process.." -SecondsRemaining 0 -Completed
}

Start-Sleep 300 # Giving the OS enough time to safely shutdown.
#endregion

#region Now to update the Content Catalog
# First we need to delete the original VM Templates since there isn't an update feature for VMs.
Remove-ContentLibraryItem -ContentLibraryItem $Server2019STD -Confirm:$false
Remove-ContentLibraryItem -ContentLibraryItem $Server2019DC -Confirm:$false
Remove-ContentLibraryItem -ContentLibraryItem $Win10GI -Confirm:$false
New-ContentLibraryItem -ContentLibrary $ContentLibrary -Name $Server2019DC -VM $Server2019DCVM -Location $vmhost -VMTemplate -InventoryLocation $vmfolder
New-ContentLibraryItem -ContentLibrary $ContentLibrary -Name $Server2019STD -VM $Server2019STDVM -VMTemplate -Location $vmhost -InventoryLocation $vmfolder
New-ContentLibraryItem -ContentLibrary $ContentLibrary -Name $Win10GI -VM $Win10GIVM -VMTemplate -Location $vmhost -InventoryLocation $vmfolder
#endregion

Be on the look out for future TAM Lab YouTube series where I’ll go over these scripts. I may do a screen capture and speed up the video to prove how well this works.

Shutdown/Power VM Guest Automation

Recently I wanted to automate some processes in my homelab environment that take longer than 5 minutes to do. One of the tasks is I have about 8 VMs that I deemed “non-essential” that I don’t need running 24/7 so I’d like to let that compute resources and power to be saved for those other VMs that I do want running, in this case I’ve been a part of the VMware Folding@Home team. I have one, 2 vCPU VM running per Intel NUC that if I power on all my VMs in that cluster I end up starving each other and tend to take longer to do work.

Below is the script to shutdown the guest cleanly based upon the VM folder “Non-Essential-Services” that I created within vCenter. It starts off with creating a credential file that protects my password without having to type it in every time. Then it pulls a list of VMs that are in a power on state currently and exports that to a CSV file that we will use later. Finally it sends a clean shutdown command to the guest OS.

# Script to shutdown the guest OS on several VMs - Will be used to power down homelab at night.
# Save vCenter credentials - Only needs to be ran once to create .cred file.
# $credential = Get-Credential
# $credential | Export-Clixml -path c:\path\to\folder\scripts\vcenter.cred
# Import cred file
$credential = import-clixml -path c:\path\to\folder\scripts\vcenter.cred
# Connect to vCenter with saved creds
connect-viserver -Server vcenter.fqdn.com -Credential $credential
# Get list of VMs based upon folders in vCenter
$vmservers=get-vm -location (Get-Folder -Name Non-Essential-Services) | Where {$_.PowerState -eq "PoweredOn"} 
$vmservers | select Name | export-csv c:\path\to\folder\scripts\non-essential-services.csv -NoTypeInformation
$vmservers | Shutdown-VMGuest -Confirm:$false

Below is the Power on script based only on the VMs that the shutdown script exported the night before. Again I commented out the credential part but wanted to keep it in so if something happens I can redo it easily. Then it imports that CSV file of VMs and then powers just those back on.

# Script to power on the guest OS on several VMs - Will be used to power on non-essential VMs that were powered down at night.
# Save vCenter credentials - Only needs to be ran once to create .cred file.
# $credential = Get-Credential
# $credential | Export-Clixml -path c:\path\to\folder\scripts\vcenter.cred
# Import cred file
$credential = import-clixml -path c:\path\to\folder\scripts\vcenter.cred
# Connect to vCenter with saved creds
connect-viserver -Server vcenter.fqdn.com -Credential $credential
# Import the night before list of VMs that were automatically powered off
$servers = import-csv c:\path\to\folder\scripts\non-essential-services.csv | Select -ExpandProperty name
Start-VM -VM $servers

Now you can easily run these by launching the script or what I plan on doing is using Task Scheduler from one of my Windows servers to perform this on a daily routine. If you have other ideas, hit me up.