Consolidating unmounted and open snapshots after virtual machine backup

At times, one or more disks from a virtual machine will remain mounted and snapshots will remain following the backup of a virtual machine. The cause of this is generally due to their not being enough time to unmount the disks and consolidate the snapshots when the job completes.

This would require user interaction to firstly unmount the virtual machine disks and then perform a consolidation of the virtual machine virtual disks. As we can agree, this is a manual task that is perfect to be automated.

The below solution has a number of caveats, the backup solution/appliance is a virtual machine and the virtual machine contains no hard disks that are of the disk type ‘Independent – Nonpersistent’.

The script in its entirety can be downloaded from here. However, the below will describe the logic to remove the virtual machine hard disks and consolidate the virtual machines.

Once we have established a connection to the vCenter Server and specified the virtual machine running the backup appliance/solution we will retrieve a collection of virtual machines where the ‘RunTime.ConsolidatedNeeded’ value is equal to ‘True’.

$VMs = Get-View -ViewType VirtualMachine -Property Name, RunTime | Where-Object {$_.RunTime.ConsolidationNeeded -eq $True}

Then we will connect to the virtual machine hosting the backup appliance/solution and retrieve all the virtual machine hard disks where the persistence value is equal to ‘IndependentNonPersistent’.

$HardDisks = Get-VM $ComputerName | Get-HardDisk | Where-Object {$_.Persistence -eq 'IndependentNonPersistent'} 

We will then invoke the Remove-HardDisk cmdlet to remove the filtered virtual machine hard disks. In this instance we are only removing the hard disks and not deleting them permanently.

Get-HardDisk -VM $ComputerName -Name $HardDisks.Name | Remove-HardDisk -Confirm:$False

Once the virtual machine hard disks have been removed, we will now consolidate the virtual machine disks on each of the virtual machines returned in the collection.

(Get-VM $VM.Name).ExtensionData.ConsolidateVMDisks() | Out-Null

The above script will provide console output to report the status of the invocation and can be triggered as a scheduled task returning an exit return code dependant on the status of the invocation.

Advertisements

PowerCLI: Retrieving ESXi Host System Management Network IP Address

I was recently looking at retrieving the allocated management network IP address for a large number of ESXi host systems and therefore decided to use the ‘Get-VMHostNetworkAdapter’ cmdlet  to which I would filter based on the device name value retrieved from the ESXi host system. In its simpliest form the cmdlet can be invoked as below:

Get-VMHostNetworkAdapter -VMHost esxi1.dean.local

In this instance I wanted to retrieve only the allocated IP address to the device name vmk0 (management network) for all ESXi host systems in a datacenter, which could be achieved by invoking the cmdlet with the filter to retrieve only the host network adapter where the device name was equal to ‘vmk0’

Get-Datacenter Edinburgh | Get-VMHost | Select-Object @{N="Name";E={$_.Name}},@{N="Management Network";E={(Get-VMHostNetworkAdapter -VMHost $_.Name  | Where-Object {$_.Name -eq "vmk0"}).IP }}

As the above retrieved both the name of the ESXi host system and the allocated IP address of the management network this provided me with the information required. However, in the future would I need to repeat this task with the same or different requirements. Therefore I decided to create a function which would retrieve the management network IP address for a filtered list of ESXi host systems to which this could be targeted at a vCenter Server System, Datacentre, Cluster or a particular list of ESXi host systems., as below.

The function will establish a connection to the vCenter Server System retrieve a collection of ESXi host systems as specified or retrieve a collection of all ESXi host systems managed by that vCenter Server and return the host name and management network IP address.

Function Get-VMHostManagementNetwork { 

Param ([Parameter(Mandatory=$true)][String[]] $VIServer,[String[]] $Datacenter,[String[]] $Cluster,[String[]] $VMHost) 

If (-not (Get-PSSnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue)) {Add-PSSnapin VMware.VimAutomation.Core | Out-Null}

Connect-VIServer $VIServer | Out-Null 

If ($Datacenter){$VMHosts = Get-Datacenter $Datacenter | Get-VMHost} 
ElseIf ($Cluster) {$VMHosts = Get-Cluster $Cluster | Get-VMHost} 
ElseIf ($VMHost) {$VMHosts = Get-VMHost $VMHost} 
Else {$VMHosts = Get-VMHost}

$VMHosts | Select-Object @{N="Name";E={$_.Name}},@{N="Management Network";E={(Get-VMHostNetworkAdapter -VMHost $_.Name  | Where-Object {$_.Name -eq "vmk0"}).IP }}
} 

The PowerShell function can be downloaded from here.

Windows Server 2012 virtual machines stop responding at splash screen

I recently discovered an issue on a restart of a virtual machine running the guest operating system Windows Server 2012 where following a restart the virtual machine would stop responding at the splash screen and required a manual shutdown and power on to restart the virtual machine. From investigation is a known issue caused by the following:

Startingwith Windows 8 / Windows 2012 Server, during its boot process the operating system will reset the TSC (TimeStampCounter, which increments by 1 for each passed cycle) on CPU0. It does not reset the TSC of the other vCPUs and the resulting discrepancy between two vCPUs’ TSC can result in the OS not booting past the Windows splash screen, and a full power off and on will fix it.

There is a workaround provided until this issue is resolved which requires the virtual machine configuration file to be modified so that the TSC for all vCPUs should be reset to zero on a soft reset of the machine, and not just CPU0. In order to apply the workaround, the virtual machine will be required to be in a powered off state to add the following line to the configuration file.

monitor_control.enable_softResetClearTSC = "TRUE"

In order to apply the configuration change to a number of virtual machines, I orchestrated a number of steps in a powershell script as follows:

Firstly, I wanted to write the progress of the orchestration to both the console session and a log file, to which I have previously discussed the creation of this function here.

$LogFile = "softResetClearTSC_11022014.log"

$Path = [System.IO.Path]::Combine($env:USERPROFILE,$LogFile)
$FileMode = [System.IO.FileMode]::Append
$FileAccess = [System.IO.FileAccess]::Write
$FileShare = [IO.FileShare]::Read
$FileStream = New-Object IO.FileStream($Path, $FileMode, $FileAccess, $FileShare)
$StreamWriter = New-Object System.IO.StreamWriter($FileStream)


Function Log($Event)
{
   Write-Host $Event
   $StreamWriter.WriteLine($Event)
}

Now we will connect to the vCenter System System and retrieve a collection of virtual machines to which the configuration change will be applied.

Connect-VIServer deanvc1.dean.local 
$VMS = "deanvm1","deanvm2","deanvm3"

For each object retrieved from the collection I will orchestrate the following steps:

  • Perform a shutdown of the guest operating system (requires VMware Tools to be installed) and wait until the Power State is returned as ‘PoweredOff’.
  • Add the key value pair ‘monitor_control.enable_softResetClearTSC = “TRUE”‘ to the virtual machine configuration file.
  • Power on the virtual machine.
ForEach ($VM in $VMS) 
    { 
    Try
        { 
        Log ("" + (Get-Date -Format s) + ": INFORMATION: Preparing guest operating system for shutdown for virtual machine " + $VM + ".") 
        Shutdown-VMGuest $VM -Confirm:$False | Out-Null 
        Do 
            { 
            Start-Sleep -Seconds 5 
            } 
        Until ((Get-VM $VM).PowerState -eq "PoweredOff")
        } 
    Catch [System.Exception]
        { 
        Log ("" + (Get-Date -Format s) + ": ERROR: Failed to shutdown the virtual machine " + $VM + ".") 
        Log $Error.Exception
        $Error.Clear() 
        Break
        } 
    Log ("" + (Get-Date -Format s) + ": INFORMATION: Successfully shutdown virtual machine " + $VM + ".") 
    Try
        { 
        Log ("" + (Get-Date -Format s) + ": INFORMATION: Modifying configuration file for the virtual machine " + $VM + ".") 
        $View = Get-VM $VM | Get-View
        $Config = New-Object VMware.Vim.VirtualMachineConfigSpec
        $Config.ExtraConfig += New-Object VMware.Vim.OptionValue
        $Config.extraConfig[0].key = "monitor_control.enable_softResetClearTSC"
        $Config.extraConfig[0].value = "TRUE"
        ($View).ReconfigVM_Task($Config) | Out-Null
        } 
    Catch [System.Exception]
        {
        Log ("" + (Get-Date -Format s) + ": ERROR: Failed to modify the configuration file for the virtual machine " + $VM + ".") 
        Log $Error.Exception
        $Error.Clear() 
        Break
        }
    Log ("" + (Get-Date -Format s) + ": INFORMATION: Successfully added the key monitor_control.enable_softResetClearTSC to the virtual machine " + $VM + ".") 
    Try
        {
        Log ("" + (Get-Date -Format s) + ": INFORMATION: Powering on the virtual machine " + $VM + ".") 
        Start-VM $VM -Confirm:$False | Out-Null 
        }
    Catch [System.Exception] 
        {
        Log ("" + (Get-Date -Format s) + ": ERROR: Failed to power on the virtual machine " + $VM + ".") 
        Log $Error.Exception
        $Error.Clear() 
        Break
        }
    Log ("" + (Get-Date -Format s) + ": INFORMATION: Succesfully powered on the virtual machine " + $VM + ".") 
    } 

Finally, we will dispose of the StreamWriter and FileStream classes used for the log function to write the progress to the console session and the log file.

$StreamWriter.Dispose()
$FileStream.Dispose()

The above script can be downloaded from https://github.com/dean1609/scripts/blob/master/Set-softResetClearTSC.ps1.

Retrieving Virtual Machine Templates as vSphere View Objects

It is possible to retreive virtual machine template information using the Get-View cmdlet, by retrieving VirtualMachine type objects based on the property value that describes the object as a template to which the value is returned as a boolean value.

To retrieve VirtualMachine objects , the property value we require to retrieve is Config.Template where the value is ‘True’, which can be achieved as follows:

Get-View -ViewType VirtualMachine -Property Name, Config.Template | Where-Object {$_.Config.Template -eq "True"}

In my example, I required not to include virtual machine templates in the collection of VirtualMachine objects and there selecting only objects where the property value was not equal to ‘True’.

Get-View -ViewType VirtualMachine -Property Name, Config.Template | Where-Object {$_.Config.Template -ne "True"}

For the Property value I am specifying only the Name and Config.Template properties of the view object as this is only what I require, but in your use case you may want to extend the properties retrieved.

Automating the creation of virtual machine hard disks and NTFS mounted volumes for a virtual machine.

As part of an automated delivery of a virtual machine running Microsoft SQL Server I had a requirement to create a number of virtual machine hard disks  and for each disk create a NTFS mounted folder.

In order to achieve this I was required to create two functions one to create the virtual machine hard disks and the second to create the NTFS mounted volume based on the SCSI Port and Target ID of the disk.

Firstly, once the virtual machine has been deployed from a base template, we want to add the virtual machine hard disks with the following requirements:

  • Create four virtual machine hard disks attached to a new VMware Paravirtual SCSI (PVSCSI) adapter.
  • Create two virtual machine hard disks attached to a new VMware Paravirtual SCSI (PVSCSI) adapter.
  • Create two virtual machine hard disks attached to a new VMware Paravirtual SCSI (PVSCSI) adapter.
  • Each group of virtual machine hard disks created are to be of the same capacity.
  • Each virtual machine hard disk will be provisioned with the thick provision eager zeroed format.
  • Each virtual machine hard disk will be created on the same datastore as  the virtual machine.

As discussed, I have created a function for the above named ‘Create-VirtualHardDisks’ to which I have configured the mandatory parameters ‘Number’ for the count of virtual machine hard disks to create and ‘CapacityGB’ for the size of the virtual machine hard disk.

Function Create-VirtualHardDisks {
Param ([Parameter(Mandatory=$true)][int] $Number, [Parameter(Mandatory=$true)][string] $VM, [Parameter(Mandatory=$true)][int] $CapacityGB)

For the number of virtual machine hard disks to be create we will invoke a ForEach statement to repeat the action a specific number of times, in this case the number of virtual machine hard disks that are required to be created.

ForEach ($HardDisk in (1..$Number))
    { 

Now we will invoke the New-HardDisk cmdlet to create the virtual machine hard disks specifying the virtual machine, CapacityGB and the storage format.

New-HardDisk -VM $VM -CapacityGB $CapacityGB -StorageFormat EagerZeroedThick

Once the virtual machine hard disks have been created we now create the VMware Paravirtual SCSI (PVSCSI) adapter and attach the virtual machine hard disks. In order to retrieve the virtual machine hard disks we will invoke the Get-HardDisk cmdlet and invoke the Select-Object cmdlet to retrieve the virtual machine hard disks created by specifying the last number of objects whihc is equal to the number of virtual machine hard disks created.

$HardDisks = Get-HardDisk -VM $VM | Select-Object -Last $Number

Once the virtual machine hard disks have been retrieved we will invoke the ‘New-ScsiController’ and create the VMware Paravirtual SCSI (PVSCSI) adapter and attach the virtual machine hard disks.

New-ScsiController -Type ParaVirtual -HardDisk $HardDisks    
}
}

Now that we have the virtual machine hard disks created, I am now required to create the NTFS mounted volumes based on the SCSI Port and SCSI Target ID of the disk, with the following requirements:

  • Create the empty folder on the parent volume.
  • Initialize the disk with the GPT partition style.
  • Create a new partition and use the maximum size of the disk.
  • Add a partition access path to the empty folder created.
  • Perform a full format of the  volume with the NTFS file system with a Unit Allocation Size of 64K and set the file system label to be that of the empty folder.

As previously, we will be creating a function to achieve this task, to which we will specify mandatory parameters for the virtual machine, NTFS mounted folder path and the SCSI Port and $SCSITargetID of the disk.

Function Create-NTFSMountedFolder { 
Param ([Parameter(Mandatory=$true)][String] $VM,[Parameter(Mandatory=$true)][String] $Folder,[Parameter(Mandatory=$true)][String] $SCSIPort,[Parameter(Mandatory=$true)][String] $SCSITargetID)

We will be invoking this action using the ‘Invoke-VMScript’ cmdlet so we create a script text string to perform the orchestration to create the NTFS mounted folder. Firstly we will invoke the ‘New-Item’ cmdlet to create the empty folder on the parent volume.

$ScriptText = "New-Item -ItemType Directory -Path $Folder;

Now we will be required to retrieve the disk based on the SCSI Port and TargetID. In order to retrieve the disk required we will invoke the ‘Get-Disk’ cmdlet where we will retrieve the disk number required by invoking the ‘Get-WMIObject’ cmdlet with the class Win32_Drive to filter the disks where the SCSI Port and TargetID are equal to the those specified in the mandatory parameter and return the Index value of the disk.

`$Disk = Get-Disk -Number (Get-WmiObject -Class Win32_DiskDrive | Where-Object {`$_.SCSIPort -eq $SCSIPort -and `$_.SCSITargetID -eq $SCSITargetID}).Index;

Once we have retrieved the disk, we will now initialize the disk with the GPT partition style by invoking the Initialize-Disk cmdlet where the disk number is retrieved from the above query by specifying the Number value.

Initialize-Disk -Number `$Disk.Number -PartitionStyle GPT -PassThru;

Next we will create a partition and use the maximum size of the disk by invoking the New-Partition cmdlet .

New-Partition -DiskNumber `$Disk.Number -UseMaximumSize;

For the empty folder created in the first step we will now create an access path by invoking the ‘Add-PartitionAccessPath’ cmdlet to create the access path on the disk and partition specified.

Add-PartitionAccessPath -DiskNumber `$Disk.Number -PartitionNumber 2 -AccessPath $Folder\;

Finally, we will retrieve the partition created using the ‘Get-Partition’ cmdlet and invoke the ‘Format-Volume’ cmdlet to perform a full format of the volume with Unit Allocation Size of 64K and set the file system label to be the name of the empty folder created.

Get-Partition -DiskNumber `$Disk.Number -PartitionNumber 2 | Format-Volume -FileSystem NTFS -AllocationUnitSize 65536  -Full -NewFileSystemLabel $Folder -Confirm:`$False"

Now we will invoke the ‘Invoke-VMScript’ cmdlet to run the above in the guest operating system to create the NTFS mounted folder.

Invoke-VMScript -VM $VM -ScriptText $ScriptText
}

So by invoking the above functions we be able to create the virtual machine hard disks and create a NTFS mounted folder. In the below example, I will create three virtual machine hard disks and then create three NTFS mounted folders with the names D:\Databases1, D:\Databases2 and D:\Databases3.

Create-VirtualHardDisks -VM vm1 -Number 3 -CapacityGB 20
Create-NTFSMountedFolder -VM vm1 -Folder D:\Databases1 -SCSIPort 4 -SCSITargetID 3
Create-NTFSMountedFolder -VM vm1 -Folder D:\Databases2 -SCSIPort 4 -SCSITargetID 4
Create-NTFSMountedFolder -VM vm1 -Folder D:\Databases3 -SCSIPort 4 -SCSITargetID 5

The above functions, can be downloaded from the below:

Create-VirtualHardDisks – https://github.com/dean1609/scripts/blob/master/Create-VirtualHardDisks.ps1

Create-NTFSMountedFolder – https://github.com/dean1609/scripts/blob/e48d13eeb13baefa71777371f265d60d55d5a4a6/Create-NTFSMountedFolder.ps1

Waiting on virtual machine Guest OS customization to complete

I recently discovered a blog article from Vitali Baruh on the PowerCLI QE team in regards to waiting for the guest OS customization to complete following a deployment of a virtual machine from a template.

http://blogs.vmware.com/PowerCLI/2012/08/waiting-for-os-customization-to-complete.html

As Vitali discusses this process can be difficult to determine if this has completed in the guest operating system and if the task had completed successfully.

Previously, following the power on of a deployed virtual machine from a template I have performed a Do…Until loop to process a condition until the ‘Guest.HostName’ value retrieved from the Get-View cmdlet matches the name of the virtual machine as below:

Do     
    {
    $GuestHostName = (Get-View $VM.Id).Guest.HostName
    Start-Sleep -Seconds 10
    } 
Until ($GuestHostName -eq $VM.Name)

Whilst I have received no issues from this, there is an argument agaisnt using this method as we are not retrieving the guest OS customization task status. Now, this is where Vitali has produced an excellent blog post to describe the steps of this function which is based on a number of virtual machine events generated by the vCenter server.

The script workflow, works as follows:

  • 1. For each VM we will look for the last VmStarting event, because the customization process starts after the VM has been powered on
    • 1.1. If such an event is found, change the status to “CustomizationNotStarted”
    • 1.2. If such an event is not found, change the status to “VmNotStarted”
  • 2. Start a loop
    • 2.1. For each VM with status “CustomizationNotStarted”
      • 2.1.1. Check for posted CustomizationStartedEvent after the last power-on operation
        • 2.1.1.1. If such an event is found, change the status to “CustomizationStarted”
    • 2.2. For each VM with status “CustomizationStarted”
      • 2.2.1. Check for Succeded or Failed Event
        • 2.2.1.1. If such an event is found, change the status to the corresponding “CustomizationSucceeded” or“CustomizationFailed” values
    • 2.3. Check is there more VMs with status “CustomizationNotStarted” or “CustomizationStarted” (we should continue to monitor these VMs)
      • 2.3.1. If no, break the loop
    • 2.4. Check whether the specified timeout has elapsed
      • 2.4.1. If yes, break the loop
  • 3. Return result

For those who are provisioning virtual machines to include a guest OS customization with PowerCLI, I would definitely recommend a read of this article.

 

Modifying guest network interface with the Invoke-VMScript cmdlet


images

Set-VMGuestNetworkInterface

With the news that the ‘Set-VMGuestNetworkInterface’ cmdlet is to be deprecated in the next release of PowerCLI (http://blogs.vmware.com/PowerCLI/2014/11/announcement-future-cmdlet-deprecation.html), the question is how do I now modify the guest network interface?

The ability to perform this action can now be moved to the Invoke-VMScript cmdlet, which is a pretty cool way to invoke a script in the guest operating system of the virtual machine when VMware Tools is installed.

As per the description of the cmdlet you will need to satisfy the below:

To run Invoke-VMScript, the user must have read access to the folder containing the virtual machine and a VirtualMachine.Interaction.Console Interaction privilege. The virtual machines must be powered on and have VMware Tools installed. Network connectivity to the ESX system hosting the virtual machine on port 902 must be present to authenticate with the host or the guest OS, one of the HostUser/HostPassword (GuestUser/GuestPassword) pair and HostCredential (GuestCredential) parameters must be provided. The guest account you use to authenticate with the guest operating system must have administrator’s privileges.

To run this cmdlet against vCenter Server/ESX/ESXi versions earlier than 5.0, you need to meet the following requirements:

*You must run the cmdlet on the 32-bit version of Windows PowerShell.
*You must have access to the ESX that hosts the virtual machine over TCP port 902.
*For vCenter Server/ESX/ESXi versions earlier than 4.1, you need VirtualMachine.Interact.ConsoleInteract privilege.  For vCenterServer/ESX/ESXi 4.1 and later, you need VirtualMachine.Interact.GuestControl privilege.

As mentioned VMware Tools is required to be installed and the virtual machine to be powered on, in my use case modifying the guest network interface is a one of a number of script blocks in deployment script to customise a virtual machine and therefore in order to confirm that VMware Tools is running following a virtual machine power on event I run a statement in a script block until the Guest.ToolsStatus is retrieved as ‘toolsOK’ from the Get-View cmdlet, where the MoRef value for the virtual machine is retrieved from an associated array.

Do     
    {
    Start-Sleep -Seconds 10
    $GuestToolsStatus = (Get-View $VM.Id -Property Guest).Guest.ToolsStatus
    } 
Until ($GuestToolsStatus -eq toolsOk)

Now we want to modify the guest network interface, with the following requirements.

  • Configure IPv4 address
  • Configure subnet mask
  • Configure default gateway
  • Configure DNS servers

So, in order to modify the guest network interface I will be replacing the soon to be deprecated ‘Set-VMGuestNetworkInterface’ cmdlet with the ‘New-NetIPAddress’ and ‘Set-DnsClientServerAddress’ cmdlets in Windows Powershell.

In order to run the script inside the guest operating system and in this example a Microsoft Windows Server 2012 operating, I will create a ScriptText variable containing the syntax of the two cmdlets I wish to invoke.  Please not the ‘;’ separator to invoke multiple lines of commands in the text of the script.

$ScriptText = "New-NetIPAddress –InterfaceAlias Ethernet0 –IPAddress 10.0.0.5 -AddressFamily IPv4 –PrefixLength 24 -DefaultGateway 10.0.0.254;
Set-DnsClientServerAddress -InterfaceAlias Ethernet0 -ServerAddresses 10.0.0.1,10.0.0.2"

In the above example, we are modifying the network interface ‘Ethernet0’ to use the IPv4 address 10.0.0.5/24 with a default gateway 10.0.0.254 and the DNS servers 10.0.0.1 and 10.0.0.2.

Now, let’s actually run the command in the guest operating operating system, using the Invoke-VMScript cmdlet, where guest credentials are retrieved from paramaters supplied for my script.

Invoke-VMScript -VM $VM -ScriptText $ScriptText -GuestUser $GuestUser -GuestPassword $GuestPassword

As you can see this is a powerful way to invoke scripts inside the guest operating system. In the default state the script type invoked on a Microsoft Windows operating system is Powershell. However, you may specify the type to be Bat or for a Linux operating system Bash.

The script above, can be downloaded from https://github.com/dean1609/scripts/commit/1bd96807a899a3f8861ca676354eb129545fa1c9.