Identifying orphoned virtual machines and hard disks in vSphere

Connect to the shell of the ESXi host system and browse to location of the datastore (/vmfs/volumes/{datastore name}) to which you want to identity orphoned virtual machine hard disks and invoke the following command. This will return all virtual machine hard disks to which the modified date is seven days in the past (-mtime +7) and return the filename pattern match (find -name “*flat.vmdk*) with the modified date timestamp (-exec stat -c “%n %y” {} \;).

find -iname "*flat.vmdk" -mtime +7 -exec stat -c "%n %y" {} \;

This approach assumes that if the virtual machine hard disk has not been touched in the specified timespan the virtual machine hard disk is orphoned. This may not be accurate, as the virtual machine may be in a powered off state for a period that exceeds the timespan and the virtual machine may be valid. The output should be similar to the below:

./vmfs/volumes/424b26ec-4294ea41/ubuntu-01/ubuntu-01-flat.vmdk 2017-01-29 10:52:03.000000000
./vmfs/volumes/424b26ec-4294ea41/ubuntu-02/ubuntu-02-flat.vmdk 2017-01-29 10:17:46.000000000
./vmfs/volumes/424b26ec-4294ea41/ubuntu03/ubuntu03-flat.vmdk 2017-01-29 11:04:18.000000000
./vmfs/volumes/424b26ec-4294ea41/ubuntu04/ubuntu04-flat.vmdk 2017-01-29 10:17:49.000000000
./vmfs/volumes/424b26ec-4294ea41/ubuntu05/ubuntu05-flat.vmdk 2017-01-29 10:52:03.000000000

An alternative method, is using RVTools which can identify orphoned virtual machines and hard disks (categorised as ‘zombie’) from the inventory scan of a vCenter Server System or ESXi Host System. On completion of the inventory scan, the vHealth tab will display any virtual machines and hard disks identified and may be exported for reference.

vCenter Server 5.5 Update 3a: Update fails with Warning 32014.

Recently during an upgrade of a an instance of a vCenter Server System to 5.5 Update 3a the installation returned an error with the following message:

Warning 32014. A utility for phone home data collector couldn’t be executed successfully Please see its log file (with name PhoneHome) and vminst.log in system temporary folder for more details.

Upon restarting the ‘VMware VirtualCenter Server’ service the following error message is returned:

Error 1053: The service did not respond to the start or control request in a timely fashion.”

This is a known issue affecting vCenter 5.5 where the installer failed to update the ‘deployPkg.dll’ file during the upgrade process, to which currently there is no resolution.

To workaround the issue we have two options. Firstly, we can rollback the instance of the vCenter Server System prior to the upgrade and remove the file from patch cache which by default is located at ‘C:\Windows\Installer\$PatchCache$\Managed\05550F1E83248734780F0115742A159D\5.5.0’ and perform the upgrade.

Alternatively, you can perform the following to provide resolution to the issue.

1) Browse to ‘C:\Program Files\VMware\Infrastructure\VirtualCenter Server\’ and remove the file ‘deployPkg.dll’.

2) Download the file ‘2134141_deployPkg.zip‘ from the official VMware site and extract to the location ‘C:\Program Files\VMware\Infrastructure\VirtualCenter Server\’.

3) Start both the ‘VMware VirtualCenter Server Service’ and ‘VMware VirtualCenter Management Webservices’ services.

4) Uninstall the Profile-Driven Storage service by running the following command with elevated privelages.

msiexec.exe /x {7BC9E9D9-3DF6-4040-B4A1-B6A3A8AE75BA} SKIPVCCHECK=1 SUPPRESS_CONFIRM_UNINSTALL="1" /qr

5) Install the vCenter Server 5.5 Update 3a version of the Profile-Driven Storage¬†by running the following command with elevated privelages. In the below example, I am using ‘vcenter.dean.local’ as the FQDN of my vCenter Server System and the installation media is mounted on D:\.

msiexec.exe /L*V "%temp%\sps_vminst.log" /I "D:\vCenter-Server\Profile-Driven Storage\VMware vSphere Profile-Driven Storage.msi" INSTALLDIR="C:\Program Files\VMware\Infrastructure\" COMPUTER_FQDN=vcenter.dean.local TOMCAT_MAX_MEMORY_OPTION="S" VC_KEYSTORE_TYPE=PKCS12 VC_KEYSTORE_PASSWORD=testpassword VC_SSL_DIR="C:\ProgramData\VMware\VMware VirtualCenter\SSL\" VC_SPS_EXTENSION_DIR="C:\Program Files\VMware\Infrastructure\VirtualCenter Server\extensions\com.vmware.vim.sps\" IS_URL="https://vcenter.domain.local:10443" ARPSYSTEMCOMPONENT=1 SKIPVCCHECK=1 /qr

Now confirm that all your services are running and you are able to access the vCenter Server System.

As this is a known issue, you can prevent the symptom of this prior to performing an upgrade by removing the file from the patch cache prior to running your vCenter Server System upgrade.

Consolidating unmounted and open snapshots after virtual machine backup

At times, one or more disks from a virtual machine will remain mounted and snapshots will remain following the backup of a virtual machine. The cause of this is generally due to their not being enough time to unmount the disks and consolidate the snapshots when the job completes.

This would require user interaction to firstly unmount the virtual machine disks and then perform a consolidation of the virtual machine virtual disks. As we can agree, this is a manual task that is perfect to be automated.

The below solution has a number of caveats, the backup solution/appliance is a virtual machine and the virtual machine contains no hard disks that are of the disk type ‘Independent – Nonpersistent’.

The script in its entirety can be downloaded from here. However, the below will describe the logic to remove the virtual machine hard disks and consolidate the virtual machines.

Once we have established a connection to the vCenter Server and specified the virtual machine running the backup appliance/solution we will retrieve a collection of virtual machines where the ‘RunTime.ConsolidatedNeeded’ value is equal to ‘True’.

$VMs = Get-View -ViewType VirtualMachine -Property Name, RunTime | Where-Object {$_.RunTime.ConsolidationNeeded -eq $True}

Then we will connect to the virtual machine hosting the backup appliance/solution and retrieve all the virtual machine hard disks where the persistence value is equal to ‘IndependentNonPersistent’.

$HardDisks = Get-VM $ComputerName | Get-HardDisk | Where-Object {$_.Persistence -eq 'IndependentNonPersistent'} 

We will then invoke the Remove-HardDisk cmdlet to remove the filtered virtual machine hard disks. In this instance we are only removing the hard disks and not deleting them permanently.

Get-HardDisk -VM $ComputerName -Name $HardDisks.Name | Remove-HardDisk -Confirm:$False

Once the virtual machine hard disks have been removed, we will now consolidate the virtual machine disks on each of the virtual machines returned in the collection.

(Get-VM $VM.Name).ExtensionData.ConsolidateVMDisks() | Out-Null

The above script will provide console output to report the status of the invocation and can be triggered as a scheduled task returning an exit return code dependant on the status of the invocation.

PowerCLI: Retrieving ESXi Host System Management Network IP Address

I was recently looking at retrieving the allocated management network IP address for a large number of ESXi host systems and therefore decided to use the ‘Get-VMHostNetworkAdapter’ cmdlet ¬†to which I would filter based on the device name value retrieved from the ESXi host system. In its simpliest¬†form the cmdlet can be invoked as below:

Get-VMHostNetworkAdapter -VMHost esxi1.dean.local

In this instance I wanted to retrieve only the allocated IP address to the device name vmk0 (management network) for all ESXi host systems in a datacenter, which could be achieved by invoking the cmdlet with the filter to retrieve only the host network adapter where the device name was equal to ‘vmk0’

Get-Datacenter Edinburgh | Get-VMHost | Select-Object @{N="Name";E={$_.Name}},@{N="Management Network";E={(Get-VMHostNetworkAdapter -VMHost $_.Name  | Where-Object {$_.Name -eq "vmk0"}).IP }}

As the above retrieved both the name of the ESXi host system and the allocated IP address of the management network this provided me with the information required. However, in the future would I need to repeat this task with the same or different requirements. Therefore I decided to create a function which would retrieve the management network IP address for a filtered list of ESXi host systems to which this could be targeted at a vCenter Server System, Datacentre, Cluster or a particular list of ESXi host systems., as below.

The function will establish a connection to the vCenter Server System retrieve a collection of ESXi host systems as specified or retrieve a collection of all ESXi host systems managed by that vCenter Server and return the host name and management network IP address.

Function Get-VMHostManagementNetwork { 

Param ([Parameter(Mandatory=$true)][String[]] $VIServer,[String[]] $Datacenter,[String[]] $Cluster,[String[]] $VMHost) 

If (-not (Get-PSSnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue)) {Add-PSSnapin VMware.VimAutomation.Core | Out-Null}

Connect-VIServer $VIServer | Out-Null 

If ($Datacenter){$VMHosts = Get-Datacenter $Datacenter | Get-VMHost} 
ElseIf ($Cluster) {$VMHosts = Get-Cluster $Cluster | Get-VMHost} 
ElseIf ($VMHost) {$VMHosts = Get-VMHost $VMHost} 
Else {$VMHosts = Get-VMHost}

$VMHosts | Select-Object @{N="Name";E={$_.Name}},@{N="Management Network";E={(Get-VMHostNetworkAdapter -VMHost $_.Name  | Where-Object {$_.Name -eq "vmk0"}).IP }}
} 

The PowerShell function can be downloaded from here.

Installing open-vm-tools deployPkg plug-in for Ubuntu Guest Operating Systems

If you are installing the open source implementation of VMware Tools ‘open-vm-tools’ into your guest operating system and require to use the virtual machine as a template or leverage Site Recovery Manager to customize virtual machines after failover, then there is a requirement to install the ‘deployPkg Tools’ plug-in.

The below details installing the plug-in on an Ubuntu operating system, however steps for other operating systems can be found here.

Firstly, we will need to obtain and import the VMware Packaging Public keys which can be downloaded from here and the files are required to be saved into a directory on the guest operating system.  For each key downloaded, we will import the key on successful completion you should receive the below notification:

sudo apt-key add /tmp/keys/VMWARE-PACKAGING-GPG-DSA-KEY.pub
OK
sudo apt-key add /tmp/keys/VMWARE-PACKAGING-GPG-RSA-KEY.pub
OK

We will now create the file ‘/etc/apt/sources.list.d/vmware-tools.list’¬†to add the below package repository and update the package index.

deb http://packages.vmware.com/packages/ubuntu precise main
sudo apt-get update

Once the package index has been updated we can invoke the following to install the ‘deployPKG’ plug-in. Once installed you should be able to customize your template or virtual machine for failover using Site Recovery Manager.

sudo apt-get install open-vm-tools-deploypkg

vSphere Web Client – Use Windows Session Authentication not available in Google Chrome

Following a recent update to the Google Chrome Browser, in my case to version¬†42.0.2311.90 I was unable to select the ‘Use Windows Session Authentication’ option when authenticating the vSphere Web Client.

In order to resolve this issue I was required to enable the use of  NPAPI plug-ins to support has been disabled by default in versions 42.0 and above, as per https://www.chromium.org/developers/npapi-deprecation.

To workaround the issue you can perform the following:

1) From the address bar, browse to ‘chrome://flags/#enable-npap’.

2) Search for ‘Enable NPAPI Mac, Windows’ and select ‘Enable’.

3) Relaunch the browser.

It is useful to note that from versions 45 and above the above workaround will not be available, the current version of the vSphere Web Client that was impacted in this instance was Version 5.5.0 Build 2026576.

In September 2015 (Chrome 45) we will remove the override and NPAPI support will be permanently removed from Chrome. Installed extensions that require NPAPI plugins will no longer be able to load those plugins.

 

Configuring vRealize Orchestrator and WinRM Host for Kerberos Authentication

In order to leverage the vRealize Orchestrator PowerShell plug-in to enable interaction between the vRealize Orchestrator appliance and Windows Powershell there is a requirement to add a PowerShell Host to your vRealize Orchestrator configuration, this will only for workflows to interact with Windows Powershell and optionally PowerCLI and the vCenter Server system.

 

4-PSHost

 

The Poweshell plug-in supports communication using the WinRM and SSH protocols, in this article we will concentrate on WinRM as the communication protocol over HTTPS using Kerberos authentication. We can enable the WinRM host to support Kerberos authentication, by invoking the following from a command prompt with elevated privileges:

winrm s winrm/config/service/auth @{Kerberos="true"}

In order to communicate with WinRM over HTTPS there is a requirement to create a server authentication certificate, in this example we will create a self signed certificate using the New-SelfSignedCertificate cmdlet and store the newly created certificate in the My certificate store for the local machine.

New-SelfSignedCertificate -DnsName dean1.dean.local -CertStoreLocation Cert:\LocalMachine\My

On generation of the certificate information, the thumbprint and the subject will be streamed to the console session to which we will make a note of the certificate thumbprint for when we create the HTTPS listener configuration.

Thumbprint                                 Subject
----------                                 -------
C9C982AF1CFDC65CCDD7A30AC661F2D9CDC3F127   CN=dean1.dean.local

From the command prompt with elevated privileges,¬†we will create an HTTPS listener for the WinRM service, to which we will specify the FDQN of the WinRM host, the certificate thumbprint for the generated self signed certificate and specify the TCP service port as ‘5986’.

winrm create winrm/config/listener?Address=*+Transport=HTTPS '@{Hostname="dean1.dean.local";CertificateThumbprint="C9C982AF1CFDC65CCDD7A30AC661F2D9CDC3F127";port="5986"}'

Following successful creation of the HTTPS listener, you should receive output similar to the below:

ResourceCreated
 Address = http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous
 ReferenceParameters
 ResourceURI = http://schemas.microsoft.com/wbem/wsman/1/config/listener
 SelectorSet
 Selector: Address = *, Transport = HTTPS

In order to allow for inbound connectivity to the remote WinRM host, where applicable we will be required to enable the Windows Remote Management (HTTP-In) rule for communications on TCP service port 5986, we can achieve this by invoking the ‘New-NetFirewallRule’ cmdlet.

New-NetFirewallRule -DisplayName "Windows Remote Management (HTTPS-In)" -Name "Windows Remote Management (HTTPS-In)" -Profile Any -LocalPort 5986 -Protocol TCP

Once we have configured the WinRM Host to use Kerberos, there is a final requirement to configure the vRealize Orchestrator appliance for¬†Kerberos¬†authentication. In order to do so, you will be required to connect using an SSH client such as PuTTY and edit/create the file ‘/usr/java/jre-vmware/lib/security/krb5.conf‘. In the below example replace the domain name DEAN.LOCAL with your domain and adhere to the case sensitivity and replace ‘dc1.dean.local’ with the FQDN of your¬†key distribution center for the provided¬†Kerberos realm.

[libdefaults]
 default_realm = DEAN.LOCAL
 udp_preference_limit = 1
[realms]
 DEAN.LOCAL = {
 kdc = dc1.dean.local
 default_domain = dean.local
 }
[domain_realms]
.dean.local=DEAN.LOCAL
dean.local=DEAN.LOCAL

Once the above file has been written, as we are using the virtual appliance for vRealize Orchestrator we will need to modify the permissions to the file so that it may be read and then restart the vRealize Orchestrator Server service to apply the configuration change.

chmod 644 /usr/java/jre-vmware/lib/security/krb5.conf

Once we have configured the WinRM host and vRealize Orchestrator applicance we now should be able to¬†invoke the ‘Add PowerShell Host’ workflow succesfully (Library > PowerShell > Configuration > Add PowerShell Host).

1-PSHost

 

2-PSHost

3-PSHost