vCenter Server 5.5 Update 3a: Update fails with Warning 32014.

Recently during an upgrade of a an instance of a vCenter Server System to 5.5 Update 3a the installation returned an error with the following message:

Warning 32014. A utility for phone home data collector couldn’t be executed successfully Please see its log file (with name PhoneHome) and vminst.log in system temporary folder for more details.

Upon restarting the ‘VMware VirtualCenter Server’ service the following error message is returned:

Error 1053: The service did not respond to the start or control request in a timely fashion.”

This is a known issue affecting vCenter 5.5 where the installer failed to update the ‘deployPkg.dll’ file during the upgrade process, to which currently there is no resolution.

To workaround the issue we have two options. Firstly, we can rollback the instance of the vCenter Server System prior to the upgrade and remove the file from patch cache which by default is located at ‘C:\Windows\Installer\$PatchCache$\Managed\05550F1E83248734780F0115742A159D\5.5.0’ and perform the upgrade.

Alternatively, you can perform the following to provide resolution to the issue.

1) Browse to ‘C:\Program Files\VMware\Infrastructure\VirtualCenter Server\’ and remove the file ‘deployPkg.dll’.

2) Download the file ‘‘ from the official VMware site and extract to the location ‘C:\Program Files\VMware\Infrastructure\VirtualCenter Server\’.

3) Start both the ‘VMware VirtualCenter Server Service’ and ‘VMware VirtualCenter Management Webservices’ services.

4) Uninstall the Profile-Driven Storage service by running the following command with elevated privelages.

msiexec.exe /x {7BC9E9D9-3DF6-4040-B4A1-B6A3A8AE75BA} SKIPVCCHECK=1 SUPPRESS_CONFIRM_UNINSTALL="1" /qr

5) Install the vCenter Server 5.5 Update 3a version of the Profile-Driven Storage¬†by running the following command with elevated privelages. In the below example, I am using ‘vcenter.dean.local’ as the FQDN of my vCenter Server System and the installation media is mounted on D:\.

msiexec.exe /L*V "%temp%\sps_vminst.log" /I "D:\vCenter-Server\Profile-Driven Storage\VMware vSphere Profile-Driven Storage.msi" INSTALLDIR="C:\Program Files\VMware\Infrastructure\" COMPUTER_FQDN=vcenter.dean.local TOMCAT_MAX_MEMORY_OPTION="S" VC_KEYSTORE_TYPE=PKCS12 VC_KEYSTORE_PASSWORD=testpassword VC_SSL_DIR="C:\ProgramData\VMware\VMware VirtualCenter\SSL\" VC_SPS_EXTENSION_DIR="C:\Program Files\VMware\Infrastructure\VirtualCenter Server\extensions\com.vmware.vim.sps\" IS_URL="https://vcenter.domain.local:10443" ARPSYSTEMCOMPONENT=1 SKIPVCCHECK=1 /qr

Now confirm that all your services are running and you are able to access the vCenter Server System.

As this is a known issue, you can prevent the symptom of this prior to performing an upgrade by removing the file from the patch cache prior to running your vCenter Server System upgrade.

Consolidating unmounted and open snapshots after virtual machine backup

At times, one or more disks from a virtual machine will remain mounted and snapshots will remain following the backup of a virtual machine. The cause of this is generally due to their not being enough time to unmount the disks and consolidate the snapshots when the job completes.

This would require user interaction to firstly unmount the virtual machine disks and then perform a consolidation of the virtual machine virtual disks. As we can agree, this is a manual task that is perfect to be automated.

The below solution has a number of caveats, the backup solution/appliance is a virtual machine and the virtual machine contains no hard disks that are of the disk type ‘Independent – Nonpersistent’.

The script in its entirety can be downloaded from here. However, the below will describe the logic to remove the virtual machine hard disks and consolidate the virtual machines.

Once we have established a connection to the vCenter Server and specified the virtual machine running the backup appliance/solution we will retrieve a collection of virtual machines where the ‘RunTime.ConsolidatedNeeded’ value is equal to ‘True’.

$VMs = Get-View -ViewType VirtualMachine -Property Name, RunTime | Where-Object {$_.RunTime.ConsolidationNeeded -eq $True}

Then we will connect to the virtual machine hosting the backup appliance/solution and retrieve all the virtual machine hard disks where the persistence value is equal to ‘IndependentNonPersistent’.

$HardDisks = Get-VM $ComputerName | Get-HardDisk | Where-Object {$_.Persistence -eq 'IndependentNonPersistent'} 

We will then invoke the Remove-HardDisk cmdlet to remove the filtered virtual machine hard disks. In this instance we are only removing the hard disks and not deleting them permanently.

Get-HardDisk -VM $ComputerName -Name $HardDisks.Name | Remove-HardDisk -Confirm:$False

Once the virtual machine hard disks have been removed, we will now consolidate the virtual machine disks on each of the virtual machines returned in the collection.

(Get-VM $VM.Name).ExtensionData.ConsolidateVMDisks() | Out-Null

The above script will provide console output to report the status of the invocation and can be triggered as a scheduled task returning an exit return code dependant on the status of the invocation.

PowerCLI: Retrieving ESXi Host System Management Network IP Address

I was recently looking at retrieving the allocated management network IP address for a large number of ESXi host systems and therefore decided to use the ‘Get-VMHostNetworkAdapter’ cmdlet ¬†to which I would filter based on the device name value retrieved from the ESXi host system. In its simpliest¬†form the cmdlet can be invoked as below:

Get-VMHostNetworkAdapter -VMHost esxi1.dean.local

In this instance I wanted to retrieve only the allocated IP address to the device name vmk0 (management network) for all ESXi host systems in a datacenter, which could be achieved by invoking the cmdlet with the filter to retrieve only the host network adapter where the device name was equal to ‘vmk0’

Get-Datacenter Edinburgh | Get-VMHost | Select-Object @{N="Name";E={$_.Name}},@{N="Management Network";E={(Get-VMHostNetworkAdapter -VMHost $_.Name  | Where-Object {$_.Name -eq "vmk0"}).IP }}

As the above retrieved both the name of the ESXi host system and the allocated IP address of the management network this provided me with the information required. However, in the future would I need to repeat this task with the same or different requirements. Therefore I decided to create a function which would retrieve the management network IP address for a filtered list of ESXi host systems to which this could be targeted at a vCenter Server System, Datacentre, Cluster or a particular list of ESXi host systems., as below.

The function will establish a connection to the vCenter Server System retrieve a collection of ESXi host systems as specified or retrieve a collection of all ESXi host systems managed by that vCenter Server and return the host name and management network IP address.

Function Get-VMHostManagementNetwork { 

Param ([Parameter(Mandatory=$true)][String[]] $VIServer,[String[]] $Datacenter,[String[]] $Cluster,[String[]] $VMHost) 

If (-not (Get-PSSnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue)) {Add-PSSnapin VMware.VimAutomation.Core | Out-Null}

Connect-VIServer $VIServer | Out-Null 

If ($Datacenter){$VMHosts = Get-Datacenter $Datacenter | Get-VMHost} 
ElseIf ($Cluster) {$VMHosts = Get-Cluster $Cluster | Get-VMHost} 
ElseIf ($VMHost) {$VMHosts = Get-VMHost $VMHost} 
Else {$VMHosts = Get-VMHost}

$VMHosts | Select-Object @{N="Name";E={$_.Name}},@{N="Management Network";E={(Get-VMHostNetworkAdapter -VMHost $_.Name  | Where-Object {$_.Name -eq "vmk0"}).IP }}

The PowerShell function can be downloaded from here.

Installing open-vm-tools deployPkg plug-in for Ubuntu Guest Operating Systems

If you are installing the open source implementation of VMware Tools ‘open-vm-tools’ into your guest operating system and require to use the virtual machine as a template or leverage Site Recovery Manager to customize virtual machines after failover, then there is a requirement to install the ‘deployPkg Tools’ plug-in.

The below details installing the plug-in on an Ubuntu operating system, however steps for other operating systems can be found here.

Firstly, we will need to obtain and import the VMware Packaging Public keys which can be downloaded from here and the files are required to be saved into a directory on the guest operating system.  For each key downloaded, we will import the key on successful completion you should receive the below notification:

sudo apt-key add /tmp/keys/
sudo apt-key add /tmp/keys/

We will now create the file ‘/etc/apt/sources.list.d/vmware-tools.list’¬†to add the below package repository and update the package index.

deb precise main
sudo apt-get update

Once the package index has been updated we can invoke the following to install the ‘deployPKG’ plug-in. Once installed you should be able to customize your template or virtual machine for failover using Site Recovery Manager.

sudo apt-get install open-vm-tools-deploypkg

vSphere Web Client – Use Windows Session Authentication not available in Google Chrome

Following a recent update to the Google Chrome Browser, in my case to version¬†42.0.2311.90 I was unable to select the ‘Use Windows Session Authentication’ option when authenticating the vSphere Web Client.

In order to resolve this issue I was required to enable the use of  NPAPI plug-ins to support has been disabled by default in versions 42.0 and above, as per

To workaround the issue you can perform the following:

1) From the address bar, browse to ‘chrome://flags/#enable-npap’.

2) Search for ‘Enable NPAPI Mac, Windows’ and select ‘Enable’.

3) Relaunch the browser.

It is useful to note that from versions 45 and above the above workaround will not be available, the current version of the vSphere Web Client that was impacted in this instance was Version 5.5.0 Build 2026576.

In September 2015 (Chrome 45) we will remove the override and NPAPI support will be permanently removed from Chrome. Installed extensions that require NPAPI plugins will no longer be able to load those plugins.


VCP5-DCV Delta Recertification Exam Experience

Well my expiry date for my current¬†VMware Certified Professional (VCP) certification was due to expire on March 10 2015, and in all honesty for one reason or another I have not been proactive in renewing the certification, ideally I wanted to¬†progress on the certification roadmap with the ‘VMware Certified Advanced Professional 5 ‚Äď Data Center Administration’ accreditation. However, through lack of study time and then discovering that the nearest available test centre for my deadline was 219 miles away I decided to renew my existing certification through the Delta Exam offered.

So, what is the focus of the exam from the outline?

This exam is focused on changes and updates from vSphere 5.1 to vSphere 5.5, and is available to existing VCP5 holders who took the VCP510 exam and need to recertify on or before March 15, 1015.

Exam Preparation 

Always a good place to start is downloading the VCP550D Exam Blueprint  which at an initial glance appears to be quite heavy for an exam that appears to be focused on changes and updates available in vSphere 5.5, but the majority of topics you should be familiar with being a current holder of the VCP-DCV certification and administrating your vSphere environment in your day to day activities.

Firstly, In order to prepare for the new material between vSphere 5.0/5.1 and vSphere 5.5 the whitepaper¬†What‚Äôs New in¬†VMware vSphere¬†5.5¬†Platform¬†and a¬†free one hour online course is also provided by VMware to help you prepare for the exam ¬†VMware vSphere: What’s New Fundamentals of V5.5¬†are a good start on your road to preparing for the recertification.¬†There is also a number of training resources available on Pluralsight¬†to which a free 10 day trial is offered to which the only limitation during this period is a maximum viewing time of the 200 minutes for the training videos.

For those who have previously undertaken the VCP-DCV examination, it has always been helpful to gain an understanding of the configuration maximums and features available in the various vSphere editions, so I familiarised myself with those:

 vSphere 5.5 Configuration Maximums

VMware vSphere Pricing Whitepaper 

Now, I decided to familiarise myself with the new features and enhancements which further extend the core capabilities of the vSphere platform in VMware vSphere 5.5, to which these are listed below:

vSphere ESXi Hypervisor Enhancements 

  • Hot-Pluggable PCIe SSD Devices
  • Support for Reliable Memory Technology
  • Enhancements to CPU C-States

Virtual Machine Enhancements

  • Virtual Machine Compatibility with VMware ESXi 5.5
  • Expanded Virtual Graphics Support
  • Graphic Acceleration for Linux Guests

VMware vCenter Server Enhancements 

  • vCenter Single Sign-On
  • vSphere Web Client
  • vCenter Server Appliance
  • vSphere App HA
  • Architecture Overview
  • vSphere App HA Policies
  • Enabling Protection for an Application Service
  • vSphere HA and DRS Virtual Machine – Virtual Machine Affinity Rules
  • VMware vSphere Data Protection Enhancements
  • vSphere Big Data Extensions

vSphere Storage Enhancements 

  • Support for 62TB VMDK
  • MCSC Updates
  • 16GB E2E Support
  • PDL Autoremove
  • vSphere Replication Interopability
  • vSphere Replication MPIT Snapshot Retention
  • VAAI UNMAP Improvements
  • VMFS Heap Improvements
  • vSphere Flash Read Cache

vSphere Network Enhancements 

  • Link Aggregation Control Protocol (LACP) Enhancements
  • Traffic Filtering
  • Quality of Service Tagging
  • SR-IOV Enhancements
  • Enhanced Host-Level Packet Capture
  • 40GB NIC Support

From reading the exam blueprint, I made a note of objectives related to the exam to which I required to refresh my knowledge or to which I have not had much exposure to in the real world to which I would also concentrate on:

  • VMware Data Protection
  • VMware vCenter Operations Manager
  • VMware Virtual SAN

Registering for the Exam

The registration process is different from the usual process as this is available online, so there is no need to visit a test centre. Firstly, you need to request authorization for the exam from VMware, once you have received your Candidate ID you can the schedule the exam with Pearson Vue. This initial process took one business day to process and once notification was received I was able to complete the registration for the exam and once payment has been processed you can take the exam within an 48 hour window.

The Actual Exam Experience

Once I completed the registration and made the payment I decided to take the exam immediately, no time like the present! So we have 65 questions to answer in the allotted time frame once the initial questionnaire was completed. In terms of time management I completed the 65 questions with approximately 15 minutes to spare, this gave my time to review my answers in particular those that I had marked for review, at no time during the exam process did I feel I was pushed for time – so one less thing to worry about!

There was two things that struck me about the exam. Firstly, unlike previous VCP exams I have taken there was no questions relating to configuration maximums and secondly the content of the questions as described in the outline are not entirely focused on updates from vSphere 5.1 to 5.5. As per the NDA, I cannot go into the content of the exam questions but a number of questions were around the topics to which I concentrated on from the exam blueprint which came in useful.

So how did I get on (drum roll…)? Well after completing the online test, I received instantly that I had passed the exam with my score so that was a relief, so I am now certified until 27th February 2017. The next day I received a notification from VMware of my result and my transcript was updated accordingly.

It is also worth noting that the user experience of the online test was excellent, very responsive and no issues experienced at all.  I am sure I am not on my own when I recount of number of bad user experiences in testing centres over the years!

Looking Back

I would definitely say that in terms of exam preparation this was probably the most unprepared I had been but time for recertification was of the essence, and I have been using VMware products for a number of years I was confident (rightly or wrongly!) that I would have a good chance of passing the exam. In my opinon, hands on experience is the most important aspect of studying for an exam so I would encourage where possible to create a lab environment in particular for features you may not use on a day to day basis as part of your job role. Alternatively, if you do not have a lab environment have a look at the VMware Hands On Labs (HOL) this was particular useful for VMWare Virtual SAN. I find just reading a text book or a document is not enough, it is a lot more beneficial to gain experience with the product features.

Also, there is some great resources at CloudCred in particular practice questions related to the exam provided by the community.

Creating VM Storage Policy returns PbmExtendedElementDescription required property summary not set

Following VM Storage Polices being enabled for a compute resource object in vSphere 5.5 and the attempt to create the first VM Storage Policy the following system error message was received:

reason = PbmExtendedElementDescription required property summary not set
inherited from com.vmware.vim.binding.vmodl.fault.SystemError: Failed to serialize response

It would appear that the above system error is generated when Tags are created using PowerCLI to which no description has been explicitly specified, as described here.

In order to resolve the issue, I used the offender PowerCLI to create my description value for the Tags, where the value would be <Category>\<Name>, as below:

$Tags = Get-Tag 
ForEach ($Tag in $Tags)
    $Tag | Set-Tag -Description $Tag 

Once the Tag description value had been populated I was able to create VM Storage policies with no issues.