vCenter Server 5.5 Update 3a: Update fails with Warning 32014.

Recently during an upgrade of a an instance of a vCenter Server System to 5.5 Update 3a the installation returned an error with the following message:

Warning 32014. A utility for phone home data collector couldn’t be executed successfully Please see its log file (with name PhoneHome) and vminst.log in system temporary folder for more details.

Upon restarting the ‘VMware VirtualCenter Server’ service the following error message is returned:

Error 1053: The service did not respond to the start or control request in a timely fashion.”

This is a known issue affecting vCenter 5.5 where the installer failed to update the ‘deployPkg.dll’ file during the upgrade process, to which currently there is no resolution.

To workaround the issue we have two options. Firstly, we can rollback the instance of the vCenter Server System prior to the upgrade and remove the file from patch cache which by default is located at ‘C:\Windows\Installer\$PatchCache$\Managed\05550F1E83248734780F0115742A159D\5.5.0’ and perform the upgrade.

Alternatively, you can perform the following to provide resolution to the issue.

1) Browse to ‘C:\Program Files\VMware\Infrastructure\VirtualCenter Server\’ and remove the file ‘deployPkg.dll’.

2) Download the file ‘2134141_deployPkg.zip‘ from the official VMware site and extract to the location ‘C:\Program Files\VMware\Infrastructure\VirtualCenter Server\’.

3) Start both the ‘VMware VirtualCenter Server Service’ and ‘VMware VirtualCenter Management Webservices’ services.

4) Uninstall the Profile-Driven Storage service by running the following command with elevated privelages.

msiexec.exe /x {7BC9E9D9-3DF6-4040-B4A1-B6A3A8AE75BA} SKIPVCCHECK=1 SUPPRESS_CONFIRM_UNINSTALL="1" /qr

5) Install the vCenter Server 5.5 Update 3a version of the Profile-Driven Storage¬†by running the following command with elevated privelages. In the below example, I am using ‘vcenter.dean.local’ as the FQDN of my vCenter Server System and the installation media is mounted on D:\.

msiexec.exe /L*V "%temp%\sps_vminst.log" /I "D:\vCenter-Server\Profile-Driven Storage\VMware vSphere Profile-Driven Storage.msi" INSTALLDIR="C:\Program Files\VMware\Infrastructure\" COMPUTER_FQDN=vcenter.dean.local TOMCAT_MAX_MEMORY_OPTION="S" VC_KEYSTORE_TYPE=PKCS12 VC_KEYSTORE_PASSWORD=testpassword VC_SSL_DIR="C:\ProgramData\VMware\VMware VirtualCenter\SSL\" VC_SPS_EXTENSION_DIR="C:\Program Files\VMware\Infrastructure\VirtualCenter Server\extensions\com.vmware.vim.sps\" IS_URL="https://vcenter.domain.local:10443" ARPSYSTEMCOMPONENT=1 SKIPVCCHECK=1 /qr

Now confirm that all your services are running and you are able to access the vCenter Server System.

As this is a known issue, you can prevent the symptom of this prior to performing an upgrade by removing the file from the patch cache prior to running your vCenter Server System upgrade.

VMware: Disable Managed Object Browser (MOB)

The managed object browser provides a way to explore the object model used by vCenter to manage the vSphere environment; it enables configurations to be changed as well.

This interface is used primarily for debugging the vSphere SDK. This interface might potentially be used to perform malicious configuration changes or actions.

In order to disable the datastore browser you will need to edit the ‚Äėvpxd.cfg‚Äô file, to ensure the‚Äė¬†enableDebugBrowse‚Äô¬†is set to false, as below:

<vpxd>
     <enableDebugBrowse>false</enableDebugBrowse> 
</vpxd>

Once the above configuration change has been made, restart the ‚ÄėVMware VirtualCenter Server‚Äô service to apply the change. Once disabled the managed object browser¬†will no longer be available for diagnostics.

For disabling the managed object browser at a host level, see http://pubs.vmware.com/vsphere-55/index.jsp?topic=%2Fcom.vmware.vsphere.security.doc%2FGUID-0EF83EA7-277C-400B-B697-04BDC9173EA3.html.

Monitoring vCenter privelage reassignment with Nagios XI

During a restart of the ‘VMware VirtualCenter Server’ service, if a user or group assigned to the Administrator Role at the root folder level could not be verified during the restart the user privelages are revoked.

As part of security hardening on the vCenter server, I created a Nagios Remote Plugin Executor (NRPE) to search for the event created in the application log and create a service status. 

Firstly, we will only require to query the application log¬†after the¬†‘VMware VirtualCenter Server’ service has started, we can retrieve this information as a date format by using the Get-Process cmdlet to return the ‘StartTime’ value of the process ‘vxpd’.

$Start= (Get-Process vpxd).StartTime

Now that we have retrieved a date value to query the application log after, we will need to filter the application log further using the ‘Get-EventLog’ cmdlet to retrieve an¬†event, which is similar to the below:

Log Name: Application
Source: VMware VirtualCenter Server
Date: M/DD/YYYY H:MM:SS PM
Event ID: 1000
Task Category: None
Level: Warning
Keywords: Classic
User: N/A
Computer: [vCenter Server]
Description:  Removing permission for entity ""<group name>"", group ""DOMAIN\Account"", role -1.  Reason: User or group not found."

We will now create a filter to pass to the ‘Get-EventLog’ cmdlet to retrieve the any results like the above and store this is a variable so that we may use the results as a count. The below will filter for the Souce ‘VMware VirtualCenter Server’, the EntryType ¬†‘Warning’ and where the message text is like ‘Removing permission*User or group not found’.

The ‘ErrorAction’ preference is required as if zero counts of the below filter are returned, an error will be passed to the console output.

$Query = Get-EventLog -LogName Application -Source "VMware VirtualCenter Server" -EntryType "Warning" -After $Start-ErrorAction SilentlyContinue | Where-Object {$_.Message -like "Removing permission*User or group not found"}  

Conditional Logic will then be used to create a service status message based on the count of results returned in the above query. If zero results are returned the service status will be set to ‘OK’ with a status information stating that no instances of privelage reassignment since the process start time have been retrieved

If one or more results are returned, the service status will be set to ‘Critical’ with the status information message that a number of instances of privelage assignment since the process start time have been retrieved.

If ($Query.Count -eq "0") 
    { 
    "No instances of privelage reassignment since " + ($Start).ToString("dd/MM/yyyy HH:mm")
    $returncode="0"
    } 
ElseIf ($Query.Count -ge "1") 
    { 
    "" + $Query.Count + " instances of privelage reassignment since " + ($Start).ToString("dd/MM/yyyy HH:mm")
    $returncode = "2"
    }

The powershell session will now exit and return an exit code.

exit $returncode

Once¬†you have configured the external script to run within Nagios (http://wp.me/p15Mdc-eC), for a service status of ‘OK’ you should receive something similar to the below:

CountVMUPR

VMware: Disable Datastore Browser from vCenter

By default, you will be able to use your web browser to find and download any files by browsing datastores in the vSphere inventory.

In order to disable the datastore browser you will need to edit the ‘vpxd.cfg’ file, to ensure the ‘enableHttpDatastoreAccess’ is set to false, as below:

<vpxd>
     <enableHttpDatastoreAccess>false</enableHttpDatastoreAccess> 
</vpxd>

Once the above configuration change has been made, restart the ‘VMware VirtualCenter Server’ service to apply the change.

This will now disable the datastore browser when connecting to the VMware vCenter server, you will still be able to connect to the datastore browser on an ESX/ESXi host.

 

 

Now that the previous limitations have been removed for the number of Hosts and VMs supported by the vCenter Server Appliance in vSphere 5.5 allowing for increased scalability and additional functionality added,  what are you currently running on or planning to deploy?

vRanger backup jobs fail with host could not be identified

I recently was investigating an issue where backup jobs in vRanger failed for all VMs with the following error message:

An internal error occurred during execution, please contact Quest support if the error persists. Error Message: <VM>'s host could not be identified

This appears to be an inventory issue between vRanger and the vCenter server which was caused by the VMware VirtualCenter Server service not responding and requiring a restart prior to the jobs being invoked.

The above issue may be resolved by performing the following from the vRanger server:

1) Stop the vRanger API and FLR services.

2) Restart the vRanger service.

3) Start the vRanger API service.

4) Start the vRanger FLR service.

After performing the above I was able to successfully run a backup job which had previously failed.

Removing vCenter plug-in using Managed Object Browser

I recently installed a third party appliance within vCenter, upon removal the installed plug-in was still present in vCenter and no uninstall package was available.

The only option was then to connect to the Managed Object Browser for that vCenter instance, and remove the plug-in.

1) Browse to ‘https://<vcenter server or IP>/mob’.

2) Select the ‘Content’ hyperlink.

3) Select the ‘ExtensionManager’ hyperlink.

4) Select the plug-in hyperlink from the ‘ExtensionList’

5) Copy the ‘key name’ string and browse back to the ‘ExtensionManager’ page

6) Select ‘UnregisterExtension’ hyperlink.

7) Paste the ‘key name’ string from Step 5 into the ‘value’ text box for the extension key.

8) Select ‘Invoke Method’

9) Refresh the ‘ExtensionManager’ web page, the plug-in should now have been removed, log in to vCenter and the plug-on should also have been removed from the list of installed/available plug-ins.