VMware Update Manager ‚Äď Part Three: Creating and Modifying Baseline Groups

By default, two dynamic baseline are available – Critical and Non-Critical. The Critical baseline will contain all patches where the severity is ‘Critical’, whilst the Non-Critical baseline will contain all other patches. ¬†In order to create a new baseline from the vSphere Web Client browse to Home > Solutions and Applications > Update Manager > Baselines and Groups¬†and selecting Baselines and Groups.

When creating a new baseline we have an option of the type of baseline that we want to use:

  • Fixed – baseline will remain the same even if new patches are added to the repository
  • Dynamic – baselines are updates when new patches meeting the criteria are added to the repository.

When selecting a dynamic baseline, we need to determine a criteria for the patches to be included in this update. In the below screenshot I am configuring the criteria to be as follows:

  • Vendor: VMware, Inc
  • Product: embeddedEsx 5.5.0
  • Severity:¬†Critial
  • Category: Security

dynamicbaseline

 

The dynamic baseline will now list the patches that meet the criteria specified where additional options are provided to permanently exclude the patches from the baseline. In this example, we will exclude the Patch ID ‘ESXi550-20131201-SG’.

excludebaseline

 

We can also permanently include remaining patches from the repository in the dynamic baseline if required.

addbaseline

 

When creating a fixed baseline we are able to select the patches from the patch repository to which we require to include in the baseline.

fixedbasline

VMware Update Manager ‚Äď Part Two: Configuring VMware Update Manager (VUM)

In order to modify the Download Settings of the VMware Update Manager (VUM) from the vSphere Client browse to Home > Solutions and Applications > Update Manager > Download Settings.

If you have a proxy server installed on your local area network to act as an intermediary for requests from the VUM component, the download settings allow for a proxy server to be configured where the proxy server IP address, port and login credentials can be specified¬†to access the proxy server and then once configured the ‘Test Connection’ link can verify the configuration.

VUMproxy

VUM can be configured to use additional download sources where it may be required to download third party patches, extensions and upgrades. For Example, you may require to download extensions from the Dell online depot by specifyin a URL address for the third party download source, optionally add a description and validate the URL to confirm the source URL.

VUMDownloadSource

 

 

 

 

 

 

 

The download schedule can be configured by browsing to Home > Solutions and Applications > Update Manager > Download¬†Schedule¬†and selecting ‘Edit Download Schedule’. ¬†From the the schedule task options you may configure the time and frequency of the task and request email notification when new patches are downloaded.


VUMSchedule1

 

 

 

 

 

 

VUMSchedule2

 

 

 

 

By default, VUM using a direct connection to the internet to download new patches and virtual appliance upgrades as specified by the Download Schedule. However as discussed in http://wp.me/p15Mdc-wd , in the scenario where VUM server does not have access to the internet, there may be a requirement to install the VUM Download Service (UMDS) on a separate server to which the internet is accessible

To configure UMDS, you will required to use the command line utility ‘C:\Program Files (x86)\VMware\Infrastructure\Update Manager\vmware-umds.exe‘ on the UMDS server, to enable or disable the download of host system and virtual appliance updates. For example, in order to enable the download of host system updates but disable virtual appliance updates you will be required to invoke the following command on the UMDS server

vmware-umds -S -enable-host -disable-va

Now that we have configured UDMS to  download patches and upgrades for host systems only in the above example, we will need to perform the initial download by invoking the following command:

vmware-umds -D

Once the initial download has been performed we can export the data to a shared repository by invoking the following command:

vmware-umds -E --export-store D:\UDMS\repository

The VUM server can now be configured to use the shared repository by placing the export store on an internal web server or file share and ¬†by browsing to Home > Solutions and Applications > Update Manager > Download Settings¬†configure the download source to use a shared repository. In order to test the configuration you may select the ‘Download Now’ option.

An offline bundle may also be downloaded and the updates imported to the VUM server by selecting the ‘Import Patches’ hyperlink in the download sources option and specifying the local path to the zip file containing the patches you require to import.

Finally, we  can modify the location of the VUM repository which is configured during installation by modifying the vci-integrity.xml file to set the patchStore value.  I have previously written a post on this procedure here.

Monitor WUInstall status in Nagios XI

I recently wrote about checking the last success time of Windows Update and reporting this to Nagios (http://wp.me/p15Mdc-mj). Now what happens if you do not use Windows Update as your patch management solution. In my case I have been managing the installation of updates using WUInstall (http://www.wuinstall.com).

When my updates are installed the registry is not updated to reflect the last success time, therefore how can I monitor the last time updates were run and the status? The command line tool WUInstall provides the functionality to write the console output to a log file, in the below example each log file is written to a shared folder.

As per the previous example, I want to report if any updates have been installed in a particular time-span and if the process was successful.

In the case of the time-span this would be dependent on the host being monitored and therefore this period would be specified as a mandatory parameter when invoking the powershell script which was to be used as the check plugin within Nagios.

Param ([parameter(Mandatory = $true)][string] $Days)

Now I want to return the most recent log file for the host being monitored from the shared folder, where the log file name is that of the host and is contained in a parent folder based on the date the process was invoked. By using the Get-ChildItem cmdlet with the recurse option I am able to retrieve the most recent log file by sorting by the LastWriteTime in descending order and selecting the first file. I will return the full name of the file to a variable to pass to the Get-Content cmdlet for reading.

$LogFile = (Get-ChildItem "\\Server\Share\Logs" -Recurse | Where-Object {$_.Name -like "$env:computername*"} | Sort LastWriteTime -Descending | Select -First 1).FullName

In order to read the content of the log file we will need to encode this in Unicode format, and then search for the string ‘Overall’. In order to retrieve the overall result code we will return the next line from the content and store this as a variable.

$Log = Get-Content $LogFile -Encoding Unicode | Select-String "Overall" -Context 0,1 | % {$_.Context.PostContext}

In order to get the last run time the datetime function will be used to parse the date in the string to return the first ten¬†characters which contain the date and the modify the date string to be in the format ‘dd/MM/yyyy’.

$LastRun = [datetime]::ParseExact($Log.Substring(0,10) , "yyyy/MM/dd", $null)
$LastRun = $LastRun.ToString("dd/MM/yyyy")

Now that we have the last run time and the result code in the log variable we can use conditional logic to set the status of the service. Firstly we will check to see if the last run date is greater or equal to the timespan value specified in the mandatory days field by subtracting this from the current date and if the result code is like ‘Succeeded’ return the service status as ‘OK’

If ($LastRun -ge (get-date).AddDays(-$Days) -and $Log -like "*Succeeded*")
   { 
   $resultcode = "0"
   }

If the last run date is less than the time-span value but the result code is like ‘Succeeded’ we will return the service status as ‘OK’.

ElseIf ($LastRun -lt (get-date).AddDays(-$Days) -and $Log -like "*Succeeded*")
   { 
   $resultcode = "1"
   }

If the result code is not like ‘Succeeded’ we will return the service status as ‘Critical’.

ElseIf ($Log -notlike "*Succeeded*")
   { 
   $resultcode = "2"
   }

If we are unable to retrieve any information required in the conditional logic the service status will be returned as ‘Unknown’.

Else
   { 
   $resultcode = "3"
   }

Finally, we will set a status information meessage based on the result code where the substring function is invoked on the  log variable to remove the first twenty characters of the string which contains timestamp information and to include the last run time and terminate the powershell session to return the exit code for the service status.

$Log.Substring(20,$Log.Length-20) + " at " + $LastRun
exit $returncode

Once a check_nrpe command has been configured in Nagios (http://wp.me/p15Mdc-eC) and you begin to monitor your host(s), you should see a service check as below:

WindowsUpdateStatusNagios

You can also run the above in Windows Powershell, to return the status information as below:

PowershellStatusWindowsUpdate

The full powershell script can be downloaded from: https://app.box.com/s/cc2pkvx9a10g5ww6ha0i 

Patch Management for guest VMs with Windows Update Server and WuInstall on vSphere

I previously detailed steps to automate approved updates from a WSUS server with WUInstall (https://deangrant.wordpress.com/2013/09/24/patch-management-with-windows-update-server-and-wuinstall/) which focused on installing updates on Amazon Web Services EC2 instances.

I have recently modified the process to take into account VM guests in a vSphere environment ,using the same process of client side targeting and executing WuInstall on the remote machine, also with the following requirements:

  • Create a snapshot of the guest VM specified in the collection and add the name ‘Windows Update on ddMMyyyyHHmm’.
  • Once the snapshot is completed, invoke WuInstall to install the approved updates.

In order to target the guest VMs to which I wish to install the approved updates I am using custom attributes to determine the environment and if updates are enabled, as below:

Name Value
Environment DEV | TST | PRE | PRD
Windows Update Yes | No

As the script will target guest VMs in a number of environments, the script defines a mandatory parameter for an expected Environment value.

Param ([Parameter(Mandatory=$true)][string] $Environment)

In order to invoke the script there is a dependency on the Mware vSphere PowerCLI snap-in (https://www.vmware.com/support/developer/PowerCLI/) and for this to be imported into the current powershell session.

If (-not (Get-PSSnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue)) 
{
Add-PSSnapin VMware.VimAutomation.Core > $null
}

I will also generate a date string to be used in the snapshot name.

$Date = (get-date).toString('ddMMyyyyHHmm')

Now, we will establish a connection to the vCenter server.

Connect-VIServer <server name or ip address> > $null

We will now be required to filter the guest VMs where the custom attribute ‘Environment’ is equal to the mandatory parameter specified.

$VMs = Get-VM | Get-Annotation -CustomAttribute "Environment" | Where-Object {$_.Value -eq $Environment}

For each guest VM that is returned in the collection we will compare the ‘WindowsUpdate’ custom attribute and if this is equal to ‘Yes’ create a snapshot of the ¬†guest VM with the name ‘Windows Update on ddMMyyyyHHmm’ and invoke the¬†WuInstall to install the approved updates.

ForEach ($VM in $VMs)
{ 
$WindowsUpdate = Get-VM $VM.AnnotatedEntity.Name | Get-Annotation -CustomAttribute "WindowsUpdate" | Where-Object {$_.Value -eq "Yes"}
If ($WindowsUpdate.Value -eq "Yes")
{
New-Snapshot -VM $VM.AnnotatedEntity.Name -Name ("Windows Update on " + $Date)
$Hostname = $VM.AnnotatedEntity.Name
$Command =  "& 'C:\Program Files (x86)\SysinternalSuite\PsExec.exe' \\$Hostname -c -f -s \\<Server>\>Share>\Tools\WUInstall.exe /install /autoaccepteula /reboot_if_needed /logfile \\<Server>\<Share>\Logs\WUInstall_$Hostname.log"
Invoke-Expression $Command
}
}

Once completed the connection to the vCenter server will be terminated.

Disconnect-VIServer -Server <server name or ip address> -Confirm:$False

In order to invoke the script run the following:

./Invoke-WindowsUpdate.ps1 - Environment <Environment Attribute>

The script will require the user account invoking the script to have ‘Virtual Machine Power Users’ role, to which I cloned the built-in role and local administrator privelages on each guest VM to install the approved updates.

The full Windows Powershell script can be downloaded from the below link:

https://app.box.com/s/d7fp1qapfmp9un05f0x5

Patch Management with Windows Update Server and WuInstall

I was recently looking to schedule approved updates from Windows Update Server and schedule updates with finer granularity than those provided by the Windows Update Services client. I discovered a command line utility called WuInstall  (http://www.wuinstall.com/index.php/en) which allows for updates to be installed on demand.

As part of this solution, I still required updates to be approved by my Windows Update Server and to be automatically downloaded by the Windows Update Services client on a daily basis.

I created a group policy object and linked to the organisational unit containing the clients and set the following group policy object settings:

Setting State Options Description
Configure Automatic Updates Enabled 3 – Auto download and notify for install Specifies that the instance will download approved updates from the WSUS server, but will only notify for install.
Specify intranet Microsoft Update Service location Enabled http://<name of WSUS server> Specifies the WSUS server as the intranet server to host updates.
Allow non-administrators to receive update notifications Disabled N/A Specifies that only administrators receive update notifications.
Enable client-side targeting Enabled <name of target group> Specifies the target group for the instances to receive updates.

Once the approved updates have been downloaded, the command line utility WuInstall will manage the schedule and installation.

As previously mentioned, WuInstall is a command line tool that allows the installation of Windows Updates on demand and can use the internal WSUS server to discover approved updates.

In the case of a single server, the executable can be downloaded and run with a number of command line arguments, in my case the following are to be used:

WuInstall.exe /install /autoaccepteula /reboot_if_needed /logfile <path to log file>
Usage Description
/install Searches, downloads and installs available updates.
/autoaccepteula Automatically accepts EULA on every update.
/reboot_if_needed Only restarts the instance if needed.
/logfile Creates log file

However, I was required to run the following against a number of servers. The command utility WuInstall supports the use of PSExec to run agaisnt multiple servers (http://www.wuinstall.com/index.php/faq#psexec) and therefore this was the mechanism used to launch the executable from a central management server to invoke the command on each remote server, with the following command line arguments:

PSExec.exe -c -f -s \\Server\Share\WUInstall.exe /install /autoaccepteula /reboot_if_needed /logfile <path to log file>
Usage Description
-c Copies the specific program (WuInstall.exe) to the remote instance for execution.
-f On copying the specific program overwrite the file is this already exists on the remote system.
-s Run remote process in the System account.

Now a further requirement was introduced in that updates were required to be installed on all servers in a particular environment and prior to the updates being installed a snapshot of the root device performed (Amazon Web Services). Therefore all the above would be compiled into a Windows Powershell script and

Param ([string] $Environment)

As the script was to be run against a number of environments, , the script defined parameters for the Environment name, which were called with the Environment argument. The script would required importing the Powershell for Amazon Web Services ((http://aws.amazon.com/powershell/) snap-in to the current powershell session.

If (-not (Get-Module AWSPowershell -ErrorAction SilentlyContinue))
{ 
Import-Module "C:\Program Files(x86)\AWS Tools\Powershell\AWSPowershell\AWSPowershell.psd1" > $null
}

As previously mentioned I have a collection of AWS EC2 instances (not a particularly large environment) to which I want to install the updates agaisnt, therefore this was represented as a collection in a hashtable to contain the information I required the InstanceID, VolumeID of the root device (/dev/sda1) and server name, below is an example of what this would look like;

$Instances = New-Object System.Collection.Generic.List(System.Collections.Hashtable)
$Instances.Add(@{"instanceid"="i-xxxxxxx1";"volumeid"=vol-xxxxxxx1";"name"="xxxxxxxxDEMxxx"})
$Instances.Add(@{"instanceid"="i-xxxxxxx2";"volumeid"=vol-xxxxxxx2";"name"="xxxxxxxxPRExxx"})
$Instances.Add(@{"instanceid"="i-xxxxxxx1";"volumeid"=vol-xxxxxxx1";"name"="xxxxxxxxPRDxxx"})

Once the instances have been listed as a collection we are required to run a loop process to return each instance name and then filter the environment using the substring method as conditional logic, where the naming convention contains the environment name in the 8th, 9th and 10th character of the hostname.

ForEach ($Instance in $Instances)
{
$String = $Instance.Name.ToString().Substring(8,3)
If ($String -eq $Environment)

Once this instances have been returned for the environment, the following is performed against each instance returned in the filter:

  • Stop the EC2 Instance
  • Once stopped perform an EC2 snapshot of the VolumeID specified in the collection and add the description:¬†<Host Name>: WIndows Update on ddMMyyyy_HHmm
  • Once the EC2 snapshot is complete, start the EC2 instance.
  • Once the EC2 instance is running Invoke WuInstall to install the approved updates.
{ 
Stop-EC2Instance $Instance.instanceid 
Do { 
$EC2Instance = Get-EC2Instance -Instance $instance.instanceid  
$State = $EC2Instance.RunningInstance| ForEach-Object {$_.InstanceState.Name}
{
Until ($State -eq "stopped")
New-EC2Snapshot -VolumeID $instance.volumeid -Description ($instance.name + ": Windows Update on " + $Date)
Do { 
$SnapshotStatus = (Get-EC2Snapshot | Where-Object {$_.Description -eq ($instance.name + ": Windows Update on " + $Date)}).status
}
Until ($SnapshotStatus -eq "completed")
Start-EC2Instance $instance.instanceid
Do { 
$EC2Instance = Get-EC2Instance -Instance $instance.instanceid  
$State = $EC2Instance.RunningInstance| ForEach-Object {$_.InstanceState.Name}
}
Until ($State -eq "running")
Start-Sleep -Seconds 300 
$Hostname = $instance.Name
$Command =  "& 'C:\Program Files\SysinternalSuite\PsExec.exe' \\$Hostname -c -f -s \\Server\Share\WUInstall.exe /install /autoaccepteula /reboot_if_needed /logfile \\Server\Share\Logs\logfile.log"
Invoke-Expression $Command
}
}

A couple of issues I experienced with the script, was that when the EC2 instance was reported as running the instance would not be contactable as the status checks had not been completed, so a the script is therefore suspended for a period of five minutes as a workaround.

There is currently a known issue with Powershell for Amazon Web Services snap-in (3.0.512.0) where the Get-EC2InstanceStatus cmdlet fails to return stopped instances. This requires a workaround to be performed where the instance state is returned using the Get-EC2Instance cmdlet and returning the value of the RunningInstance.InstanceState.Name object.

I plan to update this script to remove the dependency on a collection as a hashtable and return this information using the Powershell for Amazon Web Services Tools snap-in. Also, I will hopefully address the issue of reporting the EC2 instance as being in a running status where the status checks have not completed and remove the need to suspend the script.

The script will require AWS credentials to run the various cmdlets above, in this process I store the credentials on the local computer where the scheduled task is invoked.

The full Windows Powershell script can be downloaded from the below link:

https://app.box.com/s/7t2b5zqgbp4kus34rtwf