Update: Patch Management for Guest VMS with Windows Update Server and WUInstall on vSphere

I previously described steps to automate approved updates from a WSUS server with WUInstall for virtual machines managed by VMware vCenter Server (https://deangrant.wordpress.com/2014/01/24/patch-management-for-guest-vms-with-windows-update-server-and-wuinstall-on-vsphere/).

I have recently updated this process to filter virtual machines objects by tagging objects in the new functionality in the vSphere Web Client. Also, the previous process had no concept of  task parallelism, which has now also been addressed.

The script requires two mandatory parameters to be specified, we need to specify the vCenter server to which we require to establish a connection and the the category tag that has been assigned to the virtual machine to specify the infrastructure environment.

Param ([Parameter(Mandatory=$true)][string] $vCenter, [Parameter(Mandatory=$true)][string] $Environment)

Now we will register the ‘VMware.VimAutomation.Core’ snap-in and establish the connection to the vCenter server.

If (-not (Get-PSSnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue)) 
    {
    Add-PSSnapin VMware.VimAutomation.Core | Out-Null 
    }
Connect-VIServer $vCenter

Once connected, we will be required to store the current session details of the established connection to pass as a variable later in the script block to create a snapshot for the virtual machine prior to invoking the WUInstall executable.

$Session = ($global:DefaultVIServer).SessionId 

In order to filter virtual machines, we will firstly build a collection of virtual machine objects by invoking the Get-View cmdlet to retrieve only the name property values.

$VMs = Get-View -ViewType VirtualMachine -Property Name

For the collection of virtual machine objects we will now perform an operation on each object to filter the virtual machine collection to retreive only virtual machines to which the Tag Assignment ‘Environment’ is equal to the mandatory parameter specified and ‘Updates Enabled’ is equal to ‘Yes’.

ForEach ($VM in $VMS) 
    { 
    If (((Get-TagAssignment -Entity $VM.Name -Category Environment).Tag).Name -eq $Environment -and ((Get-TagAssignment -Entity $VM.Name -Category "Updates Enabled").Tag).Name -eq "Yes")
        { 

Before we invoke the automated process on each virtual machine the Get-Date cmdlet will retrieve the DateTime object to the format ‘yyyy-MM-dd’, to append to the log file generated by the WUInstall executable.

$Date = (get-date).toString('dd-MM-yyyy-HHmm')

The WUInstall executable will be invoked remotely by leveraging the PSExec utility and therefore we will specify the arguments to the executable in a variable to passed in the script block. The WUInstall will be located on the virtual machine in the location C:\Program Files\WUInstall and will create a log file in the directory location.

$WUInstall = "\\$VM -s 'C:\Program Files\WUInstall\WUInstall.exe' /install /autoaccepteula /reboot_if_needed /logfile 'C:\Program Files\WUInstall\wuinstall_$Date.log'"

In order to leverage task parallelism the script block create will be invoked to run each job as a background job on the local computer using the Start-Job cmdlet. For each background job we will require to establish a connection to the vCenter server using the session ID rather than reconnecting.

One of the requirements it to create a virtual machine snapshot prior to the installation of the approved updates, where the name will be ‘Windows Update on yyyy-MM-dd’ and then we will invoke the WUInstall on the virtual machine using the PSExec utility.

    $ScriptBlock = [scriptblock]::create("C:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe -Command {
                Add-PSSnapin VMware.VimAutomation.Core | Out-Null 
                Connect-VIServer $vCenter -Session $Session 
                New-Snapshot -VM $VM -Name 'Windows Update on $Date'
                & 'C:\Program Files (x86)\SysinternalsSuite\PsExec.exe ' $WUInstall
                }")

Now we will invoke the background task to initiate the process.

Start-Job -ScriptBlock $ScriptBlock   
        }
    }

Depending on the number of virtual machines to which you are invoking the command you may wish to limit the number of background jobs that are invoked in parallel. The below will limit the number of backgorund jobs run to be 10 and once all background jobs have the state of completed deletes the backgound jobs invoked.

While((Get-Job -State 'Running').Count -ge "10")
    {
        Start-Sleep -Milliseconds 30
    }
    
While (Get-Job -State "Running") 
{ 
    Start-Sleep -Milliseconds 30
}

Remove-Job * -Confirm:$False

WSUS client failing to find updates with error code 80244019

I was recently looking into an error for specific WSUS clients which were unable to find updates on the WSUS server reporting the following error message:

Failed to find updates with error code 80244019

On investigating the log file (C:\Windows\WindowsUpdateLog.log) on one of the affected clients I could see that the send request to the WSUS server was failing  and that the SelfUpdate check failed to download the package information, as below:

Misc WARNING: WinHttp: SendRequestToServerForFileInformation failed with 0x80190194
Misc WARNING: WinHttp: ShouldFileBeDownloaded failed with 0x80190194
Misc WARNING: DownloadFileInternal failed for http://<server name>/selfupdate/wuident.cab: error 0x80190194
Setup WARNING: SelfUpdate check failed to download package information, error = 0x80244019
Setup FATAL: SelfUpdate check failed, err = 0x80244019

It would appear that the WSUS server had been upgraded and the original TCP port used for client connectivity was no longer in use. As the Windows Update configuration of the clients is managed via Group Policy this required the update location to be modified to include the updated URL following the TCP port change to 8053, as below:

Setting Path: Computer Configuration>Policies>Administrative Templates\Windows Components\Windows Update 
Setting: Specify intranet Microsoft update service location.
State: Enabled
Options: Set the intranet update service for detecting updates - http:<server name>:8530

Once the update to the configuration had been applied and the group policy setting refreshed, the client was able to download and install approved updates from the WSUS server.

Monitor WUInstall status in Nagios XI

I recently wrote about checking the last success time of Windows Update and reporting this to Nagios (http://wp.me/p15Mdc-mj). Now what happens if you do not use Windows Update as your patch management solution. In my case I have been managing the installation of updates using WUInstall (http://www.wuinstall.com).

When my updates are installed the registry is not updated to reflect the last success time, therefore how can I monitor the last time updates were run and the status? The command line tool WUInstall provides the functionality to write the console output to a log file, in the below example each log file is written to a shared folder.

As per the previous example, I want to report if any updates have been installed in a particular time-span and if the process was successful.

In the case of the time-span this would be dependent on the host being monitored and therefore this period would be specified as a mandatory parameter when invoking the powershell script which was to be used as the check plugin within Nagios.

Param ([parameter(Mandatory = $true)][string] $Days)

Now I want to return the most recent log file for the host being monitored from the shared folder, where the log file name is that of the host and is contained in a parent folder based on the date the process was invoked. By using the Get-ChildItem cmdlet with the recurse option I am able to retrieve the most recent log file by sorting by the LastWriteTime in descending order and selecting the first file. I will return the full name of the file to a variable to pass to the Get-Content cmdlet for reading.

$LogFile = (Get-ChildItem "\\Server\Share\Logs" -Recurse | Where-Object {$_.Name -like "$env:computername*"} | Sort LastWriteTime -Descending | Select -First 1).FullName

In order to read the content of the log file we will need to encode this in Unicode format, and then search for the string ‘Overall’. In order to retrieve the overall result code we will return the next line from the content and store this as a variable.

$Log = Get-Content $LogFile -Encoding Unicode | Select-String "Overall" -Context 0,1 | % {$_.Context.PostContext}

In order to get the last run time the datetime function will be used to parse the date in the string to return the first ten¬†characters which contain the date and the modify the date string to be in the format ‘dd/MM/yyyy’.

$LastRun = [datetime]::ParseExact($Log.Substring(0,10) , "yyyy/MM/dd", $null)
$LastRun = $LastRun.ToString("dd/MM/yyyy")

Now that we have the last run time and the result code in the log variable we can use conditional logic to set the status of the service. Firstly we will check to see if the last run date is greater or equal to the timespan value specified in the mandatory days field by subtracting this from the current date and if the result code is like ‘Succeeded’ return the service status as ‘OK’

If ($LastRun -ge (get-date).AddDays(-$Days) -and $Log -like "*Succeeded*")
   { 
   $resultcode = "0"
   }

If the last run date is less than the time-span value but the result code is like ‘Succeeded’ we will return the service status as ‘OK’.

ElseIf ($LastRun -lt (get-date).AddDays(-$Days) -and $Log -like "*Succeeded*")
   { 
   $resultcode = "1"
   }

If the result code is not like ‘Succeeded’ we will return the service status as ‘Critical’.

ElseIf ($Log -notlike "*Succeeded*")
   { 
   $resultcode = "2"
   }

If we are unable to retrieve any information required in the conditional logic the service status will be returned as ‘Unknown’.

Else
   { 
   $resultcode = "3"
   }

Finally, we will set a status information meessage based on the result code where the substring function is invoked on the  log variable to remove the first twenty characters of the string which contains timestamp information and to include the last run time and terminate the powershell session to return the exit code for the service status.

$Log.Substring(20,$Log.Length-20) + " at " + $LastRun
exit $returncode

Once a check_nrpe command has been configured in Nagios (http://wp.me/p15Mdc-eC) and you begin to monitor your host(s), you should see a service check as below:

WindowsUpdateStatusNagios

You can also run the above in Windows Powershell, to return the status information as below:

PowershellStatusWindowsUpdate

The full powershell script can be downloaded from: https://app.box.com/s/cc2pkvx9a10g5ww6ha0i 

Monitor Windows Update Last Success Time with Nagios XI

I was recently asked to provide a monitoring service for the last success success time of Windows Update and to provide status information within Nagios XI. Basically, the last success time was compared to be run a time span period of a number of days in the past to provide a service state.

In the case of the time span this was dependent on the host being monitored and therefore this period would be specified as a mandatory parameter when invoking the powershell script which was to be used as the check plugin within Nagios.

Param ([parameter(Mandatory = $true)][string] $Days)

We can read the last success time reported from Windows Update by querying the registry to retrieve the date and then format the date string to be in the format ¬†‘dd/MM/yyyy’.

$LastRun = (Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\Results\Install").LastSuccessTime
$LastRun = Get-Date $LastRun -Format 'dd/MM/yyyy'

We will now to generate a date in the past based on the value specified by the mandatory Days parameter using the subtraction method from the current date.

In order to generate the service status, we will use conditional logic to compare the current date to that specified in the timespan and to provide a return code and status information message. If the date values cannot be compared this will return an unknown service status.

If ([datetime]::parse($LastRun) -ge [datetime]::parse($TimeSpan))
   {
   $returncode = 0
   "Windows Update last run successfully on " + $LastRun
   } 
   ElseIf ([datetime]::parse($LastRun) -lt [datetime]::parse($TimeSpan))
   {
   $returncode = 1
   "Windows Update last run successfully on " + $LastRun
   }
Else 
   { 
   $returncode = 3 
   }

Finally, we will exit the powershell session by providing the return code.

exit $returncode

Patch Management for guest VMs with Windows Update Server and WuInstall on vSphere

I previously detailed steps to automate approved updates from a WSUS server with WUInstall (https://deangrant.wordpress.com/2013/09/24/patch-management-with-windows-update-server-and-wuinstall/) which focused on installing updates on Amazon Web Services EC2 instances.

I have recently modified the process to take into account VM guests in a vSphere environment ,using the same process of client side targeting and executing WuInstall on the remote machine, also with the following requirements:

  • Create a snapshot of the guest VM specified in the collection and add the name ‘Windows Update on ddMMyyyyHHmm’.
  • Once the snapshot is completed, invoke WuInstall to install the approved updates.

In order to target the guest VMs to which I wish to install the approved updates I am using custom attributes to determine the environment and if updates are enabled, as below:

Name Value
Environment DEV | TST | PRE | PRD
Windows Update Yes | No

As the script will target guest VMs in a number of environments, the script defines a mandatory parameter for an expected Environment value.

Param ([Parameter(Mandatory=$true)][string] $Environment)

In order to invoke the script there is a dependency on the Mware vSphere PowerCLI snap-in (https://www.vmware.com/support/developer/PowerCLI/) and for this to be imported into the current powershell session.

If (-not (Get-PSSnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue)) 
{
Add-PSSnapin VMware.VimAutomation.Core > $null
}

I will also generate a date string to be used in the snapshot name.

$Date = (get-date).toString('ddMMyyyyHHmm')

Now, we will establish a connection to the vCenter server.

Connect-VIServer <server name or ip address> > $null

We will now be required to filter the guest VMs where the custom attribute ‘Environment’ is equal to the mandatory parameter specified.

$VMs = Get-VM | Get-Annotation -CustomAttribute "Environment" | Where-Object {$_.Value -eq $Environment}

For each guest VM that is returned in the collection we will compare the ‘WindowsUpdate’ custom attribute and if this is equal to ‘Yes’ create a snapshot of the ¬†guest VM with the name ‘Windows Update on ddMMyyyyHHmm’ and invoke the¬†WuInstall to install the approved updates.

ForEach ($VM in $VMs)
{ 
$WindowsUpdate = Get-VM $VM.AnnotatedEntity.Name | Get-Annotation -CustomAttribute "WindowsUpdate" | Where-Object {$_.Value -eq "Yes"}
If ($WindowsUpdate.Value -eq "Yes")
{
New-Snapshot -VM $VM.AnnotatedEntity.Name -Name ("Windows Update on " + $Date)
$Hostname = $VM.AnnotatedEntity.Name
$Command =  "& 'C:\Program Files (x86)\SysinternalSuite\PsExec.exe' \\$Hostname -c -f -s \\<Server>\>Share>\Tools\WUInstall.exe /install /autoaccepteula /reboot_if_needed /logfile \\<Server>\<Share>\Logs\WUInstall_$Hostname.log"
Invoke-Expression $Command
}
}

Once completed the connection to the vCenter server will be terminated.

Disconnect-VIServer -Server <server name or ip address> -Confirm:$False

In order to invoke the script run the following:

./Invoke-WindowsUpdate.ps1 - Environment <Environment Attribute>

The script will require the user account invoking the script to have ‘Virtual Machine Power Users’ role, to which I cloned the built-in role and local administrator privelages on each guest VM to install the approved updates.

The full Windows Powershell script can be downloaded from the below link:

https://app.box.com/s/d7fp1qapfmp9un05f0x5

Patch Management with Windows Update Server and WuInstall

I was recently looking to schedule approved updates from Windows Update Server and schedule updates with finer granularity than those provided by the Windows Update Services client. I discovered a command line utility called WuInstall  (http://www.wuinstall.com/index.php/en) which allows for updates to be installed on demand.

As part of this solution, I still required updates to be approved by my Windows Update Server and to be automatically downloaded by the Windows Update Services client on a daily basis.

I created a group policy object and linked to the organisational unit containing the clients and set the following group policy object settings:

Setting State Options Description
Configure Automatic Updates Enabled 3 – Auto download and notify for install Specifies that the instance will download approved updates from the WSUS server, but will only notify for install.
Specify intranet Microsoft Update Service location Enabled http://<name of WSUS server> Specifies the WSUS server as the intranet server to host updates.
Allow non-administrators to receive update notifications Disabled N/A Specifies that only administrators receive update notifications.
Enable client-side targeting Enabled <name of target group> Specifies the target group for the instances to receive updates.

Once the approved updates have been downloaded, the command line utility WuInstall will manage the schedule and installation.

As previously mentioned, WuInstall is a command line tool that allows the installation of Windows Updates on demand and can use the internal WSUS server to discover approved updates.

In the case of a single server, the executable can be downloaded and run with a number of command line arguments, in my case the following are to be used:

WuInstall.exe /install /autoaccepteula /reboot_if_needed /logfile <path to log file>
Usage Description
/install Searches, downloads and installs available updates.
/autoaccepteula Automatically accepts EULA on every update.
/reboot_if_needed Only restarts the instance if needed.
/logfile Creates log file

However, I was required to run the following against a number of servers. The command utility WuInstall supports the use of PSExec to run agaisnt multiple servers (http://www.wuinstall.com/index.php/faq#psexec) and therefore this was the mechanism used to launch the executable from a central management server to invoke the command on each remote server, with the following command line arguments:

PSExec.exe -c -f -s \\Server\Share\WUInstall.exe /install /autoaccepteula /reboot_if_needed /logfile <path to log file>
Usage Description
-c Copies the specific program (WuInstall.exe) to the remote instance for execution.
-f On copying the specific program overwrite the file is this already exists on the remote system.
-s Run remote process in the System account.

Now a further requirement was introduced in that updates were required to be installed on all servers in a particular environment and prior to the updates being installed a snapshot of the root device performed (Amazon Web Services). Therefore all the above would be compiled into a Windows Powershell script and

Param ([string] $Environment)

As the script was to be run against a number of environments, , the script defined parameters for the Environment name, which were called with the Environment argument. The script would required importing the Powershell for Amazon Web Services ((http://aws.amazon.com/powershell/) snap-in to the current powershell session.

If (-not (Get-Module AWSPowershell -ErrorAction SilentlyContinue))
{ 
Import-Module "C:\Program Files(x86)\AWS Tools\Powershell\AWSPowershell\AWSPowershell.psd1" > $null
}

As previously mentioned I have a collection of AWS EC2 instances (not a particularly large environment) to which I want to install the updates agaisnt, therefore this was represented as a collection in a hashtable to contain the information I required the InstanceID, VolumeID of the root device (/dev/sda1) and server name, below is an example of what this would look like;

$Instances = New-Object System.Collection.Generic.List(System.Collections.Hashtable)
$Instances.Add(@{"instanceid"="i-xxxxxxx1";"volumeid"=vol-xxxxxxx1";"name"="xxxxxxxxDEMxxx"})
$Instances.Add(@{"instanceid"="i-xxxxxxx2";"volumeid"=vol-xxxxxxx2";"name"="xxxxxxxxPRExxx"})
$Instances.Add(@{"instanceid"="i-xxxxxxx1";"volumeid"=vol-xxxxxxx1";"name"="xxxxxxxxPRDxxx"})

Once the instances have been listed as a collection we are required to run a loop process to return each instance name and then filter the environment using the substring method as conditional logic, where the naming convention contains the environment name in the 8th, 9th and 10th character of the hostname.

ForEach ($Instance in $Instances)
{
$String = $Instance.Name.ToString().Substring(8,3)
If ($String -eq $Environment)

Once this instances have been returned for the environment, the following is performed against each instance returned in the filter:

  • Stop the EC2 Instance
  • Once stopped perform an EC2 snapshot of the VolumeID specified in the collection and add the description:¬†<Host Name>: WIndows Update on ddMMyyyy_HHmm
  • Once the EC2 snapshot is complete, start the EC2 instance.
  • Once the EC2 instance is running Invoke WuInstall to install the approved updates.
{ 
Stop-EC2Instance $Instance.instanceid 
Do { 
$EC2Instance = Get-EC2Instance -Instance $instance.instanceid  
$State = $EC2Instance.RunningInstance| ForEach-Object {$_.InstanceState.Name}
{
Until ($State -eq "stopped")
New-EC2Snapshot -VolumeID $instance.volumeid -Description ($instance.name + ": Windows Update on " + $Date)
Do { 
$SnapshotStatus = (Get-EC2Snapshot | Where-Object {$_.Description -eq ($instance.name + ": Windows Update on " + $Date)}).status
}
Until ($SnapshotStatus -eq "completed")
Start-EC2Instance $instance.instanceid
Do { 
$EC2Instance = Get-EC2Instance -Instance $instance.instanceid  
$State = $EC2Instance.RunningInstance| ForEach-Object {$_.InstanceState.Name}
}
Until ($State -eq "running")
Start-Sleep -Seconds 300 
$Hostname = $instance.Name
$Command =  "& 'C:\Program Files\SysinternalSuite\PsExec.exe' \\$Hostname -c -f -s \\Server\Share\WUInstall.exe /install /autoaccepteula /reboot_if_needed /logfile \\Server\Share\Logs\logfile.log"
Invoke-Expression $Command
}
}

A couple of issues I experienced with the script, was that when the EC2 instance was reported as running the instance would not be contactable as the status checks had not been completed, so a the script is therefore suspended for a period of five minutes as a workaround.

There is currently a known issue with Powershell for Amazon Web Services snap-in (3.0.512.0) where the Get-EC2InstanceStatus cmdlet fails to return stopped instances. This requires a workaround to be performed where the instance state is returned using the Get-EC2Instance cmdlet and returning the value of the RunningInstance.InstanceState.Name object.

I plan to update this script to remove the dependency on a collection as a hashtable and return this information using the Powershell for Amazon Web Services Tools snap-in. Also, I will hopefully address the issue of reporting the EC2 instance as being in a running status where the status checks have not completed and remove the need to suspend the script.

The script will require AWS credentials to run the various cmdlets above, in this process I store the credentials on the local computer where the scheduled task is invoked.

The full Windows Powershell script can be downloaded from the below link:

https://app.box.com/s/7t2b5zqgbp4kus34rtwf

Clients not appearing in WSUS console

I recently had an issue where a number of clients were not appearing in the WSUS console even though they were being managed and receiving updates from WSUS.

This was due a number of clients having the registry value for SusClientID populated before it is deployed as part of an AMI image within Amazon Web Services EC2 and therefore a duplicate value existed.

In order to resolve the issue and remove the duplicate SuSClientID properties, I had to perform the following.

 

1) Stop the Windows Update Service Рnet stop wuauserv.

2) Browse to the following key –¬†HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate

3) Delete the following:

Name: SusClientId                              Type: REG_SZ

Name: SusClientIdValidation             Type: REG_BINARY

4) Start the Windows Update Service Рnet start wuauserv.

5) Run the resetauthorization option in combination with the detect now from the command line –¬†¬†wuauclt.exe /resetauthorization /detectnow.

The client should now appear in the WSUS console and in the correct computer group.