Update: Patch Management for Guest VMS with Windows Update Server and WUInstall on vSphere

I previously described steps to automate approved updates from a WSUS server with WUInstall for virtual machines managed by VMware vCenter Server (https://deangrant.wordpress.com/2014/01/24/patch-management-for-guest-vms-with-windows-update-server-and-wuinstall-on-vsphere/).

I have recently updated this process to filter virtual machines objects by tagging objects in the new functionality in the vSphere Web Client. Also, the previous process had no concept of  task parallelism, which has now also been addressed.

The script requires two mandatory parameters to be specified, we need to specify the vCenter server to which we require to establish a connection and the the category tag that has been assigned to the virtual machine to specify the infrastructure environment.

Param ([Parameter(Mandatory=$true)][string] $vCenter, [Parameter(Mandatory=$true)][string] $Environment)

Now we will register the ‘VMware.VimAutomation.Core’ snap-in and establish the connection to the vCenter server.

If (-not (Get-PSSnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue)) 
    {
    Add-PSSnapin VMware.VimAutomation.Core | Out-Null 
    }
Connect-VIServer $vCenter

Once connected, we will be required to store the current session details of the established connection to pass as a variable later in the script block to create a snapshot for the virtual machine prior to invoking the WUInstall executable.

$Session = ($global:DefaultVIServer).SessionId 

In order to filter virtual machines, we will firstly build a collection of virtual machine objects by invoking the Get-View cmdlet to retrieve only the name property values.

$VMs = Get-View -ViewType VirtualMachine -Property Name

For the collection of virtual machine objects we will now perform an operation on each object to filter the virtual machine collection to retreive only virtual machines to which the Tag Assignment ‘Environment’ is equal to the mandatory parameter specified and ‘Updates Enabled’ is equal to ‘Yes’.

ForEach ($VM in $VMS) 
    { 
    If (((Get-TagAssignment -Entity $VM.Name -Category Environment).Tag).Name -eq $Environment -and ((Get-TagAssignment -Entity $VM.Name -Category "Updates Enabled").Tag).Name -eq "Yes")
        { 

Before we invoke the automated process on each virtual machine the Get-Date cmdlet will retrieve the DateTime object to the format ‘yyyy-MM-dd’, to append to the log file generated by the WUInstall executable.

$Date = (get-date).toString('dd-MM-yyyy-HHmm')

The WUInstall executable will be invoked remotely by leveraging the PSExec utility and therefore we will specify the arguments to the executable in a variable to passed in the script block. The WUInstall will be located on the virtual machine in the location C:\Program Files\WUInstall and will create a log file in the directory location.

$WUInstall = "\\$VM -s 'C:\Program Files\WUInstall\WUInstall.exe' /install /autoaccepteula /reboot_if_needed /logfile 'C:\Program Files\WUInstall\wuinstall_$Date.log'"

In order to leverage task parallelism the script block create will be invoked to run each job as a background job on the local computer using the Start-Job cmdlet. For each background job we will require to establish a connection to the vCenter server using the session ID rather than reconnecting.

One of the requirements it to create a virtual machine snapshot prior to the installation of the approved updates, where the name will be ‘Windows Update on yyyy-MM-dd’ and then we will invoke the WUInstall on the virtual machine using the PSExec utility.

    $ScriptBlock = [scriptblock]::create("C:\Windows\SysWOW64\WindowsPowerShell\v1.0\powershell.exe -Command {
                Add-PSSnapin VMware.VimAutomation.Core | Out-Null 
                Connect-VIServer $vCenter -Session $Session 
                New-Snapshot -VM $VM -Name 'Windows Update on $Date'
                & 'C:\Program Files (x86)\SysinternalsSuite\PsExec.exe ' $WUInstall
                }")

Now we will invoke the background task to initiate the process.

Start-Job -ScriptBlock $ScriptBlock   
        }
    }

Depending on the number of virtual machines to which you are invoking the command you may wish to limit the number of background jobs that are invoked in parallel. The below will limit the number of backgorund jobs run to be 10 and once all background jobs have the state of completed deletes the backgound jobs invoked.

While((Get-Job -State 'Running').Count -ge "10")
    {
        Start-Sleep -Milliseconds 30
    }
    
While (Get-Job -State "Running") 
{ 
    Start-Sleep -Milliseconds 30
}

Remove-Job * -Confirm:$False

A very important patch for Windows 8.1 and Server 2012 R2

It would appear that Microsoft Windows 8.1 and Server 2012 R2 are only thirty days away from not receiving security updates, or more precisely from the beginning of patch Tuesday in May 2014 .

Well that is not entirely true, but instances running these operating systems are required to have installed KB2919355 (http://support.microsoft.com/kb/2919355) before the above date to continue to receive security updates.

According to Microsoft, this update was released to simplify servicing across both Windows Server 2012 R2, Windows 8.1 RT and Windows 8.1 and that his update will be considered as a new servicing/support baseline.

The new baseline only exists for Windows 8.1 and Windows Server 2012 R2, so those currently with Windows 8 and Windows Server 2012 operating systems will be unaffected. 

For more information on the above:

http://blogs.technet.com/b/gladiatormsft/archive/2014/04/12/information-regarding-the-latest-update-for-windows-8-1.aspx.

Happy patching!

WSUS client failing to find updates with error code 80244019

I was recently looking into an error for specific WSUS clients which were unable to find updates on the WSUS server reporting the following error message:

Failed to find updates with error code 80244019

On investigating the log file (C:\Windows\WindowsUpdateLog.log) on one of the affected clients I could see that the send request to the WSUS server was failing  and that the SelfUpdate check failed to download the package information, as below:

Misc WARNING: WinHttp: SendRequestToServerForFileInformation failed with 0x80190194
Misc WARNING: WinHttp: ShouldFileBeDownloaded failed with 0x80190194
Misc WARNING: DownloadFileInternal failed for http://<server name>/selfupdate/wuident.cab: error 0x80190194
Setup WARNING: SelfUpdate check failed to download package information, error = 0x80244019
Setup FATAL: SelfUpdate check failed, err = 0x80244019

It would appear that the WSUS server had been upgraded and the original TCP port used for client connectivity was no longer in use. As the Windows Update configuration of the clients is managed via Group Policy this required the update location to be modified to include the updated URL following the TCP port change to 8053, as below:

Setting Path: Computer Configuration>Policies>Administrative Templates\Windows Components\Windows Update 
Setting: Specify intranet Microsoft update service location.
State: Enabled
Options: Set the intranet update service for detecting updates - http:<server name>:8530

Once the update to the configuration had been applied and the group policy setting refreshed, the client was able to download and install approved updates from the WSUS server.

Monitor WUInstall status in Nagios XI

I recently wrote about checking the last success time of Windows Update and reporting this to Nagios (http://wp.me/p15Mdc-mj). Now what happens if you do not use Windows Update as your patch management solution. In my case I have been managing the installation of updates using WUInstall (http://www.wuinstall.com).

When my updates are installed the registry is not updated to reflect the last success time, therefore how can I monitor the last time updates were run and the status? The command line tool WUInstall provides the functionality to write the console output to a log file, in the below example each log file is written to a shared folder.

As per the previous example, I want to report if any updates have been installed in a particular time-span and if the process was successful.

In the case of the time-span this would be dependent on the host being monitored and therefore this period would be specified as a mandatory parameter when invoking the powershell script which was to be used as the check plugin within Nagios.

Param ([parameter(Mandatory = $true)][string] $Days)

Now I want to return the most recent log file for the host being monitored from the shared folder, where the log file name is that of the host and is contained in a parent folder based on the date the process was invoked. By using the Get-ChildItem cmdlet with the recurse option I am able to retrieve the most recent log file by sorting by the LastWriteTime in descending order and selecting the first file. I will return the full name of the file to a variable to pass to the Get-Content cmdlet for reading.

$LogFile = (Get-ChildItem "\\Server\Share\Logs" -Recurse | Where-Object {$_.Name -like "$env:computername*"} | Sort LastWriteTime -Descending | Select -First 1).FullName

In order to read the content of the log file we will need to encode this in Unicode format, and then search for the string ‘Overall’. In order to retrieve the overall result code we will return the next line from the content and store this as a variable.

$Log = Get-Content $LogFile -Encoding Unicode | Select-String "Overall" -Context 0,1 | % {$_.Context.PostContext}

In order to get the last run time the datetime function will be used to parse the date in the string to return the first ten characters which contain the date and the modify the date string to be in the format ‘dd/MM/yyyy’.

$LastRun = [datetime]::ParseExact($Log.Substring(0,10) , "yyyy/MM/dd", $null)
$LastRun = $LastRun.ToString("dd/MM/yyyy")

Now that we have the last run time and the result code in the log variable we can use conditional logic to set the status of the service. Firstly we will check to see if the last run date is greater or equal to the timespan value specified in the mandatory days field by subtracting this from the current date and if the result code is like ‘Succeeded’ return the service status as ‘OK’

If ($LastRun -ge (get-date).AddDays(-$Days) -and $Log -like "*Succeeded*")
   { 
   $resultcode = "0"
   }

If the last run date is less than the time-span value but the result code is like ‘Succeeded’ we will return the service status as ‘OK’.

ElseIf ($LastRun -lt (get-date).AddDays(-$Days) -and $Log -like "*Succeeded*")
   { 
   $resultcode = "1"
   }

If the result code is not like ‘Succeeded’ we will return the service status as ‘Critical’.

ElseIf ($Log -notlike "*Succeeded*")
   { 
   $resultcode = "2"
   }

If we are unable to retrieve any information required in the conditional logic the service status will be returned as ‘Unknown’.

Else
   { 
   $resultcode = "3"
   }

Finally, we will set a status information meessage based on the result code where the substring function is invoked on the  log variable to remove the first twenty characters of the string which contains timestamp information and to include the last run time and terminate the powershell session to return the exit code for the service status.

$Log.Substring(20,$Log.Length-20) + " at " + $LastRun
exit $returncode

Once a check_nrpe command has been configured in Nagios (http://wp.me/p15Mdc-eC) and you begin to monitor your host(s), you should see a service check as below:

WindowsUpdateStatusNagios

You can also run the above in Windows Powershell, to return the status information as below:

PowershellStatusWindowsUpdate

The full powershell script can be downloaded from: https://app.box.com/s/cc2pkvx9a10g5ww6ha0i 

Monitor Windows Update Last Success Time with Nagios XI

I was recently asked to provide a monitoring service for the last success success time of Windows Update and to provide status information within Nagios XI. Basically, the last success time was compared to be run a time span period of a number of days in the past to provide a service state.

In the case of the time span this was dependent on the host being monitored and therefore this period would be specified as a mandatory parameter when invoking the powershell script which was to be used as the check plugin within Nagios.

Param ([parameter(Mandatory = $true)][string] $Days)

We can read the last success time reported from Windows Update by querying the registry to retrieve the date and then format the date string to be in the format  ‘dd/MM/yyyy’.

$LastRun = (Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update\Results\Install").LastSuccessTime
$LastRun = Get-Date $LastRun -Format 'dd/MM/yyyy'

We will now to generate a date in the past based on the value specified by the mandatory Days parameter using the subtraction method from the current date.

In order to generate the service status, we will use conditional logic to compare the current date to that specified in the timespan and to provide a return code and status information message. If the date values cannot be compared this will return an unknown service status.

If ([datetime]::parse($LastRun) -ge [datetime]::parse($TimeSpan))
   {
   $returncode = 0
   "Windows Update last run successfully on " + $LastRun
   } 
   ElseIf ([datetime]::parse($LastRun) -lt [datetime]::parse($TimeSpan))
   {
   $returncode = 1
   "Windows Update last run successfully on " + $LastRun
   }
Else 
   { 
   $returncode = 3 
   }

Finally, we will exit the powershell session by providing the return code.

exit $returncode

Windows Server 2008 R2 Service Pack 1 fails with 800F0818

I was recently attempting to run Windows Server 2008 R2 Service Pack 1, which would fail with the error code 800F0818 due to Windows Update corruption.

In order to resolve the issue, run the System Update Readiness tool and then install the service pack.

The System Update Readiness tool is available from http://support.microsoft.com/kb/947821.

Patch Management for guest VMs with Windows Update Server and WuInstall on vSphere

I previously detailed steps to automate approved updates from a WSUS server with WUInstall (https://deangrant.wordpress.com/2013/09/24/patch-management-with-windows-update-server-and-wuinstall/) which focused on installing updates on Amazon Web Services EC2 instances.

I have recently modified the process to take into account VM guests in a vSphere environment ,using the same process of client side targeting and executing WuInstall on the remote machine, also with the following requirements:

  • Create a snapshot of the guest VM specified in the collection and add the name ‘Windows Update on ddMMyyyyHHmm’.
  • Once the snapshot is completed, invoke WuInstall to install the approved updates.

In order to target the guest VMs to which I wish to install the approved updates I am using custom attributes to determine the environment and if updates are enabled, as below:

Name Value
Environment DEV | TST | PRE | PRD
Windows Update Yes | No

As the script will target guest VMs in a number of environments, the script defines a mandatory parameter for an expected Environment value.

Param ([Parameter(Mandatory=$true)][string] $Environment)

In order to invoke the script there is a dependency on the Mware vSphere PowerCLI snap-in (https://www.vmware.com/support/developer/PowerCLI/) and for this to be imported into the current powershell session.

If (-not (Get-PSSnapin VMware.VimAutomation.Core -ErrorAction SilentlyContinue)) 
{
Add-PSSnapin VMware.VimAutomation.Core > $null
}

I will also generate a date string to be used in the snapshot name.

$Date = (get-date).toString('ddMMyyyyHHmm')

Now, we will establish a connection to the vCenter server.

Connect-VIServer <server name or ip address> > $null

We will now be required to filter the guest VMs where the custom attribute ‘Environment’ is equal to the mandatory parameter specified.

$VMs = Get-VM | Get-Annotation -CustomAttribute "Environment" | Where-Object {$_.Value -eq $Environment}

For each guest VM that is returned in the collection we will compare the ‘WindowsUpdate’ custom attribute and if this is equal to ‘Yes’ create a snapshot of the  guest VM with the name ‘Windows Update on ddMMyyyyHHmm’ and invoke the WuInstall to install the approved updates.

ForEach ($VM in $VMs)
{ 
$WindowsUpdate = Get-VM $VM.AnnotatedEntity.Name | Get-Annotation -CustomAttribute "WindowsUpdate" | Where-Object {$_.Value -eq "Yes"}
If ($WindowsUpdate.Value -eq "Yes")
{
New-Snapshot -VM $VM.AnnotatedEntity.Name -Name ("Windows Update on " + $Date)
$Hostname = $VM.AnnotatedEntity.Name
$Command =  "& 'C:\Program Files (x86)\SysinternalSuite\PsExec.exe' \\$Hostname -c -f -s \\<Server>\>Share>\Tools\WUInstall.exe /install /autoaccepteula /reboot_if_needed /logfile \\<Server>\<Share>\Logs\WUInstall_$Hostname.log"
Invoke-Expression $Command
}
}

Once completed the connection to the vCenter server will be terminated.

Disconnect-VIServer -Server <server name or ip address> -Confirm:$False

In order to invoke the script run the following:

./Invoke-WindowsUpdate.ps1 - Environment <Environment Attribute>

The script will require the user account invoking the script to have ‘Virtual Machine Power Users’ role, to which I cloned the built-in role and local administrator privelages on each guest VM to install the approved updates.

The full Windows Powershell script can be downloaded from the below link:

https://app.box.com/s/d7fp1qapfmp9un05f0x5