Windows Server 2012 virtual machines stop responding at splash screen

I recently discovered an issue on a restart of a virtual machine running the guest operating system Windows Server 2012 where following a restart the virtual machine would stop responding at the splash screen and required a manual shutdown and power on to restart the virtual machine. From investigation is a known issue caused by the following:

Startingwith Windows 8 / Windows 2012 Server, during its boot process the operating system will reset the TSC (TimeStampCounter, which increments by 1 for each passed cycle) on CPU0. It does not reset the TSC of the other vCPUs and the resulting discrepancy between two vCPUs’ TSC can result in the OS not booting past the Windows splash screen, and a full power off and on will fix it.

There is a workaround provided until this issue is resolved which requires the virtual machine configuration file to be modified so that the TSC for all vCPUs should be reset to zero on a soft reset of the machine, and not just CPU0. In order to apply the workaround, the virtual machine will be required to be in a powered off state to add the following line to the configuration file.

monitor_control.enable_softResetClearTSC = "TRUE"

In order to apply the configuration change to a number of virtual machines, I orchestrated a number of steps in a powershell script as follows:

Firstly, I wanted to write the progress of the orchestration to both the console session and a log file, to which I have previously discussed the creation of this function here.

$LogFile = "softResetClearTSC_11022014.log"

$Path = [System.IO.Path]::Combine($env:USERPROFILE,$LogFile)
$FileMode = [System.IO.FileMode]::Append
$FileAccess = [System.IO.FileAccess]::Write
$FileShare = [IO.FileShare]::Read
$FileStream = New-Object IO.FileStream($Path, $FileMode, $FileAccess, $FileShare)
$StreamWriter = New-Object System.IO.StreamWriter($FileStream)


Function Log($Event)
{
   Write-Host $Event
   $StreamWriter.WriteLine($Event)
}

Now we will connect to the vCenter System System and retrieve a collection of virtual machines to which the configuration change will be applied.

Connect-VIServer deanvc1.dean.local 
$VMS = "deanvm1","deanvm2","deanvm3"

For each object retrieved from the collection I will orchestrate the following steps:

  • Perform a shutdown of the guest operating system (requires VMware Tools to be installed) and wait until the Power State is returned as ‘PoweredOff’.
  • Add the key value pair ‘monitor_control.enable_softResetClearTSC = “TRUE”‘ to the virtual machine configuration file.
  • Power on the virtual machine.
ForEach ($VM in $VMS) 
    { 
    Try
        { 
        Log ("" + (Get-Date -Format s) + ": INFORMATION: Preparing guest operating system for shutdown for virtual machine " + $VM + ".") 
        Shutdown-VMGuest $VM -Confirm:$False | Out-Null 
        Do 
            { 
            Start-Sleep -Seconds 5 
            } 
        Until ((Get-VM $VM).PowerState -eq "PoweredOff")
        } 
    Catch [System.Exception]
        { 
        Log ("" + (Get-Date -Format s) + ": ERROR: Failed to shutdown the virtual machine " + $VM + ".") 
        Log $Error.Exception
        $Error.Clear() 
        Break
        } 
    Log ("" + (Get-Date -Format s) + ": INFORMATION: Successfully shutdown virtual machine " + $VM + ".") 
    Try
        { 
        Log ("" + (Get-Date -Format s) + ": INFORMATION: Modifying configuration file for the virtual machine " + $VM + ".") 
        $View = Get-VM $VM | Get-View
        $Config = New-Object VMware.Vim.VirtualMachineConfigSpec
        $Config.ExtraConfig += New-Object VMware.Vim.OptionValue
        $Config.extraConfig[0].key = "monitor_control.enable_softResetClearTSC"
        $Config.extraConfig[0].value = "TRUE"
        ($View).ReconfigVM_Task($Config) | Out-Null
        } 
    Catch [System.Exception]
        {
        Log ("" + (Get-Date -Format s) + ": ERROR: Failed to modify the configuration file for the virtual machine " + $VM + ".") 
        Log $Error.Exception
        $Error.Clear() 
        Break
        }
    Log ("" + (Get-Date -Format s) + ": INFORMATION: Successfully added the key monitor_control.enable_softResetClearTSC to the virtual machine " + $VM + ".") 
    Try
        {
        Log ("" + (Get-Date -Format s) + ": INFORMATION: Powering on the virtual machine " + $VM + ".") 
        Start-VM $VM -Confirm:$False | Out-Null 
        }
    Catch [System.Exception] 
        {
        Log ("" + (Get-Date -Format s) + ": ERROR: Failed to power on the virtual machine " + $VM + ".") 
        Log $Error.Exception
        $Error.Clear() 
        Break
        }
    Log ("" + (Get-Date -Format s) + ": INFORMATION: Succesfully powered on the virtual machine " + $VM + ".") 
    } 

Finally, we will dispose of the StreamWriter and FileStream classes used for the log function to write the progress to the console session and the log file.

$StreamWriter.Dispose()
$FileStream.Dispose()

The above script can be downloaded from https://github.com/dean1609/scripts/blob/master/Set-softResetClearTSC.ps1.

Advertisements

Switch between Server Core and GUI on Windows Server 2012

With the release of Windows Server 2012 it is now possible to switch between the GUI and Server Core installations using the GUI and Powershell cmdlets, with only a restart required to apply the changes.

The default installation of Windows Server 2012 is now available as Server Core, if you require to switch to the GUI installation of the operating system you can do using the following powershell cmdlets;

Install-WindowsFeature Server-Gui-Shell, Server-Gui-Mgmt-Infra -Restart

If you have a Windows Server 2012 GUI installation, you can switch to a Server Core installation, either by using the GUI to remove the Graphical Management Tools Infrastructure and Server Graphical components of the User Interfaces and Infrastructure feature or the powershell cmdlets.

Uninstall-WindowsFeature Server-Gui-Shell, Server-Gui-Mgmt-Infras -Restart

Error “CRITICAL_STRUCTURE_CORRUPTION” on Windows Server 2012 R2 using ESXi

I recently experienced BSOD with the error message “CRITICAL_STRUCTURE_CORRUPTION” for a virtual machine running Windows Server 2012 R2 running on a ESXi Host 5.0.0, 623860.

This is a known issue and was resolved in ESXi 5.0 Update 3.

There is also a workaround to create a CPUID mask for the virtual machines with the above symptom.

1) Power down the virtual machine.

2) Edit Settings for the virtual machine.

3) Select the ‘Options’ tab and browse to ‘Advanced>CPUID Mask’ and select ‘Advanced’.

4) Paste the below into the edx register field under Level 80000001.

----:0---:----:----:----:----:----:----

5) Apply the settings and power on the virtual machine