Upgrading Exchange 2019 does not work

[raandree]

Problem description

Upgrading Exchange does not work for the simple reason that the Exchange setup is killing and disabling the winmgmt (Windows Management Instrumentation) service. By killing the winmgmt service it is also killing the LCM.

One can easily see this form the event log:

$start = (Get-Date).AddMinutes(-20)
$events = Get-EventLog -LogName system -After $start -InstanceId 1073748864 | Where-Object { $_.ReplacementStrings[2] -eq 'disabled' }
$events | Format-Table -Property @{ Label = 'Service Name'; Exp = { $_.ReplacementStrings[0] } }, @{ Label = 'Old State'; Exp = { $_.ReplacementStrings[1] } }, @{ Label = 'Old State'; Exp = { $_.ReplacementStrings[2] } }

Service Name                                                  Old State    Old State
------------                                                  ---------    ---------
Microsoft Exchange Server Extension for Windows Server Backup demand start disabled 
Windows Management Instrumentation                            auto start   disabled 
World Wide Web Publishing Service                             auto start   disabled 
Tracing Service for Search in Exchange                        auto start   disabled 
Remote Registry                                               auto start   disabled 
Performance Logs & Alerts                                     auto start   disabled 
Microsoft Exchange Transport Log Search                       auto start   disabled 
Microsoft Exchange Transport                                  auto start   disabled 
Microsoft Exchange Throttling                                 auto start   disabled 
Microsoft Exchange Mailbox Transport Submission               auto start   disabled 
Microsoft Exchange Service Host                               auto start   disabled 
Microsoft Exchange RPC Client Access                          auto start   disabled 
Microsoft Exchange Replication                                auto start   disabled 
Microsoft Exchange POP3 Backend                               demand start disabled 
Microsoft Exchange POP3                                       demand start disabled 
Microsoft Exchange Mailbox Replication                        auto start   disabled 
Microsoft Exchange Mailbox Assistants                         auto start   disabled 
Microsoft Exchange Information Store                          auto start   disabled 
Microsoft Exchange IMAP4 Backend                              demand start disabled 
Microsoft Exchange IMAP4                                      demand start disabled 
Microsoft Exchange Health Manager Recovery                    auto start   disabled 
Microsoft Exchange Health Manager                             auto start   disabled 
Microsoft Exchange Frontend Transport                         auto start   disabled 
Microsoft Exchange Search                                     auto start   disabled 
Microsoft Exchange EdgeSync                                   auto start   disabled 
Microsoft Exchange Diagnostics                                auto start   disabled 
Microsoft Exchange Mailbox Transport Delivery                 auto start   disabled 
Microsoft Exchange DAG Management                             auto start   disabled 
Microsoft Exchange Anti-spam Update                           auto start   disabled 
Microsoft Exchange Active Directory Topology                  auto start   disabled 
IIS Admin Service                                             auto start   disabled 
Microsoft Exchange Search Host Controller                     auto start   disabled 
Microsoft Filtering Management Service                        auto start   disabled 
Performance Logs & Alerts                                     demand start disabled 

In this case, the LCM throws this error:

The WS-Management service cannot process the request. The WMI service or the WMI provider returned an unknown error: HRESULT 0x80041033 
    + CategoryInfo          : ResourceUnavailable: (root/Microsoft/...gurationManager:String) [], CimException
    + FullyQualifiedErrorId : HRESULT 0x80041033
    + PSComputerName        : localhost 

Exchange enables and starts the winmgmt service later but when starting the LCM again using Start-DscConfiguration -UseExisting -Wait -Force -Wait, things did not finish as expected. In some cases I no longer had an Exchange service on the machine and DSC could no longer start because it could no longer find an installed Exchange. All this does not look very reliable.

It looks to me like upgrading Exchange with this DSC resource module never worked because the code did not compare the installed version and the desired version until recently: #410.

When removing write access to the winmgmt service, the Exchange setup complains:

Error:
The following error was generated when "$error.Clear(); 
          if (Get-Service WinMgmt* | ?{$_.Name -eq 'WinMgmt'})
          {
            Set-Service WinMgmt -StartupType Automatic
            Start-SetupService -ServiceName WinMgmt
          }
        " was run: "Microsoft.PowerShell.Commands.ServiceCommandException: Service 'Windows Management Instrumentation (WinMgmt)' cannot be configured due to the following error: Access is denied ---> System.ComponentModel.Win32Exception: Access is denied
   --- End of inner exception stack trace ---".

Verbose logs

NA

DSC configuration

Configuration Test1
{
    Import-DscResource -ModuleName xExchange, ComputerManagementDsc, StorageDsc

    Node E1Ex1
    {
        xExchMaintenanceMode PutExchServersInMM
        {
            Enabled = $true
            Credential = $ConfigurationData.Credential
        }

        MountImage ExchangeIso
        {
            ImagePath   = 'd:\ExchangeServer2019-x64-CU12.ISO'
            DriveLetter = 'E'
            DependsOn = '[xExchMaintenanceMode]PutExchServersInMM'
        }

        WaitForVolume WaitForExchangeIso
        {
            DependsOn = '[MountImage]ExchangeIso'
            DriveLetter      = 'E'
            RetryIntervalSec = 5
            RetryCount       = 10
        }

        # Check if a reboot is needed before installing Exchange
        PendingReboot BeforeExchangeInstall
        {
            Name       = 'BeforeExchangeInstall'
            DependsOn  = '[WaitForVolume]WaitForExchangeIso'
        }

        xExchInstall InstallExchange
        {
            Path       = 'E:\Setup.EXE'
            Arguments  = '/mode:Upgrade /IAcceptExchangeServerLicenseTerms_DiagnosticDataON'
            Credential = $ConfigurationData.Credential
            DependsOn  = '[PendingReboot]BeforeExchangeInstall'
        }

        # See if a reboot is required after installing Exchange
        PendingReboot AfterExchangeInstall
        {
            Name      = 'AfterExchangeInstall'
            DependsOn = '[xExchInstall]InstallExchange'
        }

        Script SetReboot {
            DependsOn = '[xExchMaintenanceMode]PutExchServersInMM'
            TestScript = {
                return (Test-Path HKLM:\SOFTWARE\MyMainKey\RebootKey)
            }
            SetScript  = {
                New-Item -Path HKLM:\SOFTWARE\MyMainKey\RebootKey -Force
                $global:DSCMachineStatus = 1 
            }
            GetScript  = { 
                return @{
                    result = 'result'
                }
            }
        }

        PendingReboot AfterExchangeInstallation
        {
            DependsOn = '[Script]SetReboot'
            Name      = 'AfterExchangeInstallation'
        }

        xExchMaintenanceMode TakeExchangeServerOutOfMM
        {
            #DependsOn = '[PendingReboot]AfterExchangeInstall'
            Enabled = $false
            Credential = $ConfigurationData.Credential
        }
        
    }
}

$configData = @{
    AllNodes   = @(
        @{
            NodeName                    = 'E1Ex1'
            PSDscAllowPlainTextPassword = $true
            PSDscAllowDomainUser        = $true
        }
    )

    Credential = New-Object pscredential('contoso\administrator', ('Somepass1' | ConvertTo-SecureString -AsPlainText -Force))
}

<#
Remove-DscConfigurationDocument -Stage Current, Pending, Previous
#>

Set-Item -Path WSMan:\localhost\MaxEnvelopeSizekb -Value 8192

Test1 -OutputPath C:\DSC -ConfigurationData $configData

Start-DscConfiguration -Path C:\DSC -Wait -Verbose -Force

Suggested solution

This needs more investigation.

Operating system the target node is running

Windows Server 2022

Exchange Server edition and version the target node is running

Exchange 2019 CU10 trying to upgrade to CU12

PowerShell version and build the target node is running

Windows PowerShell 5.1

xExchange version

1.34.0

[raandree]

@johlju, this is not a bug in the resource module. As mentioned above it is "the Exchange setup is killing and disabling the winmgmt (Windows Management Instrumentation) service. By killing the winmgmt service it is also killing the LCM".

When installing or upgrading Exchange, the setup files are created in / copied to C:\Windows\Temp. The code to stop and disable Windows services temporarily is in C:\Windows\Temp\ExchangeSetup\ServiceControl.ps1. This is nothing we can fix in the module IMHO. This issue is rather for making people aware of that and gathering ideas or workarounds.

[johlju]

I labeled it external for that case - maybe it is not a bug doing interactive or upgrade from command line but it is a bug doing it trough LCM. Unless there is a way to tell Exchange to wait for the restart to later (so that we can control it) the we can’t fix anything here, agree.

Guessing we closing this issue until something changes that can make this work in LCM.

[mhincapie]

I use Chef to install Exchange and also do the CU every time they are out. The CU process has to be done through a PowerShell resource such as this one instead of using the one that is provided to install it:

powershell_script 'upgrade_exchange' do
  code '.\setup.exe /IAcceptExchangeServerLicenseTerms_DiagnosticDataON /Mode:Upgrade'
  guard_interpreter :default
  elevated elevate
  cwd base_dir
  sensitive false
  timeout 10800
  user username
  password userpswd
end

Not sure if this helps but that is our workaround.

[raandree]

I think that any other automation product except DSC will do. We know that the Exchange installation stops the Winmgmt service and therefore LCM. Chef will not be affected by this. Actually quite sad that a Microsoft product makes its own automation engine useless.