Ubuntu image update causes Kudu and node app to break when using azure cli for App Service setup

[aviita]

Description

Our very simple node app (image conversion service based on nextJS) started responding with HTTP 500 server error and Kudu for the related Azure app service stopped working after ubuntu image was updated to have powershell 7.4. It seems to me there is a backwards compatibility issue in powershell 7.4 over 7.2. I verified that powershell script from gist shared in steps.

It seems that the script compatibility between windows and ubuntu powershell suddenly broke on Powershell 7.4.

I know this should likely be reported as powershell issue, but we need to mitigate it also in Azure DevOps and if this is actually a bug in powershell we need to follow up on when it is fixed in Azure DevOps images.

Did not create powershell issue regarding this yet.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20240201.1/images/ubuntu/Ubuntu2204-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240201.1

Is it regression?

yes

Expected behavior

Kudu can be used and node app server requests without issues.

Actual behavior

Kudu fails with exception.

Server Error in '/' Application.

Illegal characters in path.

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.ArgumentException: Illegal characters in path.

Source Error:

An unhandled exception was generated during the execution of the current web request. Information regarding the origin and location of the exception can be identified using the exception stack trace below.

Stack Trace:

[ArgumentException: Illegal characters in path.] System.IO.Path.CheckInvalidPathChars(String path, Boolean checkAdditional) +14607857 System.IO.Path.Combine(String path1, String path2, String path3, String path4) +89 Kudu.Core.Infrastructure.PathWindowsUtility.IsNodeVersionInstalled(String version) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\PathUtils\PathWindowsUtility.cs:336 Kudu.Core.Infrastructure.PathWindowsUtility.ResolveNodeVersion(Boolean& fromAppSetting) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\PathUtils\PathWindowsUtility.cs:292 Kudu.Core.Infrastructure.PathWindowsUtility.ResolveNodeNpmPaths() in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\PathUtils\PathWindowsUtility.cs:343 Kudu.Core.Infrastructure.PathUtilityBase.GetPathFolders(IEnvironment environment) in C:\Kudu Files\Private\src\master\Kudu.Core\Infrastructure\PathUtils\PathUtilityBase.cs:34 Kudu.Services.Web.App_Start.NinjectServices.PrependFoldersToPath(IEnvironment environment) in C:\Kudu Files\Private\src\master\Kudu.Services.Web\App_Start\NinjectServices.cs:835 Kudu.Services.Web.App_Start.NinjectServices.RegisterServices(IKernel kernel) in C:\Kudu Files\Private\src\master\Kudu.Services.Web\App_Start\NinjectServices.cs:160 Kudu.Services.Web.App_Start.NinjectServices.CreateKernel() in C:\Kudu Files\Private\src\master\Kudu.Services.Web\App_Start\NinjectServices.cs:132 Ninject.Web.Common.Bootstrapper.Initialize(Func1 createKernelCallback) +23 Kudu.Services.Web.App_Start.NinjectServices.Start() in C:\Kudu Files\Private\src\master\Kudu.Services.Web\App_Start\NinjectServices.cs:97 [TargetInvocationException: Exception has been thrown by the target of an invocation.] System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor) +0 System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments) +269 System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) +146 WebActivatorEx.BaseActivationMethodAttribute.InvokeMethod() +81 WebActivatorEx.ActivationManager.RunActivationMethods(Boolean designerMode) +708 WebActivatorEx.ActivationManager.Run() +132 [InvalidOperationException: The pre-application start initialization method Run on type WebActivatorEx.ActivationManager threw an exception with the following error message: Exception has been thrown by the target of an invocation..] System.Web.Compilation.BuildManager.InvokePreStartInitMethodsCore(ICollection1 methods, Func1 setHostingEnvironmentCultures) +890 System.Web.Compilation.BuildManager.InvokePreStartInitMethods(ICollection1 methods) +167 System.Web.Compilation.BuildManager.CallPreStartInitMethods(String preStartInitListPath, Boolean& isRefAssemblyLoaded) +133 System.Web.Compilation.BuildManager.ExecutePreAppStart() +178 System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException) +746 [HttpException (0x80004005): The pre-application start initialization method Run on type WebActivatorEx.ActivationManager threw an exception with the following error message: Exception has been thrown by the target of an invocation..] System.Web.HttpRuntime.FirstRequestInit(HttpContext context) +552 System.Web.HttpRuntime.EnsureFirstRequestInit(HttpContext context) +122 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) +737

Repro steps

Option 1:

1. Create Azure resource group and Azure App Service Plan in Azure CLI:

az group create -g my-rg-kudu-error-test -l westeurope
az appservice plan create -n my-asp-kudu-error-test -g my-rg-kudu-error-test --sku S1

2. Run PS1 script from this gist in windows to create the webapp with node runtime env, with latest Azure CLI (2.56.0) and latest powershell (7.4.1):
   https://gist.github.com/aviita/b235133e40a55e6ab53b95edf75da907/edit
3. Test kudu site for the deployed app:
   https://my-app-kudu-error-test.scm.azurewebsites.net
4. Run the same ps script from gist from ubuntu 22.04 with powershell 7.4 and Azure cli 2.56.0.
5. Test kudu site for the deployed app:
   https://my-app-kudu-error-test.scm.azurewebsites.net

Expected:

• Kudu site works after each step

Actual:

• Kudu site works after step 3, but not after step 4.

Following command shows config value "~18" for config WEBSITE_NODE_DEFAULT_VERSION after step 3.
az webapp config appsettings list -n my-app-kudu-error-test -g my-asp-kudu-error-test

After step 5 it shows value "\"~18\"" and kudu fails with exception.

[mikhailkoliada]

Hey! Have you checked that the downgrade path described here makes your workflow functional again?

[aviita]

Hey! Have you checked that the downgrade path described here makes your workflow functional again?

Thanks for linking this. I was planning to check for workaround next. For now I just disabled this part of our pipeline.

[mikhailkoliada]

Hey, thanks for the feedback, if you see that the problem is not in pwsh update, feel free to report the problem back, closing it for now as nothing actable from our side can be done at the moment.