Publishing to Lambda is extremely slow on a Windows 7 system with .NET 7 installed

[RobotHead5000]

Update - This issue was a problem where .NET 7 was automatically installed into a Windows 7 system (not through the Toolkit), and the Toolkit then became very slow to publish projects to AWS Lambda.

.NET 7 is not supported on Windows 7, and the CLI command dotnet publish becomes slow and encounters errors. For details and mitigation, see #314 (comment)

---

Original issue:

Describe the bug

When opening an AWS Lambda project in Visual Studio 2022 with AWS Toolkit installed and enabled, memory use increases to 99% and the system becomes unusably slow. There is also a huge amount of disk activity and CPU usage use is very high (though not 100%).

To Reproduce

1. Open any AWS Lambda C# solution in VS2022 with AWS Toolkit enabled.
2. Memory use shoots up to 99% for several minutes with a huge performance drop. It recovers after a few minutes of the system being unusable.
3. Right click on the project in Solution Explorer and on 'Publish to AWS Lambda...'.
4. Click the 'Upload' button.
5. Log gets to 'Determining which projects to explore'.
6. Memory use shoots up to 99%, system becomes unusable and it remains this way permanently (necessitating the use of the physical reset button).

Expected behavior

I expect VS to open quickly and for AWS Toolkit to package and upload the project, like it did before December.

Computer (please complete the following information):

• Windows Version: Windows 7 SP1
• Visual Studio Version: 17.4.4
• AWS Toolkit for Visual Studio Version: 1.38.0.0

Additional context

• It has behaved this way since early December. It first occurred when I right clicked on the project to publish it and accidentally clicked on something in the context menu - I don't know what - but the high memory use and system slowdown occurred then for the first time.
• I've tried uninstalling and reinstalling both VS and AWS Toolkit.
• Downgrading the Toolkit makes no difference.
• I've tried deleting the "\bin" and "\obj" folders.
• There is nothing I can see in "%localappdata%\AWSToolkit\logs\visualstudio" that sheds any light on the problem.
• I don't get the problem with any other VS solution, only AWS Lambda projects. I get the problem with both .NET Core 3.1 and .NET 6 Lambda projects.
• Disabling the AWS Toolkit extension (or uninstalling it) makes the problem go away.
• Some instinct is telling me there's some strange interaction with MSBuild.exe but I couldn't say what's happening.
• Nothing appears in the process list in Task Manager that accounts for either the levels of memory or CPU use (i.e. the entries don't add up 16GB).
• I noticed today that when opening a Lambda project, the background tasks show that it spends some time installing the Lambda Mock Test Tool. I don't know if it always did that, but seems to take some time.

[shruti0085]

Hello @RobotHead5000,
Thanks for bringing this issue to our attention, and for sharing all of the investigation you've done.
I have a few questions that would help investigate this further:

• Can you share your toolkit logs file for the instance for which you encountered this issue with? The log file includes timestamps that can help us understand if some portion of the toolkit is taking longer than expected.
  • From Visual Studio, click Extensions in the main menu. Select AWS Toolkit then View Toolkit Logs.
• Can you check and share the version of the Lambda Test Tool installed on your system?
  • From the command line, run dotnet tool list -g . This should list all dotnet tools installed on your system including one with the Package-Id amazon.lambda.testtool-6.0 or amazon.lambda.testtool-3.1 . Please check and share the version corresponding to these package-ids.
  • From the command line, try installing the Lambda Test Tool dotnet tool install -g Amazon.Lambda.TestTool-6.0. If it is already installed, try dotnet tool update -g Amazon.Lambda.TestTool-6.0. How long does this take to install and is there any unexpected output?
• Please run dotnet nuget list source from the command line and share the output. We'd be interested to know if you are using a package source other than nuget.org.
• How many projects are in the .NET solution that is being loaded in Visual Studio and published to Lambda?

[RobotHead5000]

Thanks for responding. I'm going to answer the first and fourth bullets now and I'll come back to the second and third ones as I need to finish now.

In answer to the first bullet: I opened the project, let it finish its faffing around, then tried to publish it to Lambda. This time I left it alone and went and had dinner. Unusually, the output in the Toolkit window didn't get as far as the 'Determining which projects to explore' line, only to:

publish: MSBuild version 17.4.1 + 9a89d02ff for .NET

(I thought the latest version of MSBuild was 18 but I'm not sure that matters.)

Anyway, when I came back from dinner it seemed to have finished, but from the log you can see it took ~41 minutes (whereas normally it would take a few seconds). I'm afraid I haven't had time to test the function since.

log_2023-01-18_19-48-41_5280.txt

There do appear to be some errors in the log. The 19:50:20 line is when it finished faffing about and performance settled down after opening the solution.

The lines from 19:51:52 are after I clicked the Upload button. That line has the error:

ERROR Amazon.AWSToolkit.Clients.ServiceClientCreator - Error creating service client: Amazon.SecurityToken.IAmazonSecurityTokenService
System.Exception: No constructor found that accepts AWSCredentials and RegionEndpoint

It is logged into correctly as I can see everything I'd expect to see in the AWS Explorer window, and the 'Upload to AWS Lambda' publishing wizard picked up the environment variables I've set on AWS Console for this Lambda function.

Hopefully this log will give you some clues.

In answer to the fourth bullet, there's only one project in the solution.

As I say, I'll come back to the others, hopefully tomorrow (I've a few jobs on at the moment!).

[RobotHead5000]

I've now had a chance to look at bullets 2 and 3.

Bullet 2:

• It shows both amazon.lambda.testtool-3.1 and amazon.lambda.testtool-6.0. Do the numbers correspond to the .NET frameworks I've used?
• It says version 0.12.7 for both of them.
• Running the update with dotnet tool update -g Amazon.Lambda.TestTool-6.0 took about 14 seconds. Nothing unexpected - reinstalled with latest stable version '0.12.7'.

Bullet 3:

• Running dotnet nuget list source shows:

1. nuget.org [Enabled]
   https://api.nuget.org/v3/index.json
2. Microsoft Visual Studio Offline Packages [Enabled]
   C:\Program Files (x86)\Microsoft SDKs\NuGetPackages\

• This is what I expected though.

[shruti0085]

Thank you for checking and sharing all the details. That helps us rule out a couple of things.

Can you try a publish to Lambda again and share the logs from the Output Window with the timestamp? The toolkit logs do not contain those logs and it would be useful to get more information about the publish steps that are taking longer to execute.

• Before publishing the Lambda, open the View menu, and select Output. From the output pane's show output from dropdown, select Amazon Web Services. Then enable the clock icon in the Output window toolbar so that the publishing logs will be timestamped.
  

Once the Lambda is deployed, can you share the value of CodeSize field found when viewing the function?

One more thing that you could try is publishing the same project to Lambda using the dotnet aws lambda tool from the command line and check how long that takes?

• Install : dotnet tool install -g Amazon.Lambda.Tools
• Navigate to the root directory of your project
• Deploy to Lambda (using the same function configuration details as defined in the aws-lambda-tools-default.json): dotnet lambda deploy-function <YourFunctionName>

To get more insights, would you mind sharing your internet speed(upload and download)?
Also wanted to confirm if your system is running on Windows 7? It would be interesting to know if you have access to a Windows 10(or newer) system and if it also results in a similar experience while publishing.

[RobotHead5000]

I've attached the output from this evening's attempt to publish. Interestingly, this time it got to the 'Restored ...[project]' stage immediately, but it generally behaved in the same way.

Output.txt

There is an error in there following this stage, logged 53 minutes later. Failed to create CoreCLR, HRESULT: 0x8007000E

• Code size is 23,208,467 bytes.
• dotnet tool install -g Amazon.Lambda.Tools installs version 5.6.3 successfully and takes about 15s.
• Deploying using the command line behaved the same way, i.e. it took aaages.

For the last couple of things:

• Download speed is 930Mbps; upload is 110Mbps (I ran speedtest.net to confirm).
• And yes, I am running Windows 7. I knew you'd pick up on that! I don't have access to a Win10 system for these purposes.

[shruti0085]

Hi @RobotHead5000,
Thanks for sharing the output and the details.

Can you check the .NET configuration of your system?

• Run dotnet --info from the command line
• If the .NET SDK version indicates .NET 7, we suspect that the issue might be related to lack of official .NET 7 support for Windows 7 dotnet/announcements#226
  • Related issues: dotnet/runtime#78277, dotnet/runtime#78278

If it indicates .NET 7, can you try setting up a global.json file in the root of your project with latest version of .NET 6 SDK(or lower)? (At the time of writing the latest version is 6.0.405).

• Instructions to setup a global.json file: https://learn.microsoft.com/en-us/dotnet/core/tools/global-json
• You might have to install .NET 6 SDK if it is not present on your system. You can download and install the latest version from https://dotnet.microsoft.com/en-us/download/dotnet/6.0

Once the above setup is complete, can you try publishing your application again using the command line?

• Publish command taken from the logs: dotnet publish "F:\Programming Projects\Portoptimo\Portoptimo\Portoptimo_Primary" --output "F:\Programming Projects\Portoptimo\Portoptimo\Portoptimo_Primary\bin\Release\netcoreapp3.1\publish" --configuration "Release" --framework "netcoreapp3.1" /p:GenerateRuntimeConfigurationFiles=true --runtime linux-x64 --self-contained False
• This is the underlying command that the dotnet aws lambda CLI tool seems to be slowing down on when publishing.
  With the above updates it would be interesting to see if this helps speed this step up.

[RobotHead5000]

That fixed it! Thank you ever so much.

It's interesting that it suddenly started behaving like this a month ago - I can only think to blame an automatic update that replaced .NET 6 with .NET 7. At least the global.json file should force it to behave now.

Thanks again!

[awschristou]

Thank you for the update @RobotHead5000 , we appreciate your patience and cooperation through the investigation, and I'm glad we were able to find the cause and find a mitigation. I've updated the issue title and description to more accurately describe the problem, and will close out this issue.