Lots of UI delays inside Visual Studio when using the project system

[davkean]

We're seeing lots of UI delays, more than usual when using the project system, I've seen UI delays when:

1. Expanding nodes in Solution Explorer
2. Add new and removing items to and from the tree
3. Opening C# source files
4. Opening XML files
5. Randomly just pausing while typing

I suspect there's two underlying issues based on the data that I've quickly looked over:

• In my UI delays, lots of GCs everywhere - something is causing a bunch of garbage
• ExecuteSynchronously blocks - maybe ThreadPool is running out of threads?

Please look at mine, @davidfowl and @danroth27 delays when investigating this. We do not yet have any traces of these delays.

We've identified the following issues:

• Should remove synchronous block from LaunchSettingsProvider (PR: Remove ExecuteSynchronously from LaunchSettingsProvider #2317)
• [Regression] Visual Studio becomes unusable while building if you build with diagnostic results turned on (PR: Remove synchronous call in DependencySubscriptionsHost #2410)
• [Regression] Visual Studio becomes unresponsive for a long time after opening a large project trying to populate Dependency node (PR: Fix VS becomes unresponsive opening large project #2400)
• Opening pinvoke, we're blocking >20 threads in DependenciesProjectTreeProvider.GetRule due to a JoinableTaskFactory.Run call (PR: Remove JoinableTaskFactory.Run usage in GetRule #2392)
• Extreme contention in OnceInitializedOnceDisposed.EnsureInitialized(bool) (PR: remove OnceInitializedOnceDisposed and instead import services via MEF #2429)
• Investigate GCs

[davkean]

tag @srivatsn @lifengl

[davkean]

@srivatsn This has got worse in today's build, I'm hitting blocks all the time - trying to grab a trace of it...

[lifengl]

Then, it would be easy to capture a trace.

…

Sent from my phone On May 29, 2017, at 7:40 PM, David Kean <notifications@github.com<mailto:notifications@github.com>> wrote: @srivatsn<https://github.com/srivatsn> This has got worse, I'm hitting blocks all the time - trying to grab a trace of it... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#2297 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ALGWwuat7FxWaM-qisvPMDA1RFjZ2XlZks5r-4GMgaJpZM4NnCmt>.

[davkean]

Note, I suspect the non-GC blocks are caused by:

1. Should remove synchronous block from LaunchSettingsProvider #1342 (PR in review: Remove ExecuteSynchronously from LaunchSettingsProvider #2317)
2. [Regression] Visual Studio becomes unusable while building if you build with diagnostic results turned on #2379
3. [Regression] Visual Studio becomes unresponsive for a long time after opening a large project trying to populate Dependency node #2383

[davkean]

#2 was introduced in 15.3 - hence why it feels like it's got worse in recent builds.

[davkean]

A great example of this is: https://github.com/aarnott/pinvoke. This project is unusable for minutes after it's opened.

[davkean]

I filed #2383 to represent the pinvoke. Looks like that is dependency node population.

[lifengl]

#2383 has somewhat a different reason. The code ran into a scalability problem inside JTF library in the Semaphore related JTF code.

[davkean]

@davidfowl @CyrusNajmabadi @Pilchie @sharwell

Can you do the following to see if performance of the project system is much better for you until we resolve this:

1. Create a file called Directory.Build.targets with the following contents:

<Project>
  <ItemGroup>
    <ProjectCapability Remove="DependenciesTree"/>
  </ItemGroup>
</Project>

2. Drop it in the root folder that contains all projects (make sure the solution is closed, as you'll hit another bug)

This will turn off the Dependencies node.

[davkean]

We now have PRs out for all known issues, with exception to the GC delays.

[Pilchie]

Tagging @jaredpar here too. He saw a hang in #2432 that might be related.

[lifengl]

@davkean really nice to hear that. really want to try the product once your fixes are in.

[davkean]

@lifengl @sharwell @Pilchie @davidfowl @danroth27 @CyrusNajmabadi
All known UI delays are now in. They should be available in Friday's build out of PreRel.

There are still three issues that I'm tracking for 15.3 that is affecting general responsiveness of the project system for certain types of projects:

• Visual Studio constantly consumes ~50 - 90% CPU after opening a large project
• Test Window is causing lots of UI delays due to excessive GCs
• Visual Studio is running out of memory opening Roslyn.sln

I'm going to close this bug in lieu of those specific issues.

[davkean]

To update the the thread, next week's build should have resolved all known performance regressions. There's still a lot wins to be gained (looks like the new dependency node does a lot more than the previous one) so we'll keep on plowing away on those, please file them as individual issues.

[aelij]

@davkean Were these fixes included in the latest preview (July 12)? I've recently converted a large solution (99 projects) to the new project system, and I'm seeing a very degraded experience (in 15.2): the solution loads much slower, and so is opening C# files and building. In the latest preview there seems to be some improvement but I haven't really measured it so I can't be sure. I can take memory dumps or collect PerfView data if you like.

[davkean]

Most of them made into that build - there's still a few that will come in the next build.

Are you saying that 15.3 is better than 15.2? Or are you saying that it's just as bad? Can you file a bug? Traces would be great, even better would be if you could share the source (privately). Can you also "Report a Problem" and put "davkean" in the description - doesn't have to say anything, just lets me see your VS sessions, solution load times, UI delays, etc and figure out what's going on.

[aelij]

Yes, 15.3 seems better, but still slower than the previous project system. I'll send a report right now. Should I do it after the solution is open?

[davkean]

It doesn't matter when you send the report - it just lets me grab your user data and look at past sessions. Please make sure you open a new bug - so that we're tracking this and we can use to discuss.