Performance issue causing VS freezes and Out of memory crashes

[blumu]

NOTE: This reproes on a private solution so I cannot share unfortunately.

We encounter perf issues with our VS solution (hangs, and freezes). The F# team looked into perfview traces that we shared with them and have identified a performance issue in the FSharp.Data type provider causing the large object heap to be filled up, and eventually triggering long gen2 GC pauses (>30s).
The following perfview stack trace shows 6GB of memory related to assemblies being loaded for the type providers in FSharp.Data.
There is about 1GB for each type provider exposed by FSharp.Data, even though our solution only makes use of two of them (JSON, and XML).

(Visual Studio 2019 Preview 4.2 [28712.124.d16.0])

The proper fix might involve fixes to the TypeProvider SDK (ProvidedType.fs). (Will Smith submitted a PR to address this at fsprojects/FSharp.TypeProviders.SDK#283 but the proposed solution has drawbacks and was therefore not approved.)

• Given the perf impact of this bug, it may be a good idea to consider having FSHarp.Data consume the patch version of ProvidedTypes.fs from Will Smith's PR.
• Additionally, as a temporary mitigation, it would also be good to have FSHarp.Data only load the type providers that are consumed (in our cases, that's just XML and JSON but not WorldBank, CSV, and Html)

[dsyme]

@blumu @TIHan We'll get this fixed in the TPSDK tomorrow and then we can update FSharp.Data

[dsyme]

@blumu Could you check again with FSHarp.Data 3.0.1? Thanks

[blumu]

@dsyme Thanks a lot for quickly releasing this fix! I will try it in a moment.
For reference: I now have reliable and simple repro steps on FSharp.Data 3.0.0:

• Start a new VS 2019 instance
• Open our project solution (shared internally with F# team),
• Open file PrevalidationConsole.fs
• Right click on **Listener.registerFileAndConsoleTracerWithConfiguration on line 306
• Click on "Find all references"
• VS will hand and eventually crash
  (Note: I actually never use 'Find all references' because I find it too slow; but that the quickest way we found with Will Smith to repro the hang.)
  CC: @TIHan

[blumu]

@dsyme @TIHan The fix worked, awesome :-)
The "Find all references" took a while (building the whole solution I guess), process private memory increased to 2.7GB, but VS did not hang and did not crash! Best of all: "Find all references" actually worked and after crunching for 3minutes returned ... a single reference ;-)

My team is going to be delighted! Thanks again.

[TIHan]

This is great to hear @blumu , this fully confirms this was your issue. It seems the fix is intermediate as we need to find a proper solution.

@dsyme we should discuss what the next path forward should be. I do agree with @davkean that we should properly tie this in the lifecycle of something, preferably Roslyn's metadata bits.