[WIP] fsc.exe - Parallel Parsing #10208
Conversation
| results.[i] <- | ||
| let (filename: string, isLastCompiland) = sourceFiles.[i] | ||
| let pathOfMetaCommandSource = Path.GetDirectoryName filename | ||
| match ParseOneInputFile(tcConfig, lexResourceManager, ["COMPILED"], filename, (isLastCompiland, isExe), errorLogger, (*retryLocked*)false) with |
There was a problem hiding this comment.
The errorLogger is probably not concurrent safe; will need to solve that.
|
How does this stack up in benchmarks? |
Co-authored-by: Saul Rennison <saul@users.noreply.github.com>
|
I will get some benchmarks soon. My gut feel is that it will only shave a few seconds off of compiling the compiler as an example. |
|
This optimization will benefit fcs the most i believe, not fsc. IDE's like VS, Rider already can build stuff really quickly with heuristics and so on, but while working with code fcs is a thing that takes too much time. Can this optimization be turned on for fcs? |
|
@En3Tho F# parsing is usually not a problem in IDEs. To make IDE work faster we need optimizations for type-checking, which is a subsequent phase after the parsing one. Moreover, FCS keeps only few file parsing results in its cache to save memory, so there's no need to parse files until they're needed by type checker or some IDE feature (and type checker doesn't run in parallel anyway). Previously parsing would wait in the FCS queue if there's a long type checking process, and it has been improved in #3601, now IDE features are already allowed to parse files in parallel. |
|
@auduchinok @TIHan Thank you for an answer. Any plans to further parallelize compiler stages? It feels quite sad to wait for fcs every time while having 8/16 core/threads processor and 64 gigs of mem |
|
No current plans, but @TIHan has looked into it before. It should be possible in theory without having to rewrite everything, but it won't be easy |
|
@auduchinok is right. Type-checking is where almost all the time is spent. Parsing is really fast in F# - I'm not even sure if parallel parsing files will give us that much benefit (haven't made benchmarks yet). @En3Tho, @dsyme and I discussed parallel type-checking in FCS last week. We went over a variety of things, and it should be possible. However, specifically in VS, parallelism could make things worse as more objects can be allocated at one time. When I did the compiler server prototype, I made a lot of fixes that supported the work being done in parallel, such as forcing all type provider calls on the same thread. Perhaps I can re-use some of that work and make small PRs that will inch our way towards parallelism. Another idea that's interesting is parallel type-checking when compiling: #10214 -- The caveat here with parallelism is that the compiler has not been fully proven to be safe in parallel at certain stages. So, what we really need to do is design for correctness first over speed. |
I imagine it would help parsing our FSharp.Core test suite, since other threads could parse stuff while one is working on this chonky boi: https://github.com/dotnet/fsharp/blob/main/tests/FSharp.Core.UnitTests/FSharp.Core/ComparersRegression.fs |
|
@TIHan , it looks like you are not convinced about the performance benefits of this change: #10208 (comment). So I will close it, please reactivate the issue if you change your mind and get an opportunity to develope it further. Thanks |
This enables parallel parsing when running
fsc.exe. I think this should be fairly straight forward to make safe.