Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[wasm] Compile .bc->.o in parallel, before passing to the linker #54053

Merged
merged 22 commits into from
Jun 23, 2021

Commits on Jun 15, 2021

  1. Utils.RunProcess: some refactoring

    - this cleans up `Utils.RunProcess` a bit, but doesn't change the
      behavior for the existing non-wasm users (Apple, and Android build
      tasks)
    
    - New overloads are added to support getting the output, and exit code,
      and throwing only when really needed
    - And running shell commands, appropriately on windows, and unix.
    radical committed Jun 15, 2021
    Configuration menu
    Copy the full SHA
    8848879 View commit details
    Browse the repository at this point in the history
  2. [wasm] Compile .bc -> .o files, in parallel

    Currently, we use `mono-aot-cross` to compile the managed assemblies
    into bitcode files. Then we pass the `.bc` files to `emcc` for linking.
    We can save some amount of work, if precompile `.bc` files to `.o`
    files, and pass them for linking.
    
    This would also allow us to rebuild the `.o` files only when needed,
    thus improving incremental builds.
    Also, emscripten has recently removed some amount of parallelization, so
    this allows us to do some pre-linking work in parallel, and speed up the
    builds a little bit.
    
    This is implemented via a new `EmccCompile` task, that is meant *only*
    for compiling, and not as a general purpose `emcc` invocation task.
    radical committed Jun 15, 2021
    Configuration menu
    Copy the full SHA
    aefb115 View commit details
    Browse the repository at this point in the history
  3. Disable failing tests

    `System.IO.MemoryMappedFiles.Tests` - Issue: dotnet#54194
    radical committed Jun 15, 2021
    Configuration menu
    Copy the full SHA
    1909fcc View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    24338c6 View commit details
    Browse the repository at this point in the history
  5. [wasm] EmccCompile task: Use msbuild's new resource management API

    - `IBuildEngine9.RequestCores`
    - This is available only in 16.10+, so bump msbuild used for building
      the tests
    radical committed Jun 15, 2021
    Configuration menu
    Copy the full SHA
    e7b638c View commit details
    Browse the repository at this point in the history
  6. cleanup

    radical committed Jun 15, 2021
    Configuration menu
    Copy the full SHA
    9f23826 View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2021

  1. Configuration menu
    Copy the full SHA
    f87f705 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    093e133 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d7f867e View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    f562d2e View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    f20240b View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    6e90a22 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2021

  1. Configuration menu
    Copy the full SHA
    d816231 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2021

  1. Disable Microsoft.NETCore.Platforms.Tests on wasm, for local builds too

    - This is meant to be disabled for CI (it has `IgnoreForCI=true` in the
      csproj)
    radical committed Jun 22, 2021
    Configuration menu
    Copy the full SHA
    1be69d6 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    76d370a View commit details
    Browse the repository at this point in the history
  3. Undo 16.10.0 msbuild bump

    .. because we keep running into the following on browser, and other
    platforms:
    
    ```
    /__w/1/s/src/libraries/Microsoft.NETCore.Platforms/tests/Microsoft.NETCore.Platforms.Tests.csproj : error NU1201: Project System.Security.AccessControl is not compatible with net472 (.NETFramework,Version=v4.7.2). Project System.Security.AccessControl supports: net6.0 (.NETCoreApp,Version=v6.0) [/__w/1/s/Build.proj]
    \#\#[error]src/libraries/Microsoft.NETCore.Platforms/tests/Microsoft.NETCore.Platforms.Tests.csproj(0,0): error NU1201: (NETCORE_ENGINEERING_TELEMETRY=Restore) Project System.Security.AccessControl is not compatible with net472 (.NETFramework,Version=v4.7.2). Project System.Security.AccessControl supports: net6.0 (.NETCoreApp,Version=v6.0)
    ```
    radical committed Jun 22, 2021
    Configuration menu
    Copy the full SHA
    5694644 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    9374c61 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    b7f52e4 View commit details
    Browse the repository at this point in the history
  6. disable Microsoft.NETCore.Platforms.Tests on wasm

    This is already not being sent to CI with `IgnoreForCI=true`.
    It fails on CI with:
    
    ```
    /_/src/Shared/NativeMethodsShared.cs(855,29): error IL1005: Microsoft.Build.Shared.NativeMethodsShared.SystemInformation.get: Error processing method 'Microsoft.Build.Shared.NativeMethodsShared.SystemInformationData.SystemInformationData()' in assembly 'Microsoft.Build.Utilities.Core.dll' [/__w/1/s/src/libraries/Microsoft.NETCore.Platforms/tests/Microsoft.NETCore.Platforms.Tests.csproj]
    \##[error]/_/src/Shared/NativeMethodsShared.cs(855,29): error IL1005: (NETCORE_ENGINEERING_TELEMETRY=Build) Microsoft.Build.Shared.NativeMethodsShared.SystemInformation.get: Error processing method 'Microsoft.Build.Shared.NativeMethodsShared.SystemInformationData.SystemInformationData()' in assembly 'Microsoft.Build.Utilities.Core.dll'
      Mono.Linker.LinkerFatalErrorException: /_/src/Shared/NativeMethodsShared.cs(855,29): error IL1005: Microsoft.Build.Shared.NativeMethodsShared.SystemInformation.get: Error processing method 'Microsoft.Build.Shared.NativeMethodsShared.SystemInformationData.SystemInformationData()' in assembly 'Microsoft.Build.Utilities.Core.dll'
       ---> System.NotImplementedException: switch
         at Mono.Linker.Steps.UnreachableBlocksOptimizer.BodyReducer.IsConstantBranch(OpCode opCode, Int32 operand) in illink.dll:token 0x6000770+0x41
         at Mono.Linker.Steps.UnreachableBlocksOptimizer.BodyReducer.RemoveConditions() in illink.dll:token 0x6000768+0x22a
         at Mono.Linker.Steps.UnreachableBlocksOptimizer.BodyReducer.RewriteBody() in illink.dll:token 0x6000766+0xa
         at Mono.Linker.Steps.UnreachableBlocksOptimizer.ProcessStack() in illink.dll:token 0x60004e9+0x102
         at Mono.Linker.Steps.UnreachableBlocksOptimizer.ProcessMethod(MethodDefinition method) in illink.dll:token 0x60004e5+0x44
         at Mono.Linker.Steps.MarkStep.ProcessMethod(MethodDefinition method, DependencyInfo& reason, Scope& scope) in illink.dll:token 0x600040e+0x9b
         at Mono.Linker.Steps.MarkStep.ProcessQueue() in illink.dll:token 0x6000399+0x24
         --- End of inner exception stack trace ---
         at Mono.Linker.Steps.MarkStep.ProcessQueue() in illink.dll:token 0x6000399+0xaa
         at Mono.Linker.Steps.MarkStep.ProcessPrimaryQueue() in illink.dll:token 0x6000396+0xa
         at Mono.Linker.Steps.MarkStep.Process() in illink.dll:token 0x6000393+0x0
         at Mono.Linker.Steps.MarkStep.Process(LinkContext context) in illink.dll:token 0x600038a+0x41
         at Mono.Linker.Pipeline.ProcessStep(LinkContext context, IStep step) in illink.dll:token 0x6000264+0x0
         at Mono.Linker.Pipeline.Process(LinkContext context) in illink.dll:token 0x6000263+0xf
         at Mono.Linker.Driver.Run(ILogger customLogger) in illink.dll:token 0x60000fd+0x20
      Optimizing assemblies for size, which may change the behavior of the app. Be sure to test after publishing. See: https://aka.ms/dotnet-illink
    /__w/1/s/.dotnet/sdk/6.0.100-preview.4.21255.9/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.ILLink.targets(80,5): error NETSDK1144: Optimizing assemblies for size failed. Optimization can be disabled by setting the PublishTrimmed property to false. [/__w/1/s/src/libraries/Microsoft.NETCore.Platforms/tests/Microsoft.NETCore.Platforms.Tests.csproj]
    \##[error].dotnet/sdk/6.0.100-preview.4.21255.9/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.ILLink.targets(80,5): error NETSDK1144: (NETCORE_ENGINEERING_TELEMETRY=Build) Optimizing assemblies for size failed. Optimization can be disabled by setting the PublishTrimmed property to false.
    ```
    radical committed Jun 22, 2021
    Configuration menu
    Copy the full SHA
    f3a45d7 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    38e4fb8 View commit details
    Browse the repository at this point in the history

Commits on Jun 23, 2021

  1. Configuration menu
    Copy the full SHA
    ad2255c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    e677e8a View commit details
    Browse the repository at this point in the history