Wasm: Reenable the CI for Wasm

[yowl]

Seems like the only way to test this is to create a PR and keep editing it until it works....

First commit just to see if I'm on the right track and can add a Wasm platform to the matrix and see what it runs. Have commented out other platforms, I will redo the push squashing commits when it works.

What's required is to install emscripten 1.39.4 (or later) activate it, set up the env, then run build wasm {configuration} ... I think.

Have changed the running of test from Firefox to Node (which I use from the Emscripten source) with the hope that will alleviate some of the timeout problems that used to occur in Jenkins.

Not sure I've got the matrix logic in the right place, but couldn't see somewhere better.

build wasm release is not in as its broken - #8012

There was an issue with the SSL certificate for googleapis.com and downloading from there using python. Emscripten does this to start its install. The error was intermittent but if it occurred, retrying did not help. I understand python (later versions of v2 and v3) on windows use the machine certificate store, so my theory is that depending on the agent the store is different and it either fails or succeeds, just a theory. I added some powershell to do the same download and that always seems to succeed, and if the python download failed, and the powershell download is performed, then the python download succeeds. So it could be that the powershell download using System.Net.Security.SslStream is changing the machine certs, at least that's my best theory, and that's the approach here which has succeed 6 times consequitively. Upgrading python is not necessary so I've left it out. The native tools download of python is not used by the emscripten install as it just pulls python from the path, and the native tools download does not change the path.

[MichalStrehovsky]

The buildpipeline directory controls official builds and package publishing. For CI, you only need to worry about the YAML files in eng.

[yowl]

having various problems . One might be xunit and just recording this link in case its relevant
dotnet/standard#481

[MichalStrehovsky]

I'm wondering if the RyuJIT restore issue you're getting now is fallout from #8001 (I touched the project). Not sure what RIDs we set up when restoring the RyuJIT.depproj. Maybe master is broken for wasm right now in general?

[yowl]

@MichalStrehovsky you suspect https://github.com/dotnet/corert/pull/8001/files#diff-d7bea50ef95c95c2ed222c77540b7f8a ?

It does seem to be taking for ever now when build wasm is run on a clean clone. It seems to get stuck on "dotnet.exe" exec E:\GitHub\corerttest\Tools\dotnetcli\sdk\2.1.510\MSBuild.dll -maxcpucount -verbosity:m E:\GitHub\corerttest\buildscripts\..\build.proj /nologo /t:Restore /flp:v=normal;LogFile=E:\GitHub\corerttest\buildscripts\..\bin\Logs\msbuild_WebAssembly__wasm__Debug.log /p:NuPkgRid=win-x64 /maxcpucount /p:OSGroup=WebAssembly /p:Configuration=Debug /p:Platform=wasm which doesn't appear to have hung, in that in ProcExp, the threads tab is moving but its only clocking a tiny amount of cpu, e.g. 0.02 %

Virtual memory of this dotnet.exe looks like its 2TB which may or may not be interesting:

[yowl]

A red herring perhaps, its just errored with

  Failed to download package 'runtime.win-x64.Microsoft.Private.CoreFx.NETCoreApp.5.0.0-alpha.1.19563.6' from 'https://dotnetfeed.blob.core.windows.net/dotnet-core/flatcontainer/runtime.win-x64.microsoft.private.corefx.netcoreapp/5.0.0-alpha.1.19563.6/runtime.win-x64.microsoft.private.corefx.netcoreapp.5.0.0-alpha.1.19563.6.nupkg'.
  Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
    An existing connection was forcibly closed by the remote host

Although it did then go on to write

E:\GitHub\corerttest\Tools\depProj.targets(83,5): error : Error no assets were resolved from NuGet packages. [E:\GitHub\corerttest\src\ILCompiler\RyuJIT\RyuJIT.depproj]

If I run build wasm again without cleaning the repo, it fails immediately with the RyuJIT.depproj problem.

[yowl]

I tried again today on a clean repo and the HTTP error was just transient and has gone, but the RyuJIT problem persists.

E:\GitHub\corerttest\Tools\depProj.targets(83,5): error : Error no assets were resolved from NuGet packages. [E:\GitHub\corerttest\src\ILCompiler\RyuJIT\RyuJIT.depproj]

[MichalStrehovsky]

I think it's because I used TargetsUnix in the RyuJIT.depproj to decide whether we want to grab clrjit or libclrjit. Can you try switching that to Condition="'$(OSEnvironment)'!='Windows_NT'" (and ==, respectively)?

[yowl]

@MichalStrehovsky Yes, that fixes it, thanks

[yowl]

Want me to create a separate PR for that, would seem like a good idea?

[MichalStrehovsky]

Want me to create a separate PR for that, would seem like a good idea?

Yes, thanks. And sorry for breaking it in the first place.

[yowl]

When this downloads emscripten it just puts it in the root, should I move that to Tools?

[yowl]

WIP again, left some hard coded stuff

[yowl]

/azp run

[azure-pipelines]

Commenter does not have sufficient privileges for PR 7998 in repo dotnet/corert

[yowl]

I've finished with this and I ran 5 consecutive tests so I think this approach is going to work. I only have this empirical evidence that this is ok, if we put it in and it starts failing I can always pull it again?

[MichalStrehovsky]

Looks good otherwise, thanks!

[MichalStrehovsky]

Could you move the files you added in eng/common to eng?

CoreRT repo is not on Arcade yet, but this layout is kind of a prep for that. The runtime repo has explanation: https://github.com/dotnet/runtime/tree/master/eng/common#dont-touch-this-folder

[filipnavara]

Thanks, @yowl!