Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge from dotnet/runtime #898

Merged
merged 99 commits into from
Apr 1, 2021
Merged

Conversation

jkotas
Copy link
Member

@jkotas jkotas commented Apr 1, 2021

No description provided.

jkoritzinsky and others added 30 commits March 27, 2021 00:11
* Handle the fact that on Android, more validation failures are treated as PartialChain when doing CMS signing.

* Fix copy-paste error for RSA SHA-512 PKCS1 padding

* Change condition so we have it in netstandard builds.
* Resolve remaining Xslt-related warnings

* Addressing final linker warning and adding annotations to ref assembly

* Update src/libraries/System.Private.Xml/src/System/Xml/Xsl/IlGen/GenerateHelper.cs

Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com>

* Update src/libraries/System.Private.Xml/src/System/Xml/Xsl/Runtime/XmlExtensionFunction.cs

Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com>

* Addressing PR feedback

* kept -> preserved

* Address remaining feedback

Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com>
When generating disasm for an SPMI replay, we can call
`eeGetMethodFullName` for callees for which we are missing data.
Fix `MethodContext::repGetArgNext` to raise an appropriate exception,
and not crash, when it encounters missing data.
* [mono] Add a separate lock for the method_to_dyn_method hashtable.

* Use the jit memory manager lock instead of the domain lock in the arm64 patch code.

* Remove usage of the domain lock from the interpreter code.

* Use a separate lock to protect the jit info table.

* Fix the build.
[main] Update dependencies from dotnet/arcade dotnet/xharness


 - reintroduce publishing branch workarounds. 

Required while dotnet/arcade#6987 is fixed

 - fix whitespace in publish-using-darc script

 - re-introduce publishing branch workaround

dotnet/arcade#698
* Add documentation on ILLink files

* Add link to ILLink-files from project-guidelines
…lContext (#49945)

* catch exceptions in callbacks from native code

* style update

* add traces for failures

* add comment
* Increase test coverage for Exception.TargetSite

* Apply suggestions from code review

Co-authored-by: Stephen Toub <stoub@microsoft.com>

Co-authored-by: Stephen Toub <stoub@microsoft.com>
- SslStream was holding onto a 4K byte[] after the handshake was complete. This was because the ArrayBuffer struct doesn't clear the local buffer field in dispose. This changes that.
- LoggerMessage.Define with overload to disable IsEnabled check
- Hoist skipEnabledCheck
The failures do not look jit-related. If they fail again please open another issue.
We should not blow up jitting just because edge counts are inconsistent.

Also, make sure we're not doing any guarded devirt if we disable pgo.

And, add an option JitEnablePgoRange to selectively enable use of PGO for
problem isolation.
Co-authored-by: Stephen Toub <stoub@microsoft.com>
* Update dependencies from https://github.com/mono/linker build 20210324.1

Microsoft.NET.ILLink.Tasks
 From Version 6.0.100-preview.2.21172.2 -> To Version 6.0.100-preview.2.21174.1

* Update dependencies from https://github.com/mono/linker build 20210326.1

Microsoft.NET.ILLink.Tasks
 From Version 6.0.100-preview.2.21172.2 -> To Version 6.0.100-preview.2.21176.1

* Update dependencies from https://github.com/mono/linker build 20210327.1

Microsoft.NET.ILLink.Tasks
 From Version 6.0.100-preview.2.21172.2 -> To Version 6.0.100-preview.2.21177.1

* Add missing annotations

* Always keep ValueType constructor

* Fix exceptions descriptors to match what native lookup expects

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Marek Safar <marek.safar@gmail.com>
It seems like we are allocating memory, but never releasing it.
…50357)

It allocates a delegate/closure for each counter group each time the loop runs.
* [wasm] Wasm.Build.Tests: add support for shared builds

- Essentially, we want to share builds wherever possible. Example cases:

    - Same build, but run with different hosts like v8/chrome/safari, as
      separate test runs
    - Same build but run with different command line arguments

- Sharing builds especially helps when we are AOT'ing, which is slow!

- This is done by caching the builds with the key:

    `public record BuildArgs(string ProjectName, string Config, bool AOT, string ProjectFileContents, string? ExtraBuildArgs);`

- Also, ` SharedBuildClassFixture` is added, so that the builds can be
  cleaned up after all the tests in a particular class have finished
  running.

- Each test run gets a randomly generated test id. This is used for
  creating:
  1. build paths, like `artifacts/bin/Wasm.Build.Tests/net6.0-Release/browser-wasm/xharness-output/logs/n1xwbqxi.ict`
  2. and the log for running with xharness, eg. for Chrome, are in
     `artifacts/bin/Wasm.Build.Tests/net6.0-Release/browser-wasm/xharness-output/logs/n1xwbqxi.ict/Chrome/`

- split `WasmBuildAppTest.cs` into : `BuildTestBase.cs`, and
  `MainWithArgsTests.cs`.

* [wasm] Wasm.Build.Tests: Add test for relinking with InvariantGlobalization

* [wasm] Wasm.Build.Tests - Check `CultureInfo` for invariant culture

.. tests. Code stolen from @maximlipin's dotnet/runtime#49204

* fix invariant+aot test

* [wasm] Fix the order of include paths

For AOT we generate `pinvoke-table.h` in the obj directory. But there is
one present in the runtime pack too.

In my earlier changes the order in which these were passed as include
search paths was changed from:

`"-I/runtime/pack/microsoft.netcore.app.runtime.browser-wasm/Release/runtimes/browser-wasm/native/include/wasm" "-Iartifacts/obj/mono/Wasm.Console.Sample/wasm/Release/browser-wasm/wasm/"`

.. which meant that the one from the runtime pack took precedence, and
got used. So, fix the order!

And change the property names to indicate where they are sourced from.

* [wasm] Only test with Release config on CI

* [wasm] Fallback to `dotnet xharness` if `XHARNESS_CLI_PATH` is not set.

The environment variable is set on helix. During local testing it can be
useful when using a locally built xharness.

* [wasm] fix invariant test - 'en-ES' -> 'es-ES'

* [wasm] RunWithEmSdkEnv: log the working directory also

* [wasm] Re-enable wasm build tests

* [wasm] Add regression test for issue #49588

* fix test

* [wasm] Cleanup, and add more tests

* Update tests to track wasm relinking being default in some cases

* Fix InvariantGlobalization to track change in wasm relinking defaults

* [wasm] Update emsdk check message to track changes

* [wasm] Update tests to track changes

* [wasm] Move Scenario=BuildWasmApps to be submitted first

TLDR;
- this might help with the job getting scheduled first, and thus having
  a chance at completing at the same time as others.

Reasoning:

The problem this is trying to fix is:

1. The helix step submits 3 jobs:
    a. library tests to be run with v8
    b. library tests to be run with browser (scenario=wasmtestonbrowser)
    c. Wasm.Build.Tests (scenario=buildwasmapps)

2. The 3 jobs, individually take roughly 30mins each
3. And they get submitted at roughly the same time
4. But .. the first two seem to complete earlier, and the 3rd one
   completes 25-30mins later.

The hypothesis is that all the machines might be busy processing the
200+ work items from each of the first two steps, and so
Wasm.Build.Tests get scheduled pretty late.

So, here we move that to be submitted first, in the hope that it would
be able to run in parallel with the other jobs.
* Convert all configuration options from REGUTIL to CLRConfig.

* Remove uses of REGUTIL outside of CLRConfig impl.
* versioning

* no need to support 2.0 bundle version

* enable compression

* fixes after rebase

* build fix for Unix

* Fix build with GCC

* disable a test temporarily to see what else breaks

* Add EnableCompression option to Bundler + PR feedback

* Couple more places should use version 6.0

* PR feedback (header versioning, more tests, fixed an assert)

* Suggestion from PR review

Co-authored-by: Vitek Karas <vitek.karas@microsoft.com>

* sorted usings

* should be bundle_major_version in two more places.

* More PR feedback

Co-authored-by: Vitek Karas <vitek.karas@microsoft.com>
Adjust the error code determination for asm diffs to not report
diffs if all the compilation failures were due to missing data
in the MCH files.

Update the SuperPMI end status line to report the count of MC
compilation failures due to missing data (and adjust parallel
SuperPMI to handle it as well).

As a result, with `superpmi.py asmdiffs` you won't see the message
"Asm diffs" if all the failures were due to missing data.
- Split browser from non-browser to avoid CA1416 suppressions, UnsupportedOSPlatform attributes, and PNSEs when used from browser
- Cache domain name and local addresses for use in IsLocal, along with clearing of the caches upon network change detection
- Removing redundant IPAddress.IsLoopback check that's covered by previous Uri.IsLoopback check
- Change IsMatchInBypassList to use string interpolation, such that the non-default port case will get better implicitly with C# 10 interpolated string improvements
vargaz and others added 25 commits March 31, 2021 13:21
… (), the name might be different at runtime if DISABLE_JIT is set. (#50482)
* Support private SuperPMI MCH file stores

In addition to the default Azure Storage SuperPMI location,
support private data stores via the new `-private_store` argument
to `replay` and `asmdiffs`. Private stores are typically file
system locations, or, on Windows, UNC paths.

There already exists the `-mch_files` argument, which allows you
to use any specified set of MCH files. However, when specifying
`-mch_files`, only those files are used and the normal stores are
ignored.

The private store mechanism is more convenient in some scenarios,
as it treats the specified private stores equivalently to the default
Azure Storage store.

In addition, both private stores and `-mch_files` handle downloading
ZIP files, un-ZIPping, and caching their contents.

Finally, a new top-level command `upload-private` is introduced to
help create private stores by specifying a set of MCH files to
use. It ZIP compresses the files to upload.

* Fix `-base_jit_path`

If `-base_jit_path` is specified on the command line, ensure it is a
full path before using it.
… (#50437)

- Build the concept of a cacheable resolution failure
- Plumb it through the Ecma type loader and the required public api surfaces
- Use it within the Mibc parser to avoid throwing
* Add option for building a test exe as single file

* Remove left over test

* Add target to exclude references from single-file

* First attempt at adding a CI job

* Opt-in specific libraries for single-file testing support

Start with System.Collections as all tests pass.

* Config testing using single-file in build.cmd

* Change yml suffix name to SingleFile

* Windows_NT_x64 -> windows_x64

* Fix for helix queueing

* Respond to host rename

* Change TargetOS to check for windows

* chmod test exe on linux

* Direct singlefilehost to the locally built copy

* Adjust singlefilehost copy+call

* Add .exe suffix on Windows

* Move libraries after hosts build to allow for libs.test to depend on hosts build

* Split up host and libs packaging and tests

* Move host packaging

* Move pretest up

* Move packages up as well

* Reorder libs pretest and libs.packages

* Add isSingleFile build parameter to limit Linux Helix jobs

* Typo

* Change conditional check

* Fix yml

* Fix yml

* Fix neq

* Fix subsets

* Typo

* Fix

* Adjust assert

* Include code from Michal to skip failing test

* Remove empty ItemGroup

* Update src/libraries/Common/tests/SingleFileTestRunner/SingleFileTestRunner.cs

Co-authored-by: Michal Strehovský <MichalStrehovsky@users.noreply.github.com>

* Update eng/testing/tests.singlefile.targets

Co-authored-by: Michal Strehovský <MichalStrehovsky@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Michal Strehovský <MichalStrehovsky@users.noreply.github.com>

* Use ProjectExclusions

* Update eng/testing/tests.singlefile.targets

Co-authored-by: Michal Strehovský <MichalStrehovsky@users.noreply.github.com>

* Revert changes

* Remove host build from tests

Co-authored-by: Michal Strehovský <MichalStrehovsky@users.noreply.github.com>
I previously disabled publishing all but Microsoft.NETCore.App.sfxproj,
which publishes the dotnet-runtime-internal package. This is almost, but
not quite, the right package. It doesn't contain the host dependencies,
although it does contain the right runtime files. The right package is
Microsoft.NETCore.App.bundleproj, which builds dotnet-runtime. Unfortunately,
I didn't catch this in manual testing because I think copying over the temp
bits reused the host dependencies from the previous install location. I've
wiped this from scratch and it looks fine now. (crossed fingers)
* Prohibit BF unless the app opts in

* Address feedback

* First unit test

* 2nd unit test. Checkpoint

* sq

* sq

* sq

* Address Levi's feedback

* sq

* Address nits

* Linker changes and tests

* sq

* sq

* Linker warnings ids

* Address comments

* Change trimming test so linker can detect a pattern

* sq

* Address comments

* sq

Co-authored-by: Eric Erhardt <eric.erhardt@microsoft.com>
* Fix usage of process_vm_readv in createdump; fallback to pread64

* No-op logging
* Reenable Android.Device_Emulator.Aot.Test.csproj

Closes dotnet/runtime#49757

* Update tests.proj
…ess dotnet/llvm-project (#50416)

[main] Update dependencies from dotnet/arcade dotnet/icu dotnet/xharness dotnet/llvm-project


 - Revert publishing branch changes

dotnet/arcade#6987

 - reintroduce publishing branch workaround

dotnet/arcade#6987
And sprinkle some "static" around to help avoid this in the future.
* Big-endian fix: Accessing OBJECTHEADER in ImageConverter

The GetBitmapStream routine in ImageConverter is supposed to
skip an embedded OBJECTHEADER structure.  However, this
structure needs to be read in little-endian mode, not in
native byte order.

Fixes the ImageConverterTest.ImageWithOleHeader test on
big-endian systems.

* Update src/libraries/System.Drawing.Common/src/System/Drawing/ImageConverter.cs

Co-authored-by: Stephen Toub <stoub@microsoft.com>
…0.4 (#50481)

Microsoft.NET.ILLink.Tasks
 From Version 6.0.100-preview.2.21180.1 -> To Version 6.0.100-preview.2.21180.4

Co-authored-by: dotnet-maestro[bot] <dotnet-maestro[bot]@users.noreply.github.com>
… (#50509)

It is skipped based on flakiness/slowness on a Checked/Debug runtime, which we are not currently
using for Android tests. Put the test suite under Android test ProjectExclusions instead.
- Add Exception._remoteStackTrace
@jkotas jkotas merged commit f1a3fc1 into dotnet:feature/NativeAOT Apr 1, 2021
@jkotas jkotas deleted the merge-master branch April 1, 2021 04:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.