AdvSimd support for System.Text.Unicode.Utf8Utility.TranscodeToUtf8 #39041

carlossanlop · 2020-07-09T22:34:01Z

Contributes to #35035

Adds AdvSimd support for
System.Text.Unicode.Utf8Utility.TranscodeToUtf8()
inside the file
runtime\src\libraries\System.Private.CoreLib\src\System\Text\Unicode\Utf8Utility.Transcoding.cs

I've been having difficulties testing this in my ARM device so I want to analyze the CI results.
I manually executed an additional "Libraries Test Run" pipeline to ensure arm64 is run in all platforms.

Dotnet-GitSync-Bot · 2020-07-09T22:34:04Z

I couldn't figure out the best area label to add to this PR. If you have write-permissions please help me learn by adding exactly one area label.

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs

ghost · 2020-07-09T22:39:32Z

Tagging subscribers to this area: @tarekgh, @krwq
Notify danmosemsft if you want to be subscribed.

ghost · 2020-07-10T16:51:05Z

Tagging subscribers to this area: @tannergooding
Notify danmosemsft if you want to be subscribed.

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs

Add AdvSimd equivalent operation to TestZ.

carlossanlop · 2020-07-14T06:27:49Z

The following unit tests hit the code added in this PR:

System.Runtime:

System.Text.Unicode.Tests.Utf8Tests.ToBytes_WithLargeValidBuffers

System.Text.Encoding:

System.Text.Encodings.Tests.EncodingVirtualTests.Encode
System.Text.Tests.EncoderConvert2.EncoderConvertRandomCharArray
System.Text.Tests.EncoderConvert2.EncoderUTF8ConvertMixedASCIIUnicodeCharArrayPartial
System.Text.Tests.EncoderConvert2.EncoderUTF8ConvertUnicodeCharArrayPartial
System.Text.Tests.EncoderGetBytes2.EncoderGetBytesMixedInput
System.Text.Tests.EncoderGetBytes2.EncoderGetBytesRandomInput
System.Text.Tests.EncoderGetBytes2.EncoderGetBytesUnicodeInput
System.Text.Tests.UTF8EncodingEncode.GetBytes_ValidASCIIUnicode

System.Utf8String.Experimental:

System.Tests.Utf8StringTests.Ctor_CharArrayOffset_ValidData_ReturnsAsUtf8

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs

…ss and use MaxPairwise instead

carlossanlop · 2020-07-16T19:17:29Z

Update:

I finally got WSL in my ARM device to execute the unit tests.

The unit tests that hit these changes are listed here: #39041 (comment)

After I ran built everything and ran the unit tests in my other PR, I switch to the branch from this PR, and since both of my branches had the same base commit, I only had to quickly build clr.corelib+libs.pretest, then run the unit tests.

The System.Text.Encoding unit tests hit some new failures which I'm now investigating:

System.Text.Encoding

calope@calopearm:~/runtime$ dotnet msbuild /t:Test /p:CoreCLRConfiguration=Release /p:RuntimeConfiguration=Debug src/libraries/System.Text.Encoding/tests/System.Text.Encoding.Tests.csproj 
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  TestUtilities -> /home/calope/runtime/artifacts/bin/TestUtilities/net5.0-Debug/TestUtilities.dll
  System.Text.Encoding.Tests -> /home/calope/runtime/artifacts/bin/System.Text.Encoding.Tests/net5.0-Debug/System.Text.Encoding.Tests.dll
  ----- start Thu Jul 16 12:09:43 PDT 2020 =============== To repro directly: =====================================================
  pushd /home/calope/runtime/artifacts/bin/System.Text.Encoding.Tests/net5.0-Debug
  /home/calope/runtime/artifacts/bin/testhost/net5.0-Linux-Debug-arm64/dotnet exec --runtimeconfig System.Text.Encoding.Tests.runtimeconfig.json --depsfile System.Text.Encoding.Tests.deps.json xunit.console.dll System.Text.Encoding.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing 
  popd
  ===========================================================================================================
  ~/runtime/artifacts/bin/System.Text.Encoding.Tests/net5.0-Debug ~/runtime/src/libraries/System.Text.Encoding/tests
    Discovering: System.Text.Encoding.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Text.Encoding.Tests (found 307 test cases)
    Starting:    System.Text.Encoding.Tests (parallel test collections = on, max threads = 8)
      System.Text.Encodings.Tests.EncodingVirtualTests.Encode(chars: "¤ÐaR|{AnGe��", index: 0, count: 12, expected: [194, 164, 195, 144, 97, ...]) [FAIL]
        Assert.Equal() Failure
        Expected: 194
        Actual:   163
        Stack Trace:
          /home/calope/runtime/src/libraries/System.Text.Encoding/tests/EncodingTestHelpers.cs(125,0): at System.Text.Tests.EncodingHelpers.VerifyGetBytes(Byte[] bytes, Int32 byteIndex, Int32 byteCount, Byte[] originalBytes, Byte[] expectedBytes)
          /home/calope/runtime/src/libraries/System.Text.Encoding/tests/EncodingTestHelpers.cs(79,0): at System.Text.Tests.EncodingHelpers.VerifyGetBytes(Encoding encoding, String source, Int32 index, Int32 count, Byte[] bytes, Int32 byteIndex, Byte[] expectedBytes)
          /home/calope/runtime/src/libraries/System.Text.Encoding/tests/EncodingTestHelpers.cs(47,0): at System.Text.Tests.EncodingHelpers.GetBytes(Encoding encoding, String source, Int32 index, Int32 count, Byte[] expectedBytes)
          /home/calope/runtime/src/libraries/System.Text.Encoding/tests/EncodingTestHelpers.cs(13,0): at System.Text.Tests.EncodingHelpers.Encode(Encoding encoding, String chars, Int32 index, Int32 count, Byte[] expected)
          /home/calope/runtime/src/libraries/System.Text.Encoding/tests/Encoding/EncodingVirtualTests.cs(16,0): at System.Text.Encodings.Tests.EncodingVirtualTests.Encode(String chars, Int32 index, Int32 count, Byte[] expected)
        System.PlatformNotSupportedException : Operation is not supported on this platform.
      System.Text.Tests.EncoderConvert2.EncoderConvertRandomCharArray(encoder: EncoderNLS { Encoding = UTF8EncodingSealed { BodyName = "utf-8", CodePage = 65001, DecoderFallback = DecoderReplacementFallback { ... }, EncoderFallback = EncoderReplacementFallback { ... }, EncodingName = "Unicode (UTF-8)", ... }, Fallback = EncoderReplacementFallback { DefaultString = "�", MaxCharCount = 1 }, FallbackBuffer = EncoderReplacementFallbackBuffer { Remaining = 0 }, MustFlush = False }) [FAIL]
      [FATAL ERROR] System.PlatformNotSupportedException
        Stack Trace:
          /home/calope/runtime/src/libraries/System.Private.CoreLib/src/System/Runtime/Intrinsics/X86/Sse2.PlatformNotSupported.cs(44,0): at System.Runtime.Intrinsics.X86.Sse2.X64.ConvertToUInt64(Vector128`1 value)
          /home/calope/runtime/src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs(1016,0): at System.Text.Unicode.Utf8Utility.TranscodeToUtf8(Char* pInputBuffer, Int32 inputLength, Byte* pOutputBuffer, Int32 outputBytesRemaining, Char*& pInputBufferRemaining, Byte*& pOutputBufferRemaining)
          /home/calope/runtime/src/libraries/System.Private.CoreLib/src/System/Text/Encoding.Internal.cs(417,0): at System.Text.Encoding.GetBytes(Char* pChars, Int32 charCount, Byte* pBytes, Int32 byteCount, EncoderNLS encoder)
          /home/calope/runtime/src/libraries/System.Private.CoreLib/src/System/Text/EncoderNLS.cs(131,0): at System.Text.EncoderNLS.GetBytes(Char* chars, Int32 charCount, Byte* bytes, Int32 byteCount, Boolean flush)
          /home/calope/runtime/src/libraries/System.Private.CoreLib/src/System/Text/EncoderNLS.cs(113,0): at System.Text.EncoderNLS.GetBytes(Char[] chars, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex, Boolean flush)
          /home/calope/runtime/src/libraries/System.Private.CoreLib/src/System/IO/StreamWriter.cs(278,0): at System.IO.StreamWriter.Flush(Boolean flushStream, Boolean flushEncoder)
          /home/calope/runtime/src/libraries/System.Private.CoreLib/src/System/IO/StreamWriter.cs(458,0): at System.IO.StreamWriter.WriteLine(String value)
          /home/calope/runtime/src/libraries/System.Private.CoreLib/src/System/IO/TextWriter.cs(880,0): at System.IO.TextWriter.SyncTextWriter.WriteLine(String value)
          C:\Dev\xunit\xunit\src\xunit.runner.utility\Extensions\MessageSinkMessageExtensions.cs(44,0): at MessageSinkMessageExtensions.Dispatch[TMessage](IMessageSinkMessage message, HashSet`1 messageTypes, MessageHandler`1 callback)
      System.Text.Tests.UTF8EncodingEncode.Encode(chars: "¤ÐaR|{AnGe��", index: 0, count: 12, expected: [194, 164, 195, 144, 97, ...]) [FAIL]
        System.PlatformNotSupportedException : Operation is not supported on this platform.
        Stack Trace:
          /home/calope/runtime/src/libraries/System.Private.CoreLib/src/System/Runtime/Intrinsics/X86/Sse2.PlatformNotSupported.cs(44,0): at System.Runtime.Intrinsics.X86.Sse2.X64.ConvertToUInt64(Vector128`1 value)
          /home/calope/runtime/src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs(1016,0): at System.Text.Unicode.Utf8Utility.TranscodeToUtf8(Char* pInputBuffer, Int32 inputLength, Byte* pOutputBuffer, Int32 outputBytesRemaining, Char*& pInputBufferRemaining, Byte*& pOutputBufferRemaining)
          /home/calope/runtime/src/libraries/System.Private.CoreLib/src/System/Text/UTF8Encoding.cs(291,0): at System.Text.UTF8Encoding.GetBytes(String s, Int32 charIndex, Int32 charCount, Byte[] bytes, Int32 byteIndex)
          /home/calope/runtime/src/libraries/System.Private.CoreLib/src/System/Text/Encoding.cs(668,0): at System.Text.Encoding.GetBytes(String s)
          /home/calope/runtime/src/libraries/System.Text.Encoding/tests/EncodingTestHelpers.cs(78,0): at System.Text.Tests.EncodingHelpers.VerifyGetBytes(Encoding encoding, String source, Int32 index, Int32 count, Byte[] bytes, Int32 byteIndex, Byte[] expectedBytes)
          /home/calope/runtime/src/libraries/System.Text.Encoding/tests/EncodingTestHelpers.cs(47,0): at System.Text.Tests.EncodingHelpers.GetBytes(Encoding encoding, String source, Int32 index, Int32 count, Byte[] expectedBytes)
          /home/calope/runtime/src/libraries/System.Text.Encoding/tests/EncodingTestHelpers.cs(13,0): at System.Text.Tests.EncodingHelpers.Encode(Encoding encoding, String chars, Int32 index, Int32 count, Byte[] expected)
          /home/calope/runtime/src/libraries/System.Text.Encoding/tests/UTF8Encoding/UTF8EncodingEncode.cs(121,0): at System.Text.Tests.UTF8EncodingEncode.Encode(String chars, Int32 index, Int32 count, Byte[] expected)
    Finished:    System.Text.Encoding.Tests
  === TEST EXECUTION SUMMARY ===
    System.Text.Encoding.Tests  Total: 11039, Errors: 1, Failed: 3, Skipped: 0, Time: 26.068s
  ~/runtime/src/libraries/System.Text.Encoding/tests
  ----- end Thu Jul 16 12:10:11 PDT 2020 ----- exit code 1 ----------------------------------------------------------
  Looking around for any Linux dump...
  ... found no dump in /home/calope/runtime/artifacts/bin/System.Text.Encoding.Tests/net5.0-Debug
/home/calope/runtime/eng/testing/tests.targets(118,5): error : One or more tests failed while running tests from 'System.Text.Encoding.Tests'. Please check /home/calope/runtime/artifacts/bin/System.Text.Encoding.Tests/net5.0-Debug/testResults.xml for details! [/home/calope/runtime/src/libraries/System.Text.Encoding/tests/System.Text.Encoding.Tests.csproj]

kunalspathak · 2020-07-16T20:55:46Z

                    possibleNonAsciiQWord = Sse2.X64.ConvertToUInt64(utf16Data.AsUInt64());

ah, missing check.

Refers to: src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs:1017 in 191ec1f. [](commit_id = 191ec1f, deletion_comment = False)

kunalspathak · 2020-07-16T21:50:43Z

@echesakovMSFT - What should be an equivalent of Sse2.X64.ConvertToUInt64(utf16Data.AsUInt64()). I believe it should be utf16Data.AsUInt64().ToLower(). Can you please confirm?

echesakov · 2020-07-16T22:00:26Z

What should be an equivalent of Sse2.X64.ConvertToUInt64(utf16Data.AsUInt64()). I believe it should be utf16Data.AsUInt64().ToLower(). Can you please confirm?

@kunalspathak No .GetLower() returns Vector64<ulong> while ConvertToUInt64 returns ulong.

What are you trying to do?

If you want to store the lower 8 bytes to memory then

StoreSelectedScalar(ptr, utf16Data.AsUInt64(), 0);

If you need the value in a GP registers then

utf16Data.AsUInt64().GetElement(0)

or

AdvSimd.Extract(utf16Data.AsUInt64(), 0)

kunalspathak · 2020-07-16T22:04:45Z

What are you trying to do?

Just an equivalent of that SSE2 instruction. Didn't realize it returns ulong. In that case , probably what you suggested.

possibleNonAsciiQWord = utf16Data.AsUInt64().GetElement(0);
// or
possibleNonAsciiQWord = utf16Data.AsUInt64().ToScalar();

carlossanlop · 2020-07-17T00:08:48Z

Fixed the unit test failures with the last two commits:

System.Runtime

calope@calopearm:~/runtime$ dotnet msbuild /t:Test /p:CoreCLRConfiguration=Release /p:RuntimeConfiguration=Debug src/libraries/System.Runtime/tests/System.Runtime.Tests.csproj 
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

TestUtilities -> /home/calope/runtime/artifacts/bin/TestUtilities/net5.0-Debug/TestUtilities.dll
TestUtilities.Unicode -> /home/calope/runtime/artifacts/bin/TestUtilities.Unicode/net5.0-Debug/TestUtilities.Unicode.dll
TestCollectibleAssembly -> /home/calope/runtime/artifacts/bin/TestCollectibleAssembly/netstandard2.0-Debug/TestCollectibleAssembly.dll
TestLoadAssembly -> /home/calope/runtime/artifacts/bin/TestLoadAssembly/netstandard2.0-Debug/TestLoadAssembly.dll
System.Reflection.TestModule -> /home/calope/runtime/artifacts/bin/System.Reflection.TestModule/netstandard2.0-Debug/System.Reflection.TestModule.dll
System.TestStructs -> /home/calope/runtime/artifacts/bin/System.TestStructs/netstandard2.0-Debug/System.TestStructs.dll
System.Runtime.Tests -> /home/calope/runtime/artifacts/bin/System.Runtime.Tests/net5.0-Unix-Debug/System.Runtime.Tests.dll
----- start Thu Jul 16 16:52:09 PDT 2020 =============== To repro directly: =====================================================
pushd /home/calope/runtime/artifacts/bin/System.Runtime.Tests/net5.0-Unix-Debug
/home/calope/runtime/artifacts/bin/testhost/net5.0-Linux-Debug-arm64/dotnet exec --runtimeconfig System.Runtime.Tests.runtimeconfig.json --depsfile System.Runtime.Tests.deps.json xunit.console.dll System.Runtime.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
~/runtime/artifacts/bin/System.Runtime.Tests/net5.0-Unix-Debug ~/runtime/src/libraries/System.Runtime/tests
    Discovering: System.Runtime.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Runtime.Tests (found 5478 of 5522 test cases)
    Starting:    System.Runtime.Tests (parallel test collections = on, max threads = 8)
    System.Tests.DateTimeOffsetTests.ToLocalTime_MaxValue [SKIP]
        Condition(s) not met: "IsMaxValuePositiveLocalOffset"
    System.Tests.ArgIteratorTests.ArgIterator_GetNextArgType [SKIP]
        Condition(s) not met: "IsArgIteratorSupported"
    System.Tests.ArgIteratorTests.ArgIterator_GetRemainingCount_GetNextArg [SKIP]
        Condition(s) not met: "IsArgIteratorSupported"
    System.Tests.DateTimeOffsetTests.ToLocalTime_Ambiguous [SKIP]
        Condition(s) not met: "IsPacificTime"
    System.Tests.StringTests.IndexOf_SingleLetter(s: "Hello", target: '\0', startIndex: 0, count: 5, expected: -1) [SKIP]
        Target \0 is not supported in ICU
    System.Tests.StringTests.LastIndexOf_NullInStrings [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
    System.Tests.StringTests.IndexOf_NullInStrings [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
    System.Tests.StringTests.EndsWith_NullInStrings_NonOrdinal [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
    System.Tests.StringTests.StartsWith_NullInStrings_NonOrdinal [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
    System.Tests.StringTests.EndsWith_NullInStrings [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
    System.Tests.StringTests.StartsWith_NullInStrings [SKIP]
        Condition(s) not met: "IsNlsGlobalization"
    System.Tests.ExceptionTests.ThrowStatementDoesNotResetExceptionStackLineOtherMethod [SKIP]
        Condition(s) not met: "IsNotArm64Process"
    Finished:    System.Runtime.Tests
=== TEST EXECUTION SUMMARY ===
    System.Runtime.Tests  Total: 35603, Errors: 0, Failed: 0, Skipped: 12, Time: 29.069s
~/runtime/src/libraries/System.Runtime/tests
----- end Thu Jul 16 16:52:42 PDT 2020 ----- exit code 0 ----------------------------------------------------------
exit code 0 means Exited Successfully

System.Text.Encoding

calope@calopearm:~/runtime$ dotnet msbuild /t:Test /p:CoreCLRConfiguration=Release /p:RuntimeConfiguration=Debug src/libraries/System.Text.Encoding/tests/System.Text.Encoding.Tests.csproj 
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

TestUtilities -> /home/calope/runtime/artifacts/bin/TestUtilities/net5.0-Debug/TestUtilities.dll
System.Text.Encoding.Tests -> /home/calope/runtime/artifacts/bin/System.Text.Encoding.Tests/net5.0-Debug/System.Text.Encoding.Tests.dll
----- start Thu Jul 16 16:53:02 PDT 2020 =============== To repro directly: =====================================================
pushd /home/calope/runtime/artifacts/bin/System.Text.Encoding.Tests/net5.0-Debug
/home/calope/runtime/artifacts/bin/testhost/net5.0-Linux-Debug-arm64/dotnet exec --runtimeconfig System.Text.Encoding.Tests.runtimeconfig.json --depsfile System.Text.Encoding.Tests.deps.json xunit.console.dll System.Text.Encoding.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
~/runtime/artifacts/bin/System.Text.Encoding.Tests/net5.0-Debug ~/runtime/src/libraries/System.Text.Encoding/tests
    Discovering: System.Text.Encoding.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Text.Encoding.Tests (found 307 test cases)
    Starting:    System.Text.Encoding.Tests (parallel test collections = on, max threads = 8)
    Finished:    System.Text.Encoding.Tests
=== TEST EXECUTION SUMMARY ===
    System.Text.Encoding.Tests  Total: 11039, Errors: 0, Failed: 0, Skipped: 0, Time: 33.524s
~/runtime/src/libraries/System.Text.Encoding/tests
----- end Thu Jul 16 16:53:39 PDT 2020 ----- exit code 0 ----------------------------------------------------------
exit code 0 means Exited Successfully

System.Utf8String.Experimental

calope@calopearm:~/runtime$ dotnet msbuild /t:Test /p:CoreCLRConfiguration=Release /p:RuntimeConfiguration=Debug src/libraries/System.Utf8String.Experimental/tests/System.Utf8String.Experimental.Tests.csproj 
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

TestUtilities -> /home/calope/runtime/artifacts/bin/TestUtilities/net5.0-Debug/TestUtilities.dll
System.Utf8String.Experimental.Tests -> /home/calope/runtime/artifacts/bin/System.Utf8String.Experimental.Tests/net5.0-Debug/System.Utf8String.Experimental.Tests.dll
----- start Thu Jul 16 16:56:16 PDT 2020 =============== To repro directly: =====================================================
pushd /home/calope/runtime/artifacts/bin/System.Utf8String.Experimental.Tests/net5.0-Debug
/home/calope/runtime/artifacts/bin/testhost/net5.0-Linux-Debug-arm64/dotnet exec --runtimeconfig System.Utf8String.Experimental.Tests.runtimeconfig.json --depsfile System.Utf8String.Experimental.Tests.deps.json xunit.console.dll System.Utf8String.Experimental.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
~/runtime/artifacts/bin/System.Utf8String.Experimental.Tests/net5.0-Debug ~/runtime/src/libraries/System.Utf8String.Experimental/tests
    Discovering: System.Utf8String.Experimental.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Utf8String.Experimental.Tests (found 179 of 188 test cases)
    Starting:    System.Utf8String.Experimental.Tests (parallel test collections = on, max threads = 8)
    Finished:    System.Utf8String.Experimental.Tests
=== TEST EXECUTION SUMMARY ===
    System.Utf8String.Experimental.Tests  Total: 380, Errors: 0, Failed: 0, Skipped: 0, Time: 0.803s
~/runtime/src/libraries/System.Utf8String.Experimental/tests
----- end Thu Jul 16 16:56:18 PDT 2020 ----- exit code 0 ----------------------------------------------------------
exit code 0 means Exited Successfully

carlossanlop · 2020-07-17T03:42:59Z

The Alpine ARM64 leg finally finished and it hit a bunch of UTF8 encoding errors, which I will investigate:
https://dev.azure.com/dnceng/public/_build/results?buildId=734549&view=logs&j=ee1d0ee6-ff1e-56e7-4a8c-87d1cd9cc235&t=6282fd0a-028a-54e0-ae3d-9a2e13527062

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs

carlossanlop · 2020-07-18T00:30:43Z

@echesakovMSFT the tests are passing after applying your suggestion. Thank you!

System.Text.Json

calope@calopearm:~/runtime$ dotnet msbuild /t:Test /p:Configuration=Release /p:TargetArchitecture=arm64 src/libraries/System.Text.Json/tests/System.Text.Json.Tests.csproj 
Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  TestUtilities -> /home/calope/runtime/artifacts/bin/TestUtilities/net5.0-Release/TestUtilities.dll
  System.Text.Json.Tests -> /home/calope/runtime/artifacts/bin/System.Text.Json.Tests/net5.0-Release/System.Text.Json.Tests.dll
  ----- start Fri Jul 17 17:23:51 PDT 2020 =============== To repro directly: =====================================================
  pushd /home/calope/runtime/artifacts/bin/System.Text.Json.Tests/net5.0-Release
  /home/calope/runtime/artifacts/bin/testhost/net5.0-Linux-Release-arm64/dotnet exec --runtimeconfig System.Text.Json.Tests.runtimeconfig.json --depsfile System.Text.Json.Tests.deps.json xunit.console.dll System.Text.Json.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing 
  popd
  ===========================================================================================================
  ~/runtime/artifacts/bin/System.Text.Json.Tests/net5.0-Release ~/runtime/src/libraries/System.Text.Json/tests
    Discovering: System.Text.Json.Tests (method display = ClassAndMethod, method display options = None)
    Discovered:  System.Text.Json.Tests (found 1957 of 1991 test cases)
    Starting:    System.Text.Json.Tests (parallel test collections = on, max threads = 8)
    Finished:    System.Text.Json.Tests
  === TEST EXECUTION SUMMARY ===
    System.Text.Json.Tests  Total: 8511, Errors: 0, Failed: 0, Skipped: 0, Time: 80.258s
  ~/runtime/src/libraries/System.Text.Json/tests
  ----- end Fri Jul 17 17:25:13 PDT 2020 ----- exit code 0 ----------------------------------------------------------
  exit code 0 means Exited Successfully

System.Formats.Asn1

``` calope@calopearm:~/runtime$ dotnet msbuild /t:Test /p:Configuration=Release /p:TargetArchitecture=arm64 src/libraries/System.Formats.Asn1/tests/System.Formats.Asn1.Tests.csproj Microsoft (R) Build Engine version 16.7.0-preview-20309-02+d6862eb21 for .NET Copyright (C) Microsoft Corporation. All rights reserved.

TestUtilities -> /home/calope/runtime/artifacts/bin/TestUtilities/net5.0-Release/TestUtilities.dll
System.Formats.Asn1.Tests -> /home/calope/runtime/artifacts/bin/System.Formats.Asn1.Tests/net5.0-Release/System.Formats.Asn1.Tests.dll
----- start Fri Jul 17 17:28:40 PDT 2020 =============== To repro directly: =====================================================
pushd /home/calope/runtime/artifacts/bin/System.Formats.Asn1.Tests/net5.0-Release
/home/calope/runtime/artifacts/bin/testhost/net5.0-Linux-Release-arm64/dotnet exec --runtimeconfig System.Formats.Asn1.Tests.runtimeconfig.json --depsfile System.Formats.Asn1.Tests.deps.json xunit.console.dll System.Formats.Asn1.Tests.dll -xml testResults.xml -nologo -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================
~/runtime/artifacts/bin/System.Formats.Asn1.Tests/net5.0-Release ~/runtime/src/libraries/System.Formats.Asn1/tests
  Discovering: System.Formats.Asn1.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Formats.Asn1.Tests (found 527 test cases)
  Starting:    System.Formats.Asn1.Tests (parallel test collections = on, max threads = 8)
  Finished:    System.Formats.Asn1.Tests
=== TEST EXECUTION SUMMARY ===
  System.Formats.Asn1.Tests  Total: 3418, Errors: 0, Failed: 0, Skipped: 0, Time: 1.853s
~/runtime/src/libraries/System.Formats.Asn1/tests
----- end Fri Jul 17 17:28:44 PDT 2020 ----- exit code 0 ----------------------------------------------------------
exit code 0 means Exited Successfully

</details>

carlossanlop · 2020-07-18T00:33:11Z

We had a conversation offline and we decided we will collect the perf data after merging. @kunalspathak @jeffhandley

kunalspathak · 2020-07-18T00:34:07Z

We had a conversation offline and we decided we will collect the perf data after merging. @kunalspathak @jeffhandley

@BruceForstall , @JulieLeeMSFT

GrabYourPitchforks · 2020-07-18T07:35:47Z

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs

+                            if (AdvSimd.IsSupported)
+                            {
+                                Vector64<byte> lower = AdvSimd.ExtractNarrowingSaturateUnsignedLower(utf16Data);
+                                AdvSimd.StoreSelectedScalar((uint*)pOutputBuffer, lower.AsUInt32(), 0);


Should comment on this line that (uint*)pOutputBuffer is unaligned.

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs

GrabYourPitchforks · 2020-07-18T07:40:38Z

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs

+                            if (AdvSimd.IsSupported)
+                            {
+                                Vector64<byte> lower = AdvSimd.ExtractNarrowingSaturateUnsignedLower(utf16Data);
+                                AdvSimd.StoreSelectedScalar((uint*)pOutputBuffer, lower.AsUInt32(), 0);


Please add a comment saying (uint*)pOutputBuffer is unaligned.

If that's ok, I can add this to a list of TODOs for the next PR we open for intrinsics changes (there are things that the FC may have to address later). The CI is taking a really long time and I would like to get this merged today. I would like to avoid resetting the CI.

carlossanlop · 2020-07-20T17:51:21Z

I posted the perf results in this comment: #39050 (comment)
It includes a comparison of SSE2 turned off vs on.
It shows the benchmarks run against each PR, and also the benchmarks run against all 3 PRs merged together.

carlossanlop · 2020-07-20T19:30:54Z

I think this is ready, @kunalspathak @echesakovMSFT @GrabYourPitchforks I already have two sign-offs. I'll wait one more hour in case you have any final comments, otherwise I'll merge.

Note that the CI failure is unrelated to my change. I ran that CI leg multiple times and it's happening consistently and is pre-existing. I also found it in other PRs.

Libraries build Windows_NT allConfigurations x64 Release

Results: https://dev.azure.com/dnceng/public/_build/results?buildId=736504&view=logs&j=e87d290e-7c43-5717-098e-deb8643034d5&t=699159aa-4b47-567c-6f4b-ee557432ed14

F:\workspace\_work\1\s\src\libraries\pkg\test\testPackages.proj(217,5): error : Validation Failed: Microsoft.Extensions.DependencyModel is harvesting assets from package version 3.1.5 which is not the latest for that package release. Latest package version from that release is 3.1.6. In order to fix this, run `dotnet msbuild Microsoft.Extensions.DependencyModel.pkgproj /t:UpdateHarvestVersionOnPackageIndex /p:UpdateStablePackageInfo=true`
##[error]src\libraries\pkg\test\testPackages.proj(217,5): error : (NETCORE_ENGINEERING_TELEMETRY=Build) Validation Failed: Microsoft.Extensions.DependencyModel is harvesting assets from package version 3.1.5 which is not the latest for that package release. Latest package version from that release is 3.1.6. In order to fix this, run `dotnet msbuild Microsoft.Extensions.DependencyModel.pkgproj /t:UpdateHarvestVersionOnPackageIndex /p:UpdateStablePackageInfo=true`

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs

…otnet#39041) * AdvSimd support for System.Text.Unicode.Utf8Utility.TranscodeToUtf8 * Readd using to prevent build failure. Add AdvSimd equivalent operation to TestZ. * Inverted condition * Address IsSupported order, improve use ExtractNarrowingSaturated usage * Rename source to result, second argument utf16Data * Improve CompareTest * Add shims causing failures in Linux * Use unsigned version of ExtractNarrowingSaturate, avoid using MinAcross and use MaxPairwise instead * Missing support check for Sse2.X64 * Add missing case for AdvSimd * Use MinPairwise for short

…#39738) * AdvSimd support for System.Text.Unicode.Utf16Utility.GetPointerToFirstInvalidChar (#39050) * AdvSimd support for System.Text.Unicode.Utf16Utility.GetPointerToFirstInvalidChar * Move using directive outside #if. Improve Arm64MoveMask. * Change overloads * UIn64 in Arm64MoveMask * Build error implicit conversion fix * Rename method and use simpler version * Use ShiftRightArithmetic instead of CompareEqual + And. * Remove unnecessary comment * Add missing shims causing Linux build to fail * AdvSimd support for System.Text.Unicode.Utf8Utility.TranscodeToUtf8 (#39041) * AdvSimd support for System.Text.Unicode.Utf8Utility.TranscodeToUtf8 * Readd using to prevent build failure. Add AdvSimd equivalent operation to TestZ. * Inverted condition * Address IsSupported order, improve use ExtractNarrowingSaturated usage * Rename source to result, second argument utf16Data * Improve CompareTest * Add shims causing failures in Linux * Use unsigned version of ExtractNarrowingSaturate, avoid using MinAcross and use MaxPairwise instead * Missing support check for Sse2.X64 * Add missing case for AdvSimd * Use MinPairwise for short * AdvSimd support for System.Text.Unicode.Utf8Utility.GetPointerToFirstInvalidByte (#38653) * AdvSimd support for System.Text.Unicode.Utf8Utility.GetPointerToFirstInvalidByte * Move comment to the top, add shims. * Little endian checks * Use custom MoveMask method for AdvSimd * Address suggestions to improve the AdvSimdMoveMask method * Define initialMask outside MoveMask method * UInt64 in Arm64MoveMask * Add unit test case to verify intrinsics improvement * Avoid casting to smaller integer type * Typo and comment * Use ShiftRightArithmetic instead of CompareEqual + And. Remove test case causing other unit tests to fail. * Use AddPairwise version of GetNotAsciiBytes * Add missing shims causing Linux build to fail * Simplify GetNonAsciiBytes to only one AddPairwise call, shorter bitmask * Respect data type returned by masking method * Address suggestions - assert trailingzerocount and bring back uint mask * Trailing zeroes in AdvSimd need to be divided by 4, and total number should not be larger than 16 * Avoid declaring static field which causes PNSE in Utf8String.Experimental (S.P.Corelib code is used for being NetStandard) * Prefer using nuint for BitConverter.TrailingZeroCount * Fix build failure in net472 debug AdvSimd Utf16Utility (#39652) Co-authored-by: Carlos Sanchez Lopez <1175054+carlossanlop@users.noreply.github.com>

…otnet#39041) * AdvSimd support for System.Text.Unicode.Utf8Utility.TranscodeToUtf8 * Readd using to prevent build failure. Add AdvSimd equivalent operation to TestZ. * Inverted condition * Address IsSupported order, improve use ExtractNarrowingSaturated usage * Rename source to result, second argument utf16Data * Improve CompareTest * Add shims causing failures in Linux * Use unsigned version of ExtractNarrowingSaturate, avoid using MinAcross and use MaxPairwise instead * Missing support check for Sse2.X64 * Add missing case for AdvSimd * Use MinPairwise for short

carlossanlop added the arch-arm64 label Jul 9, 2020

carlossanlop added this to the 5.0.0 milestone Jul 9, 2020

carlossanlop requested review from eiriktsarpalis, echesakov, pgovind and tannergooding July 9, 2020 22:34

carlossanlop self-assigned this Jul 9, 2020

carlossanlop commented Jul 9, 2020

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs Outdated Show resolved Hide resolved

carlossanlop requested review from GrabYourPitchforks and kunalspathak July 9, 2020 22:39

carlossanlop added the area-System.Text.Encoding label Jul 9, 2020

tarekgh added the utf8-impact Potentially impacts UTF-8 support in the runtime label Jul 9, 2020

carlossanlop marked this pull request as draft July 10, 2020 01:02

carlossanlop requested a review from BruceForstall July 10, 2020 16:50

carlossanlop added area-System.Runtime.Intrinsics and removed area-System.Text.Encoding labels Jul 10, 2020

AdvSimd support for System.Text.Unicode.Utf8Utility.TranscodeToUtf8

6e26a21

kunalspathak reviewed Jul 10, 2020

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs Show resolved Hide resolved

carlossanlop commented Jul 10, 2020

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs Show resolved Hide resolved

Readd using to prevent build failure.

f596eaf

Add AdvSimd equivalent operation to TestZ.

carlossanlop force-pushed the ARM-Utf8Utility.Transcoding branch from 8a19f17 to f596eaf Compare July 10, 2020 20:41

carlossanlop marked this pull request as ready for review July 11, 2020 00:33

jaredpar mentioned this pull request Jul 13, 2020

OSX machines are de-provisioned during CI / PR runs leading to failures #34472

Closed

tannergooding reviewed Jul 14, 2020

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs Outdated Show resolved Hide resolved

tannergooding reviewed Jul 14, 2020

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs Outdated Show resolved Hide resolved

tannergooding reviewed Jul 14, 2020

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs Show resolved Hide resolved

pgovind reviewed Jul 15, 2020

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs Outdated Show resolved Hide resolved

Use unsigned version of ExtractNarrowingSaturate, avoid using MinAcro…

191ec1f

…ss and use MaxPairwise instead

echesakov approved these changes Jul 15, 2020

View reviewed changes

carlossanlop added 2 commits July 16, 2020 16:19

Missing support check for Sse2.X64

1e5519c

Add missing case for AdvSimd

86ef01f

echesakov reviewed Jul 17, 2020

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs Outdated Show resolved Hide resolved

Use MinPairwise for short

101c4f1

GrabYourPitchforks approved these changes Jul 18, 2020

View reviewed changes

carlossanlop mentioned this pull request Jul 20, 2020

AdvSimd support for System.Text.Unicode.Utf16Utility.GetPointerToFirstInvalidChar #39050

Merged

kunalspathak reviewed Jul 20, 2020

View reviewed changes

src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8Utility.Transcoding.cs Show resolved Hide resolved

carlossanlop merged commit 7324741 into dotnet:master Jul 20, 2020

tannergooding mentioned this pull request Jul 21, 2020

[release/5.0-preview8] ARM64 HWIntrinsic usage in System.Text.Unicode #39738

Merged

carlossanlop deleted the ARM-Utf8Utility.Transcoding branch July 29, 2020 16:35

kunalspathak mentioned this pull request Sep 4, 2020

[ARM64] Performance regression: Utf8Encoding #41699

Closed

ghost locked as resolved and limited conversation to collaborators Dec 8, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AdvSimd support for System.Text.Unicode.Utf8Utility.TranscodeToUtf8 #39041

AdvSimd support for System.Text.Unicode.Utf8Utility.TranscodeToUtf8 #39041

carlossanlop commented Jul 9, 2020 •

edited

Loading

Dotnet-GitSync-Bot commented Jul 9, 2020

ghost commented Jul 9, 2020

ghost commented Jul 10, 2020

carlossanlop commented Jul 14, 2020

carlossanlop commented Jul 16, 2020 •

edited

Loading

kunalspathak commented Jul 16, 2020

kunalspathak commented Jul 16, 2020

echesakov commented Jul 16, 2020 •

edited

Loading

kunalspathak commented Jul 16, 2020

carlossanlop commented Jul 17, 2020

carlossanlop commented Jul 17, 2020

carlossanlop commented Jul 18, 2020

carlossanlop commented Jul 18, 2020

kunalspathak commented Jul 18, 2020 •

edited

Loading

GrabYourPitchforks Jul 18, 2020

GrabYourPitchforks Jul 18, 2020

carlossanlop Jul 20, 2020

carlossanlop commented Jul 20, 2020

carlossanlop commented Jul 20, 2020

AdvSimd support for System.Text.Unicode.Utf8Utility.TranscodeToUtf8 #39041

AdvSimd support for System.Text.Unicode.Utf8Utility.TranscodeToUtf8 #39041

Conversation

carlossanlop commented Jul 9, 2020 • edited Loading

Dotnet-GitSync-Bot commented Jul 9, 2020

ghost commented Jul 9, 2020

ghost commented Jul 10, 2020

carlossanlop commented Jul 14, 2020

carlossanlop commented Jul 16, 2020 • edited Loading

kunalspathak commented Jul 16, 2020

kunalspathak commented Jul 16, 2020

echesakov commented Jul 16, 2020 • edited Loading

kunalspathak commented Jul 16, 2020

carlossanlop commented Jul 17, 2020

carlossanlop commented Jul 17, 2020

carlossanlop commented Jul 18, 2020

carlossanlop commented Jul 18, 2020

kunalspathak commented Jul 18, 2020 • edited Loading

GrabYourPitchforks Jul 18, 2020

Choose a reason for hiding this comment

GrabYourPitchforks Jul 18, 2020

Choose a reason for hiding this comment

carlossanlop Jul 20, 2020

Choose a reason for hiding this comment

carlossanlop commented Jul 20, 2020

carlossanlop commented Jul 20, 2020

Libraries build Windows_NT allConfigurations x64 Release

carlossanlop commented Jul 9, 2020 •

edited

Loading

carlossanlop commented Jul 16, 2020 •

edited

Loading

echesakov commented Jul 16, 2020 •

edited

Loading

kunalspathak commented Jul 18, 2020 •

edited

Loading