-
Notifications
You must be signed in to change notification settings - Fork 5k
Includes type forwarding for System.Text.Unicode.Utf8 to the Microsoft.Bcl.Memory library #111292
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
This commit introduces the `System.Text.Unicode.Utf8` type to the `Microsoft.Bcl.Memory` library. It includes type forwarding for `Utf8` in `Microsoft.Bcl.Memory.Forwards.cs`, updates the documentation in `PACKAGE.md` to include `Utf8` functionality, and adds corresponding test cases in `Microsoft.Bcl.Memory.Tests.csproj`. The documentation now emphasizes `Utf8` alongside `Index`, `Range`, and `Base64Url`, highlighting its role in converting data between UTF-8 and UTF-16 encodings.
Thank you, but this is not the intent of that work item, which is to actually ship those Utf8 APIs downlevel, e.g. in addition to type forwarding, it also needs an implementation for netstandard. |
Updated `PackageDescription` to include "Utf8" support. Added new `ItemGroup` for conditional compilation of UTF-8 and Unicode handling files for non-net8.0 frameworks. Modified visibility and implementations in `Ascii.Utility.Helpers.cs`, `Utf8.cs`, and `Utf8Utility` based on `MICROSOFT_BCL_MEMORY` define.
I added Utf8 APIs to Microsoft.Bcl.Memory library. I haven't added tests yet. I'll add them too. |
Updated `Microsoft.Bcl.Memory.Tests.csproj` to include `UnicodeUtility.cs` and removed .NET 8.0 targeting condition. Modified `Utf8Tests.cs` by adjusting using directives and enhancing the `DecodeHex` method with conditional compilation for .NET 5.0+.
Added a new property `<DefineConstants>$(DefineConstants);MICROSOFT_BCL_MEMORY</DefineConstants>` to the project file to define a new compilation constant for the project.
src/libraries/Microsoft.Bcl.Memory/src/Microsoft.Bcl.Memory.csproj
Outdated
Show resolved
Hide resolved
@stephentoub I added Utf8 APIs to Microsoft.Bcl.Memory library for older .NET platforms. I've also added tests for older .NET platforms. |
src/libraries/System.Private.CoreLib/src/System/Text/Ascii.Utility.cs
Outdated
Show resolved
Hide resolved
@@ -9,7 +9,10 @@ | |||
|
|||
namespace System.Text.Unicode | |||
{ | |||
#if SYSTEM_PRIVATE_CORELIB | |||
#if SYSTEM_PRIVATE_CORELIB || MICROSOFT_BCL_MEMORY |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This file is being referenced only by System.Private.CoreLib
and Microsoft.BCL.Memory
, so the class can always be public
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm afraid to make such changes and let someone else make them as a separate PR.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can just delete the ifdefs and make this public. I don't know why the ifdefs were here in the first place.
src/libraries/System.Private.CoreLib/src/System/Text/Ascii.Utility.cs
Outdated
Show resolved
Hide resolved
src/libraries/System.Private.CoreLib/src/System/Text/Ascii.Utility.Helpers.cs
Outdated
Show resolved
Hide resolved
src/libraries/System.Private.CoreLib/src/System/Text/Ascii.Utility.Helpers.cs
Outdated
Show resolved
Hide resolved
#if !MICROSOFT_BCL_MEMORY | ||
public | ||
#else | ||
private | ||
#endif | ||
static byte[] ToUtf8(Rune rune) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#if !MICROSOFT_BCL_MEMORY | |
public | |
#else | |
private | |
#endif | |
static byte[] ToUtf8(Rune rune) | |
public static byte[] ToUtf8(Rune rune) |
Why can't this always be public?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In tests for older versions of .Net where there is no Rune class yet, I create it as an internal class, and the compiler complains that the public API cannot reference internal classes.
src/libraries/System.Runtime/tests/System.Runtime.Tests/System/Text/Unicode/Utf8Tests.cs
Outdated
Show resolved
Hide resolved
src/libraries/System.Private.CoreLib/src/System/Text/Ascii.Utility.cs
Outdated
Show resolved
Hide resolved
src/libraries/Microsoft.Bcl.Memory/src/Microsoft.Bcl.Memory.Forwards.cs
Outdated
Show resolved
Hide resolved
src/libraries/Microsoft.Bcl.Memory/src/Microsoft.Bcl.Memory.csproj
Outdated
Show resolved
Hide resolved
Co-authored-by: Theodore Tsirpanis <teo@tsirpanis.gr>
- Added polyfill for System.Numerics.BitOperations for .NET Standard 2.0.
Removed the `DefineConstants` property from the project file, which included the constant `MICROSOFT_BCL_MEMORY`. This change may impact conditional compilation within the project.
(removing
NO-MERGE
|
@dotnet/area-system-memory could you review this? Thanks. |
src/libraries/System.Private.CoreLib/src/System/Text/Unicode/Utf8.cs
Outdated
Show resolved
Hide resolved
For downlevel frameworks we add `Rune.cs` to `Microsoft.Bcl.Memory`. (cherry picked from commit 79ee05d)
(cherry picked from commit bf6f989)
CI failures are unrelated. |
I do not have any additional comments. @dotnet/area-system-memory should review and signoff. |
@dotnet/area-system-memory friendly reminder. |
src/libraries/Microsoft.Bcl.Memory/src/Microsoft.Bcl.Memory.csproj
Outdated
Show resolved
Hide resolved
@@ -1490,7 +1496,11 @@ int IComparable.CompareTo(object? obj) | |||
return this.CompareTo(other); | |||
} | |||
|
|||
#if SYSTEM_PRIVATE_CORLIB |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this corelib-specific? The Microsoft.Bcl.Memory package can't include the same resource?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Microsoft.Bcl.Memory
does not include any resources at present. And the non-corelib path is highly unlikely to be actually hit.
@@ -9,7 +9,10 @@ | |||
|
|||
namespace System.Text.Unicode | |||
{ | |||
#if SYSTEM_PRIVATE_CORELIB | |||
#if SYSTEM_PRIVATE_CORELIB || MICROSOFT_BCL_MEMORY |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can just delete the ifdefs and make this public. I don't know why the ifdefs were here in the first place.
src/libraries/System.Runtime/tests/System.Runtime.Tests/System/Text/Unicode/Utf8Tests.cs
Outdated
Show resolved
Hide resolved
src/libraries/System.Runtime/tests/System.Runtime.Tests/System/Text/Unicode/Utf8Tests.cs
Outdated
Show resolved
Hide resolved
PR feedback addressed in teo-tsirpanis@445a232. |
(cherry picked from commit 445a232)
/// <summary> | ||
/// Provides methods for transcoding between UTF-8 and UTF-16. | ||
/// Provides static methods that convert chunked data between UTF-8 and UTF-16 encodings. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Documentation copied from Learn.
CI failures are unrelated. |
// C# no-alloc optimization that directly wraps the data section of the dll (similar to string constants) | ||
// https://github.com/dotnet/roslyn/pull/24621 | ||
|
||
private static ReadOnlySpan<byte> TrailingZeroCountDeBruijn => // 32 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: this can be a local in TrailingZeroCount (single use). Slightly better on netfx as well.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can do it in a subsequent PR.
Close #52947
This commit introduces the
System.Text.Unicode.Utf8
type to theMicrosoft.Bcl.Memory
library.It includes type forwarding for
Utf8
inMicrosoft.Bcl.Memory.Forwards.cs
, updates the documentation inPACKAGE.md
to includeUtf8
functionality, and adds corresponding test cases inMicrosoft.Bcl.Memory.Tests.csproj
.The documentation now emphasizes
Utf8
alongsideIndex
,Range
, andBase64Url
, highlighting its role in converting data between UTF-8 and UTF-16 encodings.