MsQuic interop #1

ManickaP · 2022-04-08T20:03:07Z

No description provided.

nibanks · 2022-04-09T14:38:25Z

...Net.Quic/src/System/Net/Quic/Implementations/MsQuic/Interop/SafeMsQuicConfigurationHandle.cs

-            QUIC_CREDENTIAL_FLAGS flags = QUIC_CREDENTIAL_FLAGS.NONE;
+            QUIC_CREDENTIAL_FLAGS flags = QUIC_CREDENTIAL_FLAGS.QUIC_CREDENTIAL_FLAG_NONE;


@thhous-msft I wonder if we should post-process and remove the type prefix from our interop layer, so this would stay NONE.

Yeah I'd be fine doing that. It works a lot better from C# that would.

nibanks · 2022-04-09T14:41:56Z

...aries/System.Net.Quic/src/System/Net/Quic/Implementations/MsQuic/Interop/msquic_generated.cs

@@ -0,0 +1,2673 @@
+//
+// Copyright (c) Microsoft Corporation.


So you're just going to manually copy this in every time you sync to the latest MsQuic?

I also have to make all the classes/structs/enums internal, so I cannot just take it as it is.
We could publish the interop lib and consume that, but that would mean setting up infra for it and is not so straightforward.

Do you have any other way to consume this in mind?

ThadHouse · 2022-04-09T15:04:01Z

src/libraries/System.Net.Quic/src/System/Net/Quic/Implementations/MsQuic/MsQuicListener.cs

            try
            {
                Debug.Assert(!Monitor.IsEntered(_state), "!Monitor.IsEntered(_state)");
                MsQuicAlpnHelper.Prepare(applicationProtocols, out handles, out buffers);
+                // TODO: is the layout same for SocketAddress.Buffer and QuicAddr?


IIRC, SocketAddress.Buffer does not take into account macOS having the family in a different location then linux. But, QuicAddr is identical to sockaddr, so theoretically if there is an internal SocketAddress to sockaddr helper, that would work fine.

ManickaP · 2022-04-09T19:35:15Z

...System.Net.Quic/src/System/Net/Quic/Implementations/MsQuic/Interop/msquic_generated_linux.cs

@@ -0,0 +1,110 @@
+//


@nibanks Is it possible for us to change the copyright here to:

// Licensed to the .NET Foundation under one or more agreements. // The .NET Foundation licenses this file to you under the MIT license.

???

This is what the compiler/analyzer wants from me.

Of course, I can figure out where the message comes from and whether we can skip it here in case it's not possible to change due to legal reasons.

rzikm · 2022-04-11T10:44:07Z

...Net.Quic/src/System/Net/Quic/Implementations/MsQuic/Interop/SafeMsQuicConfigurationHandle.cs

+            // TODO: should they be pinned? The content is pinned, but the array holding the individual buffers isn't.
+            QUIC_BUFFER[]? buffers = null;


The code does

public static unsafe IntPtr UnsafeAddrOfPinnedArrayElement<T>(T[] arr!!, int index) { void* pRawData = Unsafe.AsPointer(ref MemoryMarshal.GetArrayDataReference(arr)); return (IntPtr)((byte*)pRawData + (uint)index * (nuint)Unsafe.SizeOf<T>()); }

So I suppose it should be pinned because if buffers is not pinned this makes (QUIC_BUFFER*)Marshal.UnsafeAddrOfPinnedArrayElement(buffers, 0) a pointer to somewhere inside an unpinned array

That's what I thought, but we had it like this since forever, which confused me.

…ons/MsQuic/MsQuicListener.cs

ManickaP · 2022-04-20T18:55:05Z

Closing in favor of dotnet#68288

This adds support for EnC on arm64. A couple of notes on the implementation compared to x64: - On x64 we get the fixed stack size from unwind info. However, for the frames we set up on arm64 for EnC it is not possible to extract the frame size from there because their prologs generally look like stp fp, lr, [sp,#-16]! mov fp, sp sub sp, sp, dotnet#144 with unwind codes like the following: set_fp; mov fp, sp save_fplr_x #1 (0x01); tp fp, lr, [sp, #-16]! As can be seen, it is not possible to get the fixed stack size from unwind info in this case. Instead we pass it through the GC info that already has a section for EnC data. - On arm64 the JIT is required to place the PSPSym at the same offset from caller-SP for both the main function and for funclets. Due to this we try to allocate the PSPSym as early as possible in the main function and we must take some care in funclets. However, this conflicts with the EnC frame header that the JIT uses to place values that must be preserved on EnC transitions. This is currently callee-saved registers and the MonitorAcquired boolean. Before this change we were allocating PSPSym above (before) the monitor acquired boolean, but we now have to allocate MonitorAcquired first, particularly because the size of the preserved header cannot change on EnC transitions, while the PSPSym can disappear or appear. This changes frame allocation slightly for synchronized functions.

These helpers are used to report names of things in warnings. The functional changes are: * For method parameters, use the parameter name if available (and only if not fallback to the #1 notation) * For property accessor methods, use the C# naming scheme, so for example Type.Property.get instead of Type.get_Property. Both of these changes are in preparation to bring NativeAOT closer in behavior to ILLink and the trim analyzers. For this I moved some of the helpers to the common shared code. Some unrelated code cleanup as well. Co-authored-by: Michal Strehovský <MichalStrehovsky@users.noreply.github.com>

* Initial implementation for contract customization fix build errors Move converter rooting to DefaultJsonTypeInfoResolver so that it can be used standalone Fix ConfigurationList.IsReadOnly Minor refactorings (#1) * Makes the following changes: * Move singleton initialization for DefaultTypeInfoResolver behind a static property. * Consolidate JsonSerializerContext & IJsonTypeInfoResolver values to a single field. * Move reflection fallback logic away from JsonSerializerContext and into JsonSerializerOptions * Update src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonSerializerOptions.cs * remove testing of removed field Simplify the JsonTypeInfo.CreateObject implemenetation (#2) * Simplify the JsonTypeInfo.CreateObject implemenetation * Update src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Metadata/JsonTypeInfoOfT.cs * Update src/libraries/System.Text.Json/src/System/Text/Json/Serialization/Metadata/JsonTypeInfoOfT.cs Co-authored-by: Krzysztof Wicher <mordotymoja@gmail.com> Co-authored-by: Krzysztof Wicher <mordotymoja@gmail.com> Tests and fixes for JsonTypeInfoKind.None TypeInfo type mismatch tests Allow setting NumberHandling on JsonTypeInfoKind.None test resolver returning wrong type of options JsonTypeInfo/JsonPropertyInfo mutability tests rename test file Move default converter rooting responsibility behind DefaultJsonTypeInfoResolver (#3) * Move default converter rooting responsibility behind DefaultJsonTypeInfoResolver * address feedback Add simple test for using JsonTypeInfo<T> with APIs directly taking it fix and tests for untyped/typed CreateObject uncomment test cases, remove todo More tests and tiny fixes Add a JsonTypeInfoResolver.Combine test for JsonSerializerContext (#4) * Fix JsonTypeInfoResolver.Combine for JsonSerializerContext * Break up failing test Fix simple scenarios for combining contexts (#6) * Fix simple scenarios for combining contexts * feedback JsonSerializerContext combine test with different camel casing Remove unneeded virtual calls & branching when accessing Get & Set delegates (#7) JsonPropertyInfo tests everything minus ShouldSerialize & NumberHandling Update src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.cs Update src/libraries/System.Text.Json/src/System/Text/Json/Serialization/JsonConverterOfT.cs throw InvalidOperationException rather than ArgumentNullException for source gen when PropertyInfo.Name is assigned through JsonPropertyInfoValues tests for duplicated property names and JsonPropertyInfo.NumberHandling Add tests for NumberHandling and failing tests for ShouldSerialize disable the failing test and add extra checks disable remainder of the failing ShouldSerialize tests, fix working one Fix ShouldSerialize and IgnoreCondition interop Add failing tests for CreateObject + parametrized constructors Fix CreateObject support for JsonConstructor types (#10) * Fix CreateObject support for JsonConstructor types * address feedback Make contexts more combinator friendly (#9) * Make contexts more combinator friendly * remove converter cache * redesign test to account for JsonConstructorAttribute * Combine unit tests * address feedback * Add acceptance tests for DataContract attributes & Specified pattern (#11) * Add private field serialization acceptance test (#13) * tests, PR feedback (#14) * PR feedback and extra tests * Shorten class name, remove incorrect check (not true for polimorphic cases) * Make parameter matching for custom properties map property Name with parameter (dotnet#16) * Test static initialization with JsonTypeInfo (dotnet#17) * Fix test failures and proper fix this time (dotnet#18) * Fix test failures and proper fix this time * reinstate ActiveIssueAttribute * PR feedback and adjust couple of tests which don't set TypeInfoResolver * fix IAsyncEnumerable tests * Lock JsonSerializerOptions in JsonTypeInfo.EnsureConfigured() Co-authored-by: Eirik Tsarpalis <eirik.tsarpalis@gmail.com> Co-authored-by: Eirik Tsarpalis <eirik.tsarpalis@gmail.com>

E.g., Update LSRA "Allocating Registers" table description. Dump nodes added during resolution, e.g.: ``` BB29 bottom (BB08->BB08): move V25 from STK to rdi (Critical) N001 ( 1, 1) [001174] ----------z t1174 = LCL_VAR int V25 cse4 rdi REG rdi ``` Dump more data in the LSRA block sequence data: ``` -BB03( 16 ) -BB04( 4 ) +BB03 ( 16 ) critical-in critical-out +BB04 ( 4 ) critical-out ``` When dumping various flow bitvectors, annotate the bitvectors better: ``` -BB25 in gen out -0000000000000000 -0000000000000003 CSE #1.c -0000000000000003 CSE #1.c +BB25 + in: 0000000000000000 +gen: 0000000000000003 CSE #1.c +out: 0000000000000003 CSE #1.c ``` Dump hoisting bitvectors using the sorting number: ``` - USEDEF (5)={V04 V00 V01 V02 V03} + USEDEF (5)={V00 V01 V02 V03 V04} ``` Also, fix various typos and formatting.

* WIP: add gRPC tests * Fix AOT and trimming * WIP * Implement IncludeNetworkSecurityConfig * Use IncludeNetworkSecurityConfig * Fix gRPC test * Avoid git checkout * Remove unnecessary code * WIP: start working on CI configuration * Remove WinHttpHandler * Fix problem with SSL * Change server host * Setup CI (#1) * Get Docker container building & exported via test build * Changes * Add missing pfx certificate * changes * cleanup Co-authored-by: Simon Rozsival <simon@rozsival.com> * Use tls * Update yml * Revert changes to the mono Android sample app * Bump android image version * Bump image version * Enable TLS * Remove hardcoded package versions * Update package versions * Update package versions * Rename pipeline * Move interop tests website dependencies versions to Versions.props * Add cred scan supression for the interop test server private key * Fix licenses * Remove dependencies * Fix path to Versions.props * Remove unnecessary dependency version * Fix building docker image * Change pfx password Co-authored-by: Jo Shields <directhex@apebox.org>

* switch to managed thread ID in Lock * fattening the lock * __declspec(selectany) * few tweaks * fairness * more room for thread ids * remove CurrentNativeThreadId * couple fixes * fix win-arm64 build * win-arm64 build , another try * Apply suggestions from code review Co-authored-by: Jan Kotas <jkotas@microsoft.com> * fix after renaming * do not report successful spin if thread has waited * keep extern and undo mangling of tls_CurrentThread in asm * use SyncTable indexer in less perf-sensitive places. * GetNewHashCode just delegate to shared random * Apply suggestions from code review Co-authored-by: Jan Kotas <jkotas@microsoft.com> * unchecked const conversion * some refactoring comments and typos * min number of spins in the backoff * moved CurrentManagedThreadIdUnchecked to ManagedThreadId * Use `-1` to report success and allow using element #1 in the SyncTable * use threadstatic for managed thread ID * check before calling RhGetProcessCpuCount * use 0 as default thread ID Co-authored-by: Jan Kotas <jkotas@microsoft.com>

* Fix the MacOS remote unwinder for VS4Mac The wrong module was being passed to the remote unwinder because the load bias for shared modules was being calculated incorrectly. Issue: dotnet#63309 * Fix native frame unwind in syscall on arm64 for VS4Mac crash report From PR in main: dotnet#63598 Add arm64 version of StepWithCompactNoEncoding for syscall leaf node wrappers that have compact encoding of 0. Fix ReadCompactEncodingRegister so it actually decrements the addr. Change StepWithCompactEncodingArm64 to match what MacOS libunwind does for framed and frameless stepping. arm64 can have frames with the same SP (but different IPs). Increment SP for this condition so createdump's unwind loop doesn't break out on the "SP not increasing" check and the frames are added to the thread frame list in the correct order. Add getting the unwind info for tail called functions like this: __ZL14PROCEndProcessPvji: 36630: f6 57 bd a9 stp x22, x21, [sp, #-48]! 36634: f4 4f 01 a9 stp x20, x19, [sp, dotnet#16] 36638: fd 7b 02 a9 stp x29, x30, [sp, dotnet#32] 3663c: fd 83 00 91 add x29, sp, dotnet#32 ... 367ac: e9 01 80 52 mov w9, #15 367b0: 7f 3e 02 71 cmp w19, dotnet#143 367b4: 20 01 88 1a csel w0, w9, w8, eq 367b8: 2e 00 00 94 bl _PROCAbort _TerminateProcess: -> 367bc: 22 00 80 52 mov w2, #1 367c0: 9c ff ff 17 b __ZL14PROCEndProcessPvji The IP (367bc) returns the (incorrect) frameless encoding with nothing on the stack (uses an incorrect LR to unwind). To fix this get the unwind info for PC -1 which points to PROCEndProcess with the correct unwind info. This matches how lldb unwinds this frame. Always address module segment to IP lookup list instead of checking the module regions. Strip pointer authentication bits on PC/LR.

…tnet#87189) This fixes a startup crash on Big Sur: > error: * Assertion at /Users/runner/work/1/s/src/mono/mono/utils/mono-hwcap-arm64.c:35, condition `res == 0' not met Because sysctl can't find some of these options: $ sysctl hw.optional.armv8_crc32 hw.optional.armv8_crc32: 1 $ sysctl hw.optional.arm.FEAT_RDM sysctl: unknown oid 'hw.optional.arm.FEAT_RDM' $ sysctl hw.optional.arm.FEAT_DotProd sysctl: unknown oid 'hw.optional.arm.FEAT_DotProd' $ sysctl hw.optional.arm.FEAT_SHA1 sysctl: unknown oid 'hw.optional.arm.FEAT_SHA1' $ sysctl hw.optional.arm.FEAT_SHA256 sysctl: unknown oid 'hw.optional.arm.FEAT_SHA256' $ sysctl hw.optional.arm.FEAT_AES sysctl: unknown oid 'hw.optional.arm.FEAT_AES' Full stack trace: * thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 2.1 * frame #0: 0x0000010ef37560 libmonosgen-2.0.dylib`monoeg_assertion_message frame #1: 0x0000010ef375cc libmonosgen-2.0.dylib`mono_assertion_message + 32 frame #2: 0x0000010ef40d6c libmonosgen-2.0.dylib`mono_hwcap_arch_init + 544 frame #3: 0x0000010ef54bd8 libmonosgen-2.0.dylib`mono_hwcap_init + 72 frame #4: 0x0000010ee14dc0 libmonosgen-2.0.dylib`parse_optimizations + 52 frame #5: 0x0000010edbed48 libmonosgen-2.0.dylib`mono_init frame #6: 0x0000010ee18968 libmonosgen-2.0.dylib`mono_jit_init_version frame #7: 0x0000010f48a300 libxamarin-dotnet-debug.dylib`xamarin_bridge_initialize + 216 frame #8: 0x0000010f4900a4 libxamarin-dotnet-debug.dylib`xamarin_main + 376

Fixes dotnet#95367. Relevant part of the JitDump: ``` Using `if true` assertions from pred BB02 Assertions in: #1 fgMorphTree BB04, STMT00021 (before) [000070] DA--------- * STORE_LCL_VAR ubyte V10 tmp9 [000057] ----------- \--* CAST int <- ubyte <- int [000006] ----------- \--* EQ int [000004] ----------- +--* LCL_VAR ref V02 tmp1 (last use) [000055] H---------- \--* CNS_INT(h) ref 'Frozen EmptyPartition`1<Int32> object' Assertion prop for index #1 in BB04: [000006] ----------- * EQ int GenTreeNode creates assertion: [000070] DA---+----- * STORE_LCL_VAR ubyte V10 tmp9 In BB04 New Local Constant Assertion: V10 == [0000000000000001], index = #2 fgMorphTree BB04, STMT00021 (after) [000070] DA---+----- * STORE_LCL_VAR ubyte V10 tmp9 [000055] H----+----- \--* CNS_INT(h) int ``` The JitDump is unfinished because the compiler crashes when trying to dump the last line. Clearly, the `CNS_INT` is no longer a handle at that point because we just bashed it to a constant 1.

CodeQL flagged various places where we're dereferencing pointers that could be NULL, this PR systematically cleans some of them up via g_assert. * g_assert result of g_build_path calls * Allocation failure handling * mono_class_inflate_generic_class_checked can return NULL

…#102133) This generalizes the indir reordering optimization (that currently only triggers for loads) to kick in for GT_STOREIND nodes. The main complication with doing this is the fact that the data node of the second indirection needs its own reordering with the previous indirection. The existing logic works by reordering all nodes between the first and second indirection that are unrelated to the second indirection's computation to happen after it. Once that is done we know that there are no uses of the first indirection's result between it and the second indirection, so after doing the necessary interference checks we can safely move the previous indirection to happen after the data node of the second indirection. Example: ```csharp class Body { public double x, y, z, vx, vy, vz, mass; } static void Advance(double dt, Body[] bodies) { foreach (Body b in bodies) { b.x += dt * b.vx; b.y += dt * b.vy; b.z += dt * b.vz; } } ``` Diff: ```diff @@ -1,18 +1,17 @@ -G_M55007_IG04: ;; offset=0x001C +G_M55007_IG04: ;; offset=0x0020 ldr x3, [x0, w1, UXTW #3] ldp d16, d17, [x3, #0x08] ldp d18, d19, [x3, #0x20] fmul d18, d0, d18 fadd d16, d16, d18 - str d16, [x3, #0x08] - fmul d16, d0, d19 - fadd d16, d17, d16 - str d16, [x3, #0x10] + fmul d18, d0, d19 + fadd d17, d17, d18 + stp d16, d17, [x3, #0x08] ldr d16, [x3, #0x18] ldr d17, [x3, #0x30] fmul d17, d0, d17 fadd d16, d16, d17 str d16, [x3, #0x18] add w1, w1, #1 cmp w2, w1 bgt G_M55007_IG04 ```

* bug #1: don't allow for values out of the SerializationRecordType enum range * bug #2: throw SerializationException rather than KeyNotFoundException when the referenced record is missing or it points to a record of different type * bug #3: throw SerializationException rather than FormatException when it's being thrown by BinaryReader (or sth else that we use) * bug #4: document the fact that IOException can be thrown * bug #5: throw SerializationException rather than OverflowException when parsing the decimal fails * bug #6: 0 and 17 are illegal values for PrimitiveType enum * bug #7: throw SerializationException when a surrogate character is read (so far an ArgumentException was thrown)

ManickaP force-pushed the mapichov/quic_interop branch from 0176c8a to b766f73 Compare April 9, 2022 11:47

nibanks reviewed Apr 9, 2022

View reviewed changes

ThadHouse reviewed Apr 9, 2022

View reviewed changes

ManickaP commented Apr 9, 2022

View reviewed changes

rzikm reviewed Apr 11, 2022

View reviewed changes

ManickaP added 11 commits April 20, 2022 18:47

MsQuicApi adjusted

c6edafb

QuicParamHelper adjusted

ea31c49

MsQuicConfiguration adjusted

8d41cbb

Added MsQuicException

55dc554

MsQuicConnection adjusted

259545e

MsQuicListener adjusted

d0fb04e

MsQuicStream adjusted

b3d76cf

It compiles!

103cd1f

Some test fixes

08b5b47

Fixed stream read event handling

7794b58

Re-enabled IPv6 tests

46a7401

ManickaP force-pushed the mapichov/quic_interop branch from 37c6fa6 to 46a7401 Compare April 20, 2022 18:21

ManickaP and others added 2 commits April 20, 2022 20:35

latest greatest

182a8f0

Update src/libraries/System.Net.Quic/src/System/Net/Quic/Implementati…

2b0172c

…ons/MsQuic/MsQuicListener.cs

ManickaP closed this Apr 20, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MsQuic interop #1

MsQuic interop #1

ManickaP commented Apr 8, 2022

nibanks Apr 9, 2022

ThadHouse Apr 9, 2022

nibanks Apr 9, 2022

ManickaP Apr 9, 2022

ThadHouse Apr 9, 2022

ManickaP Apr 9, 2022

rzikm Apr 11, 2022

ManickaP Apr 11, 2022

ManickaP commented Apr 20, 2022

		QUIC_CREDENTIAL_FLAGS flags = QUIC_CREDENTIAL_FLAGS.NONE;
		QUIC_CREDENTIAL_FLAGS flags = QUIC_CREDENTIAL_FLAGS.QUIC_CREDENTIAL_FLAG_NONE;

		// TODO: should they be pinned? The content is pinned, but the array holding the individual buffers isn't.
		QUIC_BUFFER[]? buffers = null;

MsQuic interop #1

MsQuic interop #1

Conversation

ManickaP commented Apr 8, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ManickaP commented Apr 20, 2022