Runtime developer can run Networking tests without Loopback server capability (on mobile/browser platforms)

[lewing]

Some platforms don't support listening to incoming requests but do support making outgoing requests. Most of the existing http functional tests expect to be able to start up a loopback server to respond to incoming request which means the tests cannot run on the more restricted platforms.

---

Plan:

• Make xharness extensible by loading kestrel middlewares - [wasm][testing] hosting webSocket echo server in xharness process xharness#593
• Bring outer loop tests into inner loop via xharness on Mono platforms
  • Move existing kestrel echo server into runtime repo
    • this PR is copy of davidsh's corefx-net-endpoint, with minor improvements Refresh network tests server code to Kestrel #52642
    • this server/code should be stay owned by NCL team after merge
    • [out of scope] there are known gaps as compared to obsolete CoreFxNetCloudService as documented in Gaps in kestrel version of networking test echo server (as compared to obsolete CoreFxNetCloudService) #52693
  • move [OuterLoop("Uses external servers")] tests from outer loop on Mono platforms
    • conditional [OuterLoop] by platform - implemented conditional OuterLoopAttribute arcade#7370
    • enable network tests on mono Outerloop for browser [wasm] Enable more networking tests #53027
    • conditional OuterLoop for browser [wasm][testing] hosting echo server in xharness process #52923
    • enable more tests by avoiding crypto [wasm][testing] Avoid using crypto in WASM HTTP unit tests #53925
    • [TODO]: 199x add condition [OuterLoop("Uses external servers", TestPlatforms.Android)]
    • [Consider]: 199x add condition [OuterLoop("Uses external servers", TestPlatforms.iOS)]
      • this will require TCP tunnel via USB
    • For helix on windows we will need to send windows chromeDriver as part the helix payload.
• Bring LoopbackServer tests to Mobile platforms
  • Hosting it in Xharness via middleware binary assembly
  • This remote LoopbackServer code should not go into their Azure server above.
    • per agreement with @karelz. They don't want to expose loopback to public internet.
    • therefore this will be separate project and kestrel middleware for xharness.
  • Enable WASM tests [WIP][wasm][testing] remote loop network tests via xharness and websocket #54289
  • [Consider]: Enable LoopbackServer - Socket unit tests on Android, iOS
    • on iOS this will require TCP tunnel via USB
  • [Consider]: Enable LoopbackServer - Certificate unit tests on Android, iOS

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[akoeplinger]

Some of the http cert and websocket tests use an echo server hosted in Azure, presumably we could do the same for the basic http scenarios:

runtime/src/libraries/Common/tests/System/Net/Configuration.Http.cs

Line 44 in 6072e4d

public static string EchoClientCertificateRemoteServer => GetValue("DOTNET_TEST_HTTPHOST_ECHOCLIENTCERT", "https://corefx-net-tls.azurewebsites.net/EchoClientCertificate.ashx");

[wfurt]

If the tests can run under platform emulator using containers may be better - like we do for stress and enterprise authentication. Dependency on external servers was problematic in the past and all the tests using them are marked as Outerloop.

[stephentoub]

Some of the http cert and websocket tests use an echo server hosted in Azure

Some, yes. Two problems:

1. Going off-box results in significantly more instability of the tests, which is why most of our networking tests that do so have been relegated to outerloop, and why we've pushed to move any tests that don't require going remote to not do so. Even so, there have been discussions in the past about making it configurable.
2. Lots of test use shared in-memory state to coordinate between the client and server, in some cases just as a way to pass data back and forth, but in other cases actually synchronizing execution to ensure proper ordering for the what the test is validating. That's much harder (if not impossible) to do well with a remote server.

[wfurt]

for the state, can we use the new connection hook and use transport other than TCP socket? Pipe, memory stream or something else?

[halter73]

I'm not sure what these tests are using for the server-side, but I've started using the new SocketsHttpHandler connection hook and an in-memory stream (based on System.IO.Pipelines) in some Kestrel tests.

https://github.com/dotnet/aspnetcore/blob/925926c6156e56c6414ad004e5f1e54ced8c29c4/src/Servers/Kestrel/test/InMemory.FunctionalTests/Http2/Http2EndToEndTests.cs#L49-L61

[stephentoub]

Yes, we could use the connection callback for SocketsHttpHandler to use something other than sockets for testing... but that would only apply to SocketsHttpHandler. It wouldn't address the browser-based handler that is currently the crux of @lewing's issue. If we were to expose similar callbacks on HttpClientHandler (which wraps the browser-based handler), then maybe we could, but I don't know if the browser-based handler would even be able to support it... I suspect not... but maybe in that case supplying such a connection callback would actually opt you out of the browser-based handler? I'm waving my hands.

[lewing]

We've resolved some other blocking issues for testing the networking side in CI for Browser but we still need some way to test actual network requests even if they are going to a local endpoint.

[geoffkizer]

I don't really see any alternative here other than remote server tests. These platforms don't support doing loopback and they don't support connection-level hooks. They basically only support talking to a remote server.

As mentioned above, we do have some remote server tests, but in general they are not factored in a way that would make it easy to run only these tests (and not the loopback tests). One thing that would help here would be to refactor these tests so remote server tests are separated from loopback tests; see also #30205

The remote server tests we have today are fairly limited, but there's no reason we couldn't add some more here if we wanted. I don't think we need anywhere near the coverage for mobile/browser scenarios that we want for SocketsHttpHandler, because we are mainly relying on the platform HTTP implementation in these cases, whereas SocketsHttpHandler is implementing HTTP itself and thus we want to do extensive on-the-wire validation (which, as @stephentoub pointed out above, is only possible when using a local, in-memory server).

[wfurt]

The "loopback" can possibly be implemented externally IMHO. Let say we have external helper outside of emulator proxying connection. The "loopback" can connect to it and tell client what port to connect to. We would end-up with two stream both in process so I think most of the current logic would work.

runtime/src/libraries/Common/tests/System/Net/Http/LoopbackServer.cs

Lines 27 to 47 in 4a9ae32

	public LoopbackServer(Options options = null)
	{
	_options = options ??= new Options();
	try
	{
	_listenSocket = new Socket(options.Address.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
	_listenSocket.Bind(new IPEndPoint(options.Address, 0));
	_listenSocket.Listen(options.ListenBacklog);
	var localEndPoint = (IPEndPoint)_listenSocket.LocalEndPoint;
	string host = options.Address.AddressFamily == AddressFamily.InterNetworkV6 ?
	$"[{localEndPoint.Address}]" :
	localEndPoint.Address.ToString();
	string scheme = options.UseSsl ? "https" : "http";
	if (options.WebSocketEndpoint)
	{
	scheme = options.UseSsl ? "wss" : "ws";
	}
	_uri = new Uri($"{scheme}://{host}:{localEndPoint.Port}/");

All the helper would need to do is to pick a port, give it back to the code above and splice any data to/from the first connection to it.

Is UnixDomainSocket (or something similar) viable option @lewing? We could hijack the actual TCP and we could steer it to "loopback" via other means. Once again we would end up with test process having both sides locally available. I don't know if the platforms handlers could work on UDS but curl can AFAIK so there is at least some precedent.

[geoffkizer]

I'm not quite understanding what you are proposing, and I'm not sure how it addresses the "close coordination" issue. Maybe some code examples would help?

[wfurt]

I put together crude prototype @geoffkizer
I made small changes to Loopback: https://github.com/dotnet/runtime/compare/main...wfurt:remoteLoop?expand=1

I pick test that uses LoopbackServer.CreateClientAndServerAsync(...)
before running the test I set HELPER=192.168.0.106:10000
I started the helper on 192.168.0.106.
The loopback can run successfully without any changes and without listening and incoming socket.

With that, we should be able to run existing tests without modifications. Timing my be different but if the helper runs on Helix machine where the emulator runs, everything will be done without hitting any real network. It would be up to the logic starting the emulator to also start the loopback helper and set the environment (or let tests know via some other means.

Let me know if that make some more sense.

BTW here is the simplified helper I used for testing

helper.cs

using System;
using System.Net;
using System.Net.Sockets;
using System.Threading.Tasks;

namespace helper
{
    class Program
    {
         static async Task relayData(Socket s1, Socket s2)
        {
            using (NetworkStream ns1 = new NetworkStream(s1, ownsSocket: true))
            using (NetworkStream ns2 = new NetworkStream(s2, ownsSocket: true))
            { 

                Task t1 = ns1.CopyToAsync(ns2);
                Task t2 = ns2.CopyToAsync(ns1);
                // TBD errors and propagate half-close
                Task.WaitAll(t1, t2);
                Console.WriteLine("relayData all done");
            }
        }

        static void Main(string[] args)
        {
            byte[] port = new byte[2];
            int listenPort = 10000;

            if (args.Length > 0)
            {
                listenPort = int.Parse(args[0]);
            }
            var listener = new Socket(AddressFamily.InterNetworkV6, SocketType.Stream, ProtocolType.Tcp);
            listener.DualMode = true;
            listener.Bind(new IPEndPoint(IPAddress.IPv6Any, listenPort));
            listener.Listen(1);

            while ( true )
            {
                Socket s1 = listener.Accept();
                Console.WriteLine("Got new connection from Loopback {0} {1}", s1.LocalEndPoint, s1.RemoteEndPoint);

                var l  = new Socket(s1.AddressFamily, SocketType.Stream, ProtocolType.Tcp);
                if (((IPEndPoint)s1.LocalEndPoint).Address.IsIPv4MappedToIPv6)
                {
                    l.DualMode = true;
                }

                l.Bind(new IPEndPoint(((IPEndPoint)s1.LocalEndPoint).Address, 0));
                l.Listen(1);
                // Since we binded, the new listener has port assign.
                int p = ((IPEndPoint)l.LocalEndPoint).Port;
                port[0] = (byte)(((IPEndPoint)l.LocalEndPoint).Port & 0xff);
                port[1] = (byte)(((IPEndPoint)l.LocalEndPoint).Port >> 8);

                // send it to loopback so client knows where to connect to.
                s1.Send(port);

                // This will block until HttpClient connects 
                Socket s2 = l.Accept();
                l.Close();

                // Send one byte to signal presense of connected client.
                // We may send whole RemoteEndPoint if that is ever interesting.
                s1.Send(port, 0, 1, SocketFlags.None);
                Task.Run(() => relayData(s1, s2));
            }
        }
    }
}

[pavelsavara]

Hi @wfurt, @karelz ,

I started working on solving this problem for the browser/mobile.
In the first phase I'm not solving generic loopback socket, just the echo services.
The idea is to host the echo server in the xharness process.

I have very simple draft, in which I re-implemented 2 endpoints.
It works fine but it has two problems:

• the echo server code would move away from the tested code, into xharness repo. It makes network improvements into multi-repo PRs.
• the server code would be duplicated

I would like to be able to reuse the endpoint handler code and keep it in the runtime repo.
I thought that I would extend xharness to be able to consume assembly with the echo handlers as binary assembly via commandline switch.

To be able to do that I would have to upgrade code of the existing CoreFxNetCloudService from ASP.NET 4.5.1 to .NET core/kestrel. Unfortunately the current project doesn't even compile for me.

Right now I'm thinking that I would create new kestrel library project side by side with WebServer and make it work with xharness.

As it would eventually become functionally 1:1 with the old implementation, you could take over the new code and deploy it to Azure instead of the current one.

Does that work for you ? Do you have any suggestions ?

[karelz]

@pavelsavara I don't think the source code you found is used for our Azure endpoints.
I believe the latest version of the source code now lives in https://github.com/davidsh/corefx-net-endpoint which we have longer-term plan to move into dotnet org (I don't think it happened yet).

Can you check that source code? If it is usable for your efforts, we can move it into Runtime repo. That may save you some effort ...

[wfurt]

I'm not sure if the echo belongs to xharness. Many tests still need close coordination and as far as I understand, outbound connections work so using the Azure endpoint (or what ever else) should work.
What we discussed with @lewing was idea of plumbing socket proxy to xharness so we can sort of emulate inbound connections. Perhaps we can chat on Teams to get in sync @pavelsavara ?

[premun]

I am not sure how much you are talking about mobile devices here and how much about WASM only, but please bare in mind that on Apple devices we have in Helix (iOS 14), local network access is not possible. You cannot open any connection to a local IP. This started with iOS 14 this year, so it's something new too.

We go around this by creating a TCP tunnel through the USB cabel and then XHarness and the phone connect to a port on their localhost. I don't think there is such limitation for public network.

Maybe I am off as I don't understand the whole scenario but just something to keep in mind ^^

[pavelsavara]

We spoke with @wfurt yesterday and clarified that echo in xharness is just first step. It would unblock wasm developers to be able to create new unit tests. To to create necessary server side for testing wasm specific websocket scenarios locally, without Azure deployment.

My #52642 doesn't cover full scope of the original CoreFxNetCloudService, but I don't need it now. So I created #52693 to describe the gaps I know about.

@premun we briefly spoke about iOS limitations on Mono team meeting yesterday. @marek-safar suggested that Wasm and Android platforms are good first steps. As we discussed offline with @premun, we could enable TCP tunnel as later step for iOS.

About @wfurt idea of "asking external server to create the loopback", that could work with both Azure and xharness hosted server. For wasm, the unit test is running in the browser, where we don't have full socket, nor pipe. So we would have to implement the "ask"/coordination part with websockets. Again, this is later step from my perspective.

[karelz]

@pavelsavara I am a bit lost in the overall plan. What happens when and who does it?
Would it make se to agree on a plan and then keep up-to-date version in the top post?

[pavelsavara]

@karelz I updated the issue description with my plan. I will talk to @lewing to confirm priorities. Thanks for your feedback.

[wfurt]

What exactly do you mean by 'local network' @premun? Does it mean that you can access remote networks but not directly connected one? I assume that is some limitation of the emulator, right?

[premun]

@wfurt this is for iOS devices. Simulators seem fine but I didn't try it much. I haven't tried remote networks. For those you might need some permission which you can put in the application's manifest.

Details on the local network access:
https://support.apple.com/en-mk/HT211870
https://developer.apple.com/videos/play/wwdc2020/10110/

You cannot connect to things that "don't look localhost" but I don't understand how Apple tells it's "local". We were originally having a direct connection and we were trying to connect to things like these (which ended up in the dialog):

• fe80::1%1
• 127.0.0.1
• 10.0.0.14
• fe80::f9:699f:80e1:33a2%6
• 2a00:1028:838a:5156:1889:420c:e658:43cf
• 2a00:1028:838a:5156:8194:4691:1156:90ed
• fe80::6c5f:97ff:fe0c:597c%11
• fe80::6c5f:97ff:fe0c:597c%12
• fe80::c127:785a:da15:3304%13
• fe80::563e:5f37:bc04:aba7%14

The original intention was so that you cannot scan devices around and figure out location or fingerprint people's home...

[wfurt]

good to know @premun. I think the work @pavelsavara is doing can still work. Aside from external server, he is going to do simple prototype with "Loopback", Once that ready you can perhaps check if that would work on iOS devices and provide more feedback.

[pavelsavara]

Both loopback and echo server are now also implemented as kestrel middleware for Xhareness process.
Many tests are now enabled for WASM platform.

@mdh1418 is looking at Android.

[mdh1418]

Closing this issue as the wasm side is now complete.
Separating mobile work into another issue #54626