Unable to join any multiplayer lobby on Android

[LeNitrous]

Describe the bug:
Attempting to join any multiplayer lobby on Android throws an error. On the host's side, they will momentarily see the player joining but then suddenly leaves the lobby.

osu!lazer version:
2021.815.0-lazer

Logs:
network.log
runtime.log

[frenzibyte]

Looks to be an issue with JSON deserialisation on the Android builds, happening while invoking the multiplayer hub's JoinRoom method:

2021-08-15 14:08:59 [error]: Failed to join multiplayer room.
2021-08-15 14:08:59 [error]: System.AggregateException: One or more errors occurred. (One or more errors occurred. (Error resolving type specified in JSON 'System.Collections.Generic.List`1[[osu.Game.Online.Multiplayer.MultiplayerRoomUser, osu.Game]], System.Private.CoreLib'. Path 'result.users.$type', line 1, position 691.)) ---> System.AggregateException: One or more errors occurred. (Error resolving type specified in JSON 'System.Collections.Generic.List`1[[osu.Game.Online.Multiplayer.MultiplayerRoomUser, osu.Game]], System.Private.CoreLib'. Path 'result.users.$type', line 1, position 691.) ---> Newtonsoft.Json.JsonSerializationException: Error resolving type specified in JSON 'System.Collections.Generic.List`1[[osu.Game.Online.Multiplayer.MultiplayerRoomUser, osu.Game]], System.Private.CoreLib'. Path 'result.users.$type', line 1, position 691. ---> Newtonsoft.Json.JsonSerializationException: Could not find type 'System.Collections.Generic.List`1[[osu.Game.Online.Multiplayer.MultiplayerRoomUser, osu.Game]]' in assembly 'System, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e'.
2021-08-15 14:08:59 [error]: at Newtonsoft.Json.Serialization.DefaultSerializationBinder.GetTypeFromTypeNameKey (Newtonsoft.Json.Utilities.StructMultiKey`2[T1,T2] typeNameKey) [0x000fc] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 14:08:59 [error]: at System.Collections.Concurrent.ConcurrentDictionary`2[TKey,TValue].GetOrAdd (TKey key, System.Func`2[T,TResult] valueFactory) [0x00034] in <f537a3406b1f4b35865d9fd806fd9940>:0
2021-08-15 14:08:59 [error]: at Newtonsoft.Json.Utilities.ThreadSafeStore`2[TKey,TValue].Get (TKey key) [0x00000] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 14:08:59 [error]: at Newtonsoft.Json.Serialization.DefaultSerializationBinder.GetTypeByName (Newtonsoft.Json.Utilities.StructMultiKey`2[T1,T2] typeNameKey) [0x00000] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 14:08:59 [error]: at Newtonsoft.Json.Serialization.DefaultSerializationBinder.BindToType (System.String assemblyName, System.String typeName) [0x00008] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 14:08:59 [error]: at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolveTypeName (Newtonsoft.Json.JsonReader reader, System.Type& objectType, Newtonsoft.Json.Serialization.JsonContract& contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.String qualifiedTypeName) [0x00094] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 14:08:59 [error]: --- End of inner exception stack trace ---
2021-08-15 14:08:59 [error]: at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolveTypeName (Newtonsoft.Json.JsonReader reader, System.Type& objectType, Newtonsoft.Json.Serialization.JsonContract& contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.String qualifiedTypeName) [0x000d1] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 14:08:59 [error]: at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadMetadataProperties (Newtonsoft.Json.JsonReader reader, System.Type& objectType, Newtonsoft.Json.Serialization.JsonContract& contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue, System.Object& newValue, System.String& id) [0x00159] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 14:08:59 [error]: at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) [0x000bc] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 14:08:59 [error]: at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) [0x0006d] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 14:08:59 [error]: at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues (Newtonsoft.Json.Serialization.JsonObjectContract contract, Newtonsoft.Json.Serialization.JsonProperty containerProperty, Newtonsoft.Json.JsonReader reader, System.Type objectType) [0x00128] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 14:08:59 [error]: at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters (Newtonsoft.Json.JsonReader reader, Newtonsoft.Json.Serialization.JsonObjectContract contract, Newtonsoft.Json.Serialization.JsonProperty containerProperty, Newtonsoft.Json.Serialization.ObjectConstructor`1[T] creator, System.String id) [0x000b2] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 14:08:59 [error]: at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewObject (Newtonsoft.Json.JsonReader reader, Newtonsoft.Json.Serialization.JsonObjectContract objectContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, Newtonsoft.Json.Serialization.JsonProperty containerProperty, System.String id, System.Boolean& createdFromNonDefaultCreator) [0x00026] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 14:08:59 [error]: at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) [0x00148] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 14:08:59 [error]: at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) [0x0006d] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 14:08:59 [error]: at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize (Newtonsoft.Json.JsonReader reader, System.Type objectType, System.Boolean checkAdditionalContent) [0x000db] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 14:08:59 [error]: at Newtonsoft.Json.JsonSerializer.DeserializeInternal (Newtonsoft.Json.JsonReader reader, System.Type objectType) [0x00054] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 14:08:59 [error]: at Newtonsoft.Json.JsonSerializer.Deserialize (Newtonsoft.Json.JsonReader reader, System.Type objectType) [0x00000] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 14:08:59 [error]: at Microsoft.AspNetCore.SignalR.Protocol.NewtonsoftJsonHubProtocol.ParseMessage (Microsoft.AspNetCore.SignalR.Internal.Utf8BufferTextReader textReader, Microsoft.AspNetCore.SignalR.IInvocationBinder binder) [0x00343] in <704644bbc94c4c03ad80150814b72fa4>:0
2021-08-15 14:08:59 [error]: at Microsoft.AspNetCore.SignalR.Protocol.NewtonsoftJsonHubProtocol.TryParseMessage (System.Buffers.ReadOnlySequence`1[System.Byte]& input, Microsoft.AspNetCore.SignalR.IInvocationBinder binder, Microsoft.AspNetCore.SignalR.Protocol.HubMessage& message) [0x00017] in <704644bbc94c4c03ad80150814b72fa4>:0
2021-08-15 14:08:59 [error]: at Microsoft.AspNetCore.SignalR.Client.HubConnection.ReceiveLoop (Microsoft.AspNetCore.SignalR.Client.HubConnection+ConnectionState connectionState) [0x001ad] in <9b5c746cf3e24d3daddf4fedfbec16df>:0
2021-08-15 14:08:59 [error]: at Microsoft.AspNetCore.SignalR.Client.HubConnection.InvokeCoreAsyncCore (System.String methodName, System.Type returnType, System.Object[] args, System.Threading.CancellationToken cancellationToken) [0x00218] in <9b5c746cf3e24d3daddf4fedfbec16df>:0
2021-08-15 14:08:59 [error]: at System.Threading.Tasks.ForceAsyncAwaiter`1[T].GetResult () [0x0000c] in <9b5c746cf3e24d3daddf4fedfbec16df>:0
2021-08-15 14:08:59 [error]: at Microsoft.AspNetCore.SignalR.Client.HubConnection.InvokeCoreAsync (System.String methodName, System.Type returnType, System.Object[] args, System.Threading.CancellationToken cancellationToken) [0x0009e] in <9b5c746cf3e24d3daddf4fedfbec16df>:0
2021-08-15 14:08:59 [error]: at Microsoft.AspNetCore.SignalR.Client.HubConnectionExtensions.InvokeCoreAsync[TResult] (Microsoft.AspNetCore.SignalR.Client.HubConnection hubConnection, System.String methodName, System.Object[] args, System.Threading.CancellationToken cancellationToken) [0x0008f] in <9b5c746cf3e24d3daddf4fedfbec16df>:0
2021-08-15 14:08:59 [error]: at osu.Game.Online.Multiplayer.MultiplayerClient+<>c__DisplayClass45_0.<JoinRoom>b__0 () [0x000e9] in <201b7db2f23c45c7bcd1ea5ec4fb07b2>:0

[peppy]

Your log shows your build as 2021.813 - how is this a thing?

[LeNitrous]

Just to make sure I checked the version under settings. Here's what I got

The timestamps seem to match up to today's date. I have no idea what happened there.

[peppy]

Thanks.

Are any other android users able to confirm this?

[wowcake]

Could reproduce on my device aswell in the latest version.

(can't provide logs because Android 11 doesn't let me access the osu!lazer folder)

[maotovisk]

Tested on 2021.815.0-lazer and it indeed does not let me join any rooms.

runtime.log
network.log
performance.log
database.log
Edit: Android version is 10

[bdach]

i'm pretty sure we've got enough confirmations, what we need now is a root cause.

i'll look at it later if i'm not pre-empted.

[peppy]

The likely cause is that we forced json serialisation with the recent team vs changes (see HubClientConnector's ctor). Something must be broken in that area.

[bdach]

I've poked this from a few angles but got nothing concrete yet. It doesn't help that you don't get very nice stacks on xamarin/android in not-own code, as is tradition on xamarin.

The next thing I am looking to try is setting

options.PayloadSerializerSettings.TypeNameAssemblyFormatHandling = TypeNameAssemblyFormatHandling.Full;

but I would obviously have to set that on the spectator server end too, which means that I'd have to set up local instances of spectator server and web that are reachable over my local network so that the android app can hit them, which is a not-insignificant amount of work and could take a while.

Edit: As mentioned on discord @peppy has enabled the above flag for me for a second on the dev server, but it doesn't seem to help:

2021-08-15 21:43:23 [verbose]: OnlineMultiplayerClient connection error: Newtonsoft.Json.JsonSerializationException: Error resolving type specified in JSON 'System.Collections.Generic.List`1[[osu.Game.Online.Multiplayer.MultiplayerRoomUser, osu.Game, Version=2021.814.0.0, Culture=neutral, PublicKeyToken=null]], System.Private.CoreLib, Version=5.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e'. Path 'result.users.$type', line 1, position 1054. ---> Newtonsoft.Json.JsonSerializationException: Could not find type 'System.Collections.Generic.List`1[[osu.Game.Online.Multiplayer.MultiplayerRoomUser, osu.Game, Version=2021.814.0.0, Culture=neutral, PublicKeyToken=null]]' in assembly 'System, Version=2.0.5.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e'.
2021-08-15 21:43:23 [verbose]: at Newtonsoft.Json.Serialization.DefaultSerializationBinder.GetTypeFromTypeNameKey (Newtonsoft.Json.Utilities.StructMultiKey`2[T1,T2] typeNameKey) [0x000fc] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 21:43:23 [verbose]: at System.Collections.Concurrent.ConcurrentDictionary`2[TKey,TValue].GetOrAdd (TKey key, System.Func`2[T,TResult] valueFactory) [0x00034] in /Users/builder/jenkins/workspace/archive-mono/2020-02/android/release/external/corefx/src/System.Collections.Concurrent/src/System/Collections/Concurrent/ConcurrentDictionary.cs:1002
2021-08-15 21:43:23 [verbose]: at Newtonsoft.Json.Utilities.ThreadSafeStore`2[TKey,TValue].Get (TKey key) [0x00000] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 21:43:23 [verbose]: at Newtonsoft.Json.Serialization.DefaultSerializationBinder.GetTypeByName (Newtonsoft.Json.Utilities.StructMultiKey`2[T1,T2] typeNameKey) [0x00000] in <7ca8898b690a4181a32a9cf767cedb1e>:0
2021-08-15 21:43:23 [verbose]: at Newtonsoft.Json.Serialization.DefaultSerializationBinder.BindToType (System.String assemblyName, System.String typeName) [0x00008] in <7ca8898b690a4181a32a9cf767cedb1e>:0

[bdach]

After some more digging in issue threads over at the newtonsoft repository I found this hint of adding a custom serialisation binder that resolves types supposedly coming from System.Private.CoreLib against mscorlib, and after adding that custom binder and setting

options.PayloadSerializerSettings.SerializationBinder = new DotNetCompatibleSerializationBinder();

I was able to complete a multiplayer game.

I haven't thought much about how to package that "workaround", though (if at all) - it would probably be nice to have it live in the mobile projects because I imagine that binder can't be set universally.

[bdach]

I was preparing to PR the above workaround but unfortunately there is a complication - I have found yet another instance where this same exception happens for another type, namely System.Linq.EmptyPartition.

This is likely caused by the spectator server leaking implementation-specific collection types over the wire in the $type discriminator property due to specifying TypeNameHandling.All. I'm not even sure whether EmptyPartition exists on xamarin/mono/netstandard/whatever is used on android, but I wouldn't be surprised if it didn't.

I'm wondering whether specifying TypeNameHandling.Objects might be a viable out for this, but I again need a spectator server setup to test this, as it's the server-originating type names which are tripping up newtonsoft on xamarin.

[bdach]

TypeNameHandling.Objects doesn't work on its own either. Because the spectator server serializes and sends over the wire dictionaries as well, which map to json objects. And so they come with their entire dictionary type spec which also mentions System.Private.CoreLib as the originating assembly.

As it stands probably the only options left to preserve multiplayer working on android are:

1. Hand-roll the type name handling for Match{User,Room}State by writing a custom converter (for writing) and serialization binder (for reading) on both sides.
2. Use TypeNameHandling.Objects on both sides, as well as the compatibility serialization binder mentioned above.
3. ...Or be very careful with return types on both sides (only send over List<T>/Dictionary<TKey,TValue>, never IEnumerable or IDictionary).

I'm not sure which one is the worse option at this point.

[frenzibyte]

I think solution (2) would be the least worse, not requiring to write more custom serialisation code on new structures, or become very careful with return types going forward.

[peppy]

I think (3) is the easiest / lowest overhead implementation. Will look at that today.

[bdach]

Oh, slight correction: (3) will still need the workaround serialisation binder to swap out System.Private.CoreLib for mscorlib. But it will fix cases where raw live linq iterator type names are sent over the wire, as those cannot be so easily unmapped on the xamarin side.

[peppy]

Option 4 of course is to get messagepack working again and stick with that. I think this can also be done with a custom binder, and may be our best option? At least it has consistent(*probably) behaviour across platforms, and is broken in the same way on desktop as mobile currently.

[bdach]

Yeah, if it can be somehow nudged into respecting [Union] manually, then that would indeed be ideal. It's basically what (1) is but combined with bringing back msgpack.