Mono.Net.Security.MonoTlsStream: Crash when stream appears to have already been disposed

[tdiehl]

Steps to Reproduce

Unfortunately I don't yet have a reproducible case, we are observing this crash in production release of an Xamarin iOS app.

Current Behavior

We are getting crashes MonoTlsStream when the stream is being disposed here.

I suspect the behavior may be because of a timeout or some work being cancelled, perhaps causing the stream to be disposed here first.

Expected Behavior

No crash.

On which platforms did you notice this

[x] iOS
[ ] macOS
[ ] Linux
[ ] Windows

Version Used:
Mono JIT compiler version 5.18.1.28 (2018-08/223ea7ef92e Tue May 21 12:03:39 EDT 2019)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
TLS:
SIGSEGV: altstack
Notification: kqueue
Architecture: amd64
Disabled: none
Misc: softdebug
Interpreter: yes
LLVM: yes(600)
Suspend: preemptive
GC: sgen (concurrent by default)

Stacktrace

Crashed: Thread Pool Worker
EXC_BAD_ACCESS KERN_INVALID_ADDRESS 0x0000000000000000

0  FaithlifeReaderApp             0x103db6b88 Mono_Net_Security_MonoTlsStream__CreateStreamd__17_MoveNext + 139 (MonoTlsStream.cs:139)
1  FaithlifeReaderApp             0x103db6b7c Mono_Net_Security_MonoTlsStream__CreateStreamd__17_MoveNext + 139 (MonoTlsStream.cs:139)
2  FaithlifeReaderApp             0x1044b63fc System_Runtime_CompilerServices_AsyncMethodBuilderCore_MoveNextRunner_InvokeMoveNext_object + 20745408
3  FaithlifeReaderApp             0x10443b2d0 System_Threading_ExecutionContext_RunInternal_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool + 20241300
4  FaithlifeReaderApp             0x10443b11c System_Threading_ExecutionContext_Run_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool + 20240864
5  FaithlifeReaderApp             0x1044b62bc System_Runtime_CompilerServices_AsyncMethodBuilderCore_MoveNextRunner_Run + 20745088
6  FaithlifeReaderApp             0x10445eb94 System_Threading_Tasks_AwaitTaskContinuation_RunOrScheduleAction_System_Action_bool_System_Threading_Tasks_Task_ + 20386904
7  FaithlifeReaderApp             0x1044587c0 System_Threading_Tasks_Task_FinishContinuations + 20361348
8  FaithlifeReaderApp             0x104456d80 System_Threading_Tasks_Task_FinishStageThree + 20354628
9  FaithlifeReaderApp             0x104456cc0 System_Threading_Tasks_Task_FinishStageTwo + 20354436
10 FaithlifeReaderApp             0x104456988 System_Threading_Tasks_Task_Finish_bool + 20353612
11 FaithlifeReaderApp             0x104454470 System_Threading_Tasks_Task_TrySetException_object + 20344116
12 FaithlifeReaderApp             0x104665fec System_Runtime_CompilerServices_AsyncTaskMethodBuilder_1_System_Threading_Tasks_VoidTaskResult_SetException_System_Exception + 22513840
13 FaithlifeReaderApp             0x1044b4618 System_Runtime_CompilerServices_AsyncTaskMethodBuilder_SetException_System_Exception + 20737756
14 FaithlifeReaderApp             0x103db3840 Mono_Net_Security_MobileAuthenticatedStream__ProcessAuthenticationd__62_MoveNext + 396 (MobileAuthenticatedStream.cs:396)
15 FaithlifeReaderApp             0x1044b63fc System_Runtime_CompilerServices_AsyncMethodBuilderCore_MoveNextRunner_InvokeMoveNext_object + 20745408
16 FaithlifeReaderApp             0x10443b2d0 System_Threading_ExecutionContext_RunInternal_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool + 20241300
17 FaithlifeReaderApp             0x10443b11c System_Threading_ExecutionContext_Run_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool + 20240864
18 FaithlifeReaderApp             0x1044b62bc System_Runtime_CompilerServices_AsyncMethodBuilderCore_MoveNextRunner_Run + 20745088
19 FaithlifeReaderApp             0x10445eb94 System_Threading_Tasks_AwaitTaskContinuation_RunOrScheduleAction_System_Action_bool_System_Threading_Tasks_Task_ + 20386904
20 FaithlifeReaderApp             0x1044587c0 System_Threading_Tasks_Task_FinishContinuations + 20361348
21 FaithlifeReaderApp             0x104456d80 System_Threading_Tasks_Task_FinishStageThree + 20354628
22 FaithlifeReaderApp             0x1044519e8 System_Threading_Tasks_Task_1_TResult_REF_TrySetResult_TResult_REF + 20333228
23 FaithlifeReaderApp             0x1044b4c38 System_Runtime_CompilerServices_AsyncTaskMethodBuilder_1_TResult_REF_SetResult_TResult_REF + 20739324
24 FaithlifeReaderApp             0x103dae570 Mono_Net_Security_AsyncProtocolRequest__StartOperationd__23_MoveNext + 193 (AsyncProtocolRequest.cs:193)
25 FaithlifeReaderApp             0x1044b63fc System_Runtime_CompilerServices_AsyncMethodBuilderCore_MoveNextRunner_InvokeMoveNext_object + 20745408
26 FaithlifeReaderApp             0x10443b2d0 System_Threading_ExecutionContext_RunInternal_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool + 20241300
27 FaithlifeReaderApp             0x10443b11c System_Threading_ExecutionContext_Run_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool + 20240864
28 FaithlifeReaderApp             0x1044b62bc System_Runtime_CompilerServices_AsyncMethodBuilderCore_MoveNextRunner_Run + 20745088
29 FaithlifeReaderApp             0x10445eb94 System_Threading_Tasks_AwaitTaskContinuation_RunOrScheduleAction_System_Action_bool_System_Threading_Tasks_Task_ + 20386904
30 FaithlifeReaderApp             0x1044587c0 System_Threading_Tasks_Task_FinishContinuations + 20361348
31 FaithlifeReaderApp             0x104456d80 System_Threading_Tasks_Task_FinishStageThree + 20354628
32 FaithlifeReaderApp             0x104456cc0 System_Threading_Tasks_Task_FinishStageTwo + 20354436
33 FaithlifeReaderApp             0x104456988 System_Threading_Tasks_Task_Finish_bool + 20353612
34 FaithlifeReaderApp             0x104454470 System_Threading_Tasks_Task_TrySetException_object + 20344116
35 FaithlifeReaderApp             0x104665fec System_Runtime_CompilerServices_AsyncTaskMethodBuilder_1_System_Threading_Tasks_VoidTaskResult_SetException_System_Exception + 22513840
36 FaithlifeReaderApp             0x1044b4618 System_Runtime_CompilerServices_AsyncTaskMethodBuilder_SetException_System_Exception + 20737756
37 FaithlifeReaderApp             0x103daebc0 Mono_Net_Security_AsyncProtocolRequest__ProcessOperationd__24_MoveNext + 198 (AsyncProtocolRequest.cs:198)
38 FaithlifeReaderApp             0x1044b63fc System_Runtime_CompilerServices_AsyncMethodBuilderCore_MoveNextRunner_InvokeMoveNext_object + 20745408
39 FaithlifeReaderApp             0x10443b2d0 System_Threading_ExecutionContext_RunInternal_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool + 20241300
40 FaithlifeReaderApp             0x10443b11c System_Threading_ExecutionContext_Run_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool + 20240864
41 FaithlifeReaderApp             0x1044b62bc System_Runtime_CompilerServices_AsyncMethodBuilderCore_MoveNextRunner_Run + 20745088
42 FaithlifeReaderApp             0x10445eb94 System_Threading_Tasks_AwaitTaskContinuation_RunOrScheduleAction_System_Action_bool_System_Threading_Tasks_Task_ + 20386904
43 FaithlifeReaderApp             0x1044587c0 System_Threading_Tasks_Task_FinishContinuations + 20361348
44 FaithlifeReaderApp             0x104456d80 System_Threading_Tasks_Task_FinishStageThree + 20354628
45 FaithlifeReaderApp             0x104456cc0 System_Threading_Tasks_Task_FinishStageTwo + 20354436
46 FaithlifeReaderApp             0x104456988 System_Threading_Tasks_Task_Finish_bool + 20353612
47 FaithlifeReaderApp             0x104454470 System_Threading_Tasks_Task_TrySetException_object + 20344116
48 FaithlifeReaderApp             0x103ef8f0c System_Runtime_CompilerServices_AsyncTaskMethodBuilder_1_System_Nullable_1_int_SetException_System_Exception + 662 (AsyncMethodBuilder.cs:662)
49 FaithlifeReaderApp             0x103daf06c Mono_Net_Security_AsyncProtocolRequest__InnerReadd__25_MoveNext + 260 (AsyncProtocolRequest.cs:260)
50 FaithlifeReaderApp             0x1044b63fc System_Runtime_CompilerServices_AsyncMethodBuilderCore_MoveNextRunner_InvokeMoveNext_object + 20745408
51 FaithlifeReaderApp             0x10443b2d0 System_Threading_ExecutionContext_RunInternal_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool + 20241300
52 FaithlifeReaderApp             0x10443b11c System_Threading_ExecutionContext_Run_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool + 20240864
53 FaithlifeReaderApp             0x1044b62bc System_Runtime_CompilerServices_AsyncMethodBuilderCore_MoveNextRunner_Run + 20745088
54 FaithlifeReaderApp             0x10445eb94 System_Threading_Tasks_AwaitTaskContinuation_RunOrScheduleAction_System_Action_bool_System_Threading_Tasks_Task_ + 20386904
55 FaithlifeReaderApp             0x1044587c0 System_Threading_Tasks_Task_FinishContinuations + 20361348
56 FaithlifeReaderApp             0x104456d80 System_Threading_Tasks_Task_FinishStageThree + 20354628
57 FaithlifeReaderApp             0x104456cc0 System_Threading_Tasks_Task_FinishStageTwo + 20354436
58 FaithlifeReaderApp             0x104456988 System_Threading_Tasks_Task_Finish_bool + 20353612
59 FaithlifeReaderApp             0x104454470 System_Threading_Tasks_Task_TrySetException_object + 20344116
60 FaithlifeReaderApp             0x104651f24 System_Runtime_CompilerServices_AsyncTaskMethodBuilder_1_TResult_INT_SetException_System_Exception + 22431720
61 FaithlifeReaderApp             0x103db4768 Mono_Net_Security_MobileAuthenticatedStream__InnerReadd__85_MoveNext + 747 (MobileAuthenticatedStream.cs:747)
62 FaithlifeReaderApp             0x1044b63fc System_Runtime_CompilerServices_AsyncMethodBuilderCore_MoveNextRunner_InvokeMoveNext_object + 20745408
63 FaithlifeReaderApp             0x10443b2d0 System_Threading_ExecutionContext_RunInternal_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool + 20241300
64 FaithlifeReaderApp             0x10443b11c System_Threading_ExecutionContext_Run_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool + 20240864
65 FaithlifeReaderApp             0x1044b62bc System_Runtime_CompilerServices_AsyncMethodBuilderCore_MoveNextRunner_Run + 20745088
66 FaithlifeReaderApp             0x10445eb94 System_Threading_Tasks_AwaitTaskContinuation_RunOrScheduleAction_System_Action_bool_System_Threading_Tasks_Task_ + 20386904
67 FaithlifeReaderApp             0x1044587c0 System_Threading_Tasks_Task_FinishContinuations + 20361348
68 FaithlifeReaderApp             0x104456d80 System_Threading_Tasks_Task_FinishStageThree + 20354628
69 FaithlifeReaderApp             0x104456cc0 System_Threading_Tasks_Task_FinishStageTwo + 20354436
70 FaithlifeReaderApp             0x104456988 System_Threading_Tasks_Task_Finish_bool + 20353612
71 FaithlifeReaderApp             0x104454470 System_Threading_Tasks_Task_TrySetException_object + 20344116
72 FaithlifeReaderApp             0x1046694e4 System_Threading_Tasks_TaskFactory_1_FromAsyncTrimPromise_1_TResult_INT_TInstance_REF_Complete_TInstance_REF_System_Func_3_TInstance_REF_System_IAsyncResult_TResult_INT_System_IAsyncResult_bool + 22527400
73 FaithlifeReaderApp             0x104669314 System_Threading_Tasks_TaskFactory_1_FromAsyncTrimPromise_1_TResult_INT_TInstance_REF_CompleteFromAsyncResult_System_IAsyncResult + 22526936
74 FaithlifeReaderApp             0x103e86374 System_Net_Sockets_SocketAsyncResult__c__Completeb__27_0_object + 157 (SocketAsyncResult.cs:157)
75 FaithlifeReaderApp             0x104443054 System_Threading_QueueUserWorkItemCallback_System_Threading_IThreadPoolWorkItem_ExecuteWorkItem + 20273432
76 FaithlifeReaderApp             0x104440df8 System_Threading_ThreadPoolWorkQueue_Dispatch + 20264636
77 FaithlifeReaderApp             0x10477a200 ObjCRuntime_Runtime_ThreadPoolDispatcher_System_Func_1_bool + 288 (Runtime.cs:288)
78 FaithlifeReaderApp             0x104442e88 System_Threading__ThreadPoolWaitCallback_PerformWaitCallback + 20272972
79 FaithlifeReaderApp             0x1046812e8 wrapper_runtime_invoke_object_runtime_invoke_dynamic_intptr_intptr_intptr_intptr + 22625196
80 FaithlifeReaderApp             0x102f8328c mono_jit_runtime_invoke + 3120 (mini-runtime.c:3120)
81 FaithlifeReaderApp             0x1030280ec mono_runtime_try_invoke + 2960 (object.c:2960)
82 FaithlifeReaderApp             0x103059c10 worker_callback + 284 (threadpool.c:284)
83 FaithlifeReaderApp             0x103057980 worker_thread + 476 (threadpool-worker-default.c:476)
84 FaithlifeReaderApp             0x103062110 start_wrapper_internal + 1155 (threads.c:1155)
85 FaithlifeReaderApp             0x103061f94 start_wrapper + 1217 (threads.c:1217)
86 libsystem_pthread.dylib        0x215e542c0 _pthread_body + 128
87 libsystem_pthread.dylib        0x215e54220 _pthread_start + 44
88 libsystem_pthread.dylib        0x215e57cdc thread_start + 4

[marek-safar]

/cc @steveisok

[filipnavara]

Sounds suspiciously similar to #10488. If that is the case then updating to newer version should resolve the issue.

[steveisok]

Yeah, it does. @baulig, please confirm.

[baulig]

I reviewed the code and it turns out that we're actually lacking a Dispose () call in the second catch block (which I just fixed in #16233).

Apart from that, I don't see how we could possibly throw here. Do you have any test case or some way to reproduce this?

[tdiehl]

Unfortunately, I don't have a reproducible test at the moment.

We've been running for the past couple of weeks with a custom build of Mono in our Xamarin apps that uses a simple null check, which has eliminated this particular crash. Naturally I'd prefer not to have to resort to that permanently. :)

[digvijaypundir1]

Unfortunately, I don't have a reproducible test at the moment.

We've been running for the past couple of weeks with a custom build of Mono in our Xamarin apps that uses a simple null check, which has eliminated this particular crash. Naturally I'd prefer not to have to resort to that permanently. :)

I am also having the same crash. I don't know about how we can do it in our solution. As mono is the part of .net framework. Is it open source and we can modify directly?
Or there is any other solution for that as you stated custom build for Mono? Is it simply take the latest using git and change use that in our project?

Hope solution will come in next release of mono.

[ppvictory]

Also experiencing the same crash here. It look's like we'll have to wait until the next release of Xamarin.iOS to use the latest mono and see if this fixes this crash.

[pachecosoftware]

We too are experiencing this issue. Crashyltics is reporting that it has affected over 1000 users over the past week, so it seems to be a pretty significant issue for us. @tdiehl, can you elaborate regarding how you were able to use your patched version of Mono in your Xamarin.iOS app?

[pachecosoftware]

We just upgraded to the latest and greatest versions of Xamarin iOS and Mono but are still seeing this issue. Any idea on when this fix might get included in a new build of Mono?

[steveisok]

@baulig The dispose fix should be in stable. Can you please do another pass on this issue.

[pachecosoftware]

@steveisok, thanks for the update. Does @tdiehl 's workaround of adding the null check need to be included in the next build in addition to the dispose fix that @baulig worked on in order to fully resolve this issue?

[steveisok]

@pachecosoftware according to the PR @baulig merged, the null check is unnecessary. https://github.com/mono/mono/pull/16233/files#r314017987

If I'm understanding your comments correctly, you're seeing otherwise?