`SelectAsync` stopped working even when async code block is guarded by a try...catch block

[Arkatufus]

Version Information
Version of Akka.NET? 1.5.38
Which Akka.NET Modules? Akka.Streams

Describe the bug

Related issues:

• BaseByteArrayJournalDao lacks resiliency against stream failure Akka.Persistence.Sql#497
• Stream cancellation exception can be null Akka.Persistence.Sql#498
• Stream upstream error propagation can fail in async calls #7432

This has been observed inside the Akka.Persistence.Sql plugin, where this SelectAsync stage suddenly stopped working due to ArgumentException "Cancellation cause must not be null (Parameter 'cause')":

https://github.com/akkadotnet/Akka.Persistence.Sql/blob/e034a57789a7a982fd18e385311298adc055df01/src/Akka.Persistence.Sql/Journal/Dao/BaseByteArrayJournalDao.cs#L72-L89

Note that the SelectAsync code body is running inside a try...catch block.

We've fixed the symptom here: #7433 and here: #7497 , but the problem still persists, causing the Akka.Persistence.Sql write journal to stop working completely for some user with StreamDetachedException "Stream is terminated. Materialized value is detached.", the same symptom seen in akkadotnet/Akka.Persistence.Sql#497 . So far, we tried to keep the write journal going by introducing a Restart decider, but logically, we shouldn't need it; the stream should've kept going until the process stops with no errors.

Something in Akka.Streams is stopping the stream with null exception as the cause of the stop. The only point where exceptions can start to bubble up the stream stages is here inside the GraphInterpreter class:

akka.net/src/core/Akka.Streams/Implementation/Fusing/GraphInterpreter.cs

Line 866 in 6a5c7c0

connection.OutHandler.OnDownstreamFinish(cause);

, so far this is the only place inside Akka.Streams that can cause a stage cancellation that can explain the original symptom. Code review so far hasn't found any probable site where and/or when a null can possibly be injected into this code.

[Aaronontheweb]

We need to test if this is a ConfigureAwait issue - having the task escape the execution context is one of the things that could cause this.

[Aaronontheweb]

Also impacts akkadotnet/Akka.Streams.Kafka#426