[MinIO] GetObjectAsync throws System.ObjectDisposedException

[mocsharp]

Description

With MinIO plug-in, the GetObjectAsync call throws the following exceptions:

Unhandled exception. System.ObjectDisposedException: Cannot access a disposed object.
Object name: 'System.Net.Sockets.NetworkStream'.
   at System.Net.Sockets.NetworkStream.<ThrowIfDisposed>g__ThrowObjectDisposedException|63_0()
   at System.Net.Sockets.NetworkStream.ReadAsync(Memory`1 buffer, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.FillAsync(Boolean async)
   at System.Net.Http.HttpConnection.CopyToContentLengthAsync(Stream destination, Boolean async, UInt64 length, Int32 bufferSize, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnection.ContentLengthReadStream.CompleteCopyToAsync(Task copyTask, CancellationToken cancellationToken)
   at Monai.Deploy.Storage.MinIO.MinIoStorageService.<>c__DisplayClass9_0.<<GetObjectAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
   at System.Threading.QueueUserWorkItemCallback.<>c.<.cctor>b__6_0(QueueUserWorkItemCallback quwi)
   at System.Threading.ExecutionContext.RunForThreadPoolUnsafe[TState](ExecutionContext executionContext, Action`1 callback, TState& state)
   at System.Threading.QueueUserWorkItemCallback.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
   at System.Threading.Thread.StartCallback()

or

Unhandled exception. System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.
 ---> System.TimeoutException: The operation was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
 ---> System.Net.Sockets.SocketException (125): Operation canceled
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Minio.MinioClient.ExecuteTaskCoreAsync(IEnumerable`1 errorHandlers, HttpRequestMessageBuilder requestMessageBuilder, CancellationToken cancellationToken, Boolean isSts) in /root/.q/sources/minio-dotnet/Minio/MinioClient.cs:line 526
   at Minio.MinioClient.StatObjectAsync(StatObjectArgs args, CancellationToken cancellationToken) in /root/.q/sources/minio-dotnet/Minio/ApiEndpoints/ObjectOperations.cs:line 1093
   at Minio.MinioClient.getObjectHelper(GetObjectArgs args, CancellationToken cancellationToken) in /root/.q/sources/minio-dotnet/Minio/Helper/OperationsHelper.cs:line 50
   at Monai.Deploy.Storage.MinIO.MinIoStorageService.GetObjectUsingClient(MinioClient client, String bucketName, String objectName, Action`1 callback, CancellationToken cancellationToken) in /home/vicchang/sc/monai/monai-deploy-storage/src/Plugins/MinIO/MinIoStorageService.cs:line 305
   at Monai.Deploy.Storage.MinIO.MinIoStorageService.GetObjectAsync(String bucketName, String objectName, CancellationToken cancellationToken) in /home/vicchang/sc/monai/monai-deploy-storage/src/Plugins/MinIO/MinIoStorageService.cs:line 73
   at Program.Main(String[] args) in /home/vicchang/sc/monai/monai-deploy-storage/src/test/Program.cs:line 29
   at Program.<Main>(String[] args)

Steps to reproduce

Please share a clear and concise description of the problem.

1. Call the GetObjectAsync method multiple times, >20 and it will throw an exception.

Expected behavior

GetObjectAsync shall be able to return the stream.

Actual behavior

Throws exception.

Configuration

• MONAI Deploy Storage Library version/commit: 0.1.0
• OS and version (distro if applicable): ubuntu 20.04
• Kubernetes version (if applicable): NA
• Docker version (if applicable): NA
• Installation source (NGC, Dockerhub, or something else): NuGet & source clone

Regression?

Yes, before the API change.

[mocsharp]

The WithCallbackStream(...) takes Action<Stream> which may not work with async calls which require Func.