Upload from multiple threads throws 'The session is not open' error

[anandgmenon]

This is very similar to #369

I am using 2024.1.0 version of the library and connecting to an Azure Storage SFTP server. I am trying to upload 50 files in parallel using the same connection and it throws this exception

System.InvalidOperationException: The session is not open.
   at Renci.SshNet.SubsystemSession.EnsureSessionIsOpen()
   at Renci.SshNet.Sftp.SftpSession.SendRequest(SftpRequest request)
   at Renci.SshNet.Sftp.SftpSession.RequestWrite(Byte[] handle, UInt64 serverOffset, Byte[] data, Int32 offset, Int32 length, AutoResetEvent wait, Action`1 writeCompleted)
   at Renci.SshNet.SftpClient.InternalUploadFile(Stream input, String path, Flags flags, SftpUploadAsyncResult asyncResult, Action`1 uploadCallback)
   at Renci.SshNet.SftpClient.UploadFile(Stream input, String path, Boolean canOverride, Action`1 uploadCallback)
   at FunctionApp1.Function1.<>c__DisplayClass2_2.<UploadFiles>b__1() in E:\src\SFtp-Function\FunctionApp1\Function1.cs:line 62
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.<>c.<.cctor>b__272_0(Object obj)
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at FunctionApp1.Function1.<>c__DisplayClass2_1.<<UploadFiles>b__0>d.MoveNext() in E:\src\SFtp-Function\FunctionApp1\Function1.cs:line 62
--- End of stack trace from previous location ---
   at FunctionApp1.Function1.UploadFiles() in E:\src\SFtp-Function\FunctionApp1\Function1.cs:line 68

My code looks like this:

static async Task UploadFiles()
{
    var bytes = File.ReadAllBytes("E:\\src\\ConsoleApp\\ConsoleApp1\\6 MB");
    int degreeOfParallelism = 50;

    var options = new ParallelOptions();
    options.MaxDegreeOfParallelism = degreeOfParallelism;

    try
    {
        using (var client = new SftpClient(_connectionInfo))
        {
            client.KeepAliveInterval = TimeSpan.FromMinutes(30);
            client.OperationTimeout = TimeSpan.FromMinutes(2);
            client.ConnectionInfo.MaxSessions = 100;

            client.Connect();

            var tasks = Enumerable.Range(1, 50).AsParallel().WithDegreeOfParallelism(degreeOfParallelism).Select(async i =>

            {
                Console.WriteLine($"Uploading file {i}");

                await Task.Run(() => client.UploadFile(new MemoryStream(bytes), $"upload/6MB_{i}", true));

                Console.WriteLine($"Upload file {i} completed");
                return;
            });

            await Task.WhenAll(tasks);
           
        }
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex.Message);
    }
}

I tried the smilar scenario with the WINSCP library as well to rule out any issues with server and seems to be working fine there

[mus65]

You may want to subscribe to the client.ErrorOccurred event to log any exceptions happening. This might help narrowing it down.

Also doing the upload actually async instead of using Task.Run would make sense (will probably not fix the problem though). There is no UploadFileAsync (which I also found confusing) but there is an example here: #819 (comment)

If the server actually closes the connection (for whatever reason), you have to re-connect and try again. But note that the current release has a bug with re-connecting. see #1474 . You have to explicitly disconnect before re-connecting to work around it.

[Rob-Hague]

It could also be #1253 which occasionally fails in CI (and for me, quite often locally). I did not find the root cause when I looked at it last

[anandgmenon]

@Rob-Hague for me locally it starts giving issues only when using 50 threads, but when running inside azure function, we see this issue for 2-3 parallel uploads too. Could you please maybe look into the root cause and maybe suggest a work around if possible?

[Rob-Hague]

What SSH/SFTP software is the server running? Do you have access to server logs?

[anandgmenon]

This is SFTP server on Azure Storage. Unfortunately they do not have much connection server logs

[anandgmenon]

@Rob-Hague any updates on this?

[Rob-Hague]

I left some notes in #1253 I did not get any further