FtpWebRequest not reusing ssl session on linux (FTPS)

[cluetjen]

It seems that FtpWebRequest on Linux does not reuse ssl sessions for FTPS passive data connections and for this reason does not work with the currently recommended ftp server setups.

Expected
An ftp client must reuse the same ssl session that he used for the control connection for the data connection too. That's the current default configuration for most ftps servers.

Problem
On Linux FtpWebClient creates a new ssl session for the data connection and the connection fails.

Test code

        [Fact]
        public async Task We_should_be_able_to_list_files_with_ftpwebrequest()
        {
            ServicePointManager
                    .ServerCertificateValidationCallback +=
                (sender, cert, chain, sslPolicyErrors) => true;

            // Get the object used to communicate with the server.
            var request = (FtpWebRequest)WebRequest.Create("ftp://localhost/");
            request.Method = WebRequestMethods.Ftp.ListDirectoryDetails;
            request.EnableSsl = true;
            request.UsePassive = true;

            // This example assumes the FTP site uses anonymous logon.
            request.Credentials = new NetworkCredential("test", "test");

            FtpWebResponse response = (FtpWebResponse)request.GetResponse();

            var responseStream = response.GetResponseStream();
            StreamReader reader = new StreamReader(responseStream);

            var text = reader.ReadToEnd();

            Console.WriteLine($"Directory List Complete, status {response.StatusDescription}");

            reader.Close();
            response.Close();

            Assert.NotEmpty(text);
        }

Result:

[xUnit.net 00:00:03.4876467]     FileTransfer.Tests.FileZillaTests.We_should_be_able_to_list_files_with_ftpwebrequest [FAIL]
Fehler FileTransfer.Tests.FileZillaTests.We_should_be_able_to_list_files_with_ftpwebrequest
Fehlermeldung:
 System.Net.WebException : The remote server returned an error: 150 Opening data channel for directory listing of "/"
.
---- System.IO.IOException : Authentication failed because the remote party has closed the transport stream.
Stapelverfolgung:
   at System.Net.FtpWebRequest.SyncRequestCallback(Object obj)
   at System.Net.CommandStream.InvokeRequestCallback(Object obj)
   at System.Net.CommandStream.Abort(Exception e)
   at System.Net.FtpWebRequest.FinishRequestStage(RequestStage stage)
   at System.Net.FtpWebRequest.GetResponse()
   at FileTransfer.Tests.FileZillaTests.We_should_be_able_to_list_files_with_ftpwebrequest() in /mnt/c/workspace/lib-cs-filetransfer/FileTransfer.Tests/FileZillaTests.cs:line 45
--- End of stack trace from previous location where exception was thrown ---
----- Inner Stack Trace -----
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions)
   at System.Net.Security.SslStream.AuthenticateAsClient(String targetHost, X509CertificateCollection clientCertificates, SslProtocols enabledSslProtocols, Boolean checkCertificateRevocation)
   at System.Net.TlsStream.AuthenticateAsClient()
   at System.Net.FtpControlStream.QueueOrCreateFtpDataStream(Stream& stream)
   at System.Net.FtpControlStream.PipelineCallback(PipelineEntry entry, ResponseDescription response, Boolean timeout, Stream& stream)
   at System.Net.CommandStream.PostReadCommandProcessing(Stream& stream)
   at System.Net.CommandStream.PostSendCommandProcessing(Stream& stream)
   at System.Net.CommandStream.ContinueCommandPipeline()
   at System.Net.FtpWebRequest.TimedSubmitRequestHelper(Boolean isAsync)
   at System.Net.FtpWebRequest.SubmitRequest(Boolean isAsync)

FileZilla FTP server logs:

(000015)14.11.2018 23:26:12 - (not logged in) (127.0.0.1)> Connected on port 21, sending welcome message...
(000015)14.11.2018 23:26:12 - (not logged in) (127.0.0.1)> 220-FileZilla Server 0.9.60 beta
(000015)14.11.2018 23:26:12 - (not logged in) (127.0.0.1)> 220-written by Tim Kosse (tim.kosse@filezilla-project.org)
(000015)14.11.2018 23:26:12 - (not logged in) (127.0.0.1)> 220 Please visit https://filezilla-project.org/
(000015)14.11.2018 23:26:12 - (not logged in) (127.0.0.1)> AUTH TLS
(000015)14.11.2018 23:26:12 - (not logged in) (127.0.0.1)> 234 Using authentication type TLS
(000015)14.11.2018 23:26:12 - (not logged in) (127.0.0.1)> TLS connection established
(000015)14.11.2018 23:26:13 - (not logged in) (127.0.0.1)> USER test
(000015)14.11.2018 23:26:13 - (not logged in) (127.0.0.1)> 331 Password required for test
(000015)14.11.2018 23:26:13 - (not logged in) (127.0.0.1)> PASS ****
(000015)14.11.2018 23:26:13 - test (127.0.0.1)> 230 Logged on
(000015)14.11.2018 23:26:13 - test (127.0.0.1)> PBSZ 0
(000015)14.11.2018 23:26:13 - test (127.0.0.1)> 200 PBSZ=0
(000015)14.11.2018 23:26:13 - test (127.0.0.1)> PROT P
(000015)14.11.2018 23:26:13 - test (127.0.0.1)> 200 Protection level set to P
(000015)14.11.2018 23:26:13 - test (127.0.0.1)> OPTS utf8 on
(000015)14.11.2018 23:26:13 - test (127.0.0.1)> 202 UTF8 mode is always enabled. No need to send this command.
(000015)14.11.2018 23:26:13 - test (127.0.0.1)> PWD
(000015)14.11.2018 23:26:13 - test (127.0.0.1)> 257 "/" is current directory.
(000015)14.11.2018 23:26:13 - test (127.0.0.1)> TYPE I
(000015)14.11.2018 23:26:13 - test (127.0.0.1)> 200 Type set to I
(000015)14.11.2018 23:26:13 - test (127.0.0.1)> PASV
(000015)14.11.2018 23:26:13 - test (127.0.0.1)> 227 Entering Passive Mode (127,0,0,1,212,245)
(000015)14.11.2018 23:26:13 - test (127.0.0.1)> LIST
(000015)14.11.2018 23:26:13 - test (127.0.0.1)> 150 Opening data channel for directory listing of "/"
(000015)14.11.2018 23:26:13 - test (127.0.0.1)> 450 TLS session of data connection has not resumed or the session does not match the control connection
(000015)14.11.2018 23:26:13 - test (127.0.0.1)> disconnected.

• Only Linux (tested with Ubuntu 16.04, 18.04 on WSL and native)
• Tested with FileZilla FTP server current version
• Tested with dotnet core 2.1 and 2.2 preview 3

Steps to reproduce

• Install FileZilla server (https://filezilla-project.org/)
• Add user "test", PW "test" and any shared folder in FileZilla
• Create xunit project in VS mit FluentFTP nuget package
• Add test from this issue
• open bash (or install ubuntu from windows store)
• install dotnet sdk in bash (https://www.microsoft.com/net/download)
• cd /mnt/c/... wherevery your vs project is
• dotnet test

[karelz]

Note that we consider FtpWebRequest legacy API. Here's recommended list of replacements: https://github.com/dotnet/platform-compat/blob/master/docs/DE0003.md

That said, if it makes the class entirely unusable, we would be open to taking a contribution. Or if it impacts lots of users, we could eventually investigate too.

[cluetjen]

Have to say, that I'm not a network specialist, but after reading and testing a day, it seems that the real problem is the SslStream class that seems to be used by most clients. There seems to be no way to control this reuse behavior but it's using it's own cache that will reuse sessions with some magic. For me, it feels like this is the real problem with the result, that most clients do not work on Linux.

FluentFtp
That's what we normally use but it has the same Problem. I reported it as a bug:

• My issue report: FTPS not working on Linux since SSL session resumption is not supported robinrodricks/FluentFTP#347
• Same issue (as feature request, other user, 2017): Support for SSL Session Reuse robinrodricks/FluentFTP#236

CoreFTP
Tested, same problem.

Is it a real Problem?

Most servers have this feature enabled as recommended default:

ProFTPD

As of ProFTPD 1.3.3rc1, mod_tls only accepts SSL/TLS data connections that reuse the SSL session of the control connection, as a security measure.

Can be disabled in config but not recommended: http://www.proftpd.org/docs/contrib/mod_tls.html#TLSOptions

FileZilla Server

Another option you should enable is "Require TLS session resumption on data connection when using PROTP P" as it protects against data connection theft.

https://wiki.filezilla-project.org/FTPS_using_Explicit_TLS_howto_(Server)

vsftpd

require_ssl_reuse (Default: YES)
If set to yes, all SSL data connections are required to exhibit SSL session reuse (which proves that they know the same master secret as the control channel). Although this is a secure default, it may break many FTP clients, so you may want to disable it. For a discussion of the consequences, see http://scarybeastsecurity.blogspot.com/2009/02/vsftpd-210-released.html (Added in v2.1.0).

http://vsftpd.beasts.org/vsftpd_conf.html

Notes

• Please hold in mind, that (in the hope, that I'm not totally stupid) there's currently no way to connect to those ftp servers from .net core on linux. (If the servers run with default config.)

• I tested with multiple libs but only with two different FileZilla servers (customer + my test server) - not with the other mentioned ftp servers.

[karelz]

Do you know if it is Linux specific, or are there similar problems on Windows?
IMO the primary goal should be to fix better FTP stacks (like the ones you tried and failed too). If there is something we can/should do on our layers (below FTP), we'd love to know.
If it is so impactful, I wonder why we didn't hear about it for last 3 years. Maybe most people don't use FTP protocols anymore ...

[cluetjen]

Do you know if it is Linux specific, or are there similar problems on Windows?

Only Linux. (Windows tested, works perfectly)

If it is so impactful, I wonder why we didn't hear about it for last 3 years.

Same here, but it seems, that for some time it was simply normal, that most clients can't handle this feature. But today afaik most of them can (e.g. I think it was version 8 in Java world).

Maybe most people don't use FTP protocols anymore ...

Wish I could say the same. At least in the German ecommerce world people love CSV(!) and FTP - and I'm not talking about small companies...

[davidsh]

@bartonjs - Is there some OpenSsl magic that would make this scenario work better?

[avonheimburg]

As cluetjen said, the common factor here seems to be SSLStream. According to the SSLStream docs, TLS session resumption is done automatically be the framwork.

The special thing about FTPS is that for securing the entire interaction, it requires* resuming the control connection's TLS session for the data connection while keeping the control connection open. This is different from, say, HTTPS, where a failure to resue a TLS session for a parallel connection to the same server is just a performance hit.

*) Because there is no auth mechanism on FTP data connections, the only way to be sure that the client sending the data is the same client that requested a file upload in the first place is by reusing the same TLS session. Otherwise, the first one to connect to the data port can hijack it and, if it is a connection for a file upload, upload what file they want. FTP is a really weird protocol from today's perspective.

[bartonjs]

Is there some OpenSsl magic that would make this scenario work better?

If you mean some environment variables and the like, I don't think so. OpenSSL does support session resumption, but AFAIK SslStream never makes use of it. I don't have any particular insight as to how it works, or how much work it would take to do.

https://www.openssl.org/docs/man1.1.0/ssl/SSL_CTX_set_session_cache_mode.html and https://www.openssl.org/docs/man1.1.0/ssl/SSL_set_session.html make it seem like it's a bit non-trivial (I think currently every SslStream connection gets a distinct SSL_CTX; and I don't know how FTPS data connections are supposed to indicate that they're wanting to reuse from the session pool that the control connection was in).

[jmezach]

Are there any plans to make this work? I understand that FtpWebRequest is deprecated, but as mentioned before this is also an issue for third party client libraries, unless they plan on implementing SSL themselves. I'm currently working on a project that has very high security requirements that we currently cannot meet due to this issue.

[davidsh]

Are there any plans to make this work?

This is not planned for .NET 3.0. It is a difficult thing to implement completely since it relies on the behaviors of the OS TLS/SSL stack. It isn't something solvable at the .NET layer. It's possible to make it work perhaps just on Linux using the OpenSSL support that @bartonjs mentions. But then there is the question on how to enable/disable that behavior which might involve introducing new APIs.

[avonheimburg]

But then there is the question on how to enable/disable that behavior which might involve introducing new APIs.

I don't quite follow that point. All we are asking for is that SSLStream on Linux be brought to feature parity with SSLStream on Windows, which automatically does session resumption. There is no way to disable that behavior on Windows, there would need to be no way to do so under Linux.

[karelz]

To clarify priority: This is not high on our priority list and will most certainly not happen in 3.0.
If anyone has idea how to fix it, we would welcome a contribution (I'd recommend to start with discussion how to fix it first).

[karelz]

Here are technical notes on feasibility from @bartonjs about SSL session resuing:

Windows seemingly maintains the sessions per-process. OpenSSL has support for it on SSL_CTX*, but we make a new one of those for each connection. So we’d need to move to something like a static singleton SSL_CTX* and then everywhere we touch it would need to get reassessed.

A quick glance at how we interop with SSL_CTX says that we’d need to look them up by the tuple (SslProtocols, EncryptionPolicy, what cert-and-private-key are being used, ALPN protocols) -- all of the things that get set on an SSL_CTX*. All of the ones that are cert-less could probably live forever; the cert-related ones probably need to track when they’ve run out of SSL* instances that use the SSL_CTX* and evict it from the cache and clean it up.

If some of the things are also settable on SSL* then moving them to per-connection would reduce the number and complexity of cached SSL_CTX* values. And, of course, then someone would have to actually figure out how to use the SSL resumption features… and then they’d have to figure out how to test it.

Complexity:
My expectation is “major restructuring of the Linux SslStream PAL”. I’d guess somewhere in the 3 to 6 week range for either doing it or determining it can’t be done.
It may turn out to be nearly “impossible” to implement on Linux though.

We should NOT write session ressumption on top of OpenSSL ourselves, nor write our own TLS in managed code -- such approaches are not acceptable and not conformant to our security guidelines.

cc @tmds @Drawaes if they are interested ...

[karelz]

Triage: We don't believe it is feasible to make it happen in .NET Core.
Given that it is "just" about performance on Linux of legacy protocol (FTP), we do not think it is that painful to not have it.

[cluetjen]

Given that it is "just" about performance on Linux

It's absolutely NOT about performance. It's about the fact, that you cannot establish data connections to any FTP server using any .net core library if the FTP server is configured to require session reuse for security reasons. This is the recommended setting for most FTP servers.