Improvements on client simulations

[MrSeccubus]

@dcooper16 this is my attempt to make stuff a little bit more maintainable

The client signatures are now in a separate file that can be updated from the data at SSL labs by running a single program.

I send this pull request to you because I based my work on you branch that hasn't been merged yet.

parse_client_ciphers now gets the results from the SSL labs API
Wrote a parse to conver wireshark SSL Handshake ciphers lists to openssl cipher lists

[dcooper16]

Hi @seccubus,

Thanks for this PR. I've been testing out your version of testssl.sh with client-simulation-data.sh and for the most part am getting the same results with either OpenSSL or sockets (except in the expected cases in which the OpenSSL version is connecting using an ephemeral ECDH key with a curve not supported by the client). However, the results differ in a few cases. One example is www.openssl.org.

I believe the problem is with the following line:

$OPENSSL s_client -cipher ${ciphers[i]} ${protos[i]} ${tlsvers[i]} $STARTTLS $BUGS $PROXY -connect $NODEIP:$PORT ${sni[i]}  </dev/null >$TMPFILE 2>$ERRFILE

For www.openssl.org and Android 5.0.0 this results in:

$OPENSSL s_client -cipher ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:DES-CBC3-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:DHE-DSS-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:AES128-GCM-SHA256:AES128-SHA:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:RC4-SHA:RC4-MD5 -no_ssl2 -ssl3 -tls1_2 -tls1_1 -tls1 -connect 194.97.150.234:443 -servername www.openssl.org

The problem is the protocol information:

-no_ssl2 -ssl3 -tls1_2 -tls1_1 -tls1

According OpenSSL describes these options as follows:

 -ssl2         - just use SSLv2
 -ssl3         - just use SSLv3
 -tls1_2       - just use TLSv1.2
 -tls1_1       - just use TLSv1.1
 -tls1         - just use TLSv1
 -no_tls1_2/-no_tls1_1/-no_tls1/-no_ssl3/-no_ssl2 - turn off that protocol

I looked at the OpenSSL source code, and while I couldn't figure out what it does in response to the "no_*" options, I could see that if the command line has something like -tls1_2 -tls1_1 -tls1 in it, it just uses the last one. So, in the example above, OpenSSL tries to connect using just TLSv1.0, and since www.openssl.org only supports TLSv1.1 and TLSv1.2, the client simulation reports "No connection" whereas the sockets version says that it connects with "TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256" (using a 256-bit ephemeral ECDH key).

Similarly, in the for tls in ${tlsvers[i]}; do loop, it seems unnecessary to include -no_ssl2 -no_ssl3 in the OpenSSL command line since the $tls should ensure that OpenSSL will only connect with the specified protocol (-tls1_2, -tls1_1, or -tls1).

[dcooper16]

Hi @seccubus,

Would it be possible for parse_client_ciphers.pl to also extract minDhBits, maxDhBits, minRsaBits, and maxRsaBits from https://api.dev.ssllabs.com/api/v3/getClients. I don't think the information would affect the results very much, but I noticed that for Java 6u45 maxDhBits is 1024. So, in my tests I got some incorrect results. For example, connecting to en.wikipedia.org the client simulation indicates that Java 6u45 would connect with "TLSv1.0 DHE-RSA-AES128-SHA" with a 2048-bit ephemeral DH key, but that is contrary to the maxDhBits value in https://api.dev.ssllabs.com/api/v3/getClients.

At the moment, however, I don't know what to do with the data. In the case of ECDHE cipher suites it is easy - the client indicates which curves it supports and the server limits itself to those curves. In the case of DHE cipher suites, the client can't say what DH key sizes it will accept. So, if a real Java 6u45 client tried to connect to en.wikipedia.org and the server responded with "TLSv1.0 DHE-RSA-AES128-SHA" with a 2048-bit ephemeral DH key would the connection just fail or would the client try a different ClientHello? I don't know, so I don't know how to accurately simulate the client in this case.

Thanks,

Dave

[MrSeccubus]

Dave,

If we got sockets working with startssl (wich it should if I'm not mistaken), I think we can abandon using openssl altogether.

It would be relatively easy for me to extract more info from the SSLlabs API.

I'm looking into maintaining a separate set of clients independantly from SSLlabs (mainly mail related). Do you know if there is a way to extract the inProto and maxProto from the handshake?

I want to test certain clients if certain services have been detected.

[MrSeccubus]

Dave,

The information you requested is now in the configuration file.

[MrSeccubus]

BTW, is it possible from the handshake to see what TLS/SSL protocol is min/maxProto? I want to use this for the clients we maintain ourselves.

[dcooper16]

Thanks for adding the extra information. I realized that I can check whether DH bits is between minDhBits and maxDhBits in the current version, but I was able to add that check to https://github.com/dcooper16/testssl.sh/tree/extended_tls_sockets, where parse_tls_serverhello() has been extended to extract information about ephemeral keys.

I won't be able to do any more work on this until later next week, at the earliest.

I would think that maxProto would be the value of the client_version field from the ClientHello (but not the version field from the record layer). Unfortunately, though, the client only sends the maximum version that it supports, so there is nothing in the handshake to indicate minProto. I think the client just indicates the highest version it supports in its ClientHello, the server specifies the highest version it supports (that is no higher than the one specified in the ClientHello) in the server_version field of the ServerHello, and the client simply reject the connection if server_hello is less than minProto.

[MrSeccubus]

Apparently we need the -ssl2 -ssl3 together with the -no_tls1 switches:

$ bin/openssl.Darwin.x86_64 s_client -connect mozilla-modern.badssl.com:443 -no_tls1 -no_tls1_1 -no_tls1_2
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-SHA

$ bin/openssl.Darwin.x86_64 s_client -connect mozilla-modern.badssl.com:443 -ssl2 -ssl3 -no_tls1 -no_tls1_1 -no_tls1_2
SSL-Session:
    Protocol  : SSLv3
    Cipher    : ECDHE-RSA-AES128-SHA

This is also incorrectly printed when using sockets.

OpenSSL handles its options in a funning way. And it is not even consistently wrong...

I now resorted to trying each protocol individually from high to low and stop as soon as one works.