Add a "PortableCipher" list that aims to be as compatible as possible to supplement the browser cipher lists #238

mzpqnxow · 2020-12-07T21:41:24Z

See #229 and for more detail, see #236

This is a very small and non-invasive patch that adds an additional static list of ciphers that are already implemented. The objective is to provide an easily referenced collection of cipher-suites for applications using zcrypto that is as compatible as possible given the current set of algorithms implemented in zcrypto.

This should have no impact on current users of zcrypto. If this is merged, it can be used by zgrab2 to more easily access a list of ciphers focused on portability via, e.g. --cipher-suite=portable, which would be added to the other existing options in the zgrab2 http module (firefox-only, chrome-only, safari-only, etc.)

The original "default" is kept the same. Also, in this new PortableCiphers list, for the majority of cases, the same cipher will be negotiated as would be when using the default as I started the list with the same elements in the default list, and then extended it by adding the additional ciphers, to minimize the risk of any unexpected issues for those that choose to use this over the default.

Against a specific real-world dataset that I work with regularly, which consists of ~50k HTTPS endpoints, there's a 2% increase in the amount of successful SSL/TLS negotiations compared with the default cipher list. This is relatively modest, but important depending on the use-case.

I also added a single cipher (TLS_RSA_WITH_AES_256_GCM_SHA384) to the ChromeCiphers list in this commit- this may not be appropriate, if it needs to be removed I can remove it- or feel free to do it in-place in the GH PR interface. The ChromeCiphers list is currently out of date anyway, so maybe there's not much harm leaving it in?

Thanks for your consideration!

Merge master back

…leCiphers list

dadrian

LGTM % notes on a few specific ciphers.

dadrian · 2020-12-08T15:21:58Z

tls/cipher_suites.go

+	TLS_DH_ANON_EXPORT_WITH_DES40_CBC_SHA,
+	TLS_DH_ANON_EXPORT_WITH_RC4_40_MD5,
+	TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA,
+	TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA,
+	TLS_DHE_DSS_WITH_AES_128_CBC_SHA256,
+	TLS_DHE_DSS_WITH_AES_128_GCM_SHA256,
+	TLS_DHE_DSS_WITH_AES_256_CBC_SHA,
+	TLS_DHE_DSS_WITH_AES_256_CBC_SHA256,
+	TLS_DHE_DSS_WITH_AES_256_GCM_SHA384,
+	TLS_DHE_DSS_WITH_DES_CBC_SHA,
+	TLS_DHE_DSS_WITH_RC4_128_SHA,


I don't think the ANON or DSS suites are actually capable of completing a handshake.

Yeah, I've never actually even tried to use an ANON cipher-suite. Same as the comment below, I'll do testing across all of these and see what shakes out before reporting back with some additional commits with VERIFIED completed handshakes for each. Sorry, this was really careless.

Confirmed, can't complete a handshake with DSS certs, upcoming commit will reflect that

dadrian · 2020-12-08T15:22:43Z

tls/cipher_suites.go

+	TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,
+	TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384,
+	TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256,
+	TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,
+	TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,
+	TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,


I don't think we actually support the non-ephemeral DH and ECDH suites.

Hmm good point, I didn't actually test with endpoints preferring each. I just took from the list of implemented ciphers in that file. I'll rig something quick up and confirm that what you're saying is correct (and probably more important, confirm I didn't specify any others that won't actually work if they get chosen)

dadrian · 2020-12-08T15:22:55Z

tls/cipher_suites.go

+	TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384,
+	TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256,
+	TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,
+	TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,


I don't think we actually support the non-ephemeral DH and ECDH suites.

Confirmed, no non-ephemeral DH are supported. Working on the commit with all of the unsupported stuff removed

tls/cipher_suites.go

zakird · 2020-12-08T17:38:16Z

This looks good to me once @dadrian's concerns are addressed. I agree that we shouldn't advertise ciphers we're unable to handshake for.

mzpqnxow · 2020-12-10T23:31:17Z

Yeah, apologies for not actually testing to ensure that the handshakes can complete. I jumped to conclusions based on the implemented ciphers list. I'll confirm all of them before coming back :)

dadrian · 2020-12-11T16:25:03Z

@mzpqnxow It's not unreasonable to expect the implemented ciphers list to contain ciphers that are implemented! 😆

zakird · 2021-01-24T18:31:18Z

Hey @mzpqnxow, I'd absolutely love to get this merged. Any chance you'd be willing to take a look at that list of ciphers?

mzpqnxow · 2021-02-06T23:22:57Z

Hey @zakird yes, I’m sorry this fell off of my map (or maybe I fell off of the map) I’d like to get to this in the upcoming week if you’re still interested

zakird · 2021-02-06T23:52:17Z

Absolutely!

…

On Sat, Feb 6, 2021 at 3:23 PM AG ***@***.***> wrote: Hey @zakird <https://github.com/zakird> yes, I’m sorry this fell off of my map (or maybe I fell off of the map) I’d like to get to this in the upcoming week if you’re still interested — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#238 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABREUDRON5QZYZWYE4RSU3S5XFOBANCNFSM4URBYXXQ> .

…le as well as unsupported suites from existing browser profiles (DSS, Non-anon DH, Non-ephemeral DH) and added annotations only for other unsupported suites. I only actually removed (commented out) suites that were in browser profiles. Where unsupported suites existed elsewhere, I annotated with comments- not removing because they still may be useful, if only as constants/symbols. The annotations can easily be removed if they're an eye-sore, just le met know. I regret now not breaking this up into multiple commits May be worth retrying...

…com/mzpqnxow/zcrypto into feature/portable-cipher-suite-profile

mzpqnxow · 2021-02-18T19:38:56Z

@zakird @dadrian I just pushed a rather large commit onto the existing PR...

Unfortunately, I did this all as one commit, making it pretty hard for you to review as individual sections. If you'd prefer, I can revert to the old version and include only the removal of the unsupported suites from the PortableCiphers list that I added.

What is in this:

Fixup to the PortableCiphers- commenting out the unsupported suites
Fixups for any of the browser-specific suite lists, commenting out unsupported suites- in one or two cases, there were DSS or non-ephemeral DH ciphers that were listed above a few others. I don't know how you feel about that- if the goal is to look like those browsers over the wire and you're OK with the trade-off of potentially losing a few handshakes then you probably won't want this. Well, you get the idea, let me know if you prefer I uncomment them and leave them as is
Annotations (comments) but not removal in a few of the suite lists that include unsupported suites. This is also easy for me to undo if you don't want the clutter all over the place, just let me know

If you get a chance, just let me know at a high level which (if any) of the three categories of changes made should be removed. Sorry I went a bit overboard but I figured since I was putting the effort in I might as well be as verbose and correct as possible. Not everyone agrees on that though :)

BTW- this came out of this effort... maybe useful in the future (but probably not): https://gist.github.com/mzpqnxow/b863ee8ea4b30c96c10f7ee4d101e887

EDIT: Also, forgot to mention- I spent a good deal of time confirming these. Though it's possible I made a mistake or two, I think it should be good, at least in terms of being accurate for what will complete a handshake successfully with zgrab2 http. Only those that won't work under any circumstances (e.g. even when specifying --cipher-suite <int16>) were commented out from the browser profiles

…ded in zmap/zcrypto#238

…ded in zmap/zcrypto#238 (#302) Co-authored-by: Adam Greene <copyright@mzpqnxow.com>

…d in zmap/zcrypto#238 (#302)

mzpqnxow and others added 2 commits December 6, 2020 18:44

Merge pull request #1 from zmap/master

8cac09c

Merge master back

add TLS_RSA_WITH_AES_256_GCM_SHA384 to Chrome profile, add new Portab…

0cb6bb9

…leCiphers list

dadrian approved these changes Dec 8, 2020

View reviewed changes

Merge branch 'master' into feature/portable-cipher-suite-profile

6817ce0

zakird and others added 2 commits January 24, 2021 10:31

Merge branch 'master' into feature/portable-cipher-suite-profile

c4b424a

Merge branch 'master' into feature/portable-cipher-suite-profile

df714cf

zakird and others added 3 commits February 16, 2021 17:12

Merge branch 'master' into feature/portable-cipher-suite-profile

cf6c26a

Merge branch 'feature/portable-cipher-suite-profile' of ssh://github.…

1d503e2

…com/mzpqnxow/zcrypto into feature/portable-cipher-suite-profile

Merge branch 'master' into feature/portable-cipher-suite-profile

3e00b4c

dadrian merged commit 433622c into zmap:master Feb 25, 2021

mzpqnxow mentioned this pull request Mar 4, 2021

Default cipher-suite list for zcrypto and exposing PortableCiphers list #236

Closed

mzpqnxow pushed a commit to mzpqnxow/zgrab2 that referenced this pull request Mar 4, 2021

Add support for specifying the "PortableCiphers" cipher-suite list ad…

7c7c0b9

…ded in zmap/zcrypto#238

mzpqnxow mentioned this pull request Mar 4, 2021

Feature/portable suites zmap/zgrab2#302

Merged

dadrian pushed a commit to zmap/zgrab2 that referenced this pull request Apr 6, 2021

Add support for specifying the "PortableCiphers" cipher-suite list ad…

0eb497e

…ded in zmap/zcrypto#238 (#302) Co-authored-by: Adam Greene <copyright@mzpqnxow.com>

cgreencode pushed a commit to cgreencode/zgrab2 that referenced this pull request Feb 6, 2022

Add support for specifying the PortableCiphers cipher-suite list adde…

7220e87

…d in zmap/zcrypto#238 (#302)

mzpqnxow mentioned this pull request Sep 19, 2022

Consider using zcrypto/ztls tls.PortableCiphers by default or as a flag to enhance success rates on SSL/TLS handshakes projectdiscovery/nuclei#2621

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a "PortableCipher" list that aims to be as compatible as possible to supplement the browser cipher lists #238

Add a "PortableCipher" list that aims to be as compatible as possible to supplement the browser cipher lists #238

mzpqnxow commented Dec 7, 2020

dadrian left a comment

dadrian Dec 8, 2020

mzpqnxow Dec 10, 2020

mzpqnxow Feb 18, 2021

dadrian Dec 8, 2020

mzpqnxow Dec 10, 2020

dadrian Dec 8, 2020

mzpqnxow Feb 18, 2021

zakird commented Dec 8, 2020

mzpqnxow commented Dec 10, 2020

dadrian commented Dec 11, 2020

zakird commented Jan 24, 2021

mzpqnxow commented Feb 6, 2021

zakird commented Feb 6, 2021 via email

mzpqnxow commented Feb 18, 2021 •

edited

Loading

Add a "PortableCipher" list that aims to be as compatible as possible to supplement the browser cipher lists #238

Add a "PortableCipher" list that aims to be as compatible as possible to supplement the browser cipher lists #238

Conversation

mzpqnxow commented Dec 7, 2020

dadrian left a comment

Choose a reason for hiding this comment

dadrian Dec 8, 2020

Choose a reason for hiding this comment

mzpqnxow Dec 10, 2020

Choose a reason for hiding this comment

mzpqnxow Feb 18, 2021

Choose a reason for hiding this comment

dadrian Dec 8, 2020

Choose a reason for hiding this comment

mzpqnxow Dec 10, 2020

Choose a reason for hiding this comment

dadrian Dec 8, 2020

Choose a reason for hiding this comment

mzpqnxow Feb 18, 2021

Choose a reason for hiding this comment

zakird commented Dec 8, 2020

mzpqnxow commented Dec 10, 2020

dadrian commented Dec 11, 2020

zakird commented Jan 24, 2021

mzpqnxow commented Feb 6, 2021

zakird commented Feb 6, 2021 via email

mzpqnxow commented Feb 18, 2021 • edited Loading

mzpqnxow commented Feb 18, 2021 •

edited

Loading