TLS/SSL test type(s) #3290
Comments
|
I think the openssl client is probably a good start for testing for the configuration of a server. I agree that it is pointless replicating all tests for TLS as one or two like you have proposed will give enough data to provide a Delta between TLS and non TLS tests to allow for an overhead calculation. I would say that AES is the most widely supported algo for the bulk ciper. Sha256 or higher for the hash shouldn't be controversial either. The main issue will be deciding on the certificate and initial key exchange. |
|
The two tests should be enough to extrapolate from when comparing to non-TLS. TLS should be 1.2 (1.3 isn't widely supported yet, < 1.1 is bad) Cipher Suite should be encouraged to be a common one between frameworks; as there are dramatic differences in performances and varying them would be comparing the algorithm rather than the differing effects of using TLS. Not sure what the cipher suite should be. SSL Labs Cipher Suite (in order of preference) reports for various browsers Chrome Edge Firefox Top common is |
Would an "Is edge configuration" flag be sensible? TLS is valid in non-internet facing stacks; and there are already stacks in the tests that use for example ngnix as their front-end server; so they would presumably be edge/internet facing ready. However, that can come with a performance disadvantage, so comparing non-internet facing stacks against internet facing stacks may be a little apples and oranges. While both fruit, one is more resistant to bruising? The disadvantage I can see to that is it make be taken as approval that a particular stack is internet facing ready; while it may not be. So would at least need caveats; and may not be good for that reason? |
I'm using this one: https://testssl.sh/, https://github.com/drwetter/testssl.sh I think it will tick every box: mainly shell script, non-exotic external dependencies, similar functionality to SSL Labs tests. Off-topic: there is a similar command-line tool for SSH auditing https://github.com/arthepsy/ssh-audit It is not related to the Update: |
The TLS-Plaintext will hardly test the handshake and mostly test the encoding costs. However, the encoding costs might depend more on the SSL/TLS library (OpenSSL / BearSSL etc') rather than the framework (I'm ignoring buffer consolidation at the moment, although this is definitely a consideration where pipelining is concerned). I'm wondering if the performance testing costs are worth the possible data mining that would be enabled by the TLS-Plaintext test...?
Sidenote: Technically speaking, nginx (and friends) has a known issue with forward security and session cache invalidation. I'm not sure if resolving these issues would effect performance, but I'm quite certain that security is more important than any performance performance penalty that patching these issues might incur. I'm pointing this out because this might be something we want to test (I have no idea how to do that, though). |
|
Keep in mind that this is a framework benchmark. Choosing to use openssl/bearssl is a choice of the framework of language authors. I don't see why this is any less of a framework test because of that choice. |
|
Absolutely, and from my digging in frameworks the way they use openssl etc can make a massive difference, including unneeded memory copies and locks. What I hope is that this benchmark will highlight and put pressure on these things to be changed. If every framework ends up with the same Delta cost for https overtime then this test would have achieved it's goal in my mind. |
|
Another 2 minor points:
|
@zloster That looks fantastic. We'll probably also want to ensure that the load generator uses the agreed-to cipher suite (whatever that ends up being) and only that cipher suite.
@boazsegev I agree that to whatever degree it is possible, I would like to test configurations that are as secure as current best-practices suggest. If/when there is a resolution to the issue you cite, it would be good to apply that to the test implementations.
@RX14 is correct on this point. If the framework has specified a preferred (canonical) TLS provider, we'd want to use that in these test implementations. As we have done elsewhere, in cases where a preferred provider is not specified, we'd like to use a mainstream option such as OpenSSL.
@Drawaes Agreed! This is my sentiment generally across everything we do in this project. :)
@zloster We are bringing a new environment (named "Citrine" in the results dashboard for the time being), and these servers include AES-NI. |
|
The point raised regarding testing TLS handshake performance (aka for new connections) vs. encoding/encryption costs is an interesting one, and makes me wonder whether we should configure the Fortunes variant to not allow connection re-use. Or perhaps we need a separate test altogether, e.g. TLS-Handshake, that is effectively plaintext without connection re-use (which of course implies no pipelining too). Having a different test might help avoid any conflation of traditional Fortunes results with the TLS variant. I, for one, certainly think being able to see a framework's performance characteristics for TLS handshakes vs. encryption is valuable. |
|
One point you will potentially be measuring a whole stack of areas that just aren't measured today and might tell you very little about TLS performance per se. If this is likely to be a test that is looked at first I would recommend there is a test with zero connection reuse without TLS ( unless I am mistaken and this is what Fortunes is today?) As you would want a baseline for establishing the socket etc before moving onto the handshake itself. Reason being is that if you have an equal field configuration without session resumption then the only difference in reality is the extra round trips of tiny messages. From my measurements the actual cryptographic calculations are a minimal part of the handshake. Then of course you have to decide do you allow session resumption/ticketing and which keyshares/curves etc which massively expands the problem. |
https://en.wikipedia.org/wiki/HTTP_persistent_connection#HTTP_1.1 Connection reuse AKA "keep-alive" should be the default behaviour for all tests. The |
|
Which is what I thought ;) I guess my concern people will look at this test and think... Wow TLS is expensive when in the real world keep alive is the norm and there is no unencrypted baseline to compare to. |
|
I believe the JSON test would be a better choice than the plaintext test when testing TLS. As I mentioned here, the plaintext pipelining approach exposes servers to DoS attacks through clients that pipeline large quantities of short requests (preferably requesting a large payload). A 48 byte request can be repeated ~11 times in a single TCP/IP packet when MTU=576 (which is the common limitation on the web). An attacker could make hundreds of requests, combined with a slow client approach, to starve a server of resources (for example, consider memory consumption for the buffered responses). There are two defense measures a security aware framework might implement: 1. throttling (slowing down pipelined requests); and 2. limiting (limiting the number of incomplete downloads for slow clients). This means that security aware frameworks are likely to perform slower on plaintext tests due to pipelining. This would result in an unfair advantage for frameworks that ignore this possible attack vector. Hence, I believe that the JSON test would provide a better representation where security is concerned, since it avoids this question altogether. Just my 2¢... also, I'm somewhat biased (facil.io throttles pipelined requests). |
|
Hi folks, I'm a bit of a noob regarding all this, but I have some feedback. I just ran a crude same-machine test for aspcore/json manually and put a https endpoint in. I got roughly a third rps less with ssl (47K vs 70K), which is substantial and was not on my radar when I saw the impressive rps values of the benchmarks (or the concerns raised by @boazsegev). I don't know if connections were kept alive during my test, the keep-alive header had no effect on the result. I assume the expensive part of ssl is the handshake and done only once for each connection. Is it possible to tell wrk to re-open each connection? There's no obvious option. |
This issue is for planning a new test type that exercises TLS (SSL), which was suggested as test type 10 on issue #133. I recommend reviewing the comment thread for #133 since it contains several prior thoughts concerning a TLS/SSL test type that remain applicable. Though I am not manually copying those comments over here, please feel free to reiterate points raised there.
Add any comments and questions that you believe can help us reach a consensus on reasonable requirements for this new test type.
Not all participants will agree that a TLS test is warranted in this project. Historically, we have not focused on the performance of web servers, intending our project to be about web application frameworks. However, our project is about measuring the high-water mark of performance using real-world web application technology stacks. And from that perspective, including a TLS test type is reasonable as long as there are some stacks for which TLS would be included "in-stack." And I know that is satisfied in the real-world because many stacks include nginx or another mainstream web server that has robust TLS capability.
Perhaps it is important to remember that, to whatever degree it is possible, we want the test implementations in this project to represent canonical usage of the platforms and frameworks we exercise. Many platforms and frameworks are of the opinion that TLS is an external concern to be handled by a front-end web server, load balancer, or other external system or service. That is perfectly fine and acceptable. Such platforms and frameworks may elect to bypass implementing this TLS test type. All test types in this project are optional, and omission should not be considered a failing or even evidence of lack of capability for a framework or platform. (Aside: We intend to eventually provide contributors a way to document the architectural approach that they have taken, explain the details of each test implementation they have provided, and in the case of omitting a test, explain why that test type is not applicable.)
I propose the following:
Thoughts, questions, corrections, comments appreciated!
The text was updated successfully, but these errors were encountered: