-
Notifications
You must be signed in to change notification settings - Fork 2.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Backport 3.6: test sample programs in ssl-opt.sh #9541
base: mbedtls-3.6
Are you sure you want to change the base?
Backport 3.6: test sample programs in ssl-opt.sh #9541
Conversation
11d18d0
to
2b21033
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a first pass, will make another when the pre-requisite is in, but looking pretty good to me so far!
!defined(MBEDTLS_TIMING_C) || !defined(MBEDTLS_FS_IO) || \ | ||
!defined(MBEDTLS_PEM_PARSE_C) | ||
int main(int argc, char *argv[]) | ||
#if !defined(MBEDTLS_ENTROPY_C) || !defined(MBEDTLS_CTR_DRBG_C) || \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not on this line: copy-pasta in the commit message, says pthread when it's about fork.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Changed in my latest rebase.
Note that I intend more rebases. Both after #9546 is merged, and also because my latest batch of commits has broken something (still puzzling out what).
2b21033
to
32ecece
Compare
requires_protocol_version tls13 | ||
requires_openssl_tls1_3 | ||
requires_config_enabled MBEDTLS_SSL_TLS1_3_COMPATIBILITY_MODE | ||
run_test "Sample: ssl_client1, openssl server, TLS 1.3" \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This and some other test cases are skipped in builds with only TLS 1.3. This is due to #9560, which is a preexisting bug affecting many preexisting test cases, and I will not fix it here.
32ecece
to
6a0738f
Compare
This is getting too large: 29 commits (not counting the ones from #9556). So I intend to split this in two pieces. Probably all except Last version with the |
6a0738f
to
a9b319f
Compare
tests/ssl-opt.sh
Outdated
# Check if our test programs use files. | ||
case "$CLI_CMD" in | ||
*programs/ssl/*"$DATA_FILES_PATH/"*) | ||
requires_config_enabled MBEDTLS_FS_IO;; | ||
esac | ||
case "$SRV_CMD" in | ||
*programs/ssl/*"$DATA_FILES_PATH/"*) | ||
requires_config_enabled MBEDTLS_FS_IO;; | ||
esac |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So this innocent-looking change (at least it looked innocent to me) is causing a bunch more test cases to be executed in config-suite-b
, config-ccm-psk-tls1_2.h
and config-ccm-psk-dtls1_2.h
. And some are failing because a genuine requirement isn't declared. The joys of ssl-opt.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And by the way, maybe we should enable MBEDTLS_FS_IO
in test-ref-configs
when we run ssl-opt
(we already add MBEDTLS_DEBUG_C
and MBEDTLS_ERROR_C
)?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Aw, let's not do that now, then.
The joys of ssl-opt.
Yeah, it was designed to only work in the default config, and dependency declarations were added later as an after-thought, and for some time the target was it should run in the default and full configs, then perhaps a few others that are close enough... but we never took the time to think it through, and here we are now.
And by the way, maybe we should enable MBEDTLS_FS_IO in test-ref-configs when we run ssl-opt (we already add MBEDTLS_DEBUG_C and MBEDTLS_ERROR_C)?
Yeah, I think that would make sense. Perhaps for your other PR that splits test-ref-configs? (Unless it turns out to be another can of worms.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps for your other PR that splits test-ref-configs? (Unless it turns out to be another can of worms.)
No, because a lot of the worms are test cases that currently don't run in the reference configs because they lack MBEDTLS_FS_IO
. When we turn this on, they get executed, which fails for a variety of reasons: some other obvious-in-hindsight dependency is missing, or we have unfragmentable handshake packets that go over the message limit, or the DTLS adaptation code does something wrong, or …
I've filed #9595 and added it to our tech debt pile.
I've removed all the |
I'm happy to review this PR as it is (plus the few commits needed for it to pass CI). I don't feel splitting it just for the sake of it would significantly help, unless there's a breaking point that really makes sense. However, I'm wondering about PSK-only mode, which seems to be bringing a lot of issues. Would it make sense to have a first step where we only test the programs with cert-based (or if it helps, even signature-based, that is cert-based + ephemeral as opposed to static ECDH or RSA encryption) key exchanges, and leave PSK-only for step 2? (That's a real question, the answer might be no, you know better than me at this point.) |
|
Come to think of it, the only purpose of improving MBEDTLS_FS_IO detection was to run more tests in config-ccm-psk-dtls1_2. And that was only useful for the mini_client PSK tests. While these changes are desirable, they're technical debt, not on the critical path for the deliverable. So I'm going to remove that and its consequences, file a tech debt issue with pointers, and hopefully the delivery can converge quickly. It was already passing everything except |
The *.sh files in opt-testcases cannot be executed directly: they can only be sourced by ssl-opt.sh. So don't make them executable and don't give them a shebang line. Also make sure that the first paragraph of each file is a short description. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
For better searchability and readability, call requires_config_enabled or requires_config_disabled for each option, instead of calling requires_all_configs_enabled or requires_all_configs_disabled with a long list of options. ``` perl -0777 -i -pe ' s!^requires_all_configs_(enabled|disabled) *((?:(?: \w+) *(?:\\\n)? *)+)\n! $state = $1; join("", map {"requires_config_$state $_\n"} $2 =~ /(\w+)/g) !egm' tests/ssl-opt.sh tests/opt-testcases/*.sh ``` Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The compile-time option MBEDTLS_SSL_TLS1_3_COMPATIBILITY_MODE gates both support for interoperability with a peer that uses middlebox compatibility mode, and support for activating that mode ourselves. Change code that is only needed for interoperability to be guarded by MBEDTLS_SSL_TLS1_3_ACCEPT_COMPATIBILITY_MODE. As of this commit, MBEDTLS_SSL_TLS1_3_ACCEPT_COMPATIBILITY_MODE is always enabled: there is no way to disable it, and there are no tests with it disabled. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Adapt the test cases for TLS 1.3 middlebox compatibility mode, now that we always interoperate with peers that support it, regardless of whether MBEDTLS_SSL_TLS1_3_COMPATIBILITY_MODE is enabled. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
This is necessary for the SSL sample programs: they hard-code port 4433. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Test ssl_client1 with both TLS 1.2 and TLS 1.3. Test against both OpenSSL and GnuTLS. Clean up compile-time requirements in ssl_client1.c: any certificate-based key exchange is ok, so don't insist on built-in RSA. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Test against both OpenSSL and GnuTLS. Don't use a proxy. It's not particularly useful here, and would complicate figuring out port numbers. Clean up compile-time requirements in dtls_client.c: any certificate-based key exchange is ok, so don't insist on built-in RSA. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Test ssl_server with both TLS 1.2 and TLS 1.3. Test against both OpenSSL and GnuTLS. Clean up compile-time requirements in ssl_server.c: any certificate-based key exchange is ok, so don't insist on built-in RSA. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Test ssl_pthread_server with both TLS 1.2 and TLS 1.3. Test against both OpenSSL and GnuTLS. In the server, flush more often. Otherwise, when stdout is redirected to a file, the server gets killed before it writes important information, such as the logs that we expect in the test cases. Clean up compile-time requirements in ssl_pthread_server.c: any certificate-based key exchange is ok, so don't insist on built-in RSA. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Test ssl_fork_server with both TLS 1.2 and TLS 1.3. Test against both OpenSSL and GnuTLS. In the server, flush more often. Otherwise, when stdout is redirected to a file, the server gets killed before it writes important information, such as the logs that we expect in the test cases. In the server, only write output for 10 seconds, not 100. That's enough time to start concurrent clients if desired. 100 seconds causes ssl-opt to take a very long time when the client actually listens to the whole input (which `gnutls-cli` does, but not `openssl s_client`). Clean up compile-time requirements in ssl_fork_server.c: any certificate-based key exchange is ok, so don't insist on built-in RSA. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Test against both OpenSSL and GnuTLS. Don't use a proxy. It's not particularly useful here, and would complicate figuring out port numbers. Clean up compile-time requirements dtls_server.c: any certificate-based key exchange is ok, so don't insist on built-in RSA. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
GnuTLS 3.4.x doesn't allow repeated `-p PORT` arguments. OpenSSL 1.0.2 has different logs. For TLS 1.2 test cases, use a line that is present in logs from OpenSSL 1.0.2g, 3.3.0 and presumably all versions in between. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
This is necessary when testing against OpenSSL 1.0.2g. In the server, flush more often. Otherwise, when stdout is redirected to a file, the server gets killed before it writes important information, such as the logs that we expect in the test cases. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Default to connecting to "localhost", like ssl_client1. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Mbed-TLS#9560 Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
dtls_client connects to "localhost", which is usually IPv6 on modern systems. On our CI, $OPENSSL is OpenSSL 1.0.2g which doesn't support IPv6. Pitching dtls_client against $OPENSSL works on the CI at the moment, but only because the CI runs in Docker with default network settings which has IPv6 disabled. This would stop working if we changed the CI's Docker setup, and the test case is likely to fail on a developer machine. So switch the test case to using $OPENSSL_NEXT (which is a version of OpenSSL that has IPv6 support). Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Always test all configurations. For each configuration, run all the tests unless the build fails. Print a summary of failures at the end. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
f4dd36a
to
9b01ff6
Compare
When building with `configs/config-suite-b.h`, the SSL I/O buffer size is 1024 bytes. Experimentally, this isn't quite enough for the test certificate that we use: the server aborts the handshake with `MBEDTLS_ERR_SSL_BUFFER_TOO_SMALL` raised from `mbedtls_ssl_write_certificate()`. State an ad hoc minimum output buffer size to skip testing `ssl_server` in `config-suite-b`. Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
9b01ff6
to
58ebd1c
Compare
Checking changes since my last (partial review). It seems that the following change has been removed: - # Check if test uses files
- case "$SRV_CMD $CLI_CMD" in
- *$DATA_FILES_PATH/*)
+ # Check if our test programs use files.
+ case "$CLI_CMD" in
+ *programs/ssl/*"$DATA_FILES_PATH/"*)
+ requires_config_enabled MBEDTLS_FS_IO;;
+ esac
+ case "$SRV_CMD" in
+ *programs/ssl/*"$DATA_FILES_PATH/"*) I thought that was a good change, why did you remove it? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Incremental review since my previous partial review, not complete yet, but looking pretty good so far.
# data, no matter what happens afterwards. | ||
run_test "Sample: dtls_client, ssl_server2" \ | ||
-P 4433 \ | ||
"$PROGRAMS_DIR/ssl_server2 dtls=1 server_addr=localhost" \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Out of curiosity: why do we need server_addr=localhost
here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
By default, on my machine, ssl_server2
only listens on IPv4. I actually don't understand why: I would have thought it would prioritize IPv6. dtls_client
connects to localhost
, whatever it is. I remember it took me a little while to find something that would minimize changes in the test programs, have the tests pass on my machine (where localhost
resolves to ::1
), and have the tests pass on the CI (where IPv6 is disabled).
I don't know why this isn't necessary for ssl_client1
.
-S "error" \ | ||
-C "error" | ||
|
||
run_test "Sample: ssl_client1 with ssl_server" \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor: naming is inconsistent. Not sure it matters though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are two naming patterns: “Sample: PROGRAM, PEER” for testing a sample program against a peer that isn't a sample program (OpenSSL, GnuTLS or Mbed TLS's fancy programs), and “Sample: PROGRAM1 with PROGRAM2” for testing two sample programs against each other. I'm open to other naming conventions if you have suggestions.
sub abort { | ||
system( "mv $config_h.bak $config_h" ) and warn "$config_h not restored\n"; | ||
# use an exit code between 1 and 124 for git bisect (die returns 255) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Note: this is going to conflict with the PR that removes this script. I'd suggest removing this commit to avoid the conflict.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wanted to get detailed CI feedback here for as long as test-ref-configs
failed. Changing the script is out of scope here and I'll remove it in the next rebase (there'll be a next rebase once #9563 is merged).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've completed my review, this looks good to me.
I've also checked that the new tests are passing locally on my machine, fwiw.
It turned out to be a huge can of worms. See #9541 (comment) |
Test the SSL sample programs (
ssl_client1
,dtls_client
,ssl_server
,ssl_fork_server
,ssl_pthread_server
,dtls_server
) inssl-opt.sh
. Test against both GnuTLS and OpenSSL. Test with all relevant protocol versions (TLS 1.2/1.3; DTLS 1.2 only).mini_client
is excluded because it's complicated, see this comment. It's also not part of the deliverable we want, which is primarilyssl_client1
andssl_server
interoperabiliy with TLS 1.3, and secondarily (because that part is not much more effort) those programs anddtls_client
anddtls_server
with (D)TLS 1.2. (I'm still throwing inssl_pthread_server
andssl_fork_server
because they're really easy.)The TLS 1.3 test cases are non-regression tests for #9072 and #8749.
Fixes #9072 in the sense that when this pull request passes the CI, it means the issue is fixed. (The actual fixes are in previous PR: #9501 and #9507.)
The sample programs hard-code port 4433, so these tests can't run concurrently. This may be a problem for CI reliability (although in principle it shouldn't be since each test run runs alone in a Docker instance). If it's a problem, the obvious solution is to make the port configurable in each program (and remove the forced-port hacks in
ssl-opt.sh
).Prerequisites:
ssl-opt.sh
that warrants its own pull request.Status: ready except for a final rebase once #9563 is merged.
PR checklist