Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backport 3.6: test sample programs in ssl-opt.sh #9541

Open
wants to merge 32 commits into
base: mbedtls-3.6
Choose a base branch
from

Conversation

gilles-peskine-arm
Copy link
Contributor

@gilles-peskine-arm gilles-peskine-arm commented Sep 4, 2024

Test the SSL sample programs (ssl_client1, dtls_client, ssl_server, ssl_fork_server, ssl_pthread_server, dtls_server) in ssl-opt.sh. Test against both GnuTLS and OpenSSL. Test with all relevant protocol versions (TLS 1.2/1.3; DTLS 1.2 only).

mini_client is excluded because it's complicated, see this comment. It's also not part of the deliverable we want, which is primarily ssl_client1 and ssl_server interoperabiliy with TLS 1.3, and secondarily (because that part is not much more effort) those programs and dtls_client and dtls_server with (D)TLS 1.2. (I'm still throwing in ssl_pthread_server and ssl_fork_server because they're really easy.)

The TLS 1.3 test cases are non-regression tests for #9072 and #8749.

Fixes #9072 in the sense that when this pull request passes the CI, it means the issue is fixed. (The actual fixes are in previous PR: #9501 and #9507.)

The sample programs hard-code port 4433, so these tests can't run concurrently. This may be a problem for CI reliability (although in principle it shouldn't be since each test run runs alone in a Docker instance). If it's a problem, the obvious solution is to make the port configurable in each program (and remove the forced-port hacks in ssl-opt.sh).

Prerequisites:

Status: ready except for a final rebase once #9563 is merged.

PR checklist

  • changelog not required because: test only. (I don't think the improvements in compile-time dependencies in sample programs warrant a changelog entry.)
  • development PR TODO
  • framework PR not required
  • 3.6 PR here
  • 2.28 PR TBD — useful in principle, but we might not bother at this stage
  • tests provided

@gilles-peskine-arm gilles-peskine-arm added needs-work component-tls needs-ci Needs to pass CI tests component-tls13 size-s Estimated task size: small (~2d) priority-high High priority - will be reviewed soon labels Sep 4, 2024
@mpg mpg linked an issue Sep 5, 2024 that may be closed by this pull request
@gilles-peskine-arm gilles-peskine-arm added the needs-preceding-pr Requires another PR to be merged first label Sep 6, 2024
@mpg mpg self-requested a review September 9, 2024 09:39
Copy link
Contributor

@mpg mpg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a first pass, will make another when the pre-requisite is in, but looking pretty good to me so far!

!defined(MBEDTLS_TIMING_C) || !defined(MBEDTLS_FS_IO) || \
!defined(MBEDTLS_PEM_PARSE_C)
int main(int argc, char *argv[])
#if !defined(MBEDTLS_ENTROPY_C) || !defined(MBEDTLS_CTR_DRBG_C) || \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not on this line: copy-pasta in the commit message, says pthread when it's about fork.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed in my latest rebase.

Note that I intend more rebases. Both after #9546 is merged, and also because my latest batch of commits has broken something (still puzzling out what).

tests/opt-testcases/sample.sh Show resolved Hide resolved
tests/opt-testcases/sample.sh Show resolved Hide resolved
tests/opt-testcases/sample.sh Show resolved Hide resolved
requires_protocol_version tls13
requires_openssl_tls1_3
requires_config_enabled MBEDTLS_SSL_TLS1_3_COMPATIBILITY_MODE
run_test "Sample: ssl_client1, openssl server, TLS 1.3" \
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This and some other test cases are skipped in builds with only TLS 1.3. This is due to #9560, which is a preexisting bug affecting many preexisting test cases, and I will not fix it here.

@gilles-peskine-arm
Copy link
Contributor Author

gilles-peskine-arm commented Sep 11, 2024

This is getting too large: 29 commits (not counting the ones from #9556). So I intend to split this in two pieces. Probably all except mini_client, then mini_client, which brings in issues with PSK builds. Or maybe also do mini_client in the first PR, but excluding PSK builds.

Last version with the mini_client work: https://github.com/gilles-peskine-arm/mbedtls/tree/ssl-opt-mini_client-3.6-1 . This is https://github.com/gilles-peskine-arm/mbedtls/tree/ssl-opt-sample-programs-3.6-4 with commits restructured to put all the work related to mini_client after the rest. Completely untested.

tests/ssl-opt.sh Outdated
Comment on lines 1800 to 1804
# Check if our test programs use files.
case "$CLI_CMD" in
*programs/ssl/*"$DATA_FILES_PATH/"*)
requires_config_enabled MBEDTLS_FS_IO;;
esac
case "$SRV_CMD" in
*programs/ssl/*"$DATA_FILES_PATH/"*)
requires_config_enabled MBEDTLS_FS_IO;;
esac
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this innocent-looking change (at least it looked innocent to me) is causing a bunch more test cases to be executed in config-suite-b, config-ccm-psk-tls1_2.h and config-ccm-psk-dtls1_2.h. And some are failing because a genuine requirement isn't declared. The joys of ssl-opt.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And by the way, maybe we should enable MBEDTLS_FS_IO in test-ref-configs when we run ssl-opt (we already add MBEDTLS_DEBUG_C and MBEDTLS_ERROR_C)?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aw, let's not do that now, then.

The joys of ssl-opt.

Yeah, it was designed to only work in the default config, and dependency declarations were added later as an after-thought, and for some time the target was it should run in the default and full configs, then perhaps a few others that are close enough... but we never took the time to think it through, and here we are now.

And by the way, maybe we should enable MBEDTLS_FS_IO in test-ref-configs when we run ssl-opt (we already add MBEDTLS_DEBUG_C and MBEDTLS_ERROR_C)?

Yeah, I think that would make sense. Perhaps for your other PR that splits test-ref-configs? (Unless it turns out to be another can of worms.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps for your other PR that splits test-ref-configs? (Unless it turns out to be another can of worms.)

No, because a lot of the worms are test cases that currently don't run in the reference configs because they lack MBEDTLS_FS_IO. When we turn this on, they get executed, which fails for a variety of reasons: some other obvious-in-hindsight dependency is missing, or we have unfragmentable handshake packets that go over the message limit, or the DTLS adaptation code does something wrong, or …

I've filed #9595 and added it to our tech debt pile.

@gilles-peskine-arm
Copy link
Contributor Author

I've removed all the mini_client and I'm still at 23 commits (not counting #9556), plus a few more (I think at this point all that's failing is config-suite-b.h). The commits are mostly small and focused, but it's still a lot. @mpg How do you feel about the length of this PR? I'm thinking of breaking out another PR of preliminaries (compatibility mode, consequences of finer MBEDTLS_FS_IO detection, test-ref-configs).

@mpg
Copy link
Contributor

mpg commented Sep 13, 2024

The commits are mostly small and focused, but it's still a lot. @mpg How do you feel about the length of this PR?

I'm happy to review this PR as it is (plus the few commits needed for it to pass CI). I don't feel splitting it just for the sake of it would significantly help, unless there's a breaking point that really makes sense.

However, I'm wondering about PSK-only mode, which seems to be bringing a lot of issues. Would it make sense to have a first step where we only test the programs with cert-based (or if it helps, even signature-based, that is cert-based + ephemeral as opposed to static ECDH or RSA encryption) key exchanges, and leave PSK-only for step 2? (That's a real question, the answer might be no, you know better than me at this point.)

@gilles-peskine-arm
Copy link
Contributor Author

Would it make sense to have a first step where we only test the programs with cert-based (or if it helps, even signature-based, that is cert-based + ephemeral as opposed to static ECDH or RSA encryption) key exchanges, and leave PSK-only for step 2?

mini_client is the only sample program that can work with PSK at all. So my first split was already doing that. I only realized yesterday that some of the failures I had seen earlier were not due to #9556 or to the changes I made for the sake of mini_client test cases, but due to changing MBEDTLS_FS_IO detection. Given that yesterday's CI had further failures that I hadn't expected, I'm going to split this again with another preliminary that handles MBEDTLS_FS_IO detection and related changes, plus possibly MBEDTLS_SSL_TLS1_3_COMPATIBILITY_MODE changes. At this point I'm still not planning to tackle #9560.

@gilles-peskine-arm
Copy link
Contributor Author

preliminary that handles MBEDTLS_FS_IO detection and related changes

Come to think of it, the only purpose of improving MBEDTLS_FS_IO detection was to run more tests in config-ccm-psk-dtls1_2. And that was only useful for the mini_client PSK tests. While these changes are desirable, they're technical debt, not on the critical path for the deliverable. So I'm going to remove that and its consequences, file a tech debt issue with pointers, and hopefully the delivery can converge quickly. It was already passing everything except test-ref-configs.pl at one point.

The *.sh files in opt-testcases cannot be executed directly: they can only
be sourced by ssl-opt.sh. So don't make them executable and don't give them
a shebang line.

Also make sure that the first paragraph of each file is a short description.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
For better searchability and readability, call requires_config_enabled or
requires_config_disabled for each option, instead of calling
requires_all_configs_enabled or requires_all_configs_disabled with a long
list of options.

```
perl -0777 -i -pe '
    s!^requires_all_configs_(enabled|disabled) *((?:(?: \w+) *(?:\\\n)? *)+)\n!
      $state = $1;
      join("", map {"requires_config_$state $_\n"} $2 =~ /(\w+)/g)
     !egm' tests/ssl-opt.sh tests/opt-testcases/*.sh
```

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
The compile-time option MBEDTLS_SSL_TLS1_3_COMPATIBILITY_MODE gates both
support for interoperability with a peer that uses middlebox compatibility
mode, and support for activating that mode ourselves. Change code that is
only needed for interoperability to be guarded by
MBEDTLS_SSL_TLS1_3_ACCEPT_COMPATIBILITY_MODE.

As of this commit, MBEDTLS_SSL_TLS1_3_ACCEPT_COMPATIBILITY_MODE is always
enabled: there is no way to disable it, and there are no tests with it
disabled.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Adapt the test cases for TLS 1.3 middlebox compatibility mode, now that we
always interoperate with peers that support it, regardless of whether
MBEDTLS_SSL_TLS1_3_COMPATIBILITY_MODE is enabled.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
This is necessary for the SSL sample programs: they hard-code port 4433.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Test ssl_client1 with both TLS 1.2 and TLS 1.3.
Test against both OpenSSL and GnuTLS.

Clean up compile-time requirements in ssl_client1.c: any certificate-based
key exchange is ok, so don't insist on built-in RSA.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Test against both OpenSSL and GnuTLS.

Don't use a proxy. It's not particularly useful here, and would complicate
figuring out port numbers.

Clean up compile-time requirements in dtls_client.c: any certificate-based
key exchange is ok, so don't insist on built-in RSA.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Test ssl_server with both TLS 1.2 and TLS 1.3.
Test against both OpenSSL and GnuTLS.

Clean up compile-time requirements in ssl_server.c: any certificate-based
key exchange is ok, so don't insist on built-in RSA.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Test ssl_pthread_server with both TLS 1.2 and TLS 1.3.
Test against both OpenSSL and GnuTLS.

In the server, flush more often. Otherwise, when stdout is redirected to a
file, the server gets killed before it writes important information, such as
the logs that we expect in the test cases.

Clean up compile-time requirements in ssl_pthread_server.c: any certificate-based
key exchange is ok, so don't insist on built-in RSA.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Test ssl_fork_server with both TLS 1.2 and TLS 1.3.
Test against both OpenSSL and GnuTLS.

In the server, flush more often. Otherwise, when stdout is redirected to a
file, the server gets killed before it writes important information, such as
the logs that we expect in the test cases.

In the server, only write output for 10 seconds, not 100. That's enough time
to start concurrent clients if desired. 100 seconds causes ssl-opt to take a
very long time when the client actually listens to the whole input (which
`gnutls-cli` does, but not `openssl s_client`).

Clean up compile-time requirements in ssl_fork_server.c: any certificate-based
key exchange is ok, so don't insist on built-in RSA.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Test against both OpenSSL and GnuTLS.

Don't use a proxy. It's not particularly useful here, and would complicate
figuring out port numbers.

Clean up compile-time requirements dtls_server.c: any certificate-based key
exchange is ok, so don't insist on built-in RSA.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
GnuTLS 3.4.x doesn't allow repeated `-p PORT` arguments.

OpenSSL 1.0.2 has different logs. For TLS 1.2 test cases, use a line that
is present in logs from OpenSSL 1.0.2g, 3.3.0 and presumably all versions
in between.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
This is necessary when testing against OpenSSL 1.0.2g.

In the server, flush more often. Otherwise, when stdout is redirected to a
file, the server gets killed before it writes important information, such as
the logs that we expect in the test cases.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Default to connecting to "localhost", like ssl_client1.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Mbed-TLS#9560

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
dtls_client connects to "localhost", which is usually IPv6 on modern
systems. On our CI, $OPENSSL is OpenSSL 1.0.2g which doesn't support IPv6.
Pitching dtls_client against $OPENSSL works on the CI at the moment, but
only because the CI runs in Docker with default network settings which has
IPv6 disabled. This would stop working if we changed the CI's Docker setup,
and the test case is likely to fail on a developer machine. So switch the
test case to using $OPENSSL_NEXT (which is a version of OpenSSL that has
IPv6 support).

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
Always test all configurations. For each configuration, run all the tests
unless the build fails. Print a summary of failures at the end.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
@gilles-peskine-arm gilles-peskine-arm force-pushed the ssl-opt-sample-programs-3.6 branch 2 times, most recently from f4dd36a to 9b01ff6 Compare September 14, 2024 11:22
When building with `configs/config-suite-b.h`, the SSL I/O buffer size is
1024 bytes. Experimentally, this isn't quite enough for the test certificate
that we use: the server aborts the handshake with
`MBEDTLS_ERR_SSL_BUFFER_TOO_SMALL` raised from
`mbedtls_ssl_write_certificate()`. State an ad hoc minimum output buffer
size to skip testing `ssl_server` in `config-suite-b`.

Signed-off-by: Gilles Peskine <Gilles.Peskine@arm.com>
@mpg
Copy link
Contributor

mpg commented Sep 18, 2024

Checking changes since my last (partial review). It seems that the following change has been removed:

-    # Check if test uses files
-    case "$SRV_CMD $CLI_CMD" in
-        *$DATA_FILES_PATH/*)
+    # Check if our test programs use files.
+    case "$CLI_CMD" in
+        *programs/ssl/*"$DATA_FILES_PATH/"*)
+            requires_config_enabled MBEDTLS_FS_IO;;
+    esac
+    case "$SRV_CMD" in
+        *programs/ssl/*"$DATA_FILES_PATH/"*)

I thought that was a good change, why did you remove it?

Copy link
Contributor

@mpg mpg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incremental review since my previous partial review, not complete yet, but looking pretty good so far.

# data, no matter what happens afterwards.
run_test "Sample: dtls_client, ssl_server2" \
-P 4433 \
"$PROGRAMS_DIR/ssl_server2 dtls=1 server_addr=localhost" \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity: why do we need server_addr=localhost here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By default, on my machine, ssl_server2 only listens on IPv4. I actually don't understand why: I would have thought it would prioritize IPv6. dtls_client connects to localhost, whatever it is. I remember it took me a little while to find something that would minimize changes in the test programs, have the tests pass on my machine (where localhost resolves to ::1), and have the tests pass on the CI (where IPv6 is disabled).

I don't know why this isn't necessary for ssl_client1.

-S "error" \
-C "error"

run_test "Sample: ssl_client1 with ssl_server" \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: naming is inconsistent. Not sure it matters though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two naming patterns: “Sample: PROGRAM, PEER” for testing a sample program against a peer that isn't a sample program (OpenSSL, GnuTLS or Mbed TLS's fancy programs), and “Sample: PROGRAM1 with PROGRAM2” for testing two sample programs against each other. I'm open to other naming conventions if you have suggestions.

sub abort {
system( "mv $config_h.bak $config_h" ) and warn "$config_h not restored\n";
# use an exit code between 1 and 124 for git bisect (die returns 255)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: this is going to conflict with the PR that removes this script. I'd suggest removing this commit to avoid the conflict.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to get detailed CI feedback here for as long as test-ref-configs failed. Changing the script is out of scope here and I'll remove it in the next rebase (there'll be a next rebase once #9563 is merged).

Copy link
Contributor

@mpg mpg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've completed my review, this looks good to me.

I've also checked that the new tests are passing locally on my machine, fwiw.

@gilles-peskine-arm
Copy link
Contributor Author

[FS_IO change]

I thought that was a good change, why did you remove it?

It turned out to be a huge can of worms. See #9541 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component-tls component-tls13 needs-preceding-pr Requires another PR to be merged first priority-high High priority - will be reviewed soon size-s Estimated task size: small (~2d)
Projects
Development

Successfully merging this pull request may close these issues.

ssl_client1 fails on TLS 1.3
2 participants