-
Notifications
You must be signed in to change notification settings - Fork 54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(enginenetx): add configurable HTTPS dialer #1283
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit introduces a configurable HTTPS dialer that we will use for implementing beacons, and possibly also other functionality. We need to perform TCP connect and TLS handshake as part of the same goroutine, because we cannot consider a dialing attempt successful after a successful TCP connect. Due to network interference, the dialing may also fail later during the TLS handshake. To support several possible HTTPS dialing strategies, we need to extend what happens during a LookupHost operation. Generally, one would like to have addresses to dial. Rather, here we have tactics, where a single IP address MAY be included into more than a single tactic, if we need to try different tactics with the same address. Also, tactics include a delay, which is useful to (a) avoid performing all operations in parallel, which is not gentle towards otherwise perfectly functioning networks and (b) give penalty to tactics that utilize circumvention, such that we don't even attempt them unless we need to. In turn, the DNS resolver is extended and now it is a policy for configuring dialing. Basically, the policy observes the IP addresses returned by an underlying resolver and then it will decide which tactics to produce based on that. Note that the policy could also extend the set of returned IP addresses when the domain for which we connect is such that we have known IP addresses in advance for such a domain. The default policy we introduce in this commit behaves as follows: 1. it asks the engine to create 16 goroutines for dialing; 2. it uses the DNS lookup results w/o adding any extra IP addr; 3. it produces a tactic for each IP address where we use the domain as the SNI and we add a 300 millisecond delay to the second tactic, 600 to the third, and so on--which is similar to implementing happy eyeballs. It's also worth noting that, tactics MAY override the TLS handshaker being used (for example, to use uTLS) and, also, because they may use different SNIs, the TLS verification is performed AFTER the TLS handshake. Part of ooni/probe#2531
bassosimone
commented
Sep 20, 2023
bassosimone
commented
Sep 20, 2023
if _, err := state.PeerCertificates[0].Verify(opts); err != nil { | ||
return netxlite.NewErrWrapper(netxlite.ClassifyTLSHandshakeError, netxlite.TopLevelOperation, err) | ||
} | ||
return nil |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For comparison, this is what Go code does:
if !c.config.InsecureSkipVerify {
opts := x509.VerifyOptions{
Roots: c.config.RootCAs,
CurrentTime: c.config.time(),
DNSName: c.config.ServerName,
Intermediates: x509.NewCertPool(),
}
for _, cert := range certs[1:] {
opts.Intermediates.AddCert(cert)
}
var err error
c.verifiedChains, err = certs[0].Verify(opts)
if err != nil {
c.sendAlert(alertBadCertificate)
return &CertificateVerificationError{UnverifiedCertificates: certs, Err: err}
}
}
See https://github.com/golang/go/blob/go1.21.0/src/crypto/tls/handshake_client.go#L962
Murphy-OrangeMud
pushed a commit
to Murphy-OrangeMud/probe-cli
that referenced
this pull request
Feb 13, 2024
This commit introduces a configurable HTTPS dialer that we will use for implementing beacons, and possibly also other functionality. We need to perform TCP connect and TLS handshake as part of the same goroutine, because we cannot consider a dialing attempt successful after a successful TCP connect. Due to network interference, the dialing may also fail later during the TLS handshake. To support several possible HTTPS dialing strategies, we need to extend what happens during a LookupHost operation. Generally, one would like to have addresses to dial. Rather, here we have tactics, where a single IP address MAY be included into more than a single tactic, if we need to try different tricks with the same address. Also, tactics include a delay, which is useful to (a) avoid performing all operations in parallel, which is not gentle towards otherwise perfectly functioning networks and (b) give penalty to tactics that utilize circumvention, such that we don't even attempt them unless we need to. In turn, the DNS resolver is wrapped by a policy for configuring TLS dialing. Basically, the policy observes the IP addresses returned by an underlying resolver and then it will decide which tactics to produce based on that. Note that the policy could also extend the set of returned IP addresses when the domain for which we connect is such that we have known IP addresses in advance for such a domain. The default policy we introduce in this commit behaves as follows: 1. it asks the engine to create 16 goroutines for dialing; 2. it uses the DNS lookup results w/o adding any extra IP addr; 3. it produces a tactic for each IP address where we use the domain as the SNI and we add a 300 millisecond delay to the second tactic, 600 to the third, and so on--which is similar to implementing happy eyeballs. It's also worth noting that, tactics MAY override the TLS handshaker being used (for example, to use uTLS) and, also, because they may use different SNIs, the TLS verification is performed AFTER the TLS handshake. (Because I set `InsecureSkipVerify`, I expected--or rather *demand*--GitHub to produce a security warning for this commit, which I will mark as a false positive, because TLS verification is performed a few lines of code after that.) As far as the algorithm for verification is concerned, it comes from the Go examples and it matches closely the code used by the standard library to perform verification. More details about this in comments in the commit code. Part of ooni/probe#2531
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit introduces a configurable HTTPS dialer that we will use for implementing beacons, and possibly also other functionality.
We need to perform TCP connect and TLS handshake as part of the same goroutine, because we cannot consider a dialing attempt successful after a successful TCP connect. Due to network interference, the dialing may also fail later during the TLS handshake.
To support several possible HTTPS dialing strategies, we need to extend what happens during a LookupHost operation. Generally, one would like to have addresses to dial. Rather, here we have tactics, where a single IP address MAY be included into more than a single tactic, if we need to try different tricks with the same address.
Also, tactics include a delay, which is useful to (a) avoid performing all operations in parallel, which is not gentle towards otherwise perfectly functioning networks and (b) give penalty to tactics that utilize circumvention, such that we don't even attempt them unless we need to.
In turn, the DNS resolver is wrapped by a policy for configuring TLS dialing. Basically, the policy observes the IP addresses returned by an underlying resolver and then it will decide which tactics to produce based on that. Note that the policy could also extend the set of returned IP addresses when the domain for which we connect is such that we have known IP addresses in advance for such a domain.
The default policy we introduce in this commit behaves as follows:
it asks the engine to create 16 goroutines for dialing;
it uses the DNS lookup results w/o adding any extra IP addr;
it produces a tactic for each IP address where we use the domain as the SNI and we add a 300 millisecond delay to the second tactic, 600 to the third, and so on--which is similar to implementing happy eyeballs.
It's also worth noting that, tactics MAY override the TLS handshaker being used (for example, to use uTLS) and, also, because they may use different SNIs, the TLS verification is performed AFTER the TLS handshake. (Because I set
InsecureSkipVerify
, I expected--or rather demand--GitHub to produce a security warning for this commit, which I will mark as a false positive, because TLS verification is performed a few lines of code after that.)As far as the algorithm for verification is concerned, it comes from the Go examples and it matches closely the code used by the standard library to perform verification. More details about this in comments in the commit code.
Part of ooni/probe#2531