Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(enginenetx): add configurable HTTPS dialer #1283

Merged
merged 7 commits into from
Sep 20, 2023
Merged

Conversation

bassosimone
Copy link
Contributor

@bassosimone bassosimone commented Sep 20, 2023

This commit introduces a configurable HTTPS dialer that we will use for implementing beacons, and possibly also other functionality.

We need to perform TCP connect and TLS handshake as part of the same goroutine, because we cannot consider a dialing attempt successful after a successful TCP connect. Due to network interference, the dialing may also fail later during the TLS handshake.

To support several possible HTTPS dialing strategies, we need to extend what happens during a LookupHost operation. Generally, one would like to have addresses to dial. Rather, here we have tactics, where a single IP address MAY be included into more than a single tactic, if we need to try different tricks with the same address.

Also, tactics include a delay, which is useful to (a) avoid performing all operations in parallel, which is not gentle towards otherwise perfectly functioning networks and (b) give penalty to tactics that utilize circumvention, such that we don't even attempt them unless we need to.

In turn, the DNS resolver is wrapped by a policy for configuring TLS dialing. Basically, the policy observes the IP addresses returned by an underlying resolver and then it will decide which tactics to produce based on that. Note that the policy could also extend the set of returned IP addresses when the domain for which we connect is such that we have known IP addresses in advance for such a domain.

The default policy we introduce in this commit behaves as follows:

  1. it asks the engine to create 16 goroutines for dialing;

  2. it uses the DNS lookup results w/o adding any extra IP addr;

  3. it produces a tactic for each IP address where we use the domain as the SNI and we add a 300 millisecond delay to the second tactic, 600 to the third, and so on--which is similar to implementing happy eyeballs.

It's also worth noting that, tactics MAY override the TLS handshaker being used (for example, to use uTLS) and, also, because they may use different SNIs, the TLS verification is performed AFTER the TLS handshake. (Because I set InsecureSkipVerify, I expected--or rather demand--GitHub to produce a security warning for this commit, which I will mark as a false positive, because TLS verification is performed a few lines of code after that.)

As far as the algorithm for verification is concerned, it comes from the Go examples and it matches closely the code used by the standard library to perform verification. More details about this in comments in the commit code.

Part of ooni/probe#2531

This commit introduces a configurable HTTPS dialer that we will
use for implementing beacons, and possibly also other functionality.

We need to perform TCP connect and TLS handshake as part of the
same goroutine, because we cannot consider a dialing attempt
successful after a successful TCP connect. Due to network interference,
the dialing may also fail later during the TLS handshake.

To support several possible HTTPS dialing strategies, we need to
extend what happens during a LookupHost operation. Generally, one
would like to have addresses to dial. Rather, here we have
tactics, where a single IP address MAY be included into more
than a single tactic, if we need to try different tactics with
the same address.

Also, tactics include a delay, which is useful to (a) avoid
performing all operations in parallel, which is not gentle
towards otherwise perfectly functioning networks and (b) give
penalty to tactics that utilize circumvention, such that we
don't even attempt them unless we need to.

In turn, the DNS resolver is extended and now it is a policy
for configuring dialing. Basically, the policy observes the IP
addresses returned by an underlying resolver and then it will
decide which tactics to produce based on that. Note that the
policy could also extend the set of returned IP addresses when
the domain for which we connect is such that we have known
IP addresses in advance for such a domain.

The default policy we introduce in this commit behaves as follows:

1. it asks the engine to create 16 goroutines for dialing;

2. it uses the DNS lookup results w/o adding any extra IP addr;

3. it produces a tactic for each IP address where we use the
domain as the SNI and we add a 300 millisecond delay to the
second tactic, 600 to the third, and so on--which is similar
to implementing happy eyeballs.

It's also worth noting that, tactics MAY override the TLS
handshaker being used (for example, to use uTLS) and, also,
because they may use different SNIs, the TLS verification
is performed AFTER the TLS handshake.

Part of ooni/probe#2531
if _, err := state.PeerCertificates[0].Verify(opts); err != nil {
return netxlite.NewErrWrapper(netxlite.ClassifyTLSHandshakeError, netxlite.TopLevelOperation, err)
}
return nil
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For comparison, this is what Go code does:

	if !c.config.InsecureSkipVerify {
		opts := x509.VerifyOptions{
			Roots:         c.config.RootCAs,
			CurrentTime:   c.config.time(),
			DNSName:       c.config.ServerName,
			Intermediates: x509.NewCertPool(),
		}

		for _, cert := range certs[1:] {
			opts.Intermediates.AddCert(cert)
		}
		var err error
		c.verifiedChains, err = certs[0].Verify(opts)
		if err != nil {
			c.sendAlert(alertBadCertificate)
			return &CertificateVerificationError{UnverifiedCertificates: certs, Err: err}
		}
	}

See https://github.com/golang/go/blob/go1.21.0/src/crypto/tls/handshake_client.go#L962

@bassosimone bassosimone merged commit dff7ced into master Sep 20, 2023
6 checks passed
@bassosimone bassosimone deleted the issue/2531 branch September 20, 2023 10:08
Murphy-OrangeMud pushed a commit to Murphy-OrangeMud/probe-cli that referenced this pull request Feb 13, 2024
This commit introduces a configurable HTTPS dialer that we will use for
implementing beacons, and possibly also other functionality.

We need to perform TCP connect and TLS handshake as part of the same
goroutine, because we cannot consider a dialing attempt successful after
a successful TCP connect. Due to network interference, the dialing may
also fail later during the TLS handshake.

To support several possible HTTPS dialing strategies, we need to extend
what happens during a LookupHost operation. Generally, one would like to
have addresses to dial. Rather, here we have tactics, where a single IP
address MAY be included into more than a single tactic, if we need to
try different tricks with the same address.

Also, tactics include a delay, which is useful to (a) avoid performing
all operations in parallel, which is not gentle towards otherwise
perfectly functioning networks and (b) give penalty to tactics that
utilize circumvention, such that we don't even attempt them unless we
need to.

In turn, the DNS resolver is wrapped by a policy for configuring TLS
dialing. Basically, the policy observes the IP addresses returned by an
underlying resolver and then it will decide which tactics to produce
based on that. Note that the policy could also extend the set of
returned IP addresses when the domain for which we connect is such that
we have known IP addresses in advance for such a domain.

The default policy we introduce in this commit behaves as follows:

1. it asks the engine to create 16 goroutines for dialing;

2. it uses the DNS lookup results w/o adding any extra IP addr;

3. it produces a tactic for each IP address where we use the domain as
the SNI and we add a 300 millisecond delay to the second tactic, 600 to
the third, and so on--which is similar to implementing happy eyeballs.

It's also worth noting that, tactics MAY override the TLS handshaker
being used (for example, to use uTLS) and, also, because they may use
different SNIs, the TLS verification is performed AFTER the TLS
handshake. (Because I set `InsecureSkipVerify`, I expected--or rather
*demand*--GitHub to produce a security warning for this commit, which I
will mark as a false positive, because TLS verification is performed a
few lines of code after that.)

As far as the algorithm for verification is concerned, it comes from the
Go examples and it matches closely the code used by the standard library
to perform verification. More details about this in comments in the
commit code.

Part of ooni/probe#2531
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant