Skip to content

OONI Probe CLI v3.19.0-alpha

Pre-release
Pre-release
Compare
Choose a tag to compare
@github-actions github-actions released this 10 Oct 16:36
v3.19.0-alpha
This tag was signed with the committer’s verified signature. The key has expired.
bassosimone Simone Basso
GPG key ID: 80B691277733D95B
Expired
Learn about vigilant mode.
29e8c88
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

Summary

This release is an alpha release. It is not ready for general availability and should only be used for QA.

A stable release will follow suit hopefully in a few days from today.

On a high-level, these are the most important changes:

  • ✨ Better Support for Measuring Throttling
  • ✨ Introducing OONI Probe Bridges
  • ✨ Improving Measurements Scrubbing
  • ✨ Netemx: A Better Unit/Integration Testing Framework
  • ✨ OONI Run v2 Support
  • ✨ Unit Testing Improvements (go test -short ./... does not use the host network anymore)
  • 🐛 OONI Probe Bootstrap fixes
  • 🐛 Web Connectivity v0.4 Fixes
  • 🐛 Snowflake Fixes

Read on for more specific details!

✨ New Features

Better Support for Measuring Throttling

These changes implement a better support for measuring throttling where we periodically collect download speed samples during the lifetime of downloads performed using Web Connectivity (or any other network experiment using measurexlite):

Note that, because these changes are measurexlite specific, and because only Web Connectivity LTE uses measurexlite, we would only get the benefit of better throttling measurements once we're fine switching all users to use Web Connectivity LTE. (Currently, 99% of the users are still using Web Connectivity v0.4, since we still have to finish doing A/B testing to make sure that the new version of Web Connectivity is not introducing any regressions compared to version v0.4.)

Introducing OONI Probe Bridges

We added support for OONI Probe Bridges. (We initially named this feature "beacons", which is why many pull requests below mention "beacons".) A OONI Probe Bridge is a host that allows us to connect to the OONI backend or test helpers. The current implementation knows about a single bridge, but it is possible to add more bridges (and we will add more bridges). The important concepts are that the probe knows the bridge IP address in advance and that there's the guarantee we can use pretty much any SNI with the bridge itself, even though, obviously, we use the api.ooni.io verify hostname when verifying the certificate. By default OONI Probe tries using a bunch of predefined SNIs and remembers what works to use it consistently. This change currently only modifies how we create TLS connections with the OONI backend, but we plan on extending the scope of these changes further in subsequent releases.

There is also a mechanism to bypass the probe behavior and force it to use a specific bridge. We're documenting this mechanism here with the understanding that this functionality should be considered as a technical preview and may change in future releases without any need on our end to change the major version number. With that caveat out of the way, with this release of OONI Probe, if you create $HOME/.ooniprobe/engine/bridges.conf, it will be used to override how we communicate with the backend. The file format is JSON with comments, and here is an example:

{
  // The DomainEndpoints field maps a domain's endpoint to a list of bridges
  "DomainEndpoints": {
    "api.ooni.io:443": [

      // A bridge is characterized by an IP address, a Port, a SNI to send on the wire, the
      // hostname to use for TLS certificate verification and a delay.
      //
      // The delay can be useful to stagger dial attempts such that you do not try all
      // the bridges at the same time. The unit of InitialDelay is nanoseconds.
      {
        "Address": "130.192.91.211",
        "InitialDelay": 0,
        "Port": "443",
        "SNI": "www.example.com",
        "VerifyHostname": "api.ooni.io"
      }

    ]
  }
}

There is also a new file $HOME/.ooniprobe/engine/httpsdialer.state that keeps the on-disk state used by OONI Probe to remember which bridges worked. This file also includes information about some bridges that did not work, however we periodically prune this cache to avoid it growing too large in case there are many failures. (We're more interested on what works anyway.)

Support for HTTP/HTTPS proxies

As part of work to introduce OONI Probe bridges, we also removed limitations in the codebase that prevented to use HTTP or HTTPS proxy URLs with ooniprobe --proxy URL and miniooni --proxy URL. We will update mobile apps to allow using this kind of proxies as well. Before these changes, we only supported (a) SOCKS5 proxies and (b) creating proxies using Psiphon, vanilla tor, or tor along with snowflake (respectively, --proxy socks5://<addr>:<port>, --proxy psiphon:///, --proxy tor:///, --proxy torsf:///). Assuming you have, e.g., tor exposing a proxy on port 9080, now you will be able to use ooniprobe --proxy http://127.0.0.1:9080/.

Improving Measurements Scrubbing

This set of changes ensures that we always scrub IP addresses or IP endpoints from HTTP response bodies and header values. We implemented this change because the changes required to support the bridges will make more likely, in the future and after some refactoring, that we'd discover the IPv4 user address and then use the IPv6 one for measuring (or the other way around). While improving IP addresses discovery is also a goal, with this set of changes we wanted to play safe and start making the codebase more robust ahead of future changes that may require such a robustness.

Netemx: A Better Unit/Integration Testing Framework

Building on https://github.com/ooni/netem, we created a comprehensive unit/integration testing framework that works in userspace using https://gvisor.dev/ and featuring support to emulate censorship. We ported existing integration tests using jafar, the previous QA framework we were using, to use netemx and removed Jafar. We also created an A/B testing framework that allows us to compare Web Connectivity v0.4 and Web Connectivity LTE side-by-side. This framework will, hopefully, simplify our job of performing QA and finally declare Web Connectivity LTE available for general use.

OONI Run v2 Support

  • feat(oonimkall): experimental OONI Run v2 API by @bassosimone in #1176. This change introduces minimum support to allow OONI Probe Android and iOS to implement OONI Run v2.

Unit Testing Improvements

  • ✨ refactor: go test -short does not use the host network by @bassosimone in #1352. This change ensures that tests marked as short either use the loopback interface or netemx, thus making the core body of tests reproducible and independent of the the place in which you are running tests.

🚧 Maintenance

Preliminary Refactoring to Enable Bridges

We merged the following refactoring pull requests to remove technical debt that prevented us from cleanly implementing bridges:

Dependency Upgrades

Misc Changes

  • 🚧 chore(oonimkall): re-enable TestTaskRunnerRun by @bassosimone in #1354
  • 🚧 fix: do not mention NewDefaultCertPool by @bassosimone in #1155
  • 🚧 [forwardport] fix: use correct Xcode and mingw version by @bassosimone in #1158
  • 🚧 fix(.github): avoid using deprecated ::set-output by @bassosimone in #1159
  • 🚧 fix: avoid submitting when tor binary is missing by @bassosimone in #1078
  • 🚧 fix(tutorial): avoid broken pipe caused by passing non-JSON to jq by @imw in #1193
  • 🚧 feat(MONOREPO): script to build stable android release by @bassosimone in #1188

🐛 Bug Fixes

OONI Probe Bootstrap fixes

  • 🐛 fix: make bootstrap more robust (and slower 😢) by @bassosimone in #1351. This change increases the timeout used for finding out OONI Probe's IP address and modifies the default weights for DNS-over-HTTPS (DoH) resolvers and the system resolver to allow OONI Probe to cold bootstrap in networks where most DoH resolvers are blocked. This change also makes the bootstrap slower if there are many timeouts. Solving the tension between robustness and efficiency would require more refactoring, which will happen in the next release.

Web Connectivity v0.4 Fixes

  • 🐛 fix: anomaly with android_dns_cache_no_data and inconsistent dns by @bassosimone in #1211. This change ensures we handle android_dns_cache_no_data as an anomaly when the test helper could resolve the URL's domain, thus resolving a long-standing data quality issue.

Snowflake Fixes

Misc Fixes

🗒️ Documentation

New Contributors

Full Changelog: v3.18.0-alpha...v3.19.0-alpha

Contributors

bassosimone, imw, and 3 other contributors
Assets 35
Loading