GSoC 2021: Measuring QUIC and TLS censorship with OONI Probe

[kelmenhorst]

GSoC 2021: Measuring QUIC and TLS censorship with OONI Probe

Introduction

The main objectives of my Summer of Code project were twofold: Furthering the integration HTTP/3 measurements into OONI Probe, and enabling OONI Probe to use custom TLS fingerprints in order to circumvent potential TLS fingerprint blocking. These objectives have been realized in the form of a new experiment prototype, tentatively called Websteps, which supports HTTP/3 measurements and uses uTLS to emulate popular TLS fingerprints.

In the following I will describe the deliverables of my GSoC experience, and link to the corresponding pull requests in the ooni/probe-cli repository.

Along with the main deliverables, I also fixed bugs that I discovered in the existing implementation, which are for example documented here: ooni/probe-cli#432 (comment).

The design choices are a result of the collaboration with my GSoC mentor, @bassosimone, who I am very thankful for.

Deliverable 1: Refactored OONI error classification

Merged pull request: ooni/probe-cli#386

This contribution is a refactoring of the classifier code which is responsible for translating network errors to OONI error types.

The outcome in particular:

Improved handling of QUIC errors by

a) importing QUIC error codes from quic-go,

b) importing TLS alert codes from RFC5246,

Improved handling of system errors by introducing system errno constants to detect and identify system errors. This solved the problem of classifying localized Windows error strings: #1526.

Both improvements provide more robust alternatives for string comparisons.

As part of the refactoring I also divided the classifier code by layers, so that each of the following protocol layers has it‘s own classifier with layer-specific errors: DNS, TLS over TCP, and QUIC.

A general classifier remains to handle errors that can occur on multiple layers.

This change helps with the maintainability of the classification code and improves the interface of the errorx package.

Unmerged pull request: ooni/probe-cli#377, with the first design.

Deliverable 2: utls for TLS parroting

Merged commit: ooni/probe-cli@285290a

This commit introduced UTLSConn, a new type of TLS-like connection, into the Probe engine. This enables the OONI Probe networking library to use utls, which provides ClientHello fingerprinting resistance and low-level access to TLS handshake specifics.

Limitations: net/http does not support using alternative TLS implementations apart from crypto/tls, due to an implementation that casts to (tls.Conn). This is why the http.Transport breaks when using utls (see refraction-networking/utls#16).

Unmerged pull request: ooni/probe-cli#414, with more utls integration work, which is described in the pull request description. After considering and discussing the integration design with my mentor, we decided to replace it with a different approach which is described below.

Deliverable 3: Websteps - A new OONI Probe network experiment and Testhelper

Specification ooni/spec#219 of the new testhelper design (original design idea by Simone Basso, with my contribution to the further elaboration).

Merged pull request: ooni/probe-cli#432

The OONI testhelper runs the control measurements for the Web Connectivity experiment. Control measurements replicate the probe client's measurements from an uncensored network and indicate what to expect. That way, the OONI Probe client software can detect anomalies in the client's network.

Web Connectivity is a mainstream, large-scale network censorship experiment of OONI Probe, which is deployed in the enduser application. It takes input in the form of URLs and measures whether the hosts are accessible to the probe client.

ooni/probe-cli#432 contains the new implementation of the testhelper along with comprehensive unit and integration tests, as well as the prototype of a corresponding new mainstream censorship measurement which is similar to Web Connectivity.

The new testhelper introduces HTTP/3 control measurements, and can use utls for TLS parroting. It also supports testing multiple IP endpoints per domain which the legacy design does not.

As precisely described in ooni/spec#219, the testhelper algorithm consists of three main steps:

• InitialChecks verifies that the input URL can be parsed, has an expected scheme, and contains a valid domain name.
• Explore enumerates all the URLs that it discovers by redirection from the original URL, or by detecting h3 support at the target host.
• Generate performs a step-by-step measurement of each discovered URL.

The idea of the new testhelper is that it tells the probe client what to measure, and what to expect. It produces a set of URLs that originated from the initial input URL. Follow-up URLs may be caused by HTTP redirections or when the server announces support for HTTP/3.

Unmerged pull request including the first design of a new testhelper, which we discarded due to its high complexity in regards to manual HTTP redirection and cookies handling.

Websteps (preliminary name) is a new OONI Probe network measurement, which is at the prototype stage still. Websteps is the client probe software which can work in cooperation with the above described testhelper to know what to measure and what to expect. Websteps has the same high-level measurement objective as Web Connectivity and further supports HTTP/3 measurements, TLS parroting and testing all IP endpoints of a domain.

The prototype does not yet have:

• unit and integration tests,
• a policy for choosing which endpoints to measure, out of the endpoints proposed by the testhelper,
• an analysis tool to compare the control and the probe measurement.

Websteps and the testhelper use the same code for performing network operations such as the TCP three-way-handshake, or the HTTP roundtrip.

Unmerged pull request: ooni/probe-cli#431, including the first design of a new Web Connectivity-like experiment, which we discarded due to its high complexity in regards to manual HTTP redirection and cookies handling.

Deliverable 4: Enable TLS parroting for Websteps

Merged pull request: ooni/probe-cli#442

This pull request enables Websteps and the testhelper to use uTLS for TLS parroting.

In order to solve the previously described problems with using a fork of the crypto/tls library, we have forked the net/http standard library (ooni/oohttp) and modified it so that it supports all tls-like connections, like utls.Conn. As an API, I built a wrapper for the Transport (oohttp.StdlibTransport) which looks like a "net/http".Transport but uses the "ooni/oohttp".Transport internally.

The testhelper and Websteps code uses this transport wrapper for making HTTP requests with uTLS connections. As a result, we gain access to the TLS fingerprint. Tentatively, we apply the utls.HelloChrome_Auto fingerprint, i.e. the simulated TLS fingerprint of the Google Chrome browser.

How to build and test Websteps

Build and deploy the testhelper locally

The default listening endpoint is ":8080".

go build ./cmd/oohelperd
./oohelperd [-endpoint]

Try out the testserver with curl

Minimal example, with the default testhelper endpoint ":8080".

curl -X POST -d '{"url": "http://www.google.com"}' http://localhost:8080/api/unstable/websteps

Build and run Websteps

go build ./cmd/miniooni
./miniooni -i "https://www.google.com" --reportfile="measurement.json" websteps

[bassosimone]

Thanks for writing this comprehensive report @kelmenhorst! It has been a pleasure to host you for this GSoC, and I think you definitely did great work to help us at @ooni improve our measurement tools!

I put this issue in my Sprint planning and will keep it open until I have finished the final evaluation on the GSoC website.

🐳

[bassosimone]

Thanks again, I just completed the evaluation on the GSoC website! Closing this issue now.