doc: explain how to write ooni nettests #506

bassosimone · 2021-09-24T13:14:40Z

Checklist

I have read the contribution guidelines
reference issue for this pull request: Publish documentation explaining how to write OONI tests ooni.org#361 and webconnectivity: write prototype of new experiment probe#1733.
related ooni/spec pull request: N/A

Location of the issue tracker: https://github.com/ooni/probe

Description

This is a rather advanced draft that still requires QA. I am going
to push this commit so others soon can start to review it.

This draft supersedes #505

This diff also contains a library of building blocks that is
instrumental to complete websteps. In fact, the tutorial I have
written here will eventually guide people to write their own
simplified implementation of websteps (and Web Connectivity).

I have not written any test for the new code so far. It is
also a stretch to call this new code. It is existing code that
has been written by me or @kelmenhorst that I have adapted
for the purpose of writing the tutorial.

The nice exercise of writing a tutorial is that you are more
easily frustrated if the code is difficult to use XD.

How to review the tutorials

Navigate to internal/tutorial/measurex and follow the links
inside the README.md file. Or go to https://github.com/ooni/probe-cli/tree/planc/internal/tutorial/measurex
and read starting from there, which is equivalent.

Editing is quite tricky. Most text comes from comments in
Go files. You need to edit the comments and then you need
to regenerate all the files. The README.md of the
internal/tutorial/measurex tutorial contains instructions
on how to do that.

What remains to be done before merging

Mainly adding tests and making sure I didn't lose
anything when adapting existing code. I like the interface
of the new code more, but existing code written last
August may have a better implementation of corner cases.

Plan B's branch name is "db"

Allows to cut the most significant bit of complexity which also happened to be buggy :^)

Simplifies processing with python

We need to do something different now: simpler TH and basically no need to explore too much

This commit introduce a measurement library that consists of refactored code from earlier websteps experiments. I am not going to add tests for the time being, because this library is still a bit in flux, as we finalize websteps. I will soon though commit documentation explaining in detail how to use it, which currrently is at #506 and adds a new directory to internal/tutorial. The core idea of this measurement library is to allow two measurement modes: 1. tracing, which is what we're currently doing now, and the tutorial shows how we can rewrite the measurement part of web connectivity with measurex using less code. Under a tracing approach, we construct a normal http.Client that however has tracing configured, we gather events for resolve, connect, TLS handshake, QUIC handshake, HTTP round trip, etc. and then we try to make sense of what happened from the events stream; 2. step-by-step, which is what websteps does, and basically means that after each operation you immediately write into a Measurement structure its results and immediately draw the conclusions on what seems odd (which later may become an anomaly if we see what the test helper measured). This library is also such that it produces a data format compatible with the current OONI spec. This work is part of ooni/probe#1733.

This diff adds the prototype websteps implementation that used to live at #506. The code is reasonably good already and it's pointing to a roaming test helper that I've properly configured. You can run websteps with: ``` ./miniooni -n websteps ``` This will go over the test list for your country. At this stage the mechanics of the experiment is set, but we still need to have a conversation on the following topics: 1. whether we're okay with reusing the data format used by other OONI experiments, or we would like to use a more compact data format (which may either be a more compact JSON or we can choose to always submit compressed measurements for websteps); 2. the extent to which we would like to keep the measurement as a collection of "the experiment saw this" and "the test helper saw that" and let the pipeline choose an overall score: this is clearly an option, but there is also the opposite option to build a summary of the measurement on the probe. Compared to the previous prototype of websteps, the main architectural change we have here is that we are following the point of view of the probe and the test helper is much more dumb. Basically, the probe will choose which redirection to follow and ask the test helper every time it discovers a new URL to measure it w/o redirections.

This diff adds the prototype websteps implementation that used to live at #506. The code is reasonably good already and it's pointing to a roaming test helper that I've properly configured. You can run websteps with: ``` ./miniooni -n websteps ``` This will go over the test list for your country. At this stage the mechanics of the experiment is set, but we still need to have a conversation on the following topics: 1. whether we're okay with reusing the data format used by other OONI experiments, or we would like to use a more compact data format (which may either be a more compact JSON or we can choose to always submit compressed measurements for websteps); 2. the extent to which we would like to keep the measurement as a collection of "the experiment saw this" and "the test helper saw that" and let the pipeline choose an overall score: this is clearly an option, but there is also the opposite option to build a summary of the measurement on the probe. Compared to the previous prototype of websteps, the main architectural change we have here is that we are following the point of view of the probe and the test helper is much more dumb. Basically, the probe will choose which redirection to follow and ask the test helper every time it discovers a new URL to measure it w/o redirections. Reference issue: ooni/probe#1733

This is experimental code that was part of the branch #506 and that I am not sure whether it could really be useful when running in prod, but may be useful for running some benchmarks of websteps vs webconnectivity.

bassosimone · 2021-09-30T00:11:19Z

Everything has been merged, except for #531, for which I am not sure. I suspect that code is mostly going to be useful to me when running benchmarks. Either way, this PR can now be closed, as it has been superseded by the PRs cited above, where I merged the work done here in easier-to-digest chunks.

See #506

This diff has been extracted from ooni#506. In it, we introduce wrapping constructors for types and we update the docs. These new constructures are used by the code in ooni#506. In itself, this work is part of ooni/probe#1733.

I have recently seen a data race related our way of mutating the outgoing request to set the host header. Unfortunately, I've lost track of the race output, because I rebooted my Linux box before saving it. Though, after inspecting why and and where we're mutating outgoing requets, I've found that: 1. we add the host header when logging to have it logged, which is not a big deal since we already emit the URL rather than just the URL path when logging a request, and so we can safely zap this piece of code; 2. as a result, in measurements we may omit the host header but again this is pretty much obvious from the URL itself and so it should not be very important (nonetheless, avoid surprises and keep the existing behavior); 3. when the User-Agent header is not set, we default to a `miniooni/0.1.0-dev` user agent, which is probably not very useful anyway, so we can actually remove it. Part of ooni/probe#1733 (this diff has been extracted from ooni#506).

While there, make sure netxlite has 100% coverage. Part of ooni/probe#1733 and diff has been extracted from ooni#506.

Part of ooni/probe#1733 and diff has been extracted from ooni#506.

This diff attempts to modify the errors reported by our custom resolver by matching more strings from the stdlib. Part of ooni/probe#1733 and diff has been extracted from ooni#506.

This new API call performs DNS lookups for HTTPS records. Part of ooni/probe#1733 and diff has been extracted from ooni#506.

The main tutorial will be the one at ooni#506, but it's useful to also document the primitives used by measurex. So, here's the companion tutorial, which explains how to use the features in netxlite to perform measurements. This work is part of ooni/ooni.org#361.

Part of ooni#506. In parallel with tutorials, we also need to make sure we have good documentation.

Should fix ooni#523 (comment). Work part of ooni#506.

This commit introduce a measurement library that consists of refactored code from earlier websteps experiments. I am not going to add tests for the time being, because this library is still a bit in flux, as we finalize websteps. I will soon though commit documentation explaining in detail how to use it, which currrently is at ooni#506 and adds a new directory to internal/tutorial. The core idea of this measurement library is to allow two measurement modes: 1. tracing, which is what we're currently doing now, and the tutorial shows how we can rewrite the measurement part of web connectivity with measurex using less code. Under a tracing approach, we construct a normal http.Client that however has tracing configured, we gather events for resolve, connect, TLS handshake, QUIC handshake, HTTP round trip, etc. and then we try to make sense of what happened from the events stream; 2. step-by-step, which is what websteps does, and basically means that after each operation you immediately write into a Measurement structure its results and immediately draw the conclusions on what seems odd (which later may become an anomaly if we see what the test helper measured). This library is also such that it produces a data format compatible with the current OONI spec. This work is part of ooni/probe#1733.

This diff adds the prototype websteps implementation that used to live at ooni#506. The code is reasonably good already and it's pointing to a roaming test helper that I've properly configured. You can run websteps with: ``` ./miniooni -n websteps ``` This will go over the test list for your country. At this stage the mechanics of the experiment is set, but we still need to have a conversation on the following topics: 1. whether we're okay with reusing the data format used by other OONI experiments, or we would like to use a more compact data format (which may either be a more compact JSON or we can choose to always submit compressed measurements for websteps); 2. the extent to which we would like to keep the measurement as a collection of "the experiment saw this" and "the test helper saw that" and let the pipeline choose an overall score: this is clearly an option, but there is also the opposite option to build a summary of the measurement on the probe. Compared to the previous prototype of websteps, the main architectural change we have here is that we are following the point of view of the probe and the test helper is much more dumb. Basically, the probe will choose which redirection to follow and ask the test helper every time it discovers a new URL to measure it w/o redirections. Reference issue: ooni/probe#1733

See ooni#506

bassosimone added 30 commits September 19, 2021 16:07

netx: carry over changes from plan B

a3d214c

Plan B's branch name is "db"

plan C

a4ac1a2

improve websteps prototype

e1de317

refactor: move websteps specific code outisde of measurex

ac0ee0b

feat: from a single step to multiple steps

7304342

feat: add oddity and holding logger

2b33c25

forgot to commit oddity.go

7bda89e

start doing some QA and fixing bugs

15da819

heavy refactoring

f1db43a

support parallel HTTPEndpoint measurements

82289ed

identify sharing constraints and fix

be77fd7

allow for parallel dns resolutions

1272aa2

allow running automatic "parallel" TH queries

df68e46

at last add code for parsing a URL

9a25631

compute all the redirections

e77020f

implement following redirections

f922ec5

fix: pass around the measurement ID

97519ae

Allows to cut the most significant bit of complexity which also happened to be buggy :^)

also archival now is okay

d163f46

start to commit the tutorials

57d9f5d

ignore jafar binary

91b0ae3

teach miniooni to run websteps

6dd602a

fix: compute measurement runtime in a better way

4b8f005

feat: add runtime metrics before experimental run

c9f88cc

fix: improve UX related to logging

9e11026

more clarity

0ec2e7a

improve logging after feedback and add bandwidth monitoring

a4b1cfa

fix: also save elapsed time

74e8fa5

Simplifies processing with python

implement and expose new websteps test helper

a431a0d

We need to do something different now: simpler TH and basically no need to explore too much

make sure we also parse alt-svc

2a30471

document each file's content and rationale

a75f50f

bassosimone added 4 commits September 30, 2021 01:24

Merge branch 'master' into planc

5597d4e

Merge branch 'master' into planc

f8209df

fix wrong diff with master

cc3a78e

fix(webstepsx): point code to roaming th

1744cd0

bassosimone mentioned this pull request Sep 29, 2021

feat(webstepsx): websteps using measurex #530

Merged

3 tasks

Merge branch 'master' into planc

72ef21d

bassosimone mentioned this pull request Sep 30, 2021

feat(bwmon): monitor bandwidth usage #531

Draft

3 tasks

bassosimone closed this Sep 30, 2021

bassosimone deleted the planc branch September 30, 2021 00:11

bassosimone added a commit that referenced this pull request Oct 4, 2021

fix: add missing reference to netxlite's tutorial

adbaa5f

See #506

bassosimone mentioned this pull request Oct 4, 2021

fix: add missing reference to netxlite's tutorial #535

Merged

3 tasks

bassosimone added a commit that referenced this pull request Oct 4, 2021

fix: add missing reference to netxlite's tutorial (#535)

9b52a05

See #506

ainghazal pushed a commit to ainghazal/probe-cli that referenced this pull request Mar 8, 2022

fix(netxlite/quic): wrap Close (ooni#509)

d57f2e1

While there, make sure netxlite has 100% coverage. Part of ooni/probe#1733 and diff has been extracted from ooni#506.

ainghazal pushed a commit to ainghazal/probe-cli that referenced this pull request Mar 8, 2022

fix(netxlite/errorsx): map "use of closed network connection" (ooni#510)

8679260

Part of ooni/probe#1733 and diff has been extracted from ooni#506.

ainghazal pushed a commit to ainghazal/probe-cli that referenced this pull request Mar 8, 2022

feat(errorsx): introduce NewTopLevelGenericErrWrapper (ooni#511)

88bcdb2

Part of ooni/probe#1733 and diff has been extracted from ooni#506.

ainghazal pushed a commit to ainghazal/probe-cli that referenced this pull request Mar 8, 2022

feat(netxlite): implement LookupHTTPS (ooni#514)

d7404d1

This new API call performs DNS lookups for HTTPS records. Part of ooni/probe#1733 and diff has been extracted from ooni#506.

ainghazal pushed a commit to ainghazal/probe-cli that referenced this pull request Mar 8, 2022

doc(netxlite): revamp the documentation (ooni#523)

86613be

Part of ooni#506. In parallel with tutorials, we also need to make sure we have good documentation.

ainghazal pushed a commit to ainghazal/probe-cli that referenced this pull request Mar 8, 2022

fix(resolver_test.go): remove race and make deterministic (ooni#524)

1ae0883

Should fix ooni#523 (comment). Work part of ooni#506.

ainghazal pushed a commit to ainghazal/probe-cli that referenced this pull request Mar 8, 2022

fix: add missing reference to netxlite's tutorial (ooni#535)

bcd3bc9

See ooni#506

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

doc: explain how to write ooni nettests #506

doc: explain how to write ooni nettests #506

bassosimone commented Sep 24, 2021 •

edited

Loading

bassosimone commented Sep 30, 2021

doc: explain how to write ooni nettests #506

doc: explain how to write ooni nettests #506

Conversation

bassosimone commented Sep 24, 2021 • edited Loading

Checklist

Description

How to review the tutorials

What remains to be done before merging

bassosimone commented Sep 30, 2021

bassosimone commented Sep 24, 2021 •

edited

Loading