OONI Probe CLI v3.22.0-alpha
Pre-releaseThis is an alpha release for creating a branching point for the release/3.22
branch. As usual, this release includes both an OONI Probe CLI release and the corresponding release of the measurement engine, which is hosted by the same repository.
Here's a thematic summary of what changed since v3.21.0-alpha:
- ✨ Preparing Web Connectivity LTE for use in production
- 🐛 Prioritize DNS results when communicating with backends
- 🚧 Refactor and document communication with the OONI backend
- 📖 Documenting the release process
- 🚧 Minor changes & bug fixes
In terms of testing requirements, this release mostly changed the interaction with backend services, so we need
-
QA testing of
ooniprobe
andminiooni
operating in normal and restricted operating conditions, to see how the new algorithms are faring, collect feedback, and plan for future changes (or revert patches if needed). -
We should also probably test whether we're able to trigger the Web Connectivity LTE A/B testing using the check-in API, so we're actually able to perform the A/B testing 😅.
-
We should also test the new
oohelperd
in production, for now the code has a generous 8 seconds timeout for receiving HTTP headers rather than leaving the timeout unbounded (we should be fine, but we should nonetheless test). -
We should ensure in the desktop app that we can reset the
OONI_HOME
multiple times without errors. -
We should test in the CLI that the "panics" added for handling
survey
errors have no practical implication unless something very weird like not being able to write the standard output happens.
More in detail, here's what has changed:
✨ Preparing Web Connectivity LTE for use in production
We worked towards closing all the open Web Connectivity LTE issues, such that we can finally be well placed to run an A/B testing between the v0.4 and LTE using this version of the OONI Probe Engine.
- fix(webconnectivitylte): handle too many redirects by @bassosimone in #1550
🐛 Prioritize DNS results when communicating with backends
We documented the strategy to communicate with backends, including the OONI backend, and we discussed the design as part of the #1552. During such a review, it was requested to modify the policy with which we communicate with the backends, so that the DNS results have higher priority. The main concern of this change request was to avoid long recovery periods if the currently deployed bridge becomes unavailable as well as to tilt the overall design such that uncensored OONI Probe users communicate with the backends in the simplest possible way.
- feat(enginenetx): test
bridgesPolicy
with DNS success by @bassosimone in #1554 - feat(enginenetx): add algorithms to filter, mix, and stream tactics by @bassosimone in #1556
- refactor(enginenetx): assign InitialDelay when dialing by @bassosimone in #1555
- feat(enginenetx): implement deterministic+random mixing by @bassosimone in #1558
- refactor(enginenetx): split generation and mixing by @bassosimone in #1591
- fix(enginenetx): give priority to DNS by @bassosimone in #1592
- cleanup(enginenetx): remove previous policies by @bassosimone in #1593
- fix(enginenetx): compute zero time when we start dialing by @bassosimone in #1594
- doc(enginenetx): add design document by @bassosimone in #1595
🚧 Refactor and document communication with the OONI backend
We wanted to document the interaction with the OONI backend and specifically with the "probe services" (i.e., the collection of APIs used by OONI Probe), per ooni/probe#2700. To this end, we seized the opportunity to simplify the code interactions, reduce the number of used packages, and refactor the code to improve its reliability and future usability.
- cleanup(engine): remove redundant APIs by @bassosimone in #1536
- cleanup(engine): do not expose LookupLocationContext by @bassosimone in #1537
- cleanup(oonimkall): use (*Session).CheckIn directly by @bassosimone in #1538
- cleanup: use (*Session).NewSubmitter() by @bassosimone in #1539
- cleanup: SaveMeasurement doesn't belong to Experiment by @bassosimone in #1541
- refactor: move Saver, Submitter, and InputProcessor to oonirun by @bassosimone in #1543
- feat(testingx): add log collector by @bassosimone in #1562
- feat(testingx): add HTTPHandlerResetWhileReadingBody by @bassosimone in #1563
- chore(enginelocate): improve ubuntu and cloudflare tests by @bassosimone in #1564
- doc(oonirun): document v2.go and v2_test.go by @bassosimone in #1565
- chore(probeservices): better tests for bouncer, check-in, measurement-meta by @bassosimone in #1566
- feat(testingx): model OONI backend register/login flows by @bassosimone in #1567
- chore(probeservices): better tests for psiphon and tor by @bassosimone in #1568
- feat: introduce the httpclientx package by @bassosimone in #1569
- refactor(enginelocate): switch to httpclientx by @bassosimone in #1570
- oonirun: use httpclientx by @bassosimone in #1571
- chore(probeservices): improve tests for /api/v1/login by @bassosimone in #1572
- chore(probeservices): improve tests for /api/v1/register by @bassosimone in #1573
- feat(testingx): add OONI collector for testing by @bassosimone in #1574
- chore(probeservices): improve tests for measurement submission by @bassosimone in #1575
- refactor(probeservices): httpx -> httpclientx by @bassosimone in #1576
- fix(httpclientx): add support for cloudfronting by @bassosimone in #1577
- cleanup(probeservices): remove residual httpx.Client dependency by @bassosimone in #1578
- cleanup: remove unused httpx package by @bassosimone in #1579
- refactor(probeservices): use httpclientx for check-in by @bassosimone in #1580
- feat(session): implement CallWebConnectivityTestHelper by @bassosimone in #1581
- refactor(webconnectivity*): don't call httpapi directly by @bassosimone in #1582
- cleanup: only define ErrNoAvailableTestHelpers once by @bassosimone in #1583
- refactor(httpclientx): couple URL and cloudfront host header by @bassosimone in #1584
- httpclientx: implement fast recovery and return endpoint index by @bassosimone in #1586
- multierror: implement BuildErrorString and mark as legacy by @bassosimone in #1587
- feat(httpclientx): allow configuring max-response-body size by @bassosimone in #1588
- refactor(webconnectivityqa): use httpclientx by @bassosimone in #1589
- cleanup: remove httpapi and ooapi by @bassosimone in #1560
- doc: document probe services interactions by @bassosimone in #1590
📖 Documenting the release process
We were asked to document the release process in ooni/probe#2698, and the resulting work is this:
- doc: start improving releases documentation by @bassosimone in #1533
- doc: document the release process by @bassosimone in #1534
🚧 Releasing & maintenance
This set of changes deal with preparing a release. In particular, in this case, we took on the chore of reducing the number of warnings emitted by gosec
, such that its output can actually be informative and actionable (there was lots of noise). In terms of behavioral changes in the ooniprobe
binary, these are the two most "impactful" changes:
-
I explicitly ignore errors when resetting with
-f
theOONI_HOME
directory, to keep the previous idempotent behavior, because I think that otherwise the desktop app may emit errors if asked to delete theOONI_HOME
multiple times: https://github.com/ooni/probe-cli/pull/1603/files#diff-2e841e5388e3f75b0ab808e74d5b34a9ebb72466a508daecda35625a7eab5c7fR29 -
I am now panicking if
survey
fails (which should not happen unless we misconfiguresurvey
in the code or we're not able to write to the standard output): https://github.com/ooni/probe-cli/pull/1603/files#diff-3e09d6a095d686f531f06efe21b6e430b311340af5ce74989f110dc05424cbcdR24 -
I am now reporting an error if we fail to update the database to set the upload status: https://github.com/ooni/probe-cli/pull/1603/files#diff-2a4cd822f78eb174dc7c6b88e3ab71be4810624b358b98109c10bcb5371e4844R249
-
I have now configured timeouts for receiving headers in
oohelperd
: https://github.com/ooni/probe-cli/pull/1603/files#diff-663f720a0015bc642e1957e965f9bc9115cea500fc0268bf5a6e22b5ad7cb44dR111
Here's the full list of pull requests:
- chore: we're now hacking on v3.22.0-alpha by @bassosimone in #1532
- chore(deps): bump golang.org/x/net from 0.22.0 to 0.23.0 by @dependabot in #1559
- fix(gobash.yml): drop macos-latest checks by @bassosimone in #1561
- chore: upgrade to go1.21.9 by @bassosimone in #1548
- chore(deps): bump github.com/quic-go/quic-go from 0.40.1 to 0.42.0 by @dependabot in #1535
- chore: update to go1.21.10 by @bassosimone in #1596
- chore(NDKVERSION): upgrade to latest LTS release by @bassosimone in #1597
- chore: upgrade most dependencies by @bassosimone in #1598
- chore: upgrade C dependencies by @bassosimone in #1599
- chore: upgrade assets and definitions by @bassosimone in #1600
- chore: minor changes before alpha release by @bassosimone in #1601
- chore: start addressing gosec warnings by @bassosimone in #1602
- chore: continue to address gosec warnings by @bassosimone in #1604
- chore: address additional gosec warnings by @bassosimone in #1605
- chore: finish addressing ~easy gosec warnings by @bassosimone in #1603
🚧 Minor changes & bug fixes
- feat: add health check response for oohelperd by @DecFox in #1540
- feat(miniooni): add --software-name and --software-version by @bassosimone in #1544
- fix obvious typo in quicping experiment by @ainghazal in #1542
- fix(oonimkall.HTTPRequest):
URL
=>Url
to fix iOS by @aanorbel in #1545 - fix(oohelperd): disable QUIC by default by @bassosimone in #1549
- fix(measurexlite): add robust RemoteAddr accessors by @bassosimone in #1551
Full Changelog: v3.21.0-alpha...v3.22.0-alpha