OONI Probe CLI v3.16.1

Overview

This patch release fixes a bug in web_connectivity@v0.5.

Please, see v3.16.0's release notes for more details on what changed since v3.15.x.

What's changed

• [backport] fix(webconnectivity@v0.5): DoH failure shouldn't set flags (#948): 68cfc70

Full Changelog: v3.16.0...v3.16.1

OONI Probe CLI v3.16.0

Highlights

• web connectivity's long-term evolution #882 #870: we have implemented an improved version of Web Connectivity that improves TLS and DNS censorship detection while retaining backward compatibility. In particular, we are now able to detect duplicate DNS over UDP responses when some middlebox replies to DNS queries but also allows replies from legitimate servers to reach users (which is what happens, for example, in China). This new version of Web Connectivity is only available to miniooni users, who can invoke it using miniooni web_connectivity@v0.5. We look forward to collect more data and perform A/B comparison with the current stable version. We aim to make this new version of Web Connectivity the default in the next release of ooniprobe. To help us collect more data, if you are a miniooni user, please run miniooni web_connectivity@v0.5 periodically, check the resulting JSON, and reach out to us in case you see unexpected results. Thank you!

• call getaddrinfo directly #764 #908: previous releases of ooniprobe and miniooni used the Go resolver to lookup domain names, but this strategy was problematic because it hid the real system resolver error. A particularly problematic case was Android (see ooni/probe#2029 for more info). To workaround this issue, we ensured that we always link with the C library for officially built binaries and we implemented saving the raw error from libc's getaddrinfo in such a case. This change makes the resulting measurement more transparent and enables third parties to more easily vet our results.

• oohelperd improvements #890 #886: this release includes improvements in the Web Connectivity test helper that allow Web Connectivity's long term evolution's code to perform better TLS blocking detection decisions. In addition to this change, the improved test helper also includes in its response additional data useful to determine whether IP addresses resolved by any version of Web Connectivity (including the stable version) are legitimate or invalid for the requested domain.

• build ooniprobe for armv6 #904: we're now building ooniprobe for armv6, as requested by the community.

• implement rolling builds #910 #920: we created a new (fake) release called rolling. Every night a build task builds the master branch and publishes binaries in such a release. Additionally, we have a way to manually force specific builds to publish packages to be tested in such a release. If you want to run very recent binaries with all the latest changes, you should download from this release. This set of changes is an initial response to community requests regarding facilitating using ooniprobe and miniooni for research purposes in unattended, censored environments.

• make miniooni CLI more user friendly #913 #932: we modified miniooni's CLI to make miniooni [experiment] [options] work exactly as miniooni [options] [experiment]. Previously, only the latter was supported. Additionally, we revamped the output emitted with --help to provide experiment-specific contextual information, including a list of all supported options. These changes should hopefully simplify using miniooni for research purposes. Additionally, now you can pass --emoji as a command line flag to get emoji output (which simplifies quickly eyeballing logs).

• build miniooni and ooniprobe for android #907: we're now building miniooni and ooniprobe for Android, such that they could be run from Termux's CLI, as requested by the community.

• oonirun v2 #842 #843 #844 #916: the miniooni included in this release includes a preview of OONI Run v2 links which allows anyone to execute arbitrary OONI Run v2 descriptors by using the miniooni oonirun -i URL command. Please, read the provisional design document for this OONI Run v2 preview for more information: https://github.com/ooni/probe-cli/blob/v3.16.0/docs/design/dd-004-minioonirunv2.md.

• use ooni/oocrypto instead of ooni/go #751 #872: anticipating future changes in which we will make our TLS fingerprint more similar to Chrome's own fingerprint, this release switches to our fork of Go's crypto/tls library.

• dnscheck: lower the default timeouts #917: we lowered the default timeout of dnscheck, following a community request.

• generally expose more low-level fields: we improved the data model to include extra information useful to treat each kind of event separately (see ooni/spec#261). This change is the initial step to implement a community request to provide OONI measurements in tabular data, where each table is a specific kind of event (e.g. DNS lookup, TLS handshake).

• step-by-step design document #814: we designed and implemented a new strategy for organizing experiment's code that simplify collecting better data and ultimately enables providing self-sufficient events in tabular format.

What's Changed in detail

• chore: we're now hacking on v3.16.0-alpha by @bassosimone in #749
• chore: upgrade to go1.18.2 by @bassosimone in #750
• refactor: use ooni/oocrypto instead of ooni/go by @bassosimone in #751
• fix(oonimkall): export CheckInConfig.RunType by @bassosimone in #752
• fix({simplequic,tls}ping): default SNI to URL's hostname by @bassosimone in #753
• refactor: only use shaping dialer for ndt7 and dash by @bassosimone in #754
• cleanup: mark more integration tests as !short mode by @bassosimone in #755
• cleanup(geolocate): use netxlite rather than netx by @bassosimone in #756
• cleanup: minor Readme.md tweaks and changes by @bassosimone in #757
• cleanup: merge httpheader and httpfailure into model by @bassosimone in #758
• cleanup: move legacy from internal/engine to internal by @bassosimone in #759
• Spring cleanup: remove unused/unneded code by @bassosimone in #761
• refactor: DNSTransport I/Os DNS messages by @bassosimone in #760
• feat(netxlite): observe additional DNS-over-UDP responses by @bassosimone in #762
• fix(dnsoverudp): allow to cancel async round trip immediately by @bassosimone in #763
• netxlite: call getaddrinfo and handle platform-specific oddities by @bassosimone in #764
• getaddrinfo: fix CGO_ENABLED=0 and record resolver type by @bassosimone in #765
• netxlite: do not call netgo the CGO_ENABLED=0 resolver by @bassosimone in #766
• refactor(session.go): replace engine/netx with netxlite by @bassosimone in #767
• refactor(ndt7): use netxlite rather than netx by @bassosimone in #768
• refactor: continue to simplify engine/netx by @bassosimone in #769
• refactor(netxlite): allow easy dialer chain customization by @bassosimone in #770
• refactor(netxlite): allow easy QUIC dialer chain customization by @bassosimone in #771
• refactor(netx): merge archival, trace, and the savers by @bassosimone in #772
• refactor(tracex): start applying recent code conventions by @bassosimone in #773
• refactor(netxlite): better integration with tracex by @bassosimone in #774
• refactor(netxlite): introduce the getaddrinfo transport by @bassosimone in #775
• feat(netxlite): implement DNSTransport wrapping by @bassosimone in #776
• refactor(tracex): do not depend on strings for event names by @bassosimone in #777
• tracex: prepare HTTP code for future refactoring by @bassosimone in #778
• fix: pin to gopkg.in/yaml.v3 v3.0.1 by @bassosimone in #779
• fix(tracex): generate archival from single transaction-done event by @bassosimone in #780
• refactor(tracex): convert to unit testing by @bassosimone in #781
• refactor: move tracex outside of engine/netx by @bassosimone in #782
• chore: upgrade oocrypto, oohttp, probe-assets by @bassosimone in #783
• hotfix: disable oocrypto until we investigate ciphers selection by @bassosimone in #784
• [forwardport] fix(dns...

OONI Probe CLI v3.16.0-alpha.3

What's Changed

• chore: run go generate ./... by @bassosimone in #929
• chore: upgrade to db-ip.com 2022-09 release by @bassosimone in #930
• chore: upgrade user-agent used for measuring by @bassosimone in #931
• feat(miniooni): optionally log using emojis by @bassosimone in #932
• feat(webconnectivity@v0.5): use TLS info from TH by @bassosimone in #933
• fix(measurexlite): expose TCP connect event by @bassosimone in #934
• fix(webconnectivity@v0.5): fetch HTTP only using system-resolver addrs by @bassosimone in #935

Full Changelog: v3.16.0-alpha.2...v3.16.0-alpha.3

OONI Probe CLI v3.16.0-alpha.2

What's Changed

• chore: we're now hacking on v3.16.0-alpha.2 by @bassosimone in #901
• fix: repair broken linux-static build by @bassosimone in #902
• fix(oohelperd): metrics improvements after design review by @bassosimone in #903
• feat: build ooniprobe for armv6 by @bassosimone in #904
• fix: repair the Android build by @bassosimone in #905
• fix(oohelperd): measurement -> wctask by @bassosimone in #906
• feat: build miniooni and ooniprobe for android by @bassosimone in #907
• feat: stop cross compiling miniooni by @bassosimone in #908
• fix: attempt to make android builds faster by @bassosimone in #909
• feat: implement rolling builds by @bassosimone in #910
• fix: attempt to make linux builds faster by @bassosimone in #911
• refactor(miniooni): divide et impera by @bassosimone in #912
• feat(miniooni): make CLI much more user friendly by @bassosimone in #913
• fix(miniooni): handle panics with --repeat-every by @bassosimone in #914
• feat(oonirun): improve tests by @bassosimone in #915
• doc: document the minioonirunv2 functionality by @bassosimone in #916
• fix(dnscheck): lower the default timeouts by @bassosimone in #917
• dnscheck: bump patch version number by @bassosimone in #919
• feat(rolling): build fresh binaries every night by @bassosimone in #920
• fix(dnscheck): record whether residual is enabled by @bassosimone in #922
• fix(dnscheck): trim the static input list by @bassosimone in #923
• cleanup: doh.powerdns.org is not working anymore by @bassosimone in #924
• fix(tracex): use HTTP transaction end time for t by @bassosimone in #925
• fix(probeservices): use api.ooni.io by @bassosimone in #926
• feat(ghpublish): choose (pre)release depending on tag by @bassosimone in #928

Full Changelog: v3.16.0-alpha.1...v3.16.0-alpha.2

Rolling release

This tag and release contains binaries we build from release branches and development branches. You can grab these binaries if you would like to use "recent" miniooni, ooniprobe, and oohelperd binaries. Binaries for macOS and Windows will not work out of the box because they're not signed, while binaries for Linux and Android should instead work when run from the CLI.

OONI Probe CLI v3.16.0-alpha.1

What's Changed

• chore: we're now hacking on v3.16.0-alpha.1 by @bassosimone in #899
• fix: repair releasing miniooni and ooniprobe-windows by @bassosimone in #900

Full Changelog: v3.16.0-alpha...v3.16.0-alpha.1

OONI Probe CLI v3.16.0-alpha

What's Changed

• chore: we're now hacking on v3.16.0-alpha by @bassosimone in #749
• chore: upgrade to go1.18.2 by @bassosimone in #750
• refactor: use ooni/oocrypto instead of ooni/go by @bassosimone in #751
• fix(oonimkall): export CheckInConfig.RunType by @bassosimone in #752
• fix({simplequic,tls}ping): default SNI to URL's hostname by @bassosimone in #753
• refactor: only use shaping dialer for ndt7 and dash by @bassosimone in #754
• cleanup: mark more integration tests as !short mode by @bassosimone in #755
• cleanup(geolocate): use netxlite rather than netx by @bassosimone in #756
• cleanup: minor Readme.md tweaks and changes by @bassosimone in #757
• cleanup: merge httpheader and httpfailure into model by @bassosimone in #758
• cleanup: move legacy from internal/engine to internal by @bassosimone in #759
• Spring cleanup: remove unused/unneded code by @bassosimone in #761
• refactor: DNSTransport I/Os DNS messages by @bassosimone in #760
• feat(netxlite): observe additional DNS-over-UDP responses by @bassosimone in #762
• fix(dnsoverudp): allow to cancel async round trip immediately by @bassosimone in #763
• netxlite: call getaddrinfo and handle platform-specific oddities by @bassosimone in #764
• getaddrinfo: fix CGO_ENABLED=0 and record resolver type by @bassosimone in #765
• netxlite: do not call netgo the CGO_ENABLED=0 resolver by @bassosimone in #766
• refactor(session.go): replace engine/netx with netxlite by @bassosimone in #767
• refactor(ndt7): use netxlite rather than netx by @bassosimone in #768
• refactor: continue to simplify engine/netx by @bassosimone in #769
• refactor(netxlite): allow easy dialer chain customization by @bassosimone in #770
• refactor(netxlite): allow easy QUIC dialer chain customization by @bassosimone in #771
• refactor(netx): merge archival, trace, and the savers by @bassosimone in #772
• refactor(tracex): start applying recent code conventions by @bassosimone in #773
• refactor(netxlite): better integration with tracex by @bassosimone in #774
• refactor(netxlite): introduce the getaddrinfo transport by @bassosimone in #775
• feat(netxlite): implement DNSTransport wrapping by @bassosimone in #776
• refactor(tracex): do not depend on strings for event names by @bassosimone in #777
• tracex: prepare HTTP code for future refactoring by @bassosimone in #778
• fix: pin to gopkg.in/yaml.v3 v3.0.1 by @bassosimone in #779
• fix(tracex): generate archival from single transaction-done event by @bassosimone in #780
• refactor(tracex): convert to unit testing by @bassosimone in #781
• refactor: move tracex outside of engine/netx by @bassosimone in #782
• chore: upgrade oocrypto, oohttp, probe-assets by @bassosimone in #783
• hotfix: disable oocrypto until we investigate ciphers selection by @bassosimone in #784
• [forwardport] fix(dnscheck): stop emitting progress by @bassosimone in #785
• refactor(tracex): internally represent errors as strings by @bassosimone in #786
• refactor(tracex): internally store just the raw certificate by @bassosimone in #787
• cleanup(netx): remove subpackages and unnecessary code by @bassosimone in #788
• cleanup(netx): another batch of small/simple cleanups by @bassosimone in #789
• refactor(netx): use netxlite to build TLSDialer by @bassosimone in #790
• cleanup(netx): remove redundant config options by @bassosimone in #791
• cleanup(jafar): do not depend on netx and urlgetter by @bassosimone in #792
• chore: add gosec step by @ainghazal in #793
• chore: improve testing and increase coverage by @bassosimone in #794
• fix(netx): ensure we create ~same HTTP3 and HTTP2 transports by @bassosimone in #795
• cleanup(netxlite): drop the DefaultDialer legacy name by @bassosimone in #796
• cleanup(netx): stop using most netxlite resolver internals by @bassosimone in #797
• refactor(netx): move construction logic outside package by @bassosimone in #798
• cleanup: move caching resolvers from netx to netxlite by @bassosimone in #799
• refactor(netx): reorganize by topic by @bassosimone in #800
• cleanup: netx does not use netxlite legacy names by @bassosimone in #801
• doc(netx): reference issue mentioning future improvements by @bassosimone in #802
• [forwardport] fix(dnscheck): remove apparently-broken static input by @bassosimone in #804
• [forwardport] fix(stunreachability): don't emit spurious progress events by @bassosimone in #803
• refactor(oohelper): use netxlite rather than netx by @bassosimone in #805
• refactor(oohelperd): use netxlite rather than netx by @bassosimone in #806
• refactor: start refactoring session resolver by @bassosimone in #807
• hotfix(sessionresolver): prevent data race inside http3 by @bassosimone in #809
• cleanup: use NewHTTPTransportWithResolver more often by @bassosimone in #808
• refactor(sessionresolver): minor changes in files and types naming by @bassosimone in #810
• refactor(sessionresolver): replace dnsclientmaker with a function by @bassosimone in #811
• refactor(netxlite): more abstract proxy-enabled dialer construction by @bassosimone in #812
• refactor(netxlite): expose useful HTTPTransport/DNSTransport factories by @bassosimone in #813
• doc: publish the step-by-step design document by @bassosimone in #814
• chore: use go1.18.3 by @bassosimone in #816
• chore: avoid duplicating the code of conduct by @bassosimone in #818
• doc(step-by-step): readability improvements by @ainghazal in #820
• fix(oonimkall): only set annotations on success by @bassosimone in #821
• fix(workflows): elevate GITHUB_TOKEN permissions when needed by @bassosimone in #822
• feat: tlsping and tcpping using step-by-step by @bassosimone in #815
• doc(step-by-step): further improvements on design doc by @ainghazal in #830
• fix(oohelperd): use throw-away HTTPClient, Dialer, Resolver by @bassosimone in #833
• refactor(oohelperd): flatten package hierarchy by @bassosimone in #834
• refactor(oohelperd): improve tests implementation by @bassosimone in #835
• refactor(engine): *http.Client -> model.HTTPClient by @bassosimone in #836
• refactor(engine): set options from any value by @bassosimone in #837
• measurexlite: fix flaky tls_test.go by @bassosimone in #839
• refactor(engine): more abstract Experiment{,Builder} by @bassosimone in #838
• cleanup(miniooni): remove --limit option by @bassosimone in #840
• fix(engine): repair broken integration test by @bassosimone in #841
• feat: start sketching out the oonirun package by @bassosimone in #842
• feat(oonirun): implement OONIRun v1 by @bassosimone in #843
• feat(oonirun): add support for OONIRun v2 links by @bassosimone in #844
• feat(miniooni): introduce the --repeat-every command line flag by @bassosimone in https://github.com/ooni/pr...

OONI Probe CLI v3.15.3

This release is exactly equal to v3.15.2 except for an oohelperd fix.

As such, we're not going to publish any mobile or debian package but for oohelperd.

What's Changed

• fix(oohelperd): use throw-away HTTPClient, Dialer, Resolver by @bassosimone in #832

Full Changelog: v3.15.2...v3.15.3

OONI Probe CLI v3.15.2

What's Changed

• [backport] fix(oonimkall): only set annotations on success 0728027
• [backport] fix(workflows): elevate GITHUB_TOKEN permissions when needed by @bassosimone in #823
• chore: update go dependencies by @bassosimone in #824
• chore: update db-ip.com's databases by @bassosimone in #825
• chore: pin to ooni/oohttp@v0.1.0 by @bassosimone in #826
• chore: use go1.17.11 by @bassosimone in #827
• chore(mk): update the Android NDK version we use by @bassosimone in #828
• chore(ios): use setup-xcode to select Xcode version by @bassosimone in #829

Full Changelog: v3.15.1...v3.15.2

OONI Probe CLI v3.15.1

Highlights

• ✨ We now run the experimental group for automatic/unattended ooniprobe runs
• ✨ We reintroduced the vanilla_tor experiment
• ✨ We added vanilla_tor to the experimental group
• ✨ Since vanilla_tor and torsf could run for a few minutes, now we only run them for automatic/unattended runs
• ✨ We now avoid leaving empty directories behind when we successfully submit measurements
• ✨ We added the new quicping experiment
• ✨ We added the new tcpping experiment
• ✨ We added the new tlsping experiment
• ✨ We added the new simplequicping experiment
• ✨ We added the new dnsping experiment
• ✨ We improved torsf to submit additional metrics related to the tor progress
• ✨ We added several more static inputs for the dnscheck experiment
• ✨ We now include the endpoint being used by TLS handshakes
• ✨ We significantly improved our DNS (and especially our DNS-over-UDP) implementation
• ✨ We significantly improved our code for detecting bogons
• 🐛 We fixed a bug where no experiment was submitting the backend / test helpers information
• 🐛 We fixed a bug where the last measurement in webconnectivity was sometimes not submitted on mobile
• 🚧 We now mark automatic/unattended runs as ooniprobe-cli-unattended
• 🚧 We retired the Summer 2021 implementation of the websteps experiment prototype
• 🚧 We removed three IP lookuppers (avast, ipconfig, ipinfo) which were blocked by sanctions in some countries

Changes since v3.15.0

• 🐛 fix(stunreachability): don't emit spurious progress events 4fb5f7d
• 🐛 fix(dnscheck): remove apparently-broken static input 261d1a4

Full Changelog: v3.15.0...v3.15.1