v0.13.2

Packages

• ios pod 'oonimkall', :podspec => 'https://dl.bintray.com/ooni/ios/oonimkall-2020.06.24-091116.podspec'

• android implementation 'org.ooni:oonimkall:2020.06.24-091612'

Improvements

• probeservices: use ps{1,2}.ooni.io

v0.13.1

Packages

• ios: pod 'oonimkall', :podspec => 'https://dl.bintray.com/ooni/ios/oonimkall-2020.06.19-105225.podspec'

• android: implementation 'org.ooni:oonimkall:2020.06.19-105356'

Updates

• Update ASN country database

v0.13.0

Packages

• ios: pod 'oonimkall', :podspec => 'https://dl.bintray.com/ooni/ios/oonimkall-2020.06.10-122307.podspec'

• android: implementation 'org.ooni:oonimkall:2020.06.10-122505'

Improvements

• android: publish aar at JCenter

• stun_reachability: new experiment to measure reachability of STUN servers

• session: we now create a unique subdirectory below config.TempDir. Such subdirectory is recursively unlinked from the file system when you call session.Close, thus avoiding wasting disk space.

• telegram: rewritten using the urlgetter experiment API. This means the implementation is significantly more compact and we reuse much more code.

• urlgetter: expose the failed_operation as a nullable string on an experimental basis

• urlgetter: add Multi handle to perform a bunch of urlgetter operations in parallel

• urlgetter: allow to override the default GET method

• urlgetter: add to TestKeys additional fields that help analysis. These fields are not exported to JSON.

• all: start using ps.ooni.io wherever possible

• probeservices: implement support for choosing the best https probe service instance (in terms of least time to fetch the test helper) and to fallback to cloudfronted otherwise

API Changes

• session: the config.TempDir field can now be empty. If empty we use the temporary dir used by default by the ioutil.TempDir function in the Go standard library, which may not work as intended on mobile.

• oonimkall: the temp_dir field can now be empty. If empty we use the temporary dir used by default by the ioutil.TempDir function in the Go standard library, which may not work as intended on mobile

• libminiooni: remove support for -P option, which conflicted with a OONI Legacy v2.x

• netx/errorx: moved here netx/internal/errwrapper

• netx/archival: expose NewFailedOperation to compute the failed operation from an error

• netx/errorx: map context.Canceled to interrupted error (also added to the spec)

• session: include ooniprobe-engine name and version into user agent

Updates

• Update psiphon and its dependencies, by tracking the staging-client branch

• Update ASN country database

Bug fixes

• ios: make sure we can use the framework in OONI for iOS

• experiment: make sure we can add assets version to annotations with 32 bit systems

• urlgetter: wrap errors at toplevel, so that we handle the case where the failure happens before hitting at our modified transports, dialers, resolvers, etc.

• urlgetter: make sure we include tcp_connect in the exported JSON measurement

• session: fix the case where we're using --proxy URL, which have been previously broken

v0.12.0

Improvements

• We now publish iOS frameworks at BinTray

• We now include the assets version as an annotation called assets_version

• We now choose a probe service and stick to it for the lifetime of the session, rather than using separate bouncer and collector (this change is possible because we migrated to the new probe services model where a single host provides to probes most OONI APIs)

• Implement netx/selfcensor package that allow you to trigger some forms of censorship and see how the code reacts. You can trigger self censorship using miniooni's --self-censor-spec cmdline flag.

• experiment/urlgetter: allow the programmer to choose a specific TLS version

• netx/archival: annotate all DNS resolutions with ASN and org name

• experiment/urlgetter: accept and forward all cookies

• Specify and use a new data format that describes measurements using a tunnel (e.g. psiphon)

• probeservices: add support for cloud fronted probe services

• session: by default use Cloudflare's DNS over HTTPS resolver and fallback to the system resolver if the former resolver does not work, thus working around simple system resolver censorship

• session: add support for saving logs of tor tunnel to a well know file inside the temporary directory

• session: add support for contacting probe-services using an already running tor instance

• netx/dialer/proxy.go: allow to specify extemporary proxy via a context

• netx/dialer/tls.go: rewrite TLS handshake so we don't have the confusing "usage of closed network connection" when there is a timeout during the TLS handshake

session.go: significantly change initialisation: now it's not possible to modify most aspects of the session once the session has been initialised. Specifically: AddAvailableHTTPBouncer and AddAvailableHTTPCollector have been removed; SetIncludeProbe{ASN,CC,IP} have been removed. You should now properly configure SessionConfig before creating a Session instance.

API Changes

• Packages bouncer and collector have been merged into package probeservices

• netx/httpclient: factor here common DNSClient code from experiment/urlgetter, which allows to create a top-level DNS client object, useful for example also to the Session

• geoiplookup/mmdblookup: rename LookupASN => ASN and LookupCC => CC

• libminiooni: fail if --bouncer-base-url or --collector-base-url are used and nudge users towards using the --probe-services-url flag instead

Updates

• Update all dependencies

• Update HTTP header used to perform measurements

• Update assets (we're now using a asn.mmdb generated by us)

Bug fixes

• dash, ndt7: include missing extensions dictionary

• ndt7: make sure we don't depend on any m-lab code, thus reducing dependencies

• libminiooni: make sure experiments with optional input run as intended when no input is provided

• libooniffi: make sure tests don't write outside of topdir

• model/model.go: now use 127.0.0.2 if the resolver IP is unknown, thus allowing bulk replacement of unknown probe IP and unknown resolver IP more easily, because they don't conflict

• netx/archival: fix very-annoying bug where DNS information was not written in case of error

v0.11.0

Improvements

• Introduce a urlgetter experiment: this is the low level building block for all other experiments and it allows one basically to fetch http/https/dnslookup/tlshandshake URLs. The dnslookup:// URL indicates that we only need to perform a DNS lookup. The tlshandshake:// URL indicates that we should perform a TLS handshake. This experiment features a rich set of options, including running over a Psiphon/tor tunnel.

• The DASH and NDT experiments now save the hostname and the M-Lab "site" of the server being used.

• The NDT experiment now uses the new M-Lab location services API that includes access tokens for more granularly prevent abuse of the M-Lab platform.

• We now set explicitly the User-Agent when running NDT. This helps with abuse detection.

• The debug output of NDT is now much more informative wrt HTTP headers used by WebSockets.

• We have rewritten Psiphon as a special case of the urlgetter experiment. We will do more of that for all supported tests in the coming release, thus reducing the code lines and easing maintenance.

• For Go code, we're now using certifi/gocertifi as the default root CA

• In addition to a Psiphon tunnel, we now support also a Tor tunnel. To create a Tor tunnel, you need to specify the -OTunnel=tor option. The --tor-binary and --tor-args arguments can be used to force a specific tor binary and to provide options to tor. Otherwise, we'll attempt to execute tor in $PATH. In this vein, the SessionConfig struct contains the TorArgs and TorBinary fields.

• We can now optionally skip TLS verification when doing TLS in netx. This allows us to perform measurements that will fetch the block page served under HTTPS.

• We can now choose whether to save read/write info in netx. This allows us to more precisely choose when we want and where we don't want to save this piece of data.

• We can now optionally pre-populate a DNS cache in netx. This allows us to run the DNS query before performing an HTTP measurement, because we can prepopulate the cache.

• The DNS cache in netx now can also be read-only, when we don't want to cache new resolutions.

• We export the miniooni CLI's main as the libminiooni package. This allows us to very easily embed miniooni in other contexts, e.g., https://github.com/bassosimone/aladdin.

API changes

• ExperimentBuilder replaces the NeedsInput method with InputPolicy because now we have a more nuanced definition of whether an experiment needs input. The Psiphon experiment is the first experiment for which input is optional: if given input, it is processed, otherwise a default is used

• We don't depend anymore on M-Lab code for the NDT experiment. This prevents cases where inadvertently M-Lab people breaks our Windows builds by committing Unix only code. It also reduces dramatically the number of dependencies, and hence the time required to build probe-engine on a new system.

• We moved experiment/httpheader to internal/httpheader

• We changed HTTP data saved by new netx code to more easily generate data according the OONI data format.

Updates

• We updated dependencies and vendored assets

Bug fixes

• Traffic shaping when running integration tests to avoid hammering servers

• We now always correctly return an IP address when the input is an IP address in all the resolves that we construct inside of new netx code.

• We now scrub IP addresses from OONI failures (only for new netx code)

• The netx bogon filter now is such that we know about the bogons that triggered the error

• In oonimkall now the code will not miss specific events when the app was taking too much time to process an event (this was mostly an issue when running the Android simulator)

• In oonimkall now we honour the privacy settings

• We don't identify as OONI when doing IP lookups

v0.10.0

Improvements

• Automatically publish miniooni builds as artefacts at GitHub for every commit

• Allow to use ooni/probe-engine from C programs to a shared library

• More strictly honour data_format_version equal to 0.2.0

• Introduce the concept of data format "extensions"

• Rewrite the Go implementation of the DASH experiment

• Remove the C++ implementation of the DASH experiment: now we use Go

• Make ndt7 and DASH work consistently with miniooni --proxy

• Teach ndt7 and DASH to start a Psiphon tunnel and run using it as transport

• Make sure ndt7 data format is actionable when analysing data

• Start refactoring and simplifying netx (e.g. avoid using the context, use the decorator pattern instead)

• Introduce an optional "caching" resolver

API changes

• Remove the ndt5 experiment

• Remove the non-proxied HTTP server: the situation where some connections were proxied and some were not was very confusing. Now either all is proxied using --proxy or only the measurement is proxied by starting up a specific tunnel (currently only Psiphon is supported)

Updates

• Update direct and indirect dependencies

Bug fixes

• Make sure unit tests don't write outside the sources root directory

v0.9.0

Improvements

• Automatically publish mobile packages to bintray using the mobile-staging branch

• Count bytes consumed by experiment and session

• Make sure that we can interrupt all Go based experiments

• Rewrite ndt7 and mlab-ns to be our own implementation independent of the official client

• Heavy refactoring of netx for simplicity (see #359)

API changes

• Declare data format version 0.2.0 as part of a better strategy to handle data format versioning (see #423)

• Now "ndt" is "ndt7" by default but "ndt5" is still available in this release

• Deprecate ExperimentCallbacks.OnDataUsage callback

• Heavy refactoring of netx when now more building blocks are public (see see #359)

• oonimkall: handle all MK v0.9.0 variables (specifically reject unsupported ones)

Updates

• Update country MMDB

• Update direct and indirect dependencies

Bug fixes

• Prevent crashes with unaligned atomic access using new package atomicx/atomicx.go (see #414)

v0.8.0

This release features significant API changes.

Now you MUST close a session, e.g.:

sess, err := engine.NewSession(config)
if err != nil {
    log.Fatal(err)
}
defer sess.Close()

We should have done this before. We are doing this now, because we are now aware of this issue.

The second API change is that now an experiment implementation is represented by an interface, model.ExperimentMeasurer, which describes an experiment to the engine.

type ExperimentMeasurer interface {
    // ExperimentName returns the experiment name.
    ExperimentName() string

    // ExperimentVersion returns the experiment version.
    ExperimentVersion() string

    // Run runs the experiment with the specified context, session,
    // measurement, and experiment calbacks.
    Run(
        ctx context.Context, sess ExperimentSession,
        measurement *Measurement, callbacks ExperimentCallbacks,
    ) error
}

The third change is that the SessionConfig.TLSConfig field has been removed. You now have now way to specify a CA bundle different from the one which we chose as default.

The fourth change is that ExperimentBuilder.Build has ben renamed ExperimentBuilder.NewExperiment.

The fifth change is that log.Logger is now model.Logger.

The sixth change is that handlers.Callbacks is now model.ExperimentCallbacks.

The seventh change is that github.com/ooni/netx is now github.com/ooni/probe-engine/netx along with the following changes into the API:

1. the netx/httpx package has been merged with the netx package and all the names that were in httpx now contain the HTTP string inside them, e.g. httpx.Client => netx.HTTPClient

2. all the public functions and methods that used to take an handler as argument now don't take it anymore, and we removed the alternative methods which did not take an handler and had the NoHandler suffix

Finally, this release requires Go 1.14 and improves support for Android.

Read on for the usual classification of the PRs that make this release.

Improvements

• oonimkall: be liberal in the settings we accept (#364)

• .github/workflows: add tests using MK (#369)

• Require and use measurement-kit v0.10.11 (#374)

• Replace internal/httpx code with netx (#383)

• sniblocking: revamp classification mechanism (#397)

API changes

• Move httpx code into the internal package (#378)

• internal: remove the retrying dialer (#379)

• Vendor ooni/netx@4f8d645 and use it (#380)

• netx: unify netx and httpx packages (#381)

• netx: merge public API with internal (#382)

• remove session/session.go and experiment/experiment.go (#385)

• session.go: ensure we close idle connections (#386)

Updates

• internal/resources: update assets to 20200225143707 (#373)

• Update all dependencies and use Go 1.14 (#395)

Bug fixes

• iplookup: remove akamai IP lookup method (#368)

• mkcgo: fail if we don't have the right MK version (#370)

• orchestra: register desktop clients with AS0 and ZZ (#375)

• oonimkall: never omit mandatory event fields (#377)

• testlists.go: use naming consistent with other files (#387)

• sniblocking: use example.com as test helper (#389)

• sniblocking: cache recent results (#390)

• sniblocking: classify failures to produce a result (#391)

• tor: use targets-list-length dependent timeout (#394)

• README.md: document how to build miniooni (#398)

v0.7.0

Improvements

• Make MKAsyncTask like API available to Android developers as an AAR

• Switch to GitHub actions for CI

• Allow compiling probe-engine on ordinary linux systems where MK and deps are already installed

• Allow compiling with CGO_ENABLED and without MK

API changes

• Move the resources API to the internal package

Updates

• Updated all dependencies

Bug fixes

• resources: create directory where to download resources if not existent

v0.6.0

Improvements

• Better procedure for pinning Psiphon dependencies

• Add initial implementation of the SNI blocking experiment

• Implement summary for tor experiment

• Add support for or_port_dirauth in tor experiment

• Save the target_name in tor experiment

API changes

• Move jsonapi to the internal package to which it should belong

• Bump data format version to 0.4.0 (note that with this data format we declare being 0.2.0 for backward compatibility with the batch pipeline and we report the real version using annotations)

Updates

• Updated all dependencies

Bug fixes

• miniooni: save results/submit report even if experiment fails

• resources: download using https:// URL