From 46d75016dbe4fd47e33404c20d23dca5fad7420f Mon Sep 17 00:00:00 2001 From: Simone Basso Date: Mon, 6 May 2024 16:36:39 +0200 Subject: [PATCH 1/5] doc: document probe services interactions Closes https://github.com/ooni/probe/issues/2700 --- docs/design/dd-006-probeservices.md | 122 ++++++++++++++++++++++++++++ 1 file changed, 122 insertions(+) create mode 100644 docs/design/dd-006-probeservices.md diff --git a/docs/design/dd-006-probeservices.md b/docs/design/dd-006-probeservices.md new file mode 100644 index 0000000000..9f0a9052f2 --- /dev/null +++ b/docs/design/dd-006-probeservices.md @@ -0,0 +1,122 @@ +# Probe Services Interactions + +| | | +|:-------------|:-----------------------------------------------| +| Author | [@bassosimone](https://github.com/bassosimone) | +| Last-Updated | 2024-05-06 | +| Status | documentational | + +*Abstract.* We document the interaction with the probe services. + +## Introduction + +While running OONI Probe needs to communicate with several services, some +of which are managed by OONI, some of which by third parties This document is +about explaining the interaction with internal services. + +Services managed by OONI are divided into two distinct categories: + +1. "probe services", that is APIs that provide services to OONI probe +during its bootstrap, when fetcing inputs, or when submitting measurements; + +2. "test helpers", that is APIs invoked while measuring. + +This document is mostly concerned with the "probe services". Historically, +the probe services were implemented by separate hosts. However, during the +Summer 2019 team meeting in Stockholm, we decided to consolidate all of +them and serve them through a single entry point. Most recently, this entry +point has been implemented by the `api.ooni.io` host. + +## Software Architecture + +There is OONI Probe Mobile, OONI Probe Desktop, and OONI Probe CLI. Also, there +are two CLI clients, `ooniprobe` and `miniooni` (the research client). + +Mobile clients used the [pkg/oonimkall](../../pkg/oonimkall/) API. Desktop +clients invoke `ooniprobe`. Both `ooniprobe` and `miniooni` directly use the +[internal/engine](../../internal/engine/) API. Additionally, `minniooni` +implements the OONI Run v2 preview using [internal/oonirun](../../internal/oonirun). + +The following diagram summarizes what we said so far: + +``` + +-----------------+ +---------------+ +----------------------------+ + | Probe Mobile | | Probe Desktop | | miniooni | + +-----------------+ +---------------+ +----------------------------+ + | | | + V V | + +-----------------+ +---------------+ | +----------------+ + | oonimkall API | | ooniprobe | | | oonirun API | + +-----------------+ +---------------+ | +----------------+ + | | | | + V V V V + +------------------------------------------------------------------+ + | Engine API | + +------------------------------------------------------------------+ +``` + +In other words, the engine API mediates all interactions. (This is true not +only for communicating with the probe services, but in general.) + +## The Engine API + +The `*engine.Session` type represents a measurement session. This type +provides APIs to higher-level blocks in the software architecture, as +described above. In addition, it provides individual network experiments +with services and state. For example, it constructs an HTTP client with +possibly circumvention features, to communicate with probe services. + +From the engine's point of view, the software architecture is the following: + +``` + +------------------------------------------------------------------+ + | .../engine (Engine API) | + +-----------------+---------------------+--------------------------+ + | .../enginenetx | .../engineresolver | .../probeservices | + +-----------------+---------------------+--------------------------+ + | .../httpclientx | + +--------------------------+ +``` + +The `.../` ellipsis indicates packages inside [internal](../../internal/). + +Basically: + +1. the `engineresolver` package manages a composed DNS-over-HTTPS +resolver that fallsback to the system resolver; + +2. the `enginenetx` package manages dialing TLS connections +and is where we implement the "bridges" circumvention strategy; + +3. the `engine` API uses `engineresolver` and `enginenetx` +to create an HTTP client with extra robustness properties compared to +the one provided by the Go standard library; + +4. the `probeservices` package is where we use such an HTTP +client to communicate with the probe services (see the package's +design document for more information); + +5. in turn `probeservices` uses the `httpclientx` package implements +algorithms for communicating with the probe services (and other services), +including among them the possibility of trying a set of equivalent URLs +in ~parallel (refer to the package's design document for more information). + +In other words: + +1. `engineresolver` and `enginenetx` provide an HTTP client, that +is instantiated by the engine API; + +2. `httpclientx` provides algorithms to use such a client; + +3. `probeservices` uses `httpclientx` algorithms and the given client +to implement communication with the probe services. + +## Conclusion + +Probe services are a set of APIs used by OONI Probe through its +lifecycle. Every client communicates with these services through +the "engine" API. In turn, the "engine" API uses the support +"probeservices" package to communicate with the probe services. In +turn, "probeservices" uses algorithms defined by "httpclientx" as +well as an HTTP client created by the "engine" API using the +"engineresolver" and "enginenetx" packages. From a7b1071a099ffb4dc84e21fe0a8477e5ebabd54e Mon Sep 17 00:00:00 2001 From: Simone Basso Date: Mon, 6 May 2024 16:38:54 +0200 Subject: [PATCH 2/5] Apply suggestions from code review --- docs/design/dd-006-probeservices.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/design/dd-006-probeservices.md b/docs/design/dd-006-probeservices.md index 9f0a9052f2..5561b7b3e3 100644 --- a/docs/design/dd-006-probeservices.md +++ b/docs/design/dd-006-probeservices.md @@ -11,13 +11,13 @@ ## Introduction While running OONI Probe needs to communicate with several services, some -of which are managed by OONI, some of which by third parties This document is +of which are managed by OONI, some of which by third parties. This document is about explaining the interaction with internal services. Services managed by OONI are divided into two distinct categories: 1. "probe services", that is APIs that provide services to OONI probe -during its bootstrap, when fetcing inputs, or when submitting measurements; +during its bootstrap, when fetching inputs, or when submitting measurements; 2. "test helpers", that is APIs invoked while measuring. @@ -29,7 +29,7 @@ point has been implemented by the `api.ooni.io` host. ## Software Architecture -There is OONI Probe Mobile, OONI Probe Desktop, and OONI Probe CLI. Also, there +There are OONI Probe Mobile, OONI Probe Desktop, and OONI Probe CLI. Also, there are two CLI clients, `ooniprobe` and `miniooni` (the research client). Mobile clients used the [pkg/oonimkall](../../pkg/oonimkall/) API. Desktop From 027be8927fb781b50a62a7de3a4a6ed09a99f8a0 Mon Sep 17 00:00:00 2001 From: Simone Basso Date: Mon, 6 May 2024 16:43:30 +0200 Subject: [PATCH 3/5] x --- docs/design/dd-006-probeservices.md | 31 +++++++++++++++++++---------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/docs/design/dd-006-probeservices.md b/docs/design/dd-006-probeservices.md index 9f0a9052f2..62c0b3f780 100644 --- a/docs/design/dd-006-probeservices.md +++ b/docs/design/dd-006-probeservices.md @@ -64,18 +64,26 @@ The `*engine.Session` type represents a measurement session. This type provides APIs to higher-level blocks in the software architecture, as described above. In addition, it provides individual network experiments with services and state. For example, it constructs an HTTP client with -possibly circumvention features, to communicate with probe services. +possibly circumvention features, to communicate with test helpers. The +session also implements all the functionality to communicate with the probe +services (e.g., the functionality to invoke the check-in API). From the engine's point of view, the software architecture is the following: ``` +------------------------------------------------------------------+ | .../engine (Engine API) | - +-----------------+---------------------+--------------------------+ - | .../enginenetx | .../engineresolver | .../probeservices | - +-----------------+---------------------+--------------------------+ - | .../httpclientx | - +--------------------------+ + +------------------------------------------------------------------+ + | | | + V V V + +-----------------+ +---------------------+ +----------------------+ + | .../enginenetx | | .../engineresolver | | .../probeservices | + +-----------------+ +---------------------+ +----------------------+ + | + V + +----------------------+ + | .../httpclientx | + +----------------------+ ``` The `.../` ellipsis indicates packages inside [internal](../../internal/). @@ -83,18 +91,18 @@ The `.../` ellipsis indicates packages inside [internal](../../internal/). Basically: 1. the `engineresolver` package manages a composed DNS-over-HTTPS -resolver that fallsback to the system resolver; +resolver that falls back to the system resolver; 2. the `enginenetx` package manages dialing TLS connections -and is where we implement the "bridges" circumvention strategy; +and is where we implement the "bridges" circumvention strategy (see +the package's design document for more information); 3. the `engine` API uses `engineresolver` and `enginenetx` to create an HTTP client with extra robustness properties compared to the one provided by the Go standard library; 4. the `probeservices` package is where we use such an HTTP -client to communicate with the probe services (see the package's -design document for more information); +client to communicate with the probe services; 5. in turn `probeservices` uses the `httpclientx` package implements algorithms for communicating with the probe services (and other services), @@ -109,7 +117,8 @@ is instantiated by the engine API; 2. `httpclientx` provides algorithms to use such a client; 3. `probeservices` uses `httpclientx` algorithms and the given client -to implement communication with the probe services. +to implement communication with the probe services, but all the +OONI Probe's code uses the `probeservices` package through the engine API. ## Conclusion From 9868f050150ad7df6f7711f67f3a8b88ba6045c2 Mon Sep 17 00:00:00 2001 From: Simone Basso Date: Mon, 6 May 2024 16:45:05 +0200 Subject: [PATCH 4/5] x --- docs/design/dd-006-probeservices.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/design/dd-006-probeservices.md b/docs/design/dd-006-probeservices.md index 5edd0336c9..0ee08d7ac9 100644 --- a/docs/design/dd-006-probeservices.md +++ b/docs/design/dd-006-probeservices.md @@ -129,3 +129,7 @@ the "engine" API. In turn, the "engine" API uses the support turn, "probeservices" uses algorithms defined by "httpclientx" as well as an HTTP client created by the "engine" API using the "engineresolver" and "enginenetx" packages. + +Experiments also use the probe services (or other backend services +such as the test helpers). Also all these interactions are +always mediated through the "engine" API. From 93de404d2ab75613203726a6641d52c7454e6722 Mon Sep 17 00:00:00 2001 From: Simone Basso Date: Mon, 6 May 2024 16:45:43 +0200 Subject: [PATCH 5/5] x --- docs/design/dd-006-probeservices.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/docs/design/dd-006-probeservices.md b/docs/design/dd-006-probeservices.md index 0ee08d7ac9..5edd0336c9 100644 --- a/docs/design/dd-006-probeservices.md +++ b/docs/design/dd-006-probeservices.md @@ -129,7 +129,3 @@ the "engine" API. In turn, the "engine" API uses the support turn, "probeservices" uses algorithms defined by "httpclientx" as well as an HTTP client created by the "engine" API using the "engineresolver" and "enginenetx" packages. - -Experiments also use the probe services (or other backend services -such as the test helpers). Also all these interactions are -always mediated through the "engine" API.