Feature/web connectivity #9

hellais · 2018-03-23T11:41:39Z

No description provided.

willscott · 2018-03-23T11:52:04Z

nettests/websites/web_connectivity.go

+		orchestrateBaseURL,
+		ctl.Ctx.Location.CountryCode)
+
+	resp, err := http.Get(reqURL)


should we be able to request this through tor / domain fronting? should we have a central request manager interface to deal with that?

Yes for sure. This is just quick and dirty prototype code.

I should also add, that this functionality should actually be part of measurement-kit (that of fetching URL lists from the orchestration backend), however I implemented it inside of gooni for the time being to get the whole prototype working faster.

willscott · 2018-03-23T11:52:44Z

nettests/websites/web_connectivity.go

+
+	body, err := ioutil.ReadAll(resp.Body)
+	if err != nil {
+		return urls, errors.Wrap(err, "failed to read response body")


do we want to have a way to retry built in on failures to interact with the orchestrator, which may be transient?

willscott · 2018-03-23T11:53:10Z

nettests/websites/web_connectivity.go

+		parsed = new(URLResponse)
+		urls   []string
+	)
+	reqURL := fmt.Sprintf("%s/api/v1/urls?probe_cc=%s",


add a Todo to include other config constraints here, like which categories the user is okay with,
I guess also - in the orchestrator url request, does the client include any form of ID of itself to allow linking of the different requests its making to proteus?

Yeah for sure. The nettest class is designed to have knowledge of the config options via ctl.Ctx.Config.Foo, but currently config support is not complete. As soon as we have the chosen categories as part of the config, it will be possible to pass those to the orchestrator.

willscott · 2018-03-23T12:15:44Z

My comments were mostly to suggest adding some signpost TODO's in the areas that are going to need additional work so that in the future it's clear where to hook in the additional complexity that's missing from this first pass. This looks good though 👍

To generate the result summary we need to refactor how the SummaryMap works

Pass the geoip databases to measurement-kit

willscott · 2018-03-23T14:51:33Z

utils/geoip.go

+}
+
+// GeoIPDir returns the geoip data dir for the given OONI Home
+func GeoIPDir(home string) string {


Somethings weird about having these path things split up so that geoid is in util, the db folder is in database/database.go,
and the creation is in /ooni.go.

Consider a refactor to make this a bit easier to track down, potentially?

Yeah maybe all the path stuff should go into a path package. It can't live in ooni or database, because of circular dependencies.

willscott · 2018-03-23T14:52:42Z

utils/geoip.go


-		f, err := os.Open(out.Name())
+// DownloadGeoIPDatabaseFiles into the target directory


This may be an issue on devices with limited memory. The in-process tgz will do worse at being able to do this - in that it isn't streaming the file in the way that a native tar process would. that said, this is nice from a compatibility perspective.

These files are less than 10MB, so I don't think it's going to be an issue for this particular use case (I think OONI Probe already uses more than that memory in general).

@willscott

As per suggestion by @willscott in: #9 (comment)

Feature/web connectivity

@willscott

As per suggestion by @willscott in: ooni#9 (comment)

## Checklist - [x] I have read the [contribution guidelines](https://github.com/ooni/probe-cli/blob/master/CONTRIBUTING.md) - [x] reference issue for this pull request: ooni/probe#2650 - [x] if you changed anything related to how experiments work and you need to reflect these changes in the ooni/spec repository, please link to the related ooni/spec pull request: N/A - [x] if you changed code inside an experiment, make sure you bump its version number ## Description Closes ooni/probe#2650 Here's an example of the output now: ``` % ./miniooni dnsping -i udp://8.8.8.8:53/ -O Domains='www.example.com thepiratebay.net' [ 0.000028] <info> Current time: 2024-01-12 14:48:25 UTC [ 0.000125] <info> miniooni home directory: $HOME/.miniooni [ 0.000319] <info> ooniprobe-engine/v3.21.0-alpha 9b035a4 dirty=false go1.20.12 [ 0.001050] <info> Looking up OONI backends; please be patient... [ 0.001640] <info> httpsDialer: [#1] TCPConnect 162.55.247.208:443... started [ 0.047764] <info> httpsDialer: [#1] TCPConnect 162.55.247.208:443... ok [ 0.047788] <info> httpsDialer: [#1] TLSHandshake with 162.55.247.208:443 SNI=interactive.guim.co.uk ALPN=[h2 http/1.1]... started [ 0.100400] <info> httpsDialer: [#1] TLSHandshake with 162.55.247.208:443 SNI=interactive.guim.co.uk ALPN=[h2 http/1.1]... ok [ 0.100417] <info> httpsDialer: [#1] TLSVerifyCertificateChain api.ooni.io... started [ 0.102877] <info> httpsDialer: [#1] TLSVerifyCertificateChain api.ooni.io... ok [ 0.169226] <info> session: using probe services: {Address:https://api.ooni.io Type:https Front:} [ 0.169246] <info> Looking up your location; please be patient... [ 0.169272] <info> iplookup: using stun_google [ 0.170163] <info> sessionresolver: lookup stun.l.google.com using system:///... started [ 0.172175] <info> sessionresolver: lookup stun.l.google.com using system:///... ok [ 0.190530] <info> - country: IT [ 0.190545] <info> - network: Vodafone Italia S.p.A. (AS30722) [ 0.190549] <info> - resolver's IP: 91.80.36.88 [ 0.190552] <info> - resolver's network: Vodafone Italia S.p.A. (AS30722) [ 0.190587] <info> [1/1] running with input: udp://8.8.8.8:53/ [ 0.190728] <info> DNSPing #0 8.8.8.8:53 thepiratebay.net... started [ 0.190729] <info> DNSPing #0 8.8.8.8:53 www.example.com... started [ 0.201393] <info> DNSPing #0 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 0.212578] <info> DNSPing #0 8.8.8.8:53 thepiratebay.net... 83.224.65.41 [ 1.191531] <info> DNSPing #1 8.8.8.8:53 thepiratebay.net... started [ 1.191527] <info> DNSPing #1 8.8.8.8:53 www.example.com... started [ 1.199034] <info> DNSPing #1 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 1.208817] <info> DNSPing #1 8.8.8.8:53 thepiratebay.net... 83.224.65.41 [ 2.191812] <info> DNSPing #2 8.8.8.8:53 thepiratebay.net... started [ 2.191815] <info> DNSPing #2 8.8.8.8:53 www.example.com... started [ 2.228774] <info> DNSPing #2 8.8.8.8:53 thepiratebay.net... 83.224.65.41 [ 2.228793] <info> DNSPing #2 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 3.191340] <info> DNSPing #3 8.8.8.8:53 thepiratebay.net... started [ 3.191370] <info> DNSPing #3 8.8.8.8:53 www.example.com... started [ 3.197974] <info> DNSPing #3 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 3.211675] <info> DNSPing #3 8.8.8.8:53 thepiratebay.net... 83.224.65.41 [ 4.191878] <info> DNSPing #4 8.8.8.8:53 www.example.com... started [ 4.191889] <info> DNSPing #4 8.8.8.8:53 thepiratebay.net... started [ 4.203056] <info> DNSPing #4 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 4.211179] <info> DNSPing #4 8.8.8.8:53 thepiratebay.net... 83.224.65.41 [ 5.191830] <info> DNSPing #5 8.8.8.8:53 www.example.com... started [ 5.191830] <info> DNSPing #5 8.8.8.8:53 thepiratebay.net... started [ 5.233596] <info> DNSPing #5 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 5.261641] <info> DNSPing #5 8.8.8.8:53 thepiratebay.net... 83.224.65.41 [ 6.191596] <info> DNSPing #6 8.8.8.8:53 thepiratebay.net... started [ 6.191573] <info> DNSPing #6 8.8.8.8:53 www.example.com... started [ 6.200647] <info> DNSPing #6 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 6.208036] <info> DNSPing #6 8.8.8.8:53 thepiratebay.net... 83.224.65.41 [ 7.191875] <info> DNSPing #7 8.8.8.8:53 thepiratebay.net... started [ 7.191879] <info> DNSPing #7 8.8.8.8:53 www.example.com... started [ 7.200336] <info> DNSPing #7 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 7.213787] <info> DNSPing #7 8.8.8.8:53 thepiratebay.net... 83.224.65.41 [ 8.191866] <info> DNSPing #8 8.8.8.8:53 thepiratebay.net... started [ 8.191900] <info> DNSPing #8 8.8.8.8:53 www.example.com... started [ 8.204297] <info> DNSPing #8 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 8.214435] <info> DNSPing #8 8.8.8.8:53 thepiratebay.net... 83.224.65.41 [ 9.191857] <info> DNSPing #9 8.8.8.8:53 www.example.com... started [ 9.191863] <info> DNSPing #9 8.8.8.8:53 thepiratebay.net... started [ 9.200628] <info> DNSPing #9 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 9.216799] <info> DNSPing #9 8.8.8.8:53 thepiratebay.net... 83.224.65.41 Domain IPAddress ASN Org #Seen LateResponse -------------------------------------------------------------------------------------------------------------------------------------------------- www.example.com 2606:2800:220:1:248:1893:25c8:1946 15133 Edgecast Inc. 10 false www.example.com 93.184.216.34 15133 Edgecast Inc. 10 false thepiratebay.net 83.224.65.41 30722 Vodafone Italia S.p.A. 10 false [ 10.193633] <info> submitting measurement to OONI collector; please be patient... [ 10.262480] <info> New reportID: 20240112T144835Z_dnsping_IT_30722_n1_qbyxPwKoifjgp6AM [ 10.372279] <info> saving measurement to disk [ 10.373693] <info> experiment: recv 0.00 byte, sent 0.00 byte [ 10.375159] <info> whole session: recv 2.28 kbyte, sent 19.24 kbyte ```

## Checklist - [x] I have read the [contribution guidelines](https://github.com/ooni/probe-cli/blob/master/CONTRIBUTING.md) - [x] reference issue for this pull request: ooni/probe#2650 - [x] if you changed anything related to how experiments work and you need to reflect these changes in the ooni/spec repository, please link to the related ooni/spec pull request: N/A - [x] if you changed code inside an experiment, make sure you bump its version number ## Description Closes ooni/probe#2650 Here's an example of the output now: ``` % ./miniooni dnsping -i udp://8.8.8.8:53/ -O Domains='www.example.com thepiratebay.net' [ 0.000028] <info> Current time: 2024-01-12 14:48:25 UTC [ 0.000125] <info> miniooni home directory: $HOME/.miniooni [ 0.000319] <info> ooniprobe-engine/v3.21.0-alpha 9b035a4 dirty=false go1.20.12 [ 0.001050] <info> Looking up OONI backends; please be patient... [ 0.001640] <info> httpsDialer: [#1] TCPConnect 162.55.247.208:443... started [ 0.047764] <info> httpsDialer: [#1] TCPConnect 162.55.247.208:443... ok [ 0.047788] <info> httpsDialer: [#1] TLSHandshake with 162.55.247.208:443 SNI=interactive.guim.co.uk ALPN=[h2 http/1.1]... started [ 0.100400] <info> httpsDialer: [#1] TLSHandshake with 162.55.247.208:443 SNI=interactive.guim.co.uk ALPN=[h2 http/1.1]... ok [ 0.100417] <info> httpsDialer: [#1] TLSVerifyCertificateChain api.ooni.io... started [ 0.102877] <info> httpsDialer: [#1] TLSVerifyCertificateChain api.ooni.io... ok [ 0.169226] <info> session: using probe services: {Address:https://api.ooni.io Type:https Front:} [ 0.169246] <info> Looking up your location; please be patient... [ 0.169272] <info> iplookup: using stun_google [ 0.170163] <info> sessionresolver: lookup stun.l.google.com using system:///... started [ 0.172175] <info> sessionresolver: lookup stun.l.google.com using system:///... ok [ 0.190530] <info> - country: IT [ 0.190545] <info> - network: Vodafone Italia S.p.A. (AS30722) [ 0.190549] <info> - resolver's IP: 91.80.36.88 [ 0.190552] <info> - resolver's network: Vodafone Italia S.p.A. (AS30722) [ 0.190587] <info> [1/1] running with input: udp://8.8.8.8:53/ [ 0.190728] <info> DNSPing #0 8.8.8.8:53 thepiratebay.net... started [ 0.190729] <info> DNSPing #0 8.8.8.8:53 www.example.com... started [ 0.201393] <info> DNSPing #0 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 0.212578] <info> DNSPing #0 8.8.8.8:53 thepiratebay.net... 83.224.65.41 [ 1.191531] <info> DNSPing #1 8.8.8.8:53 thepiratebay.net... started [ 1.191527] <info> DNSPing #1 8.8.8.8:53 www.example.com... started [ 1.199034] <info> DNSPing #1 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 1.208817] <info> DNSPing #1 8.8.8.8:53 thepiratebay.net... 83.224.65.41 [ 2.191812] <info> DNSPing #2 8.8.8.8:53 thepiratebay.net... started [ 2.191815] <info> DNSPing #2 8.8.8.8:53 www.example.com... started [ 2.228774] <info> DNSPing #2 8.8.8.8:53 thepiratebay.net... 83.224.65.41 [ 2.228793] <info> DNSPing #2 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 3.191340] <info> DNSPing #3 8.8.8.8:53 thepiratebay.net... started [ 3.191370] <info> DNSPing #3 8.8.8.8:53 www.example.com... started [ 3.197974] <info> DNSPing #3 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 3.211675] <info> DNSPing #3 8.8.8.8:53 thepiratebay.net... 83.224.65.41 [ 4.191878] <info> DNSPing #4 8.8.8.8:53 www.example.com... started [ 4.191889] <info> DNSPing #4 8.8.8.8:53 thepiratebay.net... started [ 4.203056] <info> DNSPing #4 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 4.211179] <info> DNSPing #4 8.8.8.8:53 thepiratebay.net... 83.224.65.41 [ 5.191830] <info> DNSPing ooni#5 8.8.8.8:53 www.example.com... started [ 5.191830] <info> DNSPing ooni#5 8.8.8.8:53 thepiratebay.net... started [ 5.233596] <info> DNSPing ooni#5 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 5.261641] <info> DNSPing ooni#5 8.8.8.8:53 thepiratebay.net... 83.224.65.41 [ 6.191596] <info> DNSPing #6 8.8.8.8:53 thepiratebay.net... started [ 6.191573] <info> DNSPing #6 8.8.8.8:53 www.example.com... started [ 6.200647] <info> DNSPing #6 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 6.208036] <info> DNSPing #6 8.8.8.8:53 thepiratebay.net... 83.224.65.41 [ 7.191875] <info> DNSPing ooni#7 8.8.8.8:53 thepiratebay.net... started [ 7.191879] <info> DNSPing ooni#7 8.8.8.8:53 www.example.com... started [ 7.200336] <info> DNSPing ooni#7 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 7.213787] <info> DNSPing ooni#7 8.8.8.8:53 thepiratebay.net... 83.224.65.41 [ 8.191866] <info> DNSPing ooni#8 8.8.8.8:53 thepiratebay.net... started [ 8.191900] <info> DNSPing ooni#8 8.8.8.8:53 www.example.com... started [ 8.204297] <info> DNSPing ooni#8 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 8.214435] <info> DNSPing ooni#8 8.8.8.8:53 thepiratebay.net... 83.224.65.41 [ 9.191857] <info> DNSPing ooni#9 8.8.8.8:53 www.example.com... started [ 9.191863] <info> DNSPing ooni#9 8.8.8.8:53 thepiratebay.net... started [ 9.200628] <info> DNSPing ooni#9 8.8.8.8:53 www.example.com... 93.184.216.34 2606:2800:220:1:248:1893:25c8:1946 [ 9.216799] <info> DNSPing ooni#9 8.8.8.8:53 thepiratebay.net... 83.224.65.41 Domain IPAddress ASN Org #Seen LateResponse -------------------------------------------------------------------------------------------------------------------------------------------------- www.example.com 2606:2800:220:1:248:1893:25c8:1946 15133 Edgecast Inc. 10 false www.example.com 93.184.216.34 15133 Edgecast Inc. 10 false thepiratebay.net 83.224.65.41 30722 Vodafone Italia S.p.A. 10 false [ 10.193633] <info> submitting measurement to OONI collector; please be patient... [ 10.262480] <info> New reportID: 20240112T144835Z_dnsping_IT_30722_n1_qbyxPwKoifjgp6AM [ 10.372279] <info> saving measurement to disk [ 10.373693] <info> experiment: recv 0.00 byte, sent 0.00 byte [ 10.375159] <info> whole session: recv 2.28 kbyte, sent 19.24 kbyte ```

hellais added 5 commits March 22, 2018 15:58

Refactor directory structure

75d639d

Implement geoip related functionality in gooni

6ba779b

Refactor how we create the context

7fe1551

Add geoip command line tool

4bbffa9

Get web_connectivity to work with dynamic URL fetching

00859e8

willscott reviewed Mar 23, 2018

View reviewed changes

hellais added 5 commits March 23, 2018 13:17

Add support for generating the Measurement summary

f5ff3ac

To generate the result summary we need to refactor how the SummaryMap works

Add support for generating the web connectivity summary

98e95b8

Add result summary support for IM tests

e56ce73

Add support for downloading legacy geoip databases

749cc66

Pass the geoip databases to measurement-kit

On first run also download the geoip data files

9fe917f

willscott approved these changes Mar 23, 2018

View reviewed changes

hellais merged commit 79c9400 into master Mar 27, 2018

hellais deleted the feature/web_connectivity branch March 27, 2018 09:53

hellais added a commit that referenced this pull request Mar 27, 2018

Move all directory related functionality into paths utils

cffb9ea

As per suggestion by @willscott in: #9 (comment)

hellais mentioned this pull request Mar 27, 2018

Move all directory related functionality into paths utils #10

Merged

ainghazal pushed a commit to ainghazal/probe-cli that referenced this pull request Mar 8, 2022

Merge pull request ooni#9 from OpenObservatory/feature/web_connectivity

5b99727

Feature/web connectivity

ainghazal pushed a commit to ainghazal/probe-cli that referenced this pull request Mar 8, 2022

Move all directory related functionality into paths utils

0be6310

As per suggestion by @willscott in: ooni#9 (comment)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/web connectivity #9

Feature/web connectivity #9

hellais commented Mar 23, 2018

willscott Mar 23, 2018

hellais Mar 23, 2018

hellais Mar 23, 2018

willscott Mar 23, 2018

willscott Mar 23, 2018

hellais Mar 23, 2018 •

edited

Loading

willscott commented Mar 23, 2018

willscott Mar 23, 2018

hellais Mar 27, 2018

willscott Mar 23, 2018

hellais Mar 27, 2018


		f, err := os.Open(out.Name())
		// DownloadGeoIPDatabaseFiles into the target directory

Feature/web connectivity #9

Feature/web connectivity #9

Conversation

hellais commented Mar 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hellais Mar 23, 2018 • edited Loading

Choose a reason for hiding this comment

willscott commented Mar 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hellais Mar 23, 2018 •

edited

Loading