Refine the observation generation through internal testing

[hellais]

The goal of this issue is to collect initial feedback internally on how the observation generation and analysis is currently working.

There is a test instance of this running here:

• Observations: https://data.ooni.org/observations/
• Analysis: https://data.ooni.org/analysis/

For example if you want to see the observations and analysis generated for the recent nepal report measurements, you can do:

• https://data.ooni.org/analysis/m/20231129041111.939896_NP_webconnectivity_32105af63774eabf
• https://data.ooni.org/observations/m/20231129041111.939896_NP_webconnectivity_32105af63774eabf

This issue is a place to collect all relevant bugs that are spotted and identify any issues we might find. The primary focus is on observation generation, but it's also useful to start cataloging any specific cases where the analysis is grossly off.

One important caveat (which mostly applies to analysis) is that the processing is done direct in real-time when you access the page. This means that if a ground truth database does not exist, the analysis might be a bit off compared to how it will look like in a real production scenario.

[sitinurliza95]

Checking through some measurements that I found with issues so far:

• Confirmed blocking that is false positive - blocked 0.2 https://data.ooni.org/analysis/m/20230724181047.970068_IN_webconnectivity_d6410089030032d2

• A likely blocking of a CSO website that I can never confirm and confused about - blocked 0.75
  https://data.ooni.org/analysis/m/20240828134537.631369_KH_webconnectivity_9591f3f13ba876bd

• Confirmed blocking that I found suspicious - blocked 0.2
  https://data.ooni.org/analysis/m/20230327103147.775055_HK_webconnectivity_be8efe1e62303da1

Seems ok. But will need to go through more HK measurements as they have the most issues

[sitinurliza95]

An OK measurement that is a blocking - blocked 0.6 (not sure why it's been showing a lot of OK on Explorer. I've just added the SG fingerprints ooni/blocking-fingerprints#17)
https://data.ooni.org/analysis/m/20240401153454.011394_SG_webconnectivity_74d14611a801555f

[hellais]

Thanks for documenting these cases! The ones marked as blocked 0.2 are instances in which the fingerprint is inconsistent with the country hence we mark it as 0.2 blocked and use the down indicator to flag it as "down" in the sense that the probe is badly located or misconfigured.

It might make sense to support these cases to have an extra parameter which is used specifically to flag bad measurements with also some likelyhood distribution. Similar to what we do ATM with the failed flag.

For the NXDOMAIN one, I will have to look at that more carefully.

I think the one marked as blocked 0.6, given we don't have the fingerprint in our DB, the value seems reasonable, though maybe it ought to be weighed a bit differently.

All in all thanks for highlighting these cases they are super helpful!