Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

riseupvpn: relax scoring logic #745

Closed
bassosimone opened this issue Oct 11, 2023 · 0 comments · Fixed by #748
Closed

riseupvpn: relax scoring logic #745

bassosimone opened this issue Oct 11, 2023 · 0 comments · Fixed by #748
Assignees
Labels
api API data quality Describes data/measurement quality issues enhancement improving existing code or new feature interrupt Task not planned during planning ooni/api Issues related to https://github.com/ooni/api priority/medium

Comments

@bassosimone
Copy link
Contributor

In ooni/probe-cli#1363 et al. I am preparing riseupvpn to be included again. Because riseupvpn is a default-disabled experiment, it won't run on probes until check-in explicitly enables it.

When we'll enable riseupvpn, we should need to change and significantly relax the backend scoring logic. The probe currently does not score riseupvpn and never claims there was any anomaly.

I think the backend should do the same. Basically, we should collect and show riseupvpn measurements without interpreting them. This is part of what I discussed with @cyBerta in ooni/probe-cli#1125 (review).

For reference, the current scoring logic is the following:

def score_riseupvpn(msm: dict) -> dict:
    """Calculate measurement scoring for RiseUp VPN
    Returns a scores dict
    """
    # https://github.com/ooni/backend/issues/541
    scores = init_scores()
    tk = g_or(msm, "test_keys", {})
    tstatus = tk.get("transport_status") or {}
    obfs4 = tstatus.get("obfs4")
    openvpn = tstatus.get("openvpn")
    anomaly = (
        tk.get("api_status") == "blocked"
        or tk.get("ca_cert_status") is False
        or obfs4 == "blocked"
        or openvpn == "blocked"
    )
    if anomaly:
        scores["blocking_general"] = 1.0

    scores["extra"] = dict(test_runtime=msm.get("test_runtime"))
    return scores

We should drop the part that computes whether there was an anomaly. But, if possible, we should not flag the measurement as failed, which would hide it. We should just do such that the measurement appears on Explorer w/o any backend-derived inference on whether there was blocking.

@bassosimone bassosimone added enhancement improving existing code or new feature ooni/api Issues related to https://github.com/ooni/api priority/medium interrupt Task not planned during planning data quality Describes data/measurement quality issues api API labels Oct 11, 2023
bassosimone added a commit that referenced this issue Oct 17, 2023
This diff, which would require additional tweaks, attempts at relaxing the riseupvpn scoring logic as discussed by #745.

Closes #745.
bassosimone added a commit that referenced this issue Oct 17, 2023
This diff relaxes the riseupvpn scoring logic because we have
determined in ooni/probe-cli#1125 (comment)
that we want this experiment to just collect data useful to riseup.

Closes #745.
bassosimone added a commit that referenced this issue Oct 17, 2023
This diff relaxes the riseupvpn scoring logic because we have
determined in ooni/probe-cli#1125 (comment)
that we want this experiment to just collect data useful to riseup.

Note that I needed to change test_aggregation.py as a result of these
changes because the number of ok and anomaly results changed.

Closes #745.
bassosimone added a commit that referenced this issue Oct 18, 2023
This diff relaxes the riseupvpn scoring logic because we have
determined in ooni/probe-cli#1125 (comment)
that we want this experiment to just collect data useful to riseup.

Note that I needed to change test_aggregation.py as a result of these
changes because the number of ok and anomaly results changed.

Closes #745.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api API data quality Describes data/measurement quality issues enhancement improving existing code or new feature interrupt Task not planned during planning ooni/api Issues related to https://github.com/ooni/api priority/medium
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants