Anonymous credentials draft endpoints #1016

LDiazN · 2025-10-07T14:11:42Z

This PR adds support for the anonymous credentials protocol to the Ooni API:

Adds ooniauth-py as a dependency
Adds new endpoints: /sign_credential, /manifest, /submit

Note that the new measurement /submit endpoint won't replace the old one, and for now it's meant to be used mostly during development

Also updates the fastpath so that it's able to work with the new fields

Migrations

This PR will require some database migrations

Postgres

Adds the following tables to postgres:

ooniprobe_manifest: Describes the manifest that is reported to users when they register
ooniprobe_server_state: Defines the key pair (secret_key, public_parameters) that is used for authentication

You can find the new models here:

backend/ooniapi/services/ooniprobe/src/ooniprobe/models.py

Line 58 in 3b3bfb2

class OONIProbeServerState(Base):

And the migrations:
https://github.com/ooni/backend/blob/userauth-dep/ooniapi/common/src/common/alembic/versions/7e28b5d17a7f_add_server_state_table_for_anonymous_.py

Clickhouse

In clickhouse we need to add the fields necessary for:

Checking if a measurement is verified
Running the verification offline in the fastpath machine at some point in the future when we try to implement offline verification

As an example of the changes, you can look at the clickhouse_init.sql script in fastpath:

backend/fastpath/clickhouse_init.sql

Lines 42 to 46 in 3b3bfb2

    
           `is_verified` Int8, 
        
           `nym` Nullable(String), 
        
           `zkp_request` Nullable(String), 
        
           `age_range` Nullable(String), 
        
           `msm_range` Nullable(String),

These are the alter table statements required to run the migration in production:

ALTER TABLE fastpath
ADD COLUMN is_verified Int8,
ADD COLUMN nym Nullable(String),
ADD COLUMN zkp_request Nullable(String),
ADD COLUMN age_range Nullable(String),
ADD COLUMN msm_range Nullable(String);

Feedback

This is still early work to define the API for the anonymous credentials protocol. Some things that could benefit from a bit of feedback are:

Naming of the endpoints
- /sign_credential used to be named /register but it clashes with the older /register one, so I chose a different name
Parameters and usage
Lifecycle of manifest and server state: Right now we always create a new manifest and server state if none is available to keep the API working, but we might want to be more deliberate with how we work with this

closes #1014 #1015

codecov · 2025-10-10T11:43:17Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.92%. Comparing base (d943f03) to head (5deb275).
⚠️ Report is 11 commits behind head on master.

❌ Your project check has failed because the head coverage (90.92%) is below the target coverage (95.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1016      +/-   ##
==========================================
- Coverage   92.81%   90.92%   -1.90%     
==========================================
  Files          17       28      +11     
  Lines        1281     2932    +1651     
  Branches       65      247     +182     
==========================================
+ Hits         1189     2666    +1477     
- Misses         78      206     +128     
- Partials       14       60      +46

Flag	Coverage Δ
oonimeasurements	`87.60% <ø> (?)`
ooniprobe	`?`
oonirun	`99.39% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

hellais

I left some comments on small improvements and some questions.

We can discuss on slack anything which is not clear.

hellais · 2025-11-25T10:05:05Z

fastpath/fastpath/core.py

            measurement = ujson.loads(msm_jstr)
-        if sorted(measurement.keys()) == ["content", "format"]:
+
+        is_verified = g(measurement, 'is_verified', False)


It would be great if at some point we renamed this function g as something a bit more understandable. It's basically getting the key and if not it's setting to to False. Correct?

I believe modern versions of python have native support for this.

I guess we can take note of it as a future refactoring work.

We can just do that I think, the fastpath now runs in python 3.12 in AWS, this can be replaced with measurement.get('is_verified', False)

We can open another issue for that

hellais · 2025-11-25T10:06:42Z

fastpath/fastpath/db.py

    def tf(v: bool) -> str:
        return "t" if v else "f"

+    def s_or_n(x : Optional[Any]) -> Optional[str]:


Can we call this function a bit more explicitly, instead of these shorthands?

Even better would be if we just use an explicit lambda function inside of the age_range field, eg.

age_range=lambda x: ujson.dumps(x) if x is not None else None

I don't follow the lambda part. Where would you call the lambda? is there a place where you can use it as argument to automatically parse the thing?

But in the meanwhile I renamed s_or_n to serialize_optional to have a more descriptive name

hellais · 2025-11-25T10:08:29Z

ooniapi/services/ooniprobe/src/ooniprobe/routers/reports.py

+            compressed_len = len(data)
            data = zstd.decompress(data)
-            ratio = len(data) / len(data)
+            ratio = compressed_len / len(data)


Can we just leave this inlined? I think there is no need to have an extra line and variable.

hellais · 2025-11-25T10:09:32Z

ooniapi/services/ooniprobe/src/ooniprobe/routers/reports.py

            log.debug(f"Zstd compression ratio {ratio}")
-        except Exception as e:
+        except Exception:
            log.info("Failed zstd decompression")


Can we log e inside of log.info. Make sure you use {e} inside of log.info(f"") which should perform sanitisation.

Done, also did the same thing with other exceptions

hellais · 2025-11-25T10:10:41Z

ooniapi/services/ooniprobe/src/ooniprobe/routers/v1/probe_services.py

+    # time window
+    now = datetime.now(timezone.utc)
+    h = sha512(data_bin).hexdigest()[:16]
+    ts = now.strftime("%Y%m%d%H%M%S.%f")


Can we use ISO standard? Make sure it has in it T and Z to indicate timezone.

I'm not sure if that's a good idea because ts is used to compute the measurement id:

backend/ooniapi/services/ooniprobe/src/ooniprobe/routers/v1/probe_services.py

Line 762 in c40e14b

msmt_uid = f"{ts}_{cc}_{test_name}_{h}"

So we would be changing the measurement id format. I'm not too confident that this change won't break anything else. What do you think?

hellais · 2025-11-25T10:11:45Z

ooniapi/services/ooniprobe/src/ooniprobe/routers/v1/probe_services.py

+                f"[Try {t+1}/{N_RETRIES}] Error trying to send measurement to the fastpath ({settings.fastpath_url}). Error: {exc}"
+            )
+            sleep_time = random.uniform(0, min(3, 0.3 * 2 ** t))
+            await asyncio.sleep(sleep_time)


Is this necessary? cc @mmaker

hellais · 2025-11-25T10:12:21Z

ooniapi/services/ooniprobe/tests/test_anoncred.py

+    assert resp.status_code == status.HTTP_200_OK, f"Unexpected status code: {resp.status_code} - {url}. {resp.content}"
+    return resp.json()
+
+def post(client, url, data, headers=None):


Should we move these into some kind of utils.py module?

done, also refactored other tests to use the utils.py module

hellais · 2025-11-25T10:14:31Z

ooniapi/services/ooniprobe/src/ooniprobe/routers/v1/probe_services.py

+    content_encoding: str = Header(default=None),
+) -> SubmitMeasurementResponse | Dict[str, Any]:
+    """
+    Submit measurement


Can you write a better docstring?

Also, it would be good to know where we plug in the values for "valid" probe_age_ranges and probe_msm_range (can you rename these variables to that)

I renamed the variables and also added extra documentation to explain how these variables are used, but I'm not sure if this is what was expected here

LDiazN added 3 commits October 7, 2025 11:49

Fix capitalization in dockerfile

2d81a8a

Clone userauth repo into include dir

e6c5fdc

Add ooniauth-py as dependency

294af32

LDiazN requested a review from hellais October 7, 2025 14:11

LDiazN self-assigned this Oct 7, 2025

LDiazN added 24 commits October 8, 2025 10:15

Remove unused imports

6754df8

Add server state model for anonymous credentials

42b5ec5

remove unnecessary pass

7c11874

Add utility functions to work with protocol objects

0970cb5

Add initialization of server state table

c97f3ba

Add manifest endpoint

950cbe3

Fix error with postgres dep

a9990ed

Init server state dep in tests

94a7936

fix bare except

bcffac5

remove unused import

7b602c2

Add test file for anonymous credentials

50e0c47

Add register endpoint for anonymous credentials

ad14987

Add test for credential sign request

d286c76

Add line to check if user is able to verify credential

bfb5600

Improve error reporting

f582c8f

Add test for not found error

db5763a

Add test for protocol error

f06ce42

Add test for bad serialization

08413d7

Refactor to share metrics to the entire module

75eead1

Creating manifest table

ddf860a

Added foreign key to server state to fetch keys

2baff84

Change register endpoint to use manifest version

4dd14be

Use new manifest style in endpoints

9810800

Use extra fields in measurement submission

355e059

LDiazN added 13 commits October 10, 2025 16:29

Testing measurement submission

b9b9250

Improve testing of measurement submission

cff7f32

Fix broken test

ec63895

Fix broken tests

6c667f0

Fix bad fiture downgrade

985e89a

Add anonymous credentials related data to fastpath

4328d27

Merge branch 'master' into userauth-dep

f93131a

fix broken tests

28cb8e0

Add fastpath migration to oonimeasurements tests

3b3bfb2

refactor to_http_exception

e0f4432

Add new ooniauth dependency url

5b12744

Add anonymous creentials columns to ooniprobe test schema

f895dc8

Use alter table instead of directly changing the table

e7336cb

agrabeli mentioned this pull request Nov 20, 2025

OTFcreds: Activity 3.1 Integrate the server library into the OONI backend #900

Open

hellais requested changes Nov 25, 2025

View reviewed changes

LDiazN added 15 commits November 25, 2025 14:14

Inline compression ratio; rename s_or_n to serialize_optional

ca7e93d

Add error message to info log

536e774

Add error message to info log

c40e14b

Add utility functions for testing in ooniprobe

5deb275

Merge conflict

98f0025

Add utils file for tests

cb52441

Add better docstrings

0e0648e

Add better docstrings

e315218

Add initial version of credential update endpoints

d06e830

Adding tests for credential update

5fccf4a

Fix bad error message

a8519a2

Add simple credential update test

15ef04a

Refactor to move fixtures to conftest

b8a3d9f

Remove unused import

1ffd959

Add test with full workflow

c2f4e16

	`is_verified` Int8,
	`nym` Nullable(String),
	`zkp_request` Nullable(String),
	`age_range` Nullable(String),
	`msm_range` Nullable(String),

Anonymous credentials draft endpoints #1016

Are you sure you want to change the base?

Anonymous credentials draft endpoints #1016

Uh oh!

Conversation

LDiazN commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Migrations

Postgres

Clickhouse

Feedback

Uh oh!

codecov bot commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hellais left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LDiazN Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LDiazN Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

LDiazN Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

LDiazN commented Oct 7, 2025 •

edited

Loading

codecov bot commented Oct 10, 2025 •

edited

Loading

LDiazN Nov 25, 2025 •

edited

Loading

LDiazN Nov 25, 2025 •

edited

Loading

LDiazN Nov 26, 2025 •

edited

Loading