Event generation using test_mpc and IPA integration test + migrate to HTTP2 #647
Conversation
ipa_in_the_clear only uses breakdown keys from input, is not aware of `max_breakdown_key` parameter. That causes discrepancies on small inputs
tests/helper_networks.rs
Outdated
|
|
||
| #[test] | ||
| fn https_semi_honest_ipa() { | ||
| test_ipa(&[4433, 4434, 4435], IpaSecurityModel::SemiHonest, true); |
There was a problem hiding this comment.
this is a bit annoying - I know that there is no guaranteed way to pick an available port (at least on MacOS, on Linux it works pretty reliable with SO_REUSEPORT and rust appears to be using it), but even best-effort approach with TcpListener should work here. Right now it is not possible to test this in parallel
There was a problem hiding this comment.
I thought about this when I wrote the test... a possible strategy I came up with is to create TcpListeners in the test, pass them to the helper processes as /dev/stdin, and add an option to the helper that does TcpListener::from_raw_fd rather than bind a socket. I didn't do this at the time because it seemed like a fair amount of work, and it seemed useful to have at least one test that exercises the standard flow for binding the socket on startup. But if we're adding more tests that follow this framework, it becomes more important to do this work. I'll see if I can get this done next week.
|
integration tests hanging is not good: https://github.com/private-attribution/ipa/actions/runs/4964162687/jobs/8884058950?pr=647, opening an issue for that |
There was a problem hiding this comment.
@akoshelev and @andyleiserson should work out who gets to merge and who get to rebase :)
| // even bit vector takes too long to initialize for MAX_USER_ID. Need a sparse structure | ||
| // here | ||
| used: HashSet<UserId>, |
There was a problem hiding this comment.
Do we even need to track this? I mean, the problem here is that we're essentially stateless, but you could create user IDs in monotonically increasing sequence. A PRP would allow you to have the identifiers that you use appear to be randomly distributed in the target range...
| type Item = TestRawDataRecord; | ||
|
|
||
| fn next(&mut self) -> Option<Self::Item> { | ||
| const USERS_IN_FLIGHT: usize = 10; |
There was a problem hiding this comment.
This seems a little small for larger event counts. It's arbitrary, but I'd have said that you would expect hundreds of people for a million records. (It's no extra work for IPA, but I'm not sure how much we care about fidelity.)
@andyleiserson lets merge your changes first |
akoshelev
changed the title
There are 2 PRs which would need to be rebased based on what gets landed first (#654 also) :P |
That's fine with me. |
|
I am on vacation until Tuesday PT time, so I'll merge it to unblock you folks. Sorry @andyleiserson for the rebase |
There are there major changes that this PR brings:
Iterator<TestRawDataRecord>. for millions of events that is more efficient than the existing approach. It takes about 3 seconds to generate 1 million events (excluding IO)gen-ipa-inputscommandgen-ipa-inputsto generate inputs and run semi-honest IPA.--edit
Unintended change, but IPA integration tests were failing (see #650) on HTTP 1.1, so I had to include changing clients to use HTTP2