Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hybrid conversion #1382

Merged
merged 15 commits into from
Nov 4, 2024
Merged

Hybrid conversion #1382

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
399c6e9
Initial
tyurek Oct 24, 2024
d1c97b5
fix pre-commit
tyurek Oct 24, 2024
53437af
how do I fix this...
tyurek Oct 25, 2024
041c48f
almost done
tyurek Oct 28, 2024
ed9652a
ready for review
tyurek Oct 28, 2024
fe9062d
try, and not quite succeed, to get get all the runner stuff working
tyurek Oct 30, 2024
8457a3b
just give up
tyurek Oct 31, 2024
8fb9d34
Merge branch 'main' into hybrid_conversion
tyurek Oct 31, 2024
8cda0d1
fix horrific merge
tyurek Oct 31, 2024
9a6c491
finally found the issue
tyurek Oct 31, 2024
c718db7
cleanup
tyurek Oct 31, 2024
12552d7
address PR feedback
tyurek Nov 1, 2024
94c6f75
Change HybridInfo from enum to struct that has both types
tyurek Nov 1, 2024
4cc78c7
switch to TestHybridRecord
tyurek Nov 1, 2024
b597f39
remove HybridReport sharing for TestRawDataRecord
tyurek Nov 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion ipa-core/src/error.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ use thiserror::Error;
use crate::{
helpers::{Role, ZeroRecordsError},
protocol::RecordId,
report::InvalidReportError,
report::{hybrid::InvalidHybridReportError, InvalidReportError},
sharding::ShardIndex,
task::JoinError,
};
Expand Down Expand Up @@ -69,6 +69,8 @@ pub enum Error {
InvalidQueryParameter(BoxError),
#[error("invalid report: {0}")]
InvalidReport(#[from] InvalidReportError),
#[error("invalid hybrid report: {0}")]
InvalidHybridReport(#[from] InvalidHybridReportError),
#[error("unsupported: {0}")]
Unsupported(String),
#[error("Decompressing invalid elliptic curve point: {0}")]
Expand Down
86 changes: 63 additions & 23 deletions ipa-core/src/query/runner/hybrid.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ use futures::{stream::iter, StreamExt, TryStreamExt};
use crate::{
error::Error,
ff::{
boolean_array::{BooleanArray, BA20, BA3, BA8},
boolean_array::{BooleanArray, BA3, BA8},
U128Conversions,
},
helpers::{
Expand All @@ -20,31 +20,40 @@ use crate::{
step::ProtocolStep::Hybrid,
},
query::runner::reshard_tag::reshard_aad,
report::hybrid::{
EncryptedHybridReport, IndistinguishableHybridReport, UniqueTag, UniqueTagValidator,
report::{
hybrid::{
EncryptedHybridReport, IndistinguishableHybridReport, UniqueTag, UniqueTagValidator,
},
hybrid_info::HybridInfo,
},
secret_sharing::replicated::semi_honest::AdditiveShare as Replicated,
};

#[allow(dead_code)]
pub struct Query<C, HV, R: PrivateKeyRegistry> {
pub struct Query<'a, C, HV, R: PrivateKeyRegistry> {
config: HybridQueryParams,
key_registry: Arc<R>,
hybrid_info: HybridInfo<'a>,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I probably missed that on the previous review, but helper origin is assumed to be known statically to helpers. This should help to avoid the lifetime proliferation and the need to keep it inside query struct

If there is a need for having it, we can assume helper origin is &'static str

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The design doc for this project has the helper origin as part of both HybridConversionInfo and HybridImpressionInfo, so I assume it's needed? I can try static though

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it is definitely needed, but it is assumed to be per MPC ring, i.e. known in advance. We hardcoded it in our Rust implementation, but generally I think &static str would be more appropriate

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is conversion_site_domain also static? If so, we can remove the lifetime

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Conversion site is dynamic and provided by the submitter when they want to execute this query

phantom_data: PhantomData<(C, HV)>,
}

#[allow(dead_code)]
impl<C, HV, R: PrivateKeyRegistry> Query<C, HV, R> {
pub fn new(query_params: HybridQueryParams, key_registry: Arc<R>) -> Self {
impl<'a, C, HV, R: PrivateKeyRegistry> Query<'a, C, HV, R> {
pub fn new(
query_params: HybridQueryParams,
key_registry: Arc<R>,
hybrid_info: HybridInfo<'a>,
) -> Self {
Self {
config: query_params,
key_registry,
hybrid_info,
phantom_data: PhantomData,
}
}
}

impl<C, HV, R> Query<C, HV, R>
impl<'a, C, HV, R> Query<'a, C, HV, R>
where
C: UpgradableContext + Shuffle + ShardedContext,
HV: BooleanArray + U128Conversions,
Expand All @@ -60,6 +69,7 @@ where
let Self {
config,
key_registry,
hybrid_info,
phantom_data: _,
} = self;

Expand All @@ -73,13 +83,13 @@ where
));
}

let stream = LengthDelimitedStream::<EncryptedHybridReport, _>::new(input_stream)
let stream = LengthDelimitedStream::<EncryptedHybridReport<BA8, BA3>, _>::new(input_stream)
.map_err(Into::<Error>::into)
.map_ok(|enc_reports| {
iter(enc_reports.into_iter().map({
|enc_report| {
let dec_report = enc_report
.decrypt::<R, BA8, BA3, BA20>(key_registry.as_ref())
.decrypt(key_registry.as_ref(), &hybrid_info)
.map_err(Into::<Error>::into);
let unique_tag = UniqueTag::from_unique_bytes(&enc_report);
dec_report.map(|dec_report1| (dec_report1, unique_tag))
Expand Down Expand Up @@ -142,7 +152,7 @@ mod tests {

use crate::{
ff::{
boolean_array::{BA16, BA20, BA3, BA8},
boolean_array::{BA16, BA3, BA8},
U128Conversions,
},
helpers::{
Expand All @@ -151,7 +161,11 @@ mod tests {
},
hpke::{KeyPair, KeyRegistry},
query::runner::hybrid::Query as HybridQuery,
report::{OprfReport, DEFAULT_KEY_ID},
report::{
hybrid::HybridReport,
hybrid_info::{HybridConversionInfo, HybridInfo},
DEFAULT_KEY_ID,
},
secret_sharing::{replicated::semi_honest::AdditiveShare, IntoShares},
test_fixture::{
flatten3v, ipa::TestRawDataRecord, Reconstruct, RoundRobinInputDistribution, TestWorld,
Expand All @@ -162,10 +176,8 @@ mod tests {
const EXPECTED: &[u128] = &[0, 8, 5];

fn build_records() -> Vec<TestRawDataRecord> {
tyurek marked this conversation as resolved.
Show resolved Hide resolved
// TODO: When Encryption/Decryption exists for HybridReports
// update these to use that, rather than generating OprfReports
vec![
TestRawDataRecord {
/*TestRawDataRecord {
tyurek marked this conversation as resolved.
Show resolved Hide resolved
timestamp: 0,
user_id: 12345,
is_trigger_report: false,
Expand All @@ -178,7 +190,7 @@ mod tests {
is_trigger_report: false,
breakdown_key: 1,
trigger_value: 0,
},
},*/
TestRawDataRecord {
timestamp: 10,
user_id: 12345,
Expand All @@ -193,13 +205,13 @@ mod tests {
breakdown_key: 0,
trigger_value: 2,
},
TestRawDataRecord {
/*TestRawDataRecord {
timestamp: 20,
user_id: 68362,
is_trigger_report: false,
breakdown_key: 1,
trigger_value: 0,
},
},*/
TestRawDataRecord {
timestamp: 30,
user_id: 68362,
Expand All @@ -216,17 +228,27 @@ mod tests {
query_sizes: Vec<QuerySize>,
}

fn build_buffers_from_records(records: &[TestRawDataRecord], s: usize) -> BufferAndKeyRegistry {
fn build_buffers_from_records(
records: &[TestRawDataRecord],
s: usize,
info: &HybridInfo,
) -> BufferAndKeyRegistry {
let mut rng = StdRng::seed_from_u64(42);
let key_id = DEFAULT_KEY_ID;
let key_registry = Arc::new(KeyRegistry::<KeyPair>::random(1, &mut rng));

let mut buffers: [_; 3] = std::array::from_fn(|_| vec![Vec::new(); s]);
let shares: [Vec<OprfReport<BA8, BA3, BA20>>; 3] = records.iter().cloned().share();
let shares: [Vec<HybridReport<BA8, BA3>>; 3] = records.iter().cloned().share();
for (buf, shares) in zip(&mut buffers, shares) {
for (i, share) in shares.into_iter().enumerate() {
share
.delimited_encrypt_to(key_id, key_registry.as_ref(), &mut rng, &mut buf[i % s])
.delimited_encrypt_to(
key_id,
key_registry.as_ref(),
info,
&mut rng,
&mut buf[i % s],
)
.unwrap();
}
}
Expand Down Expand Up @@ -265,11 +287,16 @@ mod tests {
const SHARDS: usize = 2;
let records = build_records();

let hybrid_info = HybridInfo::Conversion(
HybridConversionInfo::new(0, "HELPER_ORIGIN", "meta.com", 1_729_707_432, 5.0, 1.1)
.unwrap(),
);

let BufferAndKeyRegistry {
buffers,
key_registry,
query_sizes,
} = build_buffers_from_records(&records, SHARDS);
} = build_buffers_from_records(&records, SHARDS, &hybrid_info);

let world: TestWorld<WithShards<SHARDS, RoundRobinInputDistribution>> =
TestWorld::with_shards(TestWorldConfig::default());
Expand All @@ -295,6 +322,7 @@ mod tests {
HybridQuery::<_, BA16, KeyRegistry<KeyPair>>::new(
query_params,
Arc::clone(&key_registry),
hybrid_info.clone(),
)
.execute(ctx, query_size, input)
})
Expand Down Expand Up @@ -329,11 +357,16 @@ mod tests {
const SHARDS: usize = 2;
let records = build_records();

let hybrid_info = HybridInfo::Conversion(
HybridConversionInfo::new(0, "HELPER_ORIGIN", "meta.com", 1_729_707_432, 5.0, 1.1)
.unwrap(),
);

let BufferAndKeyRegistry {
mut buffers,
key_registry,
query_sizes,
} = build_buffers_from_records(&records, SHARDS);
} = build_buffers_from_records(&records, SHARDS, &hybrid_info);

// this is double, since we duplicate the data below
let query_sizes = query_sizes
Expand Down Expand Up @@ -381,6 +414,7 @@ mod tests {
HybridQuery::<_, BA16, KeyRegistry<KeyPair>>::new(
query_params,
Arc::clone(&key_registry),
hybrid_info.clone(),
)
.execute(ctx, query_size, input)
})
Expand All @@ -400,11 +434,16 @@ mod tests {
const SHARDS: usize = 2;
let records = build_records();

let hybrid_info = HybridInfo::Conversion(
HybridConversionInfo::new(0, "HELPER_ORIGIN", "meta.com", 1_729_707_432, 5.0, 1.1)
.unwrap(),
);

let BufferAndKeyRegistry {
buffers,
key_registry,
query_sizes,
} = build_buffers_from_records(&records, SHARDS);
} = build_buffers_from_records(&records, SHARDS, &hybrid_info);

let world: TestWorld<WithShards<SHARDS, RoundRobinInputDistribution>> =
TestWorld::with_shards(TestWorldConfig::default());
Expand All @@ -430,6 +469,7 @@ mod tests {
HybridQuery::<_, BA16, KeyRegistry<KeyPair>>::new(
query_params,
Arc::clone(&key_registry),
hybrid_info.clone(),
)
.execute(ctx, query_size, input)
})
Expand Down
Loading