Implement Fog Ledger Router #3312
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Implement create_multi_view_store_query_data API * Address remoun's comments
* Create MultiViewStoreQuery API for Fog View Store * Implement multi_view_store_query in Fog View Store * Use mut_decryption_error
…discovery (#2200) * Add client_init and client_connect enclave methods * Move multiview query construction to ake enclave * Implement Remoun's suggestions * Implement second round of remoun's suggestions
#2189) * Remove FogViewStoreUri * Implement core Fog Router Service logic * Refactor module structure and add tests * Use where clauses * Implement nick's suggestions * Implement second round of nick's suggestions * Implement lint suggestions * Implement james's suggestions * Fix naming error * Fix whitespace
* Create helper methods that obliviously collate shard responses * Implement chris's suggestions * Fix error string * Fix lint * Sort dependencies * Run format * fix newline
* Add backend decrypt method to ake enclave * Add enclave methods for shard response collation * Refactor response to always include view store uri * Use shard responses to fill router response * Implement Nick's suggestions and fix CI errors
* Add readiness API to ShardingStrategy * Implement readiness check per ShardingStrategy * Change MVQR error to accommodate different types * Set not ready message * Pull out query processing logic to separate method
* attest::NonceMessage, EnclaveNonceMessage, add handling to attest-ake * Rustfmt fixes
* attest::NonceMessage, EnclaveNonceMessage, add handling to attest-ake * Rustfmt fixes * Minor DRY cleanups around sessions and auth messages
* Implement client message to sealed message API for backends * Update Fog View enclave to build with new AKE enclave API * Rust type aliases don't protect against passing the aliased type, use a new type instead * Re-add comment ASCII art * Additional plumbing of new SealedClientMessage type * Move SealedClientMessage, finish implementing in fog view router * Properly update the lockfiles * Fix clippy lint * Update lockfiles again after rebase Co-authored-by: Andrew Wygle <andrew@mobilecoin.com>
* Implement shard management API * Implement James's suggestions
* Reapply changes from #2473
* Add frontends * Fix traits on NonceSession
* Re-apply 2463. (#2547) * Fix traits on NonceSession * Minor comment and constant consistency/friendliness changes * Fix compile error. * Fix deleted Eq from NonceSession.
* Key Image Router Service * Remove unneeded dead_code annotation * Update fog/ledger/server/src/error.rs Co-authored-by: Nick Santana <nick@mobilecoin.com> * Update fog/ledger/server/src/key_image_router_service.rs Co-authored-by: Nick Santana <nick@mobilecoin.com> * Clean up commented-out code Co-authored-by: Nick Santana <nick@mobilecoin.com> * Fix misnamed type in a comment Co-authored-by: Nick Santana <nick@mobilecoin.com> * Address PR feedback around logging and comments. * Address error in loop termination logic. * Parameterize allowed number of retries for query loop * Update based on changes from previous PRs * Don't create 'groups' in `mod` or `use` declarations. Co-authored-by: NotGyro <gyrocoder@gmail.com> Co-authored-by: Nick Santana <nick@mobilecoin.com>
* Oblivious collation of key image store responses * Refactor tests to be more readable, per PR feedback * Update fog/ledger/enclave/impl/src/oblivious_utils.rs Co-authored-by: Nick Santana <nick@mobilecoin.com> Co-authored-by: Nick Santana <nick@mobilecoin.com>
* Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Fix redundant mod declaration
* Ledger enclave support for router and store (#2896) * Pulling changes in from milliec/ledger-router-dev * Run cargo fmt * Making requested changes and clarifications to Fog ledger router comments. * Additional comment fixes * Fix CI lint * PR feedback nits Co-authored-by: Andrew Wygle <andrew@mobilecoin.com> * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Update router config for parameterized retries * Remove dead_code declaration on router service constructor * Changes due to rebase * normalize naming - ledger router, key image store * Linting fixes * Accept code review suggestions * Updates for GRPCIO 0.12 * Remove some unwraps in ledger_router binary Co-authored-by: Emily C <gyrocoder@gmail.com>
* Ledger enclave support for router and store (#2896) * Pulling changes in from milliec/ledger-router-dev * Run cargo fmt * Making requested changes and clarifications to Fog ledger router comments. * Additional comment fixes * Fix CI lint * PR feedback nits Co-authored-by: Andrew Wygle <andrew@mobilecoin.com> * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Update router config for parameterized retries * Changes due to rebase * normalize naming - ledger router, key image store * Linting fixes * Accept code review suggestions * Updates for GRPCIO 0.12 * Remove some unwraps in ledger_router binary * Pulling changes in from milliec/ledger-router-dev * Making requested changes and clarifications to Fog ledger router comments. * PR feedback nits * Key Image Router Service * Update fog/ledger/server/src/key_image_router_service.rs Co-authored-by: Nick Santana <nick@mobilecoin.com> * Address PR feedback around logging and comments. * Parameterize allowed number of retries for query loop * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Key image store changes pulled in from milliec/ledger-router-dev * Cargo fmt * Run clippy * Sort itertools properly in Cargo.toml * Rebase and update to match current fog ledger router branch * Remove unused dependencies in fog-ledger-server Co-authored-by: Nick Santana <nick@mobilecoin.com> * Apply suggestions - Remove unused deps Co-authored-by: Nick Santana <nick@mobilecoin.com> * Apply suggestions from comments Applying suggestions from @nick-mobilecoin's review Co-authored-by: Nick Santana <nick@mobilecoin.com> * Resolving some code quality issues in direct_key_image_store_check() * Cargo fmt * key image server and key image service moved to router server and router service * Apply suggestions from code review Co-authored-by: Sam Dealy <33067698+samdealy@users.noreply.github.com> Co-authored-by: Nick Santana <nick@mobilecoin.com> --------- Co-authored-by: Andrew Wygle <andrew@mobilecoin.com> Co-authored-by: Nick Santana <nick@mobilecoin.com> Co-authored-by: Sam Dealy <33067698+samdealy@users.noreply.github.com>
* Ledger enclave support for router and store (#2896) * Pulling changes in from milliec/ledger-router-dev * Run cargo fmt * Making requested changes and clarifications to Fog ledger router comments. * Additional comment fixes * Fix CI lint * PR feedback nits Co-authored-by: Andrew Wygle <andrew@mobilecoin.com> * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Update router config for parameterized retries * Changes due to rebase * normalize naming - ledger router, key image store * Linting fixes * Accept code review suggestions * Updates for GRPCIO 0.12 * Remove some unwraps in ledger_router binary * Pulling changes in from milliec/ledger-router-dev * Making requested changes and clarifications to Fog ledger router comments. * PR feedback nits * Key Image Router Service * Update fog/ledger/server/src/key_image_router_service.rs Co-authored-by: Nick Santana <nick@mobilecoin.com> * Address PR feedback around logging and comments. * Parameterize allowed number of retries for query loop * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Key image store changes pulled in from milliec/ledger-router-dev * Cargo fmt * Run clippy * Sort itertools properly in Cargo.toml * Rebase and update to match current fog ledger router branch * Remove unused dependencies in fog-ledger-server Co-authored-by: Nick Santana <nick@mobilecoin.com> * Apply suggestions - Remove unused deps Co-authored-by: Nick Santana <nick@mobilecoin.com> * Apply suggestions from comments Applying suggestions from @nick-mobilecoin's review Co-authored-by: Nick Santana <nick@mobilecoin.com> * Resolving some code quality issues in direct_key_image_store_check() * Cargo fmt * key image server and key image service moved to router server and router service * Fog Ledger Router Admin service * Key Image Router Server + Binary * Update router config for parameterized retries * Changes due to rebase * normalize naming - ledger router, key image store * Linting fixes * Accept code review suggestions * Implement ShardingStrategy into DbFetcher * Split out DbFetcher creation and starting for ease of integration * Add a DbFetcher to the ledger store server. * Start the db_fetcher in the Fog Ledger Server to reflect the new behavior * Run cargo fmt * Run cargo clippy * Other clippy and fmt changes * Sort dependencies * Remove unimplemented! that snuck in from other branch * Update and sync with latest Fog ledger router * Remove some unwraps * Lint again * rebase fixups * PR feedback --------- Co-authored-by: Emily C <gyrocoder@gmail.com> Co-authored-by: Nick Santana <nick@mobilecoin.com>
* Ledger enclave support for router and store (#2896) * Pulling changes in from milliec/ledger-router-dev * Run cargo fmt * Making requested changes and clarifications to Fog ledger router comments. * Additional comment fixes * Fix CI lint * PR feedback nits Co-authored-by: Andrew Wygle <andrew@mobilecoin.com> * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Update router config for parameterized retries * Changes due to rebase * normalize naming - ledger router, key image store * Linting fixes * Accept code review suggestions * Updates for GRPCIO 0.12 * Pulling changes in from milliec/ledger-router-dev * Making requested changes and clarifications to Fog ledger router comments. * PR feedback nits * Key Image Router Service * Update fog/ledger/server/src/key_image_router_service.rs Co-authored-by: Nick Santana <nick@mobilecoin.com> * Address PR feedback around logging and comments. * Parameterize allowed number of retries for query loop * Fog Ledger Router Admin service * Sort itertools properly in Cargo.toml * Key Image Router Server + Binary * Key image store changes pulled in from milliec/ledger-router-dev * Cargo fmt * Run clippy * Sort itertools properly in Cargo.toml * Rebase and update to match current fog ledger router branch * Implement ShardingStrategy into DbFetcher * Add a DbFetcher to the ledger store server. * Run cargo fmt * Other clippy and fmt changes * Sort dependencies * Remove unimplemented! that snuck in from other branch * Update and sync with latest Fog ledger router * Refactor router dispatch into match statement Uses ugly names from rust-protobuf. rust-protobuf version 3.* will do the Rusty thing and put the generated enums into modules, but we're on version 2.* and they do the C thing of really_long_snake_case_names. * Add merkle proof service to router server * Add untrusted tx out service to router server * Add block service to router server * Fixups from rebase --------- Co-authored-by: Emily C <gyrocoder@gmail.com> Co-authored-by: Nick Santana <nick@mobilecoin.com>
* Port existing tests to router server binary, except key image * Port key image test in router_connection.rs to streaming API * Fixups from rebase * remove logging statements * All Ledger tests now use portpicker to select ports * Fog router support for the unary API (#3123) * Cargo fmt * Ensure unary key image service gets started for router server. * Improving comment clarity. * Apply suggestions removing unnecessary comments Co-authored-by: Nick Santana <nick@mobilecoin.com> --------- Co-authored-by: Nick Santana <nick@mobilecoin.com> * Apply suggestions from code review Co-authored-by: Nick Santana <nick@mobilecoin.com> * Pull request feedback --------- Co-authored-by: Emily C <gyrocoder@gmail.com> Co-authored-by: Nick Santana <nick@mobilecoin.com>
* Port existing tests to router server binary, except key image * Port key image test in router_connection.rs to streaming API * Fixups from rebase * remove logging statements * All Ledger tests now use portpicker to select ports * Fog router support for the unary API (#3123) * Cargo fmt * Ensure unary key image service gets started for router server. * Improving comment clarity. * Apply suggestions removing unnecessary comments Co-authored-by: Nick Santana <nick@mobilecoin.com> --------- Co-authored-by: Nick Santana <nick@mobilecoin.com> * Test key image retrieval via unary API on the router. * Cargo fmt * Fix a merging mistake. * Fix some additional merge mistakes * Sort dependencies --------- Co-authored-by: Andrew Wygle <andrew@mobilecoin.com> Co-authored-by: Nick Santana <nick@mobilecoin.com>
* Router integration test * Clippy updates after rebase * Implement Drop for TestEnvironment * Eliminate temp dirs after test * Remove unused GRPC arguments * Load environment config from JSON file. Might delete later. * Revert "Load environment config from JSON file. Might delete later." This reverts commit a677b59. * Remove magic numbers
* Remove client-facing API from key-image store * Cargo fmt * Remove fog_ledger_key_images_test from connection.rs (equivalent exists in router_connection.rs) * Cargo fmt * Pull out old code for the single-server design which will not be used anymore. * Fix cargo.toml.
* Optelemetry tracing for FLR * Cargo fmt * Apply James' suggested span names Co-authored-by: James Cape <james@mobilecoin.com> * Cargo fmt --------- Co-authored-by: James Cape <james@mobilecoin.com>
* Prometheus metrics for FLR * Cargo fmt
awygle
force-pushed
the
feature/fog-ledger-router
branch
from
e024ac6
to
fee2eb8
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
This PR contains all of the feature work related to the Fog Ledger Router. Each change has already been independently reviewed on the feature/fog-ledger-router branch, and the branch itself has been kept up-to-date with master.