Ledger router server prototype (mobilecoinfoundation#2612)

* Create helper methods to obliviously collate shard responses (mobilecoinfoundation#2252)

* Create helper methods that obliviously collate shard responses

* Implement chris's suggestions

* Fix error string

* Fix lint

* Sort dependencies

* Run format

* fix newline

* Collate shard response in constant time (obliviously) (mobilecoinfoundation#2250)

* Add backend decrypt method to ake enclave

* Add enclave methods for shard response collation

* Refactor response to always include view store uri

* Use shard responses to fill router response

* Implement Nick's suggestions and fix CI errors

* Implement epoch sharding via trait (mobilecoinfoundation#2350)

* Use sharding strategy to tell view what blocks to process (mobilecoinfoundation#2351)

* Return minimum highest processed block to client (mobilecoinfoundation#2387)

* Add config option for sharding strategy (mobilecoinfoundation#2352)

* Add readiness API to ShardingStrategy (mobilecoinfoundation#2353)

* Add readiness API to ShardingStrategy

* Implement readiness check per ShardingStrategy

* Change MVQR error to accommodate different types

* Set not ready message

* Pull out query processing logic to separate method

* Add report fetching logic to FVR (mobilecoinfoundation#2454)

* Fix responder ID usage in FVR (mobilecoinfoundation#2446)

* Create different types of View Servers for store and client-facing (mobilecoinfoundation#2447)

* Messages for noise protocol exchanges with explicit nonces (mobilecoinfoundation#2461)

* attest::NonceMessage, EnclaveNonceMessage, add handling to attest-ake

* Rustfmt fixes

* Minor DRY cleanups around sessions and auth messages (mobilecoinfoundation#2462)

* attest::NonceMessage, EnclaveNonceMessage, add handling to attest-ake

* Rustfmt fixes

* Minor DRY cleanups around sessions and auth messages

* Update FVR client for integration test (mobilecoinfoundation#2448)

* Create gRPC FogViewRouterAdminAPI (mobilecoinfoundation#2360)

* Create FogViewRouterAdminUri (mobilecoinfoundation#2361)

* Add NonceAuthRequest, NonceAuthResponse, NonceSession (mobilecoinfoundation#2463)

* Add NonceAuthRequest, NonceAuthResponse, NonceSession

* Remove EnclaveNonceMessage in favor of EnclaveMessage<NonceSession>

* Manually implement Hash to resolve clippy warning

* Implement client message to sealed message API for backends

* Update Fog View enclave to build with new AKE enclave API

* Rust type aliases don't protect against passing the aliased type, use a new type instead

* Re-add comment ASCII art

* Additional plumbing of new SealedClientMessage type

* Move SealedClientMessage, finish implementing in fog view router

* Properly update the lockfiles

* Fix clippy lint

* Starting branch for ledger router service.

* Adapted more of fog view router code for ledger router.

* Revert mistake in shard_responses_processor.rs

* Adapting copied fog view router code, continuing to fill in a first draft.

* Ledger router bin improvements.

* EnclaveCall changes

* Adding ledger_store_server and service

* Adding new enclave methods to ecall_dispatcher()

* Clarifying names

* Further disentangling of parts of the ledger, plus filling out more enclave-related code

* More reorganization

* Rewrite create_key_image_store_query to use sealed messages (without changing logic)

* Plumb decrypt_and_seal_query enclave call

* Enable enclave call for multi-query creation

* Implement shard response processing as in Fog View Router

* Enable store authentication from router

* Implement retry logic

* First-pass implementation of response collation

Co-authored-by: Sam Dealy <33067698+samdealy@users.noreply.github.com>
Co-authored-by: James Cape <james@mobilecoin.com>
Co-authored-by: Millie C <gyro@pop-os.localdomain>
Co-authored-by: NotGyro <gyrocoder@gmail.com>

fog/api/proto/ledger.proto

@@ -60,15 +60,33 @@ message MultiKeyImageStoreRequest {
     repeated attest.Message queries = 1;
 }
 
+
+/// The status associated with a MultiViewStoreQueryResponse
+enum MultiKeyImageStoreResponseStatus {
+    /// The Fog Ledger Store successfully fulfilled the request.
+    SUCCESS = 0;
+    /// The Fog Ledger Store is unable to decrypt a query within the MultiKeyImageStoreRequest. It needs to be authenticated
+    /// by the router.
+    AUTHENTICATION_ERROR = 1;
+    /// The Fog Ledger Store is not ready to service a MultiViewStoreQueryRequest. This might be because the store has
+    /// not loaded enough blocks yet.
+    NOT_READY = 2;
+}
+
 message MultiKeyImageStoreResponse {
     /// Optional field that gets set when the Fog Ledger Store is able to decrypt a query
     /// included in the MultiKeyImageStoreRequest and create a query response for that
     //  query.
     attest.Message query_response = 1;
 
-    /// Optional error that gets returned when the Fog Ledger Store
-    /// cannot decrypt the MultiKeyImageStoreRequest.
-    FogLedgerStoreDecryptionError decryption_error = 2;
+    /// The FogViewStoreUri for the specific Fog View Store that
+    /// tried to decrypt the MultiViewStoreQueryRequest and failed.
+    /// The client should subsequently authenticate with the machine
+    /// described by this URI.
+    string fog_ledger_store_uri = 2;
+
+    /// Status that gets returned when the Fog Ledger Store services a MultiKeyImageStoreRequest.
+    MultiKeyImageStoreResponseStatus status = 3;
 }
 
 ////

fog/api/src/conversions.rs

@@ -2,7 +2,9 @@
 //
 // Contains helper methods that enable conversions for Fog Api types.
 
-use crate::{fog_common, ingest_common, view::MultiViewStoreQueryRequest};
+use crate::{
+    fog_common, ingest_common, ledger::MultiKeyImageStoreRequest, view::MultiViewStoreQueryRequest,
+};
 use mc_api::ConversionError;
 use mc_attest_api::attest;
 use mc_attest_enclave_api::{EnclaveMessage, NonceSession};
@@ -31,6 +33,15 @@ impl From<Vec<attest::NonceMessage>> for MultiViewStoreQueryRequest {
     }
 }
 
+impl From<Vec<attest::Message>> for MultiKeyImageStoreRequest {
+    fn from(attested_query_messages: Vec<attest::Message>) -> MultiKeyImageStoreRequest {
+        let mut multi_key_image_store_request = MultiKeyImageStoreRequest::new();
+        multi_key_image_store_request.set_queries(attested_query_messages.into());
+
+        multi_key_image_store_request
+    }
+}
+
 impl From<&common::BlockRange> for fog_common::BlockRange {
     fn from(common_block_range: &common::BlockRange) -> fog_common::BlockRange {
         let mut proto_block_range = fog_common::BlockRange::new();

fog/ledger/connection/src/error.rs

@@ -8,6 +8,7 @@ use mc_api::ConversionError;
 
 use mc_fog_enclave_connection::Error as EnclaveConnectionError;
 use mc_fog_uri::FogLedgerUri;
+use mc_util_uri::UriConversionError;
 
 /// Error type returned by LedgerServerConn
 #[derive(Debug, Display)]
@@ -35,3 +36,9 @@ impl From<ConversionError> for Error {
         Error::Conversion(err)
     }
 }
+
+impl From<UriConversionError> for Error {
+    fn from(err: UriConversionError) -> Self {
+        Self::UriConversionError(err)
+    }
+}

fog/ledger/enclave/api/src/lib.rs

@@ -13,9 +13,11 @@ pub use crate::{
     error::{AddRecordsError, Error},
     messages::{EnclaveCall, KeyImageData},
 };
-use alloc::vec::Vec;
+use alloc::{collections::BTreeMap, vec::Vec};
 use core::result::Result as StdResult;
-use mc_attest_enclave_api::{ClientAuthRequest, ClientAuthResponse, ClientSession, EnclaveMessage};
+use mc_attest_enclave_api::{
+    ClientAuthRequest, ClientAuthResponse, ClientSession, EnclaveMessage, SealedClientMessage,
+};
 use mc_common::ResponderId;
 use mc_crypto_keys::X25519Public;
 pub use mc_fog_types::ledger::{
@@ -99,12 +101,13 @@ pub trait LedgerEnclave: ReportableEnclave {
     /// Add a key image data to the oram Using thrm -rf targete key image
     fn add_key_image_data(&self, records: Vec<KeyImageData>) -> Result<()>;
 
-
     // LEDGER ROUTER / STORE SYSTEM
 
-    /// Begin a connection to a Fog Ledger Store. The enclave calling this method,
-    /// most likely a router, will act as a client to the Fog Ledger Store.
-    fn connect_to_key_image_store(&self, ledger_store_id: ResponderId) -> Result<ClientAuthRequest>;
+    /// Begin a connection to a Fog Ledger Store. The enclave calling this
+    /// method, most likely a router, will act as a client to the Fog Ledger
+    /// Store.
+    fn connect_to_key_image_store(&self, ledger_store_id: ResponderId)
+        -> Result<ClientAuthRequest>;
 
     /// Complete the connection to a Fog Ledger Store that has accepted our
     /// ClientAuthRequest. This is meant to be called after the enclave has
@@ -114,22 +117,38 @@ pub trait LedgerEnclave: ReportableEnclave {
         ledger_store_id: ResponderId,
         ledger_store_auth_response: ClientAuthResponse,
     ) -> Result<()>;
-
+
+    /// Decrypts a client query message and converts it into a
+    /// SealedClientMessage which can be unsealed multiple times to
+    /// construct the MultiKeyImageStoreRequest.
+    fn decrypt_and_seal_query(
+        &self,
+        client_query: EnclaveMessage<ClientSession>,
+    ) -> Result<SealedClientMessage>;
+
     /// Transforms a client query request into a list of query request data.
     ///
     /// The returned list is meant to be used to construct the
-    /// MultiLedgerStoreQuery, which is sent to each shard.
-    fn create_key_image_store_query(
+    /// MultiKeyImageStoreRequest, which is sent to each shard.
+    fn create_multi_key_image_store_query_data(
         &self,
-        client_query: EnclaveMessage<ClientSession>,
+        sealed_query: SealedClientMessage,
     ) -> Result<Vec<EnclaveMessage<ClientSession>>>;
 
-    /// Used by a Ledger Store to handle an inbound encrypted ledger.proto LedgerRequest. 
-    /// Generally, these come in from a router. 
-    /// This could could be a key image request, a merkele proof 
+    /// Receives all of the shards' query responses and collates them into one
+    /// query response for the client.
+    fn collate_shard_query_responses(
+        &self,
+        sealed_query: SealedClientMessage,
+        shard_query_responses: BTreeMap<ResponderId, EnclaveMessage<ClientSession>>,
+    ) -> Result<EnclaveMessage<ClientSession>>;
+
+    /// Used by a Ledger Store to handle an inbound encrypted ledger.proto
+    /// LedgerRequest. Generally, these come in from a router.
+    /// This could could be a key image request, a merkele proof
     /// request, and potentially in the future an untrusted tx out request.
     fn handle_key_image_store_request(
-        &self, 
+        &self,
         router_query: EnclaveMessage<ClientSession>,
     ) -> Result<EnclaveMessage<ClientSession>>;
 }

fog/ledger/enclave/api/src/messages.rs

@@ -2,9 +2,13 @@
 
 //! The message types used by the ledger_enclave_api.
 use crate::UntrustedKeyImageQueryResponse;
-use alloc::vec::Vec;
+use alloc::{collections::BTreeMap, vec::Vec};
 use mc_attest_core::{Quote, Report, TargetInfo, VerificationReport};
-use mc_attest_enclave_api::{ClientAuthRequest, ClientSession, EnclaveMessage, ClientAuthResponse};
+
+use mc_attest_enclave_api::{
+    ClientAuthRequest, ClientAuthResponse, ClientSession, EnclaveMessage, SealedClientMessage,
+};
+
 use mc_common::ResponderId;
 use mc_fog_types::ledger::GetOutputsResponse;
 use mc_transaction_core::ring_signature::KeyImage;
@@ -34,7 +38,7 @@ pub struct KeyImageData {
 
 /// An enumeration of API calls and their arguments for use across serialization
 /// boundaries.
-#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)]
+#[derive(Clone, Debug, Deserialize, Serialize)]
 pub enum EnclaveCall {
     /// The [LedgerEnclave::enclave_init()] method.
     EnclaveInit(ResponderId, u64),
@@ -104,33 +108,45 @@ pub enum EnclaveCall {
 
     /// The [LedgerEnclave::connect_to_store()] method.
     ///
-    /// Begin a connection to a Fog Ledger Store. The enclave calling this method,
-    /// most likely a router, will act as a client to the Fog Ledger Store.
+    /// Begin a connection to a Fog Ledger Store. The enclave calling this
+    /// method, most likely a router, will act as a client to the Fog Ledger
+    /// Store.
+  
     ConnectToKeyImageStore(ResponderId),
 
     /// The [LedgerEnclave::finish_connecting_to_store()] method.
     ///
     /// Complete the connection to a Fog Ledger Store that has accepted our
     /// ClientAuthRequest. This is meant to be called after the enclave has
     /// initialized and discovers a new Fog Ledger Store.
-    FinishConnectingToKeyImageStore(
-        ResponderId,
-        ClientAuthResponse,
-    ),
-
-    /// The [LedgerEnclave::create_key_image_store_query()] method.
+    FinishConnectingToKeyImageStore(ResponderId, ClientAuthResponse),
+
+    /// The [LedgerEnclave::decrypt_and_seal_query()] method.
+    ///
+    /// Takes a client query message and returns a SealedClientMessage
+    /// sealed for the current enclave.
+    DecryptAndSealQuery(EnclaveMessage<ClientSession>),
+
+    /// The [LedgerEnclave::create_multi_key_image_store_query()] method.
     ///
     /// Transforms a client query request into a list of query request data.
     ///
     /// The returned list is meant to be used to construct the
     /// MultiKeyImageStoreRequest, which is sent to each shard.
-    CreateKeyImageStoreQuery(EnclaveMessage<ClientSession>),
+    CreateMultiKeyImageStoreQueryData(SealedClientMessage),
+
+    /// Collates shard query responses into a single query response for the
+    /// client.
+    CollateQueryResponses(
+        SealedClientMessage,
+        BTreeMap<ResponderId, EnclaveMessage<ClientSession>>,
+    ),
 
     /// The [LedgerEnclave::handle_key_image_store_request()] method.
     ///
-    /// Used by a Ledger Store to handle an inbound encrypted ledger.proto LedgerRequest. 
-    /// Generally, these come in from a router. 
-    /// This could could be a key image request, a merkele proof 
+    /// Used by a Ledger Store to handle an inbound encrypted ledger.proto
+    /// LedgerRequest. Generally, these come in from a router.
+    /// This could could be a key image request, a merkele proof
     /// request, and potentially in the future an untrusted tx out request.
     HandleKeyImageStoreRequest(EnclaveMessage<ClientSession>),
 }

fog/ledger/enclave/impl/Cargo.toml

@@ -12,6 +12,7 @@ license = "GPL-3.0"
 # mobilecoin
 mc-attest-core = { path = "../../../../attest/core", default-features = false }
 mc-attest-enclave-api = { path = "../../../../attest/enclave-api", default-features = false }
+mc-blockchain-types = { path = "../../../../blockchain/types" }
 mc-common = { path = "../../../../common", default-features = false }
 mc-crypto-ake-enclave = { path = "../../../../crypto/ake/enclave", default-features = false }
 mc-crypto-keys = { path = "../../../../crypto/keys", default-features = false }