Synchronize certificate chain of follower aggregator

[jpraynaud]

Why

We want the follower aggregator to retrieve an existing certificate chain from the leader aggregator when it does not have a valid certificate chain (empty or invalid). We need the mechanism to be able to detect that a re-genesis has occurred on the leader chain.

What

Implement a synchronization mechanism for the follower certificate chain from the leader certificate chain.

How

• Synchronize the certificate chain from the leader when it is empty or invalid:
  • Add a route to get the latest genesis certificate in the aggregator /certificate/genesis
    • Extend the MessageService to retrieve the last genesis certificate (the most recent genesis certificate if multiple in the store)
    • The route must be declared before the /certificate/{certificate_hash} route in the warp router declaration
    • Implement a certificate_certificate_genesis handler (increment the metric certificate_detail_total_served_since_startup)
    • Update OpenAPI
  • Create a new RemoteCertificateRetriever trait in the aggregator (see code below)
  • Implement the RemoteCertificateRetriever trait with reqwest (by pulling the aggregator certificate routes)
  • Create a new CertificateChainSynchronizer trait that fetches a full certificate chain from the leader aggregator
  • Implement the RemoteCertificateChainSynchronizer for the CertificateChainSynchronizer trait
  • When the synchronization has taken place, the RemoteCertificateChainSynchronizer creates an entry on the open message for the MithrilStakeDistribution of the current epoch and mark it has certified (does not happen in case of error)
• In the state machine of the follower aggregator, when transitioning from IDLE to READY, run the synchronization of the certificate chain:
  • If the certificate chain in invalid
  • Or if the remote certificate genesis has changed
  • With the code snippet in async fn try_transition_from_idle_to_ready (see code below)
• Add an integration test
• Update the e2e test so that follower aggregators do not bootstrap their own genesis certificate

pub trait RemoteCertificateRetriever: Sync + Send {
    /// Get [Certificate] details
    async fn get_certificate_details(
        &self,
        certificate_hash: &str,
    ) -> Result<Certificate, CertificateRetrieverError>;
   
   // Get latest certificate
  async fn get_latest_certificate_details(
        &self,
    ) -> Result<Certificate, CertificateRetrieverError>;

  // Get genesis certificate
  async fn get_genesis_certificate_details(
        &self,
    ) -> Result<Certificate, CertificateRetrieverError>
}

      let res = self
            .runner
            .is_certificate_chain_valid(&new_time_point)
            .await
            .map_err(|e| RuntimeError::KeepState {
                message: "certificate chain is invalid".to_string(),
                nested_error: e.into(),
            })?;

        if self.config.is_follower {
            self.runner
                .synchronize_follower_aggregator_certificate_chain(res)
                .await?;
        }

        res?;