Sign ancillary files with IOG key

[jpraynaud]

Why

We want to increase the security of the ancillary files we distribute (last immutable files and ledger state snapshot) by providing an additional signature of these files by an IOG owned key.

What

Create a distinct ancillary artifact which includes a signature of the embedded files in the aggregator and verify it in the client

How

• Create a distinct ancillary artifacts archive which includes a signature of the embedded files in the aggregator:
  • Create the main archive with only the certified immutable files
    • Cardano DB v1
    • Cardano DB v2
  • Sign the files in the archive (create multiple implementations of a AncillarySigner trait: AncillarySignerWithSecretKey, AncillarySignerWithGcpKms, and configuration + signature of a manifest file)
    • Cardano DB v1
    • Cardano DB v2
    • Implement AncillarySignerWithGcpKms (based upon asymmetric_sign in gcloud-kms and gcp_auth):
      • Review the implementation of gcloud-kms crate and gcp_auth crate
      • First implementation with gcloud-kms
      • Tests with a mock server for GCP KMS
      • Second implementation based on the gcloud-kms crate (only asymmetric signature)
      • Make a one-time test of the implementation with real credentials of GCP
  • Keep only the last immutable files and the last ledger state snapshot for the ancillary archive creation
    • Cardano DB v1
    • Cardano DB v2
  • Creation of ancillary files and manifest in a separate folder (to avoid files changing during the computation)
    • Cardano DB v1
    • Cardano DB v2
  • Upload the ancillary archive to remote storage
    • Cardano DB v1
    • Cardano DB v2
  • Add a new ancillary location field in the artifact (which only has the Legacy flavor)
    • Cardano DB v1
    • Cardano DB v2
• Verify the ancillary artifacts in the client:
  • By default, download only the main archive and verify it as usual (ledger, volatile and last immutable files are deleted)
    • Cardano DB v1
    • Cardano DB v2
  • Verify the files in the archive if download ancillary option is set (and verification key parameter is set)
    • All the files in the archive must be listed in the artifact (and do not restore those which are not in it)
    • Add a warning message explaining that the signature is not Mithril STM
    • Cardano DB v1
    • Cardano DB v2
  • Adapt the e2e test
    • Cardano DB v1
    • Cardano DB v2
• Update the explorer:
  • Cardano DB v1
  • Cardano DB v2
• Adapt the infrastructure:
  • Publish the new verification keys in the repository (@jpraynaud)
    • testing-preview
    • pre-release-preview
    • release-preprod
    • release-mainnet
  • Create the secret keys on KMS (@jpraynaud)
    • testing-preview
    • pre-release-preview
    • release-preprod
    • release-mainnet
  • Implement the configuration of the aggregator for the signature (@jpraynaud)
  • Use different SSH keys list for testing and production (@jpraynaud)
  • Create a service account with credentials for KMS in CI (@jpraynaud)
    • testing-preview
    • pre-release-preview
    • release-preprod
    • release-mainnet
  • Configure GitHub environments with KMS credentials:(@jpraynaud)
    • testing-preview
    • pre-release-preview
    • release-preprod
    • release-mainnet
  • Prepare KMS keys and credentials rotation policy (@jpraynaud)
• Adapt the documentation:
  • Update the network configurations (@jpraynaud)
  • Update the tutorial(s)
  • Update examples (will be done in Release 2517 distribution #2410)
  • Create a dev blog post (@jpraynaud) (will be done in Release 2517 distribution #2410)