Implement digests route for Incremental Cardano DB in aggregator REST API

[jpraynaud]

Why

We need to make the mapping of the immutable files and their digest accessible to the clients in the aggregator REST API. This will enable the verification of the Incremental Cardano DB in the clients.

What

Create a new /artifact/cardano-database/digests route which serves the digests of all the immutable files.

How

• Create a new ImmutableFileDigestRepository SQLite repository which stores the immutable file digests.
• Create a new implementation of ImmutableFileDigestCacheProvider with the ImmutableFileDigestRepository.
• Use this new implementation in the aggregator only.
• Create a new trait ImmutableFileDigestMapper implemented by ImmutableFileDigestRepository which retrieves the map of already computed immutable files and their digests.
• Create a new /artifact/cardano-database/digests route in the REST API that uses the ImmutableFileDigestMapper to provide the map of computed immutable files and their digests.
• Test the route in the end to end test.
• Update OpenAPI specifications.
• Prepare a migration script to avoid long computations on the aggregator:
  • Implement the script.
  • Add manual intervention in the next distribution issue.

Later

• Implement the creation of the digest file and uploaders in the Digest builder

[jpraynaud]

Process to import the immutable file digests from the JSON file.

Connect to the VM of the aggregator, and stop the aggregator:

docker stop mithril-aggregator

Then, prepare the environment variable of the targeted Cardano network:

NETWORK=<CARDANO_NETWORK>

Then, run the following commands to import the immutable file digests from the JSON file in the aggregator database:

IMMUTABLE_FILE_DIGESTS_IMPORT=/home/curry/data/${NETWORK}/mithril-aggregator/mithril/stores/immutables_digests_${NETWORK}.json
DATABASE_PATH=/home/curry/data/${NETWORK}/mithril-aggregator/mithril/stores/aggregator.sqlite3
IMPORT_SQL="
delete from immutable_file_digest;
insert into immutable_file_digest
  select
    json_extract(value, '$.key') as immutable_file_name,
    json_extract(value, '$.value') as digest
  from json_each(readfile('/tmp/immutable_file_digests-import.json'));
"
FETCH_SQL="select * from immutable_file_digest order by rowid asc;"

jq -r 'to_entries'  $IMMUTABLE_FILE_DIGESTS_IMPORT > /tmp/immutable_file_digests-import.json
sqlite3 -table $DATABASE_PATH "$IMPORT_SQL"

And verify that the immutable file digests have been imported correctly:

sqlite3 -table $DATABASE_PATH "$FETCH_SQL"

Then, restart the aggregator:

docker start mithril-aggregator