Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: incremental cardano database restoration with client CLI #2318

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
c85f98a
fix(client): use unique temp directory to avoid collision between tests
dlachaume Feb 21, 2025
3701fdb
feat(client-cli): scaffold `download` command for `cardano-db-v2`
dlachaume Feb 3, 2025
103a31e
refactor(cli): separate disk space verifications and prerequisite checks
dlachaume Feb 4, 2025
676de3b
feat: add `--start`, `--end`, `--include_ancillary` and `--allow_over…
dlachaume Feb 3, 2025
56ae41c
feat: implement full restoration process for incremental Cardano data…
dlachaume Feb 13, 2025
c0706eb
feat(client): implement aggregator statistics updates for Cardano dat…
dlachaume Feb 14, 2025
c532c5c
feat(cli): update restoration metrics by calling the client library
dlachaume Feb 17, 2025
0c4d882
feat(cli): add Cardano database snapshot restoration events in the fe…
dlachaume Feb 18, 2025
b96daff
feat(client): add `size` variant for `DigestDownloadStarted` event
dlachaume Feb 19, 2025
74b577b
feat(client): add started and completed event builders to `DownloadEv…
dlachaume Feb 19, 2025
c6b631c
refactor(client): moving the responsibility of event sending from the…
dlachaume Feb 19, 2025
9efa777
feat(cli): add size to `DigestDownloadStarted` event
dlachaume Feb 19, 2025
4faaf01
feat(cli): create a `MultiDownloadProgressReporter` that can handle m…
dlachaume Feb 19, 2025
961ca8d
feat(cli): implement `finish_download` and `finish_all` in the `Multi…
dlachaume Feb 20, 2025
1da90a4
feat(cli): implement `progress_download` for `MultiDownloadProgressRe…
dlachaume Feb 20, 2025
d550f84
feat(cli): allow `ProgressBarJsonFormatter` to handle both files and …
dlachaume Feb 20, 2025
3834e18
feat(cli): use the appropriate progress bar kind
dlachaume Feb 20, 2025
baecfa6
feat(cli): wire with the Cardano database events with the `MultiDownl…
dlachaume Feb 20, 2025
a4a75f8
refactor(cli): add a child bar only for ancillary file download
dlachaume Feb 21, 2025
b67ea25
feat(cli): make the display of labels optional in TTY output
dlachaume Feb 21, 2025
5ff6516
ci: verify cardano-db-v2 `download` command in the E2E test
dlachaume Feb 4, 2025
bcbab39
ci: add new verification steps for `cardano-db-v2 download` command i…
dlachaume Feb 4, 2025
f339548
chore: minor updates from PR reviews
dlachaume Feb 24, 2025
d7976db
chore: reference the feature in the CHANGELOG
dlachaume Feb 24, 2025
4126db1
refactor(cli): extract path conversion logic into dedicated function
dlachaume Feb 24, 2025
a9c87ee
chore: upgrade crate versions
dlachaume Feb 25, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/workflows/test-client.yml
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,12 @@ jobs:
working-directory: ./bin
run: ./mithril-client --unstable cardano-db-v2 snapshot show $CARDANO_DATABASE_V2_SNAPSHOT_HASH

- name: Cardano Database V2 Snapshot / download & restore latest (Full restoration)
if: steps.aggregator_capability_unix.outputs.cardano_database_v2_enabled == 'true' || steps.aggregator_capability_windows.outputs.cardano_database_v2_enabled == 'true'
shell: bash
working-directory: ./bin
run: ./mithril-client ${{ steps.prepare.outputs.debug_level }} --unstable cardano-db-v2 download $CARDANO_DATABASE_V2_SNAPSHOT_HASH

test-docker:
strategy:
fail-fast: false
Expand Down Expand Up @@ -328,6 +334,11 @@ jobs:
shell: bash
run: ${{ steps.command.outputs.mithril_client }} --unstable cardano-db-v2 snapshot show $CARDANO_DATABASE_V2_SNAPSHOT_HASH

- name: Cardano Database V2 Snapshot / download & restore latest (Full restoration)
if: steps.aggregator_capability.outputs.cardano_database_v2_enabled == 'true'
shell: bash
run: ${{ steps.command.outputs.mithril_client }} ${{ steps.prepare.outputs.debug_level }} --unstable cardano-db-v2 download $CARDANO_DATABASE_V2_SNAPSHOT_HASH --download-dir /app

test-mithril-client-wasm:
strategy:
fail-fast: false
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ As a minor extension, we have adopted a slightly different versioning convention
- **UNSTABLE** Cardano database incremental certification:

- Implement the client library for the the signed entity type `CardanoDatabase` (download and prove snapshot).
- Implement the client CLI commands for the signed entity type `CardanoDatabase` (snapshot list, snapshot show and download commands).

- Crates versions:

Expand Down
6 changes: 3 additions & 3 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion mithril-client-cli/Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "mithril-client-cli"
version = "0.11.2"
version = "0.11.3"
description = "A Mithril Client"
authors = { workspace = true }
edition = { workspace = true }
Expand Down
18 changes: 6 additions & 12 deletions mithril-client-cli/src/commands/cardano_db/download.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ use crate::{
commands::{client_builder, SharedArgs},
configuration::{ConfigError, ConfigSource},
utils::{
CardanoDbDownloadChecker, CardanoDbUtils, ExpanderUtils, IndicatifFeedbackReceiver,
self, CardanoDbDownloadChecker, CardanoDbUtils, ExpanderUtils, IndicatifFeedbackReceiver,
ProgressOutputType, ProgressPrinter,
},
CommandContext,
Expand Down Expand Up @@ -149,7 +149,7 @@ impl CardanoDbDownloadCommand {
progress_printer.report_step(step_number, "Checking local disk info…")?;

CardanoDbDownloadChecker::ensure_dir_exist(db_dir)?;
if let Err(e) = CardanoDbDownloadChecker::check_prerequisites(
if let Err(e) = CardanoDbDownloadChecker::check_prerequisites_for_archive(
db_dir,
cardano_db.size,
cardano_db.compression_algorithm,
Expand Down Expand Up @@ -319,17 +319,11 @@ impl ConfigSource for CardanoDbDownloadCommand {
let mut map = HashMap::new();

if let Some(download_dir) = self.download_dir.clone() {
let param = "download_dir".to_string();
map.insert(
"download_dir".to_string(),
download_dir
.to_str()
.ok_or_else(|| {
ConfigError::Conversion(format!(
"Could not read download directory: '{}'.",
download_dir.display()
))
})?
.to_string(),
param.clone(),
utils::path_to_string(&download_dir)
.map_err(|e| ConfigError::Conversion(param, e))?,
);
}

Expand Down
Loading