Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Add JSON format to import blocks and set it as default #5816

Merged
merged 67 commits into from
May 22, 2020
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
0bb1eb7
Add BlockStream Enum and utility fn
Apr 24, 2020
b291ff1
WIP: Modify import closure to work with BlockStream
Apr 24, 2020
49b0144
Fix trait bounds
Apr 24, 2020
cb6aded
Working prototype
Apr 24, 2020
155695d
Revamp block importing
Apr 28, 2020
e59e1ad
Add export_import_flow tests
Apr 28, 2020
7b729fe
Merge branch 'master' of github.com:paritytech/substrate into scott_d…
Apr 28, 2020
69a8bff
Add comments and clean code
Apr 28, 2020
ab173a5
Add more comments in the import fn
Apr 28, 2020
e22ed9b
Add link code to import function
Apr 28, 2020
00fbd6c
Add condition when returning Ready(Ok(()) to make sure we've imported…
Apr 28, 2020
e7a3c4d
Add check for imported blocks in JSON case
Apr 28, 2020
56d49fa
Use rest pattern
Apr 29, 2020
ff75523
Fix compilation error for undeclared variable
Apr 29, 2020
7c66af2
Add polling and waker before pending
Apr 29, 2020
419b0ab
Print read_block_count instead of count
Apr 29, 2020
be1d626
Simplify binary cli option with structopt
Apr 29, 2020
a67a75d
Update test to reflect changes in CLI api
Apr 29, 2020
e1ce1aa
Change Stream to take SignedBlock<B> instead of B
Apr 30, 2020
30f60de
Merge branch 'master' of github.com:paritytech/substrate into scott_d…
Apr 30, 2020
ecf8729
Add comments to BlockStream
Apr 30, 2020
7b8f34c
Move out logic to smaller functions for clearer code
Apr 30, 2020
1a9e910
Merge branch 'master' of github.com:paritytech/substrate into scott_d…
Apr 30, 2020
61260b5
Remove result over import_blocks return type
Apr 30, 2020
4ca8fe6
Check for error in command output rather than simply checking command…
May 1, 2020
c986b44
Revamp export/import/revert testing
May 1, 2020
8b3a17a
Fix minor typos and formatting errors
pscott May 6, 2020
c57415b
Remove unnecessary if condition in terminating condition
pscott May 6, 2020
32eb4af
Explicit error instead of returning it as a string
pscott May 6, 2020
412e942
Pass BlockStream to log_importing_status_updates and simplify matchin…
May 6, 2020
2266d67
Use .contains() instead of regex match
May 6, 2020
f62d5e0
Line break in match block; return future::ready instead of poll_fn
May 6, 2020
ad44445
Update Cargo.lock
May 6, 2020
17f4c74
Add check so that queue doesn't grow too big
May 6, 2020
994157c
Use Iterator instead of Stream
May 7, 2020
4d1234e
Remove allow dead_code
May 7, 2020
37ae67b
Merge branch 'master' of github.com:paritytech/substrate into scott_d…
May 7, 2020
757693d
Remove outdated comments
pscott May 7, 2020
9865fae
Return Errors instead of logging them
May 7, 2020
1150f12
Simplify match arms
pscott May 8, 2020
377823c
Remove check before terminating block import
May 8, 2020
576606c
Apply suggestions from code review
bkchr May 11, 2020
9b4658a
Check that queue is not full BEFORE calling
May 12, 2020
76a5954
Revert "Remove check before terminating block import"
May 12, 2020
f848a05
Improve unit tests to make sure we actually import blocks
May 13, 2020
d924935
Remove Unpin implementation for BlockIter
May 13, 2020
3d00933
Add prototype of enum for ImportStates
May 18, 2020
fb040fc
Add working prototype for StateMachine
May 18, 2020
4f9bcf8
Add comments for clearer code
May 18, 2020
9a134eb
Add sleep before calling Waker when waiting for import queue
May 19, 2020
fdd3683
Merge branch 'master' of github.com:paritytech/substrate into scott_d…
May 19, 2020
d98c92d
Add Speedometer
May 19, 2020
344286e
Merge branch 'master' of github.com:paritytech/substrate into scott_d…
May 20, 2020
5bdc4fd
add dbg!(&log) for test debugging
May 21, 2020
5d1f1e3
Fix lines with more than 100 cols
May 21, 2020
768c309
Merge branch 'master' of github.com:paritytech/substrate into scott_d…
May 21, 2020
a17cc1e
Fix regex capture for test
May 21, 2020
9d7d7a1
Update regexes to take to capture the whole number
May 21, 2020
03a5cc8
Merge branch 'master' of github.com:paritytech/substrate into scott_d…
May 21, 2020
ef02ec3
Rename Cmd to Command
pscott May 21, 2020
fb91c53
Actually rename Cmd to Command
May 21, 2020
4f067ef
Apply suggestions from code review
gnunicorn May 22, 2020
0c47a45
Merge branch 'master' of github.com:paritytech/substrate into scott_d…
May 22, 2020
196de60
Fix compilation errors for tests
May 22, 2020
731f1b7
Merge branch 'scott_default_json_import_blocks' of github.com:parityt…
May 22, 2020
f43f3de
Fix compilation errors from code review suggestion
May 22, 2020
cdc1290
Update bin/node/cli/tests/export_import_flow.rs
gavofyork May 22, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 81 additions & 0 deletions bin/node/cli/tests/export_import_flow.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
// Copyright 2020 Parity Technologies (UK) Ltd.
// This file is part of Substrate.

// Substrate is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.

// Substrate is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.

// You should have received a copy of the GNU General Public License
// along with Substrate. If not, see <http://www.gnu.org/licenses/>.
gnunicorn marked this conversation as resolved.
Show resolved Hide resolved

#![cfg(unix)]

use assert_cmd::cargo::cargo_bin;
use std::{process::Command, fs};
use tempfile::tempdir;

mod common;

#[test]
fn export_default_import_default_works() {
let base_path = tempdir().expect("could not create a temp dir");
let exported_blocks = base_path.path().join("exported_blocks");

common::run_dev_node_for_a_while(base_path.path());

let status = Command::new(cargo_bin("substrate"))
.args(&["export-blocks", "--pruning", "archive", "--dev", "-d"])
.arg(&base_path.path())
.arg(&exported_blocks)
.status()
.unwrap();
assert!(status.success());
pscott marked this conversation as resolved.
Show resolved Hide resolved

let metadata = fs::metadata(&exported_blocks).unwrap();
assert!(metadata.len() > 0, "file exported_blocks should not be empty");

let _ = fs::remove_dir_all(base_path.path().join("db"));

let status = Command::new(cargo_bin("substrate"))
.args(&["import-blocks", "--pruning", "archive", "--dev", "-d"])
.arg(&base_path.path())
.arg(&exported_blocks)
.status()
.unwrap();
assert!(status.success());
}
pscott marked this conversation as resolved.
Show resolved Hide resolved

#[test]
fn export_binary_import_binary_works() {
let base_path = tempdir().expect("could not create a temp dir");
let exported_blocks = base_path.path().join("exported_blocks");

common::run_dev_node_for_a_while(base_path.path());

let status = Command::new(cargo_bin("substrate"))
.args(&["export-blocks", "--pruning", "archive", "--binary", "true", "--dev", "-d"])
.arg(&base_path.path())
.arg(&exported_blocks)
.status()
.unwrap();
assert!(status.success());

let metadata = fs::metadata(&exported_blocks).unwrap();
assert!(metadata.len() > 0, "file exported_blocks should not be empty");

let _ = fs::remove_dir_all(base_path.path().join("db"));

let status = Command::new(cargo_bin("substrate"))
.args(&["import-blocks", "--pruning", "archive", "--binary", "true", "--dev", "-d"])
.arg(&base_path.path())
.arg(&exported_blocks)
.status()
.unwrap();
assert!(status.success());
pscott marked this conversation as resolved.
Show resolved Hide resolved
}
pscott marked this conversation as resolved.
Show resolved Hide resolved
2 changes: 1 addition & 1 deletion client/cli/src/commands/check_block_cmd.rs
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ impl CheckBlockCmd {
};

let start = std::time::Instant::now();
builder(config)?.check_block(block_id).await?;
builder(config)?.check_block(block_id)?.await?;
println!("Completed in {} ms.", start.elapsed().as_millis());

Ok(())
Expand Down
6 changes: 5 additions & 1 deletion client/cli/src/commands/import_blocks_cmd.rs
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,10 @@ pub struct ImportBlocksCmd {
#[structopt(long = "default-heap-pages", value_name = "COUNT")]
pub default_heap_pages: Option<u32>,

/// Try importing blocks from binary format rather than JSON.
#[structopt(long = "binary", value_name = "BOOL", parse(try_from_str), default_value("false"))]
pscott marked this conversation as resolved.
Show resolved Hide resolved
pscott marked this conversation as resolved.
Show resolved Hide resolved
pub binary: bool,

#[allow(missing_docs)]
#[structopt(flatten)]
pub shared_params: SharedParams,
Expand Down Expand Up @@ -77,7 +81,7 @@ impl ImportBlocksCmd {
};

builder(config)?
.import_blocks(file, false)
.import_blocks(file, false, self.binary)?
.await
.map_err(Into::into)
}
Expand Down
5 changes: 3 additions & 2 deletions client/service/src/builder.rs
Original file line number Diff line number Diff line change
Expand Up @@ -755,7 +755,8 @@ pub trait ServiceBuilderCommand {
self,
input: impl Read + Seek + Send + 'static,
force: bool,
) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send>>;
binary: bool,
) -> Result<Pin<Box<dyn Future<Output = Result<(), Error>> + Send>>, Error>;
pscott marked this conversation as resolved.
Show resolved Hide resolved

/// Performs the blocks export.
fn export_blocks(
Expand All @@ -776,7 +777,7 @@ pub trait ServiceBuilderCommand {
fn check_block(
self,
block: BlockId<Self::Block>
) -> Pin<Box<dyn Future<Output = Result<(), Error>> + Send>>;
) -> Result<Pin<Box<dyn Future<Output = Result<(), Error>> + Send>>, Error>;
}

impl<TBl, TRtApi, TBackend, TExec, TSc, TImpQu, TExPool, TRpc>
Expand Down
Loading