Skip to content

Commit

Permalink
Speed up big chainspec json(~1.5 GB) load (paritytech#10137)
Browse files Browse the repository at this point in the history
* Speed up chainspec json load

* Update client/chain-spec/src/chain_spec.rs

* Update client/chain-spec/src/chain_spec.rs

* Update client/chain-spec/src/chain_spec.rs

* Load the chainspec through `mmap`

Co-authored-by: icodezjb <icodezjb@users.noreply.github.com>
Co-authored-by: Bastian Köcher <bkchr@users.noreply.github.com>
Co-authored-by: Jan Bujak <jan@parity.io>
  • Loading branch information
4 people authored and ark0f committed Feb 27, 2023
1 parent 8ae91d1 commit 0814f6f
Show file tree
Hide file tree
Showing 3 changed files with 25 additions and 4 deletions.
16 changes: 13 additions & 3 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions client/chain-spec/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -22,3 +22,4 @@ serde_json = "1.0.68"
sp-runtime = { version = "4.0.0-dev", path = "../../primitives/runtime" }
sc-telemetry = { version = "4.0.0-dev", path = "../telemetry" }
codec = { package = "parity-scale-codec", version = "2.0.0" }
memmap2 = "0.5.0"
12 changes: 11 additions & 1 deletion client/chain-spec/src/chain_spec.rs
Original file line number Diff line number Diff line change
Expand Up @@ -285,10 +285,20 @@ impl<G, E: serde::de::DeserializeOwned> ChainSpec<G, E> {

/// Parse json file into a `ChainSpec`
pub fn from_json_file(path: PathBuf) -> Result<Self, String> {
// We mmap the file into memory first, as this is *a lot* faster than using
// `serde_json::from_reader`. See https://github.com/serde-rs/json/issues/160
let file = File::open(&path)
.map_err(|e| format!("Error opening spec file `{}`: {}", path.display(), e))?;

// SAFETY: `mmap` is fundamentally unsafe since technically the file can change
// underneath us while it is mapped; in practice it's unlikely to be a problem
let bytes = unsafe {
memmap2::Mmap::map(&file)
.map_err(|e| format!("Error mmaping spec file `{}`: {}", path.display(), e))?
};

let client_spec =
json::from_reader(file).map_err(|e| format!("Error parsing spec file: {}", e))?;
json::from_slice(&bytes).map_err(|e| format!("Error parsing spec file: {}", e))?;
Ok(ChainSpec { client_spec, genesis: GenesisSource::File(path) })
}
}
Expand Down

0 comments on commit 0814f6f

Please sign in to comment.