Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Speed up big chainspec json(~1.5 GB) load #10137

Merged
merged 5 commits into from
Nov 1, 2021
Merged

Speed up big chainspec json(~1.5 GB) load #10137

Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
3457a6b
Speed up chainspec json load
icodezjb Nov 1, 2021
4b7db48
Update client/chain-spec/src/chain_spec.rs
bkchr Nov 1, 2021
3b53ac7
Update client/chain-spec/src/chain_spec.rs
bkchr Nov 1, 2021
a2c7abd
Update client/chain-spec/src/chain_spec.rs
bkchr Nov 1, 2021
91d2a48
Load the chainspec through `mmap`
koute Nov 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions client/chain-spec/src/chain_spec.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -285,10 +285,12 @@ impl<G, E: serde::de::DeserializeOwned> ChainSpec<G, E> {

/// Parse json file into a `ChainSpec`
pub fn from_json_file(path: PathBuf) -> Result<Self, String> {
let file = File::open(&path)
.map_err(|e| format!("Error opening spec file `{}`: {}", path.display(), e))?;
// We read the entire file into memory first, as this is *a lot* faster than using
// `serde_json::from_reader`. See https://github.com/serde-rs/json/issues/160
let bytes = std::fs::read(&path).map_err(|e| format!("Error reading spec file: {}", e))?;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It'd most likely be better to mmap the file than to read it whole into memory (especially if it can be a few gigs in size). Could you try mmaping it through the memmap2 crate (we already use it transitively as a dependency through parity-db anyway) instead? (It should be roughly as fast, but at a significantly lower memory usage.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean having multiple gigabytes in this file is rather rare, however I'm fine with doing these change if we are going to change this here anyway. @koute could you maybe push the required changes to this pr?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, I know. Still, apparently some people do it, as evidenced by this PR. (:

Sure; done!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ty, if you now approve your own changes @koute we should be ready with this pr :D


let client_spec =
json::from_reader(file).map_err(|e| format!("Error parsing spec file: {}", e))?;
json::from_slice(&bytes).map_err(|e| format!("Error parsing spec file: {}", e))?;
Ok(ChainSpec { client_spec, genesis: GenesisSource::File(path) })
}
}
Expand Down