Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve git-pack encode performance. #600

Open
Hou-Xiaoxuan opened this issue Sep 14, 2024 · 2 comments
Open

Improve git-pack encode performance. #600

Hou-Xiaoxuan opened this issue Sep 14, 2024 · 2 comments

Comments

@Hou-Xiaoxuan
Copy link
Contributor

Hou-Xiaoxuan commented Sep 14, 2024

The current performance of Git pack encoding is poor and takes too long, causing operations like cloning to be excessively slow, as seen in issue #571 . The main time-consuming factor is the zlib compression process.

Possible solutions:

  1. Try alternative compression algorithms/libraries.
  2. The current encoding is sequential, consider changing it to parallel processing.
@Hou-Xiaoxuan Hou-Xiaoxuan changed the title Improve pack encode performance. Improve git-pack encode performance. Sep 14, 2024
@Hou-Xiaoxuan Hou-Xiaoxuan reopened this Sep 21, 2024
@genedna
Copy link
Member

genedna commented Sep 22, 2024

@Hou-Xiaoxuan ,

Maby try miniz_oxide, a pure rust replacement for the miniz deflate/zlib encoder/decoder using no unsafe code. It builds in no_std mode, though it requires alloc and collections crates.

@el-ev
Copy link
Contributor

el-ev commented Dec 14, 2024

Most of the encoding time is spent on compressing, and we can do little about it. Is it acceptable to lower the compression level or add a option for the user to decide it?

mercury/src/internal/pack/encode.rs:encode_one_object:

 let mut inflate = ZlibEncoder::new(Vec::new(), flate2::Compression::default()); // change to flate2::Compression::fast()

edit: git defaults to 6, same with mercury, so the former idea seems bad...

pack.compression
An integer -1..9, indicating the compression level for objects in a pack file. -1 is the zlib default. 0 means no compression, and 1..9 are various speed/size tradeoffs, 9 being slowest. If not set, defaults to core.compression. If that is not set, defaults to -1, the zlib default, which is "a default compromise between speed and compression (currently equivalent to level 6)."

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants