Skip to content

Commit

Permalink
Use at most 8 threads for the xz stream
Browse files Browse the repository at this point in the history
At preset 6, xz2 uses about 173MB of memory per thread. This adds up
quickly -- e.g. over 8GB of memory on a 48-CPU machine. If you happen to
try this in a 32-bit build, you'll get `LZMA_MEM_ERROR`.

We can limit this to a heuristic maximum number of threads to avoid
using so much memory, like xz's [`04_compress_easy_mt` example].

    // The number 8 is arbitrarily chosen and may be too low or
    // high depending on the compression preset and the computer
    // being used.

[`04_compress_easy_mt` example]: https://github.com/xz-mirror/xz/blob/de1f47b2b40e960b7bc3acba754f66dd19705921/doc/examples/04_compress_easy_mt.c#L71
  • Loading branch information
cuviper committed Aug 28, 2019
1 parent 85958b0 commit 3d7273c
Showing 1 changed file with 4 additions and 2 deletions.
6 changes: 4 additions & 2 deletions src/tarballer.rs
Original file line number Diff line number Diff line change
Expand Up @@ -47,9 +47,11 @@ impl Tarballer {
// Prepare the `.tar.gz` file.
let gz = GzEncoder::new(create_new_file(tar_gz)?, flate2::Compression::best());

// Prepare the `.tar.xz` file.
// Prepare the `.tar.xz` file. Note that preset 6 takes about 173MB of memory
// per thread, so we limit the number of threads to not blow out 32-bit hosts.
// (We could be more precise with `MtStreamBuilder::memusage()` if desired.)
let stream = xz2::stream::MtStreamBuilder::new()
.threads(num_cpus::get() as u32)
.threads(Ord::min(num_cpus::get(), 8) as u32)
.preset(6)
.encoder()?;
let xz = XzEncoder::new_stream(create_new_file(tar_xz)?, stream);
Expand Down

0 comments on commit 3d7273c

Please sign in to comment.