Allow multiple files in lrzip

[pete4abw]

A long missed feature of lrzip is the ability to zip more than one file at a time. While tar --use-compress-program=lrzip can work, the compression window gets reduced as a result. lrzip accepts wildcards for compression or decompression, each file is compressed individually. A better way would be to construct a list of files selected - whether with wildcards or multiple file entries on the command line - and add that list of files to an lrz file chaining them one by one, chaining them just like streams blocks are chained.

[demhademha]

+1 for this - I think this is a vital feature missing in lrzip. I'd love to have the ability to compress a directory (with multiple files and sub-directories in it) whilst using the maximum RAM window. This would also ensure that the hierarchy of the directory would be maintained

[pete4abw]

@demhademha writing a journaling archiver is not easy. I did a lot of tests and it turns out that piping tar to lrzip-next

tar -cf - file dir file | lrzip-next -o file.tar.lrz
allows for greater ram usage than using
tar -I lrzip-next -cf file.tar.lrz file dir file.

Consider in the example (with 16GB ram) using a pipe, lrzip-next computes a compression window of 5.2GB. When including lrzip-next in the tar command, only 2.6GB. Using lrzip-next alone allocates 10.5GB. More research is needed on this. But for now, piping tar is the best alternative for maximum memory. However, you can always fool around with the -w ## option when using tar -I 'lrzip -vvw ## ....

$ lrzip -vv software.tar
...
Heuristically Computed Compression Window: 105 = 10500MB

vs

$ tar -cf - /share/software/* | lrzip-next -vv -o software.tar.lrz
Using configuration file /home/peter/.lrzip/lrzip.conf
The following options are in effect for this COMPRESSION.
Threading is ENABLED. Number of CPUs detected: 8
Detected 16560091136 bytes ram
Nice Value: 19
Show Progress
Max Verbose
Output Filename Specified: software.tar.lrz
Temporary Directory set as: ./
Compression mode is: LZMA. LZ4 Compressibility testing enabled
Compression level 7
RZIP Compression level 7
Initial LZMA Dictionary Size: 33554432
Heuristically Computed Compression Window: 52 = 5200MB
Storage time in seconds 1374965312
tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets

vs

$ tar -I 'lrzip-next -vv' -cf software.tar.lrz  /share/software/*
tar: Removing leading `/' from member names
tar: Removing leading `/' from hard link targets
Using configuration file /home/peter/.lrzip/lrzip.conf
The following options are in effect for this COMPRESSION.
Threading is ENABLED. Number of CPUs detected: 8
Detected 16560091136 bytes ram
Nice Value: 19
Show Progress
Max Verbose
Temporary Directory set as: ./
Compression mode is: LZMA. LZ4 Compressibility testing enabled
Compression level 7
RZIP Compression level 7
Initial LZMA Dictionary Size: 33554432
Heuristically Computed Compression Window: 26 = 2600MB
Storage time in seconds 1374965350
Malloced 2760011776 for tmp_outbuf

Thanks for the feedback.

[8465231]

So is the only difference between using the tar -I and piping into LRzip the calculated memory usage? The output file will be the same?

Can you manually set the window size larger to compensate or will that cause issues?

What are the command options for extracting the resulting file? I was testing extracting some large files using the -I option in Tar but ran into some issues with can not extract in memory, using temp file. Naturally that was really slow.

[pete4abw]

@8465231 , YES, you can employ the -w## option to increase the compression window. And YES there is a difference in ram usage when using tar -I "lrzip-next"... and tar ...| lrzip-next -o filename.lrz. The reason is because in the first instance you are using both STDIN and STDOUT with lrzip-next. In the second, lrzip-next is only receiving STDIN but is using a file for output. In the first, lrzip-next will have a compression window 1/6 of total ram. In the second, 1/3. I'm actually working on ways to better determine optimal memory usage, but as you can guess, it's complicated.

See this WIKI page lrzip-next and tar.

Using the tar ...-t option will output a listing of all files in the tarfile.

[8465231]

Cool, that wiki page only shows the -I version of using tar, I tried that and it works but says it could not fit the whole file into ram and then started using a temp file instead of extracting directly to the destination?

Since the piping version is better would I simply use the same piping command but with the decompress option for tar? Would this solve the temp file issue?

Or would I need to decompress with lrzip first and pipe that into tar? Still very new to piping.

[pete4abw]

When I get a chance I'll update the wiki. Basically it would be the same as lrztar and lrzuntar. Take note of the -t option since that will allow you to display the contents of the tarred lrz file.

The tar -I option though is much easier to use.

Good luck.

[8465231]

yes, I noticed with regular lrzip if I used the lrzuntar that it did not use the temp file (also worked quicker with higher CPU usage it seemed), is there a way to find the underlying command that it is using?

Will lrzuntar work with lrzip-next?

[pete4abw]

lrztar and lrzuntar are just script files. Have a look. I believe I updated them to use lrzip-next. Using tar directly is better.

I appreciate your interest in lrzip-next, but I can't spend time doing a tutorial. I wrote the lrzip front end script to help construct command lines for lrzip-next. You can clone it along with this repo (see the README) or visit the site directly.

I am working on other enhancements now, so can't look back. Thanks.

[pete4abw]

@8465231 , see Piping with lrzip-next