A cross platform, fast, compression and decompression tool.
This is currently a proof of concept CLI tool using the gzp
crate.
Supported formats:
- Gzip
- Zlib
- Mgzip
- BGZF
- Raw Deflate
- Snap
- Homebrew / Linuxbrew
brew tap sstadick/crabz
brew install crabz
- Debian (Ubuntu)
curl -LO https://github.com/sstadick/crabz/releases/download/<latest>/crabz-linux-amd64.deb
sudo dpkg -i crabz-linux-amd64.deb
- Cargo
cargo install crabz
- Conda
conda install -c conda-forge crabz
❯ crabz -h
Compress and decompress files
USAGE:
crabz [FLAGS] [OPTIONS] [FILE]
FLAGS:
-d, --decompress
Flag to switch to decompressing inputs. Note: this flag may change in future releases
-h, --help
Prints help information
-I, --in-place
Perform the compression / decompression in place.
**NOTE** this will remove the input file at completion.
-V, --version
Prints version information
OPTIONS:
-l, --compression-level <compression-level>
Compression level [default: 6]
-p, --compression-threads <compression-threads>
Number of compression threads to use, or if decompressing a format that allow for multi-threaded
decompression, the number to use. Note that > 4 threads for decompression doesn't seem to help [default:
32]
-f, --format <format>
The format to use [default: gzip] [possible values: gzip, bgzf, mgzip,
zlib, deflate, snap]
-o, --output <output>
Output path to write to, empty or "-" to write to stdout
-P, --pin-at <pin-at>
Specify the physical core to pin threads at.
This can provide a significant performance improvement, but has the downside of possibly conflicting with
other pinned cores. If you are running multiple instances of `crabz` at once you can manually space out the
pinned cores.
# Example
- Instance 1 has `-p 4 -P 0` set indicating that it will use 4 cores pinned at 0, 1, 2, 3
- Instance 2 has `-p 4 -P 4` set indicating that it will use 4 cores pinned at 4, 5, 6, 7
ARGS:
<FILE>
Input file to read from, empty or "-" to read from stdin
These benchmarks use the data in bench-data
catted together 100 times. Run with bash ./benchmark.sh data.txt
.
Benchmark system specs: Ubuntu 20 AMD Ryzen 9 3950X 16-Core Processor w/ 64 GB DDR4 memory and 1TB NVMe Drive
pigz
v2.4 installed via apt on Ubuntu
Takeaways:
crabz
withzlib
backend is pretty much identical topigz
crabz
withzlib-ng
backend is roughly 30-50% faster thanpigz
crabz
withrust
backend is roughly 5-10% faster thanpigz
It is already known that zlib-ng
is faster than zlib
, so none of this is groundbreaking. However, I think crabz
gets an
an edge due to the following:
crabz
withdeflate_rust
backend is using all Rust only code, which is in theory more secure / safe.crabz
withzlib-ng
is easier to install thanpigz
with azlib-ng
backendcrabz
supports more formats thanpigz
crabz
is cross platform and can run on windows
With regards to block formats like Mgzip and BGZF, crabz
is using libdeflater
by default which excels at compressing and
decompression known-sized blocks. This makes block compression formats very fast at a small loss to the compression ratio.
See end of benchmarks section for comparison against bgzip
.
As crabz
is just a wrapper for the gzp
library, the most exciting thing about these benchmarks is that gzp
is on par with
best in class CLI tools for multi-threaded compression and decompression as a library.
Command | Mean [s] | Min [s] | Max [s] | Relative |
---|---|---|---|---|
crabz -p 1 -c 3 < ./data.txt |
6.450 ± 0.069 | 6.328 | 6.540 | 16.86 ± 0.24 |
pigz -p 1 -3 < ./data.txt |
11.404 ± 0.152 | 11.186 | 11.717 | 29.81 ± 0.49 |
crabz -p 2 -c 3 < ./data.txt |
3.437 ± 0.017 | 3.418 | 3.461 | 8.98 ± 0.10 |
pigz -p 2 -3 < ./data.txt |
5.868 ± 0.031 | 5.826 | 5.927 | 15.34 ± 0.17 |
crabz -p 4 -c 3 < ./data.txt |
1.741 ± 0.008 | 1.729 | 1.752 | 4.55 ± 0.05 |
pigz -p 4 -3 < ./data.txt |
2.952 ± 0.008 | 2.939 | 2.960 | 7.72 ± 0.08 |
crabz -p 8 -c 3 < ./data.txt |
0.889 ± 0.004 | 0.882 | 0.895 | 2.32 ± 0.02 |
pigz -p 8 -3 < ./data.txt |
1.505 ± 0.008 | 1.493 | 1.520 | 3.93 ± 0.04 |
crabz -p 16 -c 3 < ./data.txt |
0.485 ± 0.014 | 0.457 | 0.502 | 1.27 ± 0.04 |
pigz -p 16 -3 < ./data.txt |
0.775 ± 0.011 | 0.764 | 0.797 | 2.02 ± 0.04 |
crabz -p 32 -c 3 < ./data.txt |
0.383 ± 0.004 | 0.375 | 0.388 | 1.00 |
pigz -p 32 -3 < ./data.txt |
0.699 ± 0.029 | 0.668 | 0.770 | 1.83 ± 0.08 |
crabz -p 1 -c 6 < ./data.txt |
10.367 ± 0.211 | 10.106 | 10.642 | 27.10 ± 0.61 |
pigz -p 1 -6 < ./data.txt |
26.734 ± 0.345 | 26.234 | 27.135 | 69.89 ± 1.12 |
crabz -p 2 -c 6 < ./data.txt |
5.366 ± 0.036 | 5.299 | 5.429 | 14.03 ± 0.16 |
pigz -p 2 -6 < ./data.txt |
13.589 ± 0.083 | 13.428 | 13.679 | 35.52 ± 0.40 |
crabz -p 4 -c 6 < ./data.txt |
2.719 ± 0.021 | 2.694 | 2.757 | 7.11 ± 0.09 |
pigz -p 4 -6 < ./data.txt |
6.887 ± 0.013 | 6.871 | 6.916 | 18.00 ± 0.17 |
crabz -p 8 -c 6 < ./data.txt |
1.381 ± 0.007 | 1.372 | 1.397 | 3.61 ± 0.04 |
pigz -p 8 -6 < ./data.txt |
3.479 ± 0.008 | 3.463 | 3.488 | 9.09 ± 0.09 |
crabz -p 16 -c 6 < ./data.txt |
0.745 ± 0.022 | 0.727 | 0.804 | 1.95 ± 0.06 |
pigz -p 16 -6 < ./data.txt |
1.818 ± 0.036 | 1.765 | 1.874 | 4.75 ± 0.10 |
crabz -p 32 -c 6 < ./data.txt |
0.549 ± 0.006 | 0.538 | 0.557 | 1.44 ± 0.02 |
pigz -p 32 -6 < ./data.txt |
1.187 ± 0.011 | 1.172 | 1.210 | 3.10 ± 0.04 |
crabz -p 1 -c 9 < ./data.txt |
30.114 ± 0.196 | 29.842 | 30.420 | 78.72 ± 0.90 |
pigz -p 1 -9 < ./data.txt |
51.369 ± 0.164 | 51.246 | 51.698 | 134.29 ± 1.33 |
crabz -p 2 -c 9 < ./data.txt |
15.371 ± 0.070 | 15.202 | 15.443 | 40.18 ± 0.42 |
pigz -p 2 -9 < ./data.txt |
26.452 ± 0.085 | 26.253 | 26.576 | 69.15 ± 0.69 |
crabz -p 4 -c 9 < ./data.txt |
7.729 ± 0.022 | 7.699 | 7.768 | 20.20 ± 0.20 |
pigz -p 4 -9 < ./data.txt |
13.365 ± 0.047 | 13.271 | 13.449 | 34.94 ± 0.35 |
crabz -p 8 -c 9 < ./data.txt |
3.901 ± 0.006 | 3.889 | 3.910 | 10.20 ± 0.10 |
pigz -p 8 -9 < ./data.txt |
6.749 ± 0.014 | 6.737 | 6.781 | 17.64 ± 0.17 |
crabz -p 16 -c 9 < ./data.txt |
2.039 ± 0.024 | 1.997 | 2.071 | 5.33 ± 0.08 |
pigz -p 16 -9 < ./data.txt |
3.486 ± 0.054 | 3.426 | 3.574 | 9.11 ± 0.17 |
crabz -p 32 -c 9 < ./data.txt |
1.337 ± 0.072 | 1.220 | 1.411 | 3.49 ± 0.19 |
pigz -p 32 -9 < ./data.txt |
2.203 ± 0.114 | 2.082 | 2.378 | 5.76 ± 0.30 |
Command | Mean [s] | Min [s] | Max [s] | Relative |
---|---|---|---|---|
crabz -d < ./data.3.txt.gz |
1.422 ± 0.010 | 1.411 | 1.437 | 1.03 ± 0.02 |
pigz -d < ./data.3.txt.gz |
1.674 ± 0.031 | 1.621 | 1.705 | 1.21 ± 0.03 |
crabz -d < ./data.6.txt.gz |
1.403 ± 0.016 | 1.389 | 1.427 | 1.01 ± 0.02 |
pigz -d < ./data.6.txt.gz |
1.724 ± 0.026 | 1.697 | 1.766 | 1.24 ± 0.02 |
crabz -d < ./data.9.txt.gz |
1.385 ± 0.018 | 1.359 | 1.416 | 1.00 |
pigz -d < ./data.9.txt.gz |
1.745 ± 0.044 | 1.684 | 1.797 | 1.26 ± 0.04 |
Command | Mean [s] | Min [s] | Max [s] | Relative |
---|---|---|---|---|
crabz -p 1 -c 3 < ./data.txt |
11.248 ± 0.247 | 11.085 | 11.532 | 20.23 ± 0.45 |
pigz -p 1 -3 < ./data.txt |
11.296 ± 0.171 | 11.104 | 11.434 | 20.32 ± 0.31 |
crabz -p 2 -c 3 < ./data.txt |
5.681 ± 0.040 | 5.645 | 5.725 | 10.22 ± 0.08 |
pigz -p 2 -3 < ./data.txt |
5.926 ± 0.015 | 5.916 | 5.944 | 10.66 ± 0.04 |
crabz -p 4 -c 3 < ./data.txt |
2.891 ± 0.007 | 2.883 | 2.895 | 5.20 ± 0.02 |
pigz -p 4 -3 < ./data.txt |
2.966 ± 0.013 | 2.955 | 2.980 | 5.34 ± 0.03 |
crabz -p 8 -c 3 < ./data.txt |
1.461 ± 0.003 | 1.459 | 1.465 | 2.63 ± 0.01 |
pigz -p 8 -3 < ./data.txt |
1.509 ± 0.004 | 1.505 | 1.512 | 2.71 ± 0.01 |
crabz -p 16 -c 3 < ./data.txt |
0.784 ± 0.010 | 0.775 | 0.795 | 1.41 ± 0.02 |
pigz -p 16 -3 < ./data.txt |
0.772 ± 0.010 | 0.765 | 0.784 | 1.39 ± 0.02 |
crabz -p 32 -c 3 < ./data.txt |
0.556 ± 0.002 | 0.554 | 0.557 | 1.00 |
pigz -p 32 -3 < ./data.txt |
0.743 ± 0.047 | 0.694 | 0.786 | 1.34 ± 0.08 |
crabz -p 1 -c 6 < ./data.txt |
26.366 ± 0.154 | 26.189 | 26.469 | 47.42 ± 0.31 |
pigz -p 1 -6 < ./data.txt |
26.688 ± 0.103 | 26.579 | 26.783 | 48.00 ± 0.23 |
crabz -p 2 -c 6 < ./data.txt |
13.443 ± 0.069 | 13.400 | 13.523 | 24.18 ± 0.14 |
pigz -p 2 -6 < ./data.txt |
13.605 ± 0.059 | 13.567 | 13.673 | 24.47 ± 0.13 |
crabz -p 4 -c 6 < ./data.txt |
6.833 ± 0.005 | 6.828 | 6.837 | 12.29 ± 0.03 |
pigz -p 4 -6 < ./data.txt |
6.866 ± 0.028 | 6.834 | 6.884 | 12.35 ± 0.06 |
crabz -p 8 -c 6 < ./data.txt |
3.446 ± 0.000 | 3.445 | 3.446 | 6.20 ± 0.02 |
pigz -p 8 -6 < ./data.txt |
3.482 ± 0.002 | 3.480 | 3.483 | 6.26 ± 0.02 |
crabz -p 16 -c 6 < ./data.txt |
1.822 ± 0.012 | 1.813 | 1.835 | 3.28 ± 0.02 |
pigz -p 16 -6 < ./data.txt |
1.771 ± 0.004 | 1.767 | 1.776 | 3.19 ± 0.01 |
crabz -p 32 -c 6 < ./data.txt |
1.178 ± 0.008 | 1.171 | 1.187 | 2.12 ± 0.02 |
pigz -p 32 -6 < ./data.txt |
1.184 ± 0.001 | 1.184 | 1.185 | 2.13 ± 0.01 |
crabz -p 1 -c 9 < ./data.txt |
52.122 ± 0.288 | 51.790 | 52.293 | 93.75 ± 0.58 |
pigz -p 1 -9 < ./data.txt |
53.031 ± 0.071 | 52.951 | 53.085 | 95.39 ± 0.29 |
crabz -p 2 -c 9 < ./data.txt |
26.287 ± 0.047 | 26.249 | 26.339 | 47.28 ± 0.15 |
pigz -p 2 -9 < ./data.txt |
26.409 ± 0.238 | 26.190 | 26.662 | 47.50 ± 0.45 |
crabz -p 4 -c 9 < ./data.txt |
13.373 ± 0.051 | 13.317 | 13.419 | 24.05 ± 0.11 |
pigz -p 4 -9 < ./data.txt |
13.414 ± 0.035 | 13.383 | 13.451 | 24.13 ± 0.09 |
crabz -p 8 -c 9 < ./data.txt |
6.733 ± 0.003 | 6.731 | 6.736 | 12.11 ± 0.03 |
pigz -p 8 -9 < ./data.txt |
6.763 ± 0.004 | 6.761 | 6.767 | 12.16 ± 0.03 |
crabz -p 16 -c 9 < ./data.txt |
3.487 ± 0.034 | 3.450 | 3.517 | 6.27 ± 0.06 |
pigz -p 16 -9 < ./data.txt |
3.459 ± 0.021 | 3.434 | 3.473 | 6.22 ± 0.04 |
crabz -p 32 -c 9 < ./data.txt |
2.088 ± 0.008 | 2.081 | 2.097 | 3.76 ± 0.02 |
pigz -p 32 -9 < ./data.txt |
2.107 ± 0.023 | 2.090 | 2.133 | 3.79 ± 0.04 |
Command | Mean [s] | Min [s] | Max [s] | Relative |
---|---|---|---|---|
crabz -p 1 -c 3 < ./data.txt |
10.167 ± 0.164 | 10.050 | 10.355 | 18.57 ± 0.33 |
pigz -p 1 -3 < ./data.txt |
11.338 ± 0.071 | 11.292 | 11.420 | 20.71 ± 0.21 |
crabz -p 2 -c 3 < ./data.txt |
4.912 ± 0.013 | 4.898 | 4.920 | 8.97 ± 0.08 |
pigz -p 2 -3 < ./data.txt |
5.876 ± 0.047 | 5.826 | 5.919 | 10.73 ± 0.12 |
crabz -p 4 -c 3 < ./data.txt |
2.463 ± 0.018 | 2.447 | 2.482 | 4.50 ± 0.05 |
pigz -p 4 -3 < ./data.txt |
2.967 ± 0.008 | 2.958 | 2.972 | 5.42 ± 0.05 |
crabz -p 8 -c 3 < ./data.txt |
1.255 ± 0.005 | 1.250 | 1.261 | 2.29 ± 0.02 |
pigz -p 8 -3 < ./data.txt |
1.509 ± 0.002 | 1.507 | 1.511 | 2.76 ± 0.02 |
crabz -p 16 -c 3 < ./data.txt |
0.705 ± 0.030 | 0.673 | 0.731 | 1.29 ± 0.05 |
pigz -p 16 -3 < ./data.txt |
0.780 ± 0.015 | 0.768 | 0.797 | 1.42 ± 0.03 |
crabz -p 32 -c 3 < ./data.txt |
0.547 ± 0.004 | 0.544 | 0.552 | 1.00 |
pigz -p 32 -3 < ./data.txt |
0.755 ± 0.025 | 0.726 | 0.771 | 1.38 ± 0.05 |
crabz -p 1 -c 6 < ./data.txt |
27.064 ± 0.288 | 26.863 | 27.394 | 49.44 ± 0.66 |
pigz -p 1 -6 < ./data.txt |
27.034 ± 0.090 | 26.938 | 27.117 | 49.38 ± 0.43 |
crabz -p 2 -c 6 < ./data.txt |
12.400 ± 0.083 | 12.321 | 12.487 | 22.65 ± 0.24 |
pigz -p 2 -6 < ./data.txt |
13.619 ± 0.074 | 13.558 | 13.702 | 24.88 ± 0.24 |
crabz -p 4 -c 6 < ./data.txt |
6.279 ± 0.023 | 6.263 | 6.305 | 11.47 ± 0.10 |
pigz -p 4 -6 < ./data.txt |
6.879 ± 0.020 | 6.867 | 6.901 | 12.57 ± 0.11 |
crabz -p 8 -c 6 < ./data.txt |
3.189 ± 0.010 | 3.178 | 3.198 | 5.83 ± 0.05 |
pigz -p 8 -6 < ./data.txt |
3.477 ± 0.007 | 3.470 | 3.483 | 6.35 ± 0.05 |
crabz -p 16 -c 6 < ./data.txt |
1.756 ± 0.015 | 1.740 | 1.771 | 3.21 ± 0.04 |
pigz -p 16 -6 < ./data.txt |
1.799 ± 0.024 | 1.779 | 1.827 | 3.29 ± 0.05 |
crabz -p 32 -c 6 < ./data.txt |
1.192 ± 0.011 | 1.183 | 1.205 | 2.18 ± 0.03 |
pigz -p 32 -6 < ./data.txt |
1.196 ± 0.016 | 1.183 | 1.214 | 2.19 ± 0.03 |
crabz -p 1 -c 9 < ./data.txt |
44.907 ± 0.283 | 44.585 | 45.116 | 82.03 ± 0.84 |
pigz -p 1 -9 < ./data.txt |
53.109 ± 1.049 | 52.373 | 54.311 | 97.02 ± 2.07 |
crabz -p 2 -c 9 < ./data.txt |
19.977 ± 0.159 | 19.819 | 20.136 | 36.49 ± 0.41 |
pigz -p 2 -9 < ./data.txt |
26.562 ± 0.134 | 26.407 | 26.643 | 48.52 ± 0.46 |
crabz -p 4 -c 9 < ./data.txt |
10.397 ± 0.484 | 10.070 | 10.953 | 18.99 ± 0.90 |
pigz -p 4 -9 < ./data.txt |
13.346 ± 0.040 | 13.300 | 13.372 | 24.38 ± 0.21 |
crabz -p 8 -c 9 < ./data.txt |
5.100 ± 0.021 | 5.076 | 5.114 | 9.32 ± 0.08 |
pigz -p 8 -9 < ./data.txt |
6.754 ± 0.016 | 6.736 | 6.767 | 12.34 ± 0.10 |
crabz -p 16 -c 9 < ./data.txt |
2.716 ± 0.014 | 2.708 | 2.732 | 4.96 ± 0.05 |
pigz -p 16 -9 < ./data.txt |
3.444 ± 0.038 | 3.420 | 3.487 | 6.29 ± 0.09 |
crabz -p 32 -c 9 < ./data.txt |
1.747 ± 0.009 | 1.740 | 1.758 | 3.19 ± 0.03 |
pigz -p 32 -9 < ./data.txt |
2.086 ± 0.008 | 2.077 | 2.093 | 3.81 ± 0.03 |
Command | Mean [s] | Min [s] | Max [s] | Relative |
---|---|---|---|---|
crabz -d < ./data.3.txt.gz |
1.599 ± 0.014 | 1.573 | 1.615 | 1.00 |
pigz -d < ./data.3.txt.gz |
1.696 ± 0.020 | 1.654 | 1.725 | 1.06 ± 0.02 |
crabz -d < ./data.6.txt.gz |
1.615 ± 0.012 | 1.586 | 1.626 | 1.01 ± 0.01 |
pigz -d < ./data.6.txt.gz |
1.760 ± 0.030 | 1.687 | 1.797 | 1.10 ± 0.02 |
crabz -d < ./data.9.txt.gz |
1.613 ± 0.014 | 1.596 | 1.641 | 1.01 ± 0.01 |
pigz -d < ./data.9.txt.gz |
1.767 ± 0.012 | 1.748 | 1.787 | 1.11 ± 0.01 |
Command | Mean [s] | Min [s] | Max [s] | Relative |
---|---|---|---|---|
crabz -p 1 -d -f mgzip ./bdata.3.txt.gz > data.txt |
1.221 ± 0.164 | 1.073 | 1.397 | 2.32 ± 0.31 |
pigz -d -c ./bdata.3.txt.gz > data.txt |
2.415 ± 0.063 | 2.347 | 2.472 | 4.58 ± 0.14 |
crabz -p 1 -d -f mgzip ./bdata.6.txt.gz > data.txt |
1.256 ± 0.063 | 1.200 | 1.325 | 2.38 ± 0.13 |
pigz -d -c ./bdata.6.txt.gz > data.txt |
2.513 ± 0.052 | 2.467 | 2.569 | 4.77 ± 0.13 |
crabz -p 1 -d -f mgzip ./bdata.9.txt.gz > data.txt |
1.147 ± 0.065 | 1.094 | 1.219 | 2.18 ± 0.13 |
pigz -d -c ./bdata.9.txt.gz > data.txt |
2.394 ± 0.118 | 2.262 | 2.488 | 4.54 ± 0.24 |
crabz -p 1 -d -f mgzip ./bdata.12.txt.gz > data.txt |
1.165 ± 0.074 | 1.106 | 1.248 | 2.21 ± 0.15 |
pigz -d -c ./bdata.12.txt.gz > data.txt |
2.457 ± 0.067 | 2.408 | 2.534 | 4.66 ± 0.15 |
crabz -p 2 -d -f mgzip ./bdata.3.txt.gz > data.txt |
0.634 ± 0.008 | 0.628 | 0.642 | 1.20 ± 0.03 |
pigz -d -c ./bdata.3.txt.gz > data.txt |
2.379 ± 0.012 | 2.368 | 2.391 | 4.51 ± 0.08 |
crabz -p 2 -d -f mgzip ./bdata.6.txt.gz > data.txt |
0.645 ± 0.015 | 0.629 | 0.658 | 1.22 ± 0.03 |
pigz -d -c ./bdata.6.txt.gz > data.txt |
2.438 ± 0.073 | 2.356 | 2.497 | 4.62 ± 0.16 |
crabz -p 2 -d -f mgzip ./bdata.9.txt.gz > data.txt |
0.659 ± 0.015 | 0.644 | 0.674 | 1.25 ± 0.04 |
pigz -d -c ./bdata.9.txt.gz > data.txt |
2.451 ± 0.075 | 2.400 | 2.538 | 4.65 ± 0.16 |
crabz -p 2 -d -f mgzip ./bdata.12.txt.gz > data.txt |
0.656 ± 0.015 | 0.647 | 0.673 | 1.24 ± 0.04 |
pigz -d -c ./bdata.12.txt.gz > data.txt |
2.450 ± 0.045 | 2.412 | 2.500 | 4.65 ± 0.12 |
crabz -p 4 -d -f mgzip ./bdata.3.txt.gz > data.txt |
0.577 ± 0.024 | 0.554 | 0.603 | 1.10 ± 0.05 |
pigz -d -c ./bdata.3.txt.gz > data.txt |
2.459 ± 0.052 | 2.420 | 2.518 | 4.66 ± 0.13 |
crabz -p 4 -d -f mgzip ./bdata.6.txt.gz > data.txt |
0.559 ± 0.024 | 0.531 | 0.576 | 1.06 ± 0.05 |
pigz -d -c ./bdata.6.txt.gz > data.txt |
2.538 ± 0.044 | 2.502 | 2.587 | 4.81 ± 0.12 |
crabz -p 4 -d -f mgzip ./bdata.9.txt.gz > data.txt |
0.552 ± 0.011 | 0.539 | 0.560 | 1.05 ± 0.03 |
pigz -d -c ./bdata.9.txt.gz > data.txt |
2.402 ± 0.018 | 2.385 | 2.420 | 4.56 ± 0.08 |
crabz -p 4 -d -f mgzip ./bdata.12.txt.gz > data.txt |
0.592 ± 0.040 | 0.546 | 0.616 | 1.12 ± 0.08 |
pigz -d -c ./bdata.12.txt.gz > data.txt |
2.525 ± 0.038 | 2.484 | 2.558 | 4.79 ± 0.11 |
crabz -p 8 -d -f mgzip ./bdata.3.txt.gz > data.txt |
0.563 ± 0.013 | 0.548 | 0.571 | 1.07 ± 0.03 |
pigz -d -c ./bdata.3.txt.gz > data.txt |
2.490 ± 0.126 | 2.369 | 2.621 | 4.72 ± 0.25 |
crabz -p 8 -d -f mgzip ./bdata.6.txt.gz > data.txt |
0.552 ± 0.018 | 0.533 | 0.569 | 1.05 ± 0.04 |
pigz -d -c ./bdata.6.txt.gz > data.txt |
2.531 ± 0.115 | 2.417 | 2.647 | 4.80 ± 0.23 |
crabz -p 8 -d -f mgzip ./bdata.9.txt.gz > data.txt |
0.603 ± 0.029 | 0.583 | 0.636 | 1.14 ± 0.06 |
pigz -d -c ./bdata.9.txt.gz > data.txt |
2.483 ± 0.042 | 2.435 | 2.515 | 4.71 ± 0.11 |
crabz -p 8 -d -f mgzip ./bdata.12.txt.gz > data.txt |
0.527 ± 0.009 | 0.519 | 0.537 | 1.00 |
pigz -d -c ./bdata.12.txt.gz > data.txt |
2.524 ± 0.093 | 2.417 | 2.583 | 4.79 ± 0.19 |
crabz -p 16 -d -f mgzip ./bdata.3.txt.gz > data.txt |
0.603 ± 0.058 | 0.551 | 0.665 | 1.14 ± 0.11 |
pigz -d -c ./bdata.3.txt.gz > data.txt |
2.392 ± 0.007 | 2.384 | 2.397 | 4.54 ± 0.08 |
crabz -p 16 -d -f mgzip ./bdata.6.txt.gz > data.txt |
0.611 ± 0.065 | 0.565 | 0.686 | 1.16 ± 0.13 |
pigz -d -c ./bdata.6.txt.gz > data.txt |
2.593 ± 0.148 | 2.427 | 2.712 | 4.92 ± 0.29 |
crabz -p 16 -d -f mgzip ./bdata.9.txt.gz > data.txt |
0.564 ± 0.027 | 0.541 | 0.594 | 1.07 ± 0.05 |
pigz -d -c ./bdata.9.txt.gz > data.txt |
2.426 ± 0.023 | 2.404 | 2.450 | 4.60 ± 0.09 |
crabz -p 16 -d -f mgzip ./bdata.12.txt.gz > data.txt |
0.601 ± 0.020 | 0.582 | 0.623 | 1.14 ± 0.04 |
pigz -d -c ./bdata.12.txt.gz > data.txt |
2.528 ± 0.022 | 2.507 | 2.550 | 4.80 ± 0.09 |
crabz -p 32 -d -f mgzip ./bdata.3.txt.gz > data.txt |
0.595 ± 0.019 | 0.577 | 0.614 | 1.13 ± 0.04 |
pigz -d -c ./bdata.3.txt.gz > data.txt |
2.544 ± 0.107 | 2.422 | 2.621 | 4.83 ± 0.22 |
crabz -p 32 -d -f mgzip ./bdata.6.txt.gz > data.txt |
0.601 ± 0.021 | 0.586 | 0.626 | 1.14 ± 0.05 |
pigz -d -c ./bdata.6.txt.gz > data.txt |
2.519 ± 0.114 | 2.435 | 2.649 | 4.78 ± 0.23 |
crabz -p 32 -d -f mgzip ./bdata.9.txt.gz > data.txt |
0.565 ± 0.023 | 0.539 | 0.579 | 1.07 ± 0.05 |
pigz -d -c ./bdata.9.txt.gz > data.txt |
2.487 ± 0.064 | 2.415 | 2.540 | 4.72 ± 0.15 |
crabz -p 32 -d -f mgzip ./bdata.12.txt.gz > data.txt |
0.557 ± 0.013 | 0.548 | 0.571 | 1.06 ± 0.03 |
pigz -d -c ./bdata.12.txt.gz > data.txt |
2.505 ± 0.105 | 2.442 | 2.626 | 4.75 ± 0.22 |
These benchmarks were run on the all_train.csv
data found here
Command | Mean [s] | Min [s] | Max [s] | Relative |
---|---|---|---|---|
crabz -p 2 -P 0 -l 2 -f bgzf ./data.txt > ./data.out.txt.gz |
15.837 ± 0.137 | 15.688 | 15.959 | 5.52 ± 0.13 |
bgzip -c -@ 2 -l 2 ./data.txt > ./data.out.txt.gz |
19.471 ± 0.178 | 19.268 | 19.602 | 6.78 ± 0.16 |
crabz -p 2 -P 0 -l 2 -f gzip ./data.txt > ./data.out.txt.gz |
19.723 ± 0.632 | 19.285 | 20.448 | 6.87 ± 0.26 |
pigz -c -p 2 -2 ./data.txt > ./data.out.txt.gz |
32.249 ± 0.024 | 32.226 | 32.274 | 11.24 ± 0.24 |
crabz -p 4 -P 0 -l 2 -f bgzf ./data.txt > ./data.out.txt.gz |
8.601 ± 0.538 | 8.040 | 9.113 | 3.00 ± 0.20 |
bgzip -c -@ 4 -l 2 ./data.txt > ./data.out.txt.gz |
10.953 ± 0.033 | 10.929 | 10.990 | 3.82 ± 0.08 |
crabz -p 4 -P 0 -l 2 -f gzip ./data.txt > ./data.out.txt.gz |
10.887 ± 0.584 | 10.236 | 11.364 | 3.79 ± 0.22 |
pigz -c -p 4 -2 ./data.txt > ./data.out.txt.gz |
16.493 ± 0.323 | 16.257 | 16.861 | 5.75 ± 0.17 |
crabz -p 8 -P 0 -l 2 -f bgzf ./data.txt > ./data.out.txt.gz |
5.206 ± 0.372 | 4.780 | 5.464 | 1.81 ± 0.14 |
bgzip -c -@ 8 -l 2 ./data.txt > ./data.out.txt.gz |
6.920 ± 0.033 | 6.893 | 6.957 | 2.41 ± 0.05 |
crabz -p 8 -P 0 -l 2 -f gzip ./data.txt > ./data.out.txt.gz |
5.893 ± 0.135 | 5.777 | 6.041 | 2.05 ± 0.06 |
pigz -c -p 8 -2 ./data.txt > ./data.out.txt.gz |
8.974 ± 0.467 | 8.553 | 9.477 | 3.13 ± 0.18 |
crabz -p 16 -P 0 -l 2 -f bgzf ./data.txt > ./data.out.txt.gz |
2.870 ± 0.061 | 2.816 | 2.936 | 1.00 |
bgzip -c -@ 16 -l 2 ./data.txt > ./data.out.txt.gz |
5.124 ± 0.107 | 5.040 | 5.244 | 1.79 ± 0.05 |
crabz -p 16 -P 0 -l 2 -f gzip ./data.txt > ./data.out.txt.gz |
4.250 ± 0.323 | 3.933 | 4.579 | 1.48 ± 0.12 |
pigz -c -p 16 -2 ./data.txt > ./data.out.txt.gz |
4.767 ± 0.223 | 4.513 | 4.933 | 1.66 ± 0.09 |
crabz -p 32 -P 0 -l 2 -f bgzf ./data.txt > ./data.out.txt.gz |
3.669 ± 0.303 | 3.320 | 3.865 | 1.28 ± 0.11 |
bgzip -c -@ 32 -l 2 ./data.txt > ./data.out.txt.gz |
4.676 ± 0.038 | 4.632 | 4.701 | 1.63 ± 0.04 |
crabz -p 32 -P 0 -l 2 -f gzip ./data.txt > ./data.out.txt.gz |
4.324 ± 0.246 | 4.143 | 4.605 | 1.51 ± 0.09 |
pigz -c -p 32 -2 ./data.txt > ./data.out.txt.gz |
5.854 ± 0.070 | 5.795 | 5.931 | 2.04 ± 0.05 |
crabz -p 2 -P 0 -l 6 -f bgzf ./data.txt > ./data.out.txt.gz |
27.696 ± 0.147 | 27.593 | 27.864 | 9.65 ± 0.21 |
bgzip -c -@ 2 -l 6 ./data.txt > ./data.out.txt.gz |
30.961 ± 0.446 | 30.446 | 31.231 | 10.79 ± 0.28 |
crabz -p 2 -P 0 -l 6 -f gzip ./data.txt > ./data.out.txt.gz |
36.229 ± 0.175 | 36.092 | 36.427 | 12.62 ± 0.27 |
pigz -c -p 2 -6 ./data.txt > ./data.out.txt.gz |
97.175 ± 0.571 | 96.743 | 97.823 | 33.86 ± 0.74 |
crabz -p 4 -P 0 -l 6 -f bgzf ./data.txt > ./data.out.txt.gz |
14.802 ± 0.436 | 14.316 | 15.159 | 5.16 ± 0.19 |
bgzip -c -@ 4 -l 6 ./data.txt > ./data.out.txt.gz |
16.927 ± 0.130 | 16.789 | 17.048 | 5.90 ± 0.13 |
crabz -p 4 -P 0 -l 6 -f gzip ./data.txt > ./data.out.txt.gz |
19.192 ± 0.675 | 18.629 | 19.940 | 6.69 ± 0.27 |
pigz -c -p 4 -6 ./data.txt > ./data.out.txt.gz |
49.305 ± 0.114 | 49.203 | 49.429 | 17.18 ± 0.37 |
crabz -p 8 -P 0 -l 6 -f bgzf ./data.txt > ./data.out.txt.gz |
7.833 ± 0.065 | 7.784 | 7.907 | 2.73 ± 0.06 |
bgzip -c -@ 8 -l 6 ./data.txt > ./data.out.txt.gz |
9.858 ± 0.105 | 9.739 | 9.939 | 3.43 ± 0.08 |
crabz -p 8 -P 0 -l 6 -f gzip ./data.txt > ./data.out.txt.gz |
10.417 ± 0.979 | 9.626 | 11.511 | 3.63 ± 0.35 |
pigz -c -p 8 -6 ./data.txt > ./data.out.txt.gz |
25.276 ± 0.170 | 25.083 | 25.404 | 8.81 ± 0.20 |
crabz -p 16 -P 0 -l 6 -f bgzf ./data.txt > ./data.out.txt.gz |
4.704 ± 0.321 | 4.337 | 4.937 | 1.64 ± 0.12 |
bgzip -c -@ 16 -l 6 ./data.txt > ./data.out.txt.gz |
6.565 ± 0.155 | 6.429 | 6.734 | 2.29 ± 0.07 |
crabz -p 16 -P 0 -l 6 -f gzip ./data.txt > ./data.out.txt.gz |
5.722 ± 0.320 | 5.530 | 6.092 | 1.99 ± 0.12 |
pigz -c -p 16 -6 ./data.txt > ./data.out.txt.gz |
13.673 ± 0.129 | 13.525 | 13.762 | 4.76 ± 0.11 |
crabz -p 32 -P 0 -l 6 -f bgzf ./data.txt > ./data.out.txt.gz |
4.202 ± 0.213 | 3.957 | 4.328 | 1.46 ± 0.08 |
bgzip -c -@ 32 -l 6 ./data.txt > ./data.out.txt.gz |
5.538 ± 0.135 | 5.395 | 5.663 | 1.93 ± 0.06 |
crabz -p 32 -P 0 -l 6 -f gzip ./data.txt > ./data.out.txt.gz |
5.488 ± 0.064 | 5.423 | 5.550 | 1.91 ± 0.05 |
pigz -c -p 32 -6 ./data.txt > ./data.out.txt.gz |
9.079 ± 0.286 | 8.808 | 9.379 | 3.16 ± 0.12 |
crabz -p 2 -P 0 -l 9 -f bgzf ./data.txt > ./data.out.txt.gz |
162.875 ± 0.100 | 162.778 | 162.977 | 56.75 ± 1.20 |
bgzip -c -@ 2 -l 9 ./data.txt > ./data.out.txt.gz |
172.428 ± 0.242 | 172.207 | 172.687 | 60.08 ± 1.27 |
crabz -p 2 -P 0 -l 9 -f gzip ./data.txt > ./data.out.txt.gz |
139.245 ± 0.270 | 138.974 | 139.514 | 48.52 ± 1.03 |
pigz -c -p 2 -9 ./data.txt > ./data.out.txt.gz |
209.645 ± 0.058 | 209.580 | 209.691 | 73.05 ± 1.55 |
crabz -p 4 -P 0 -l 9 -f bgzf ./data.txt > ./data.out.txt.gz |
84.624 ± 0.185 | 84.414 | 84.762 | 29.49 ± 0.63 |
bgzip -c -@ 4 -l 9 ./data.txt > ./data.out.txt.gz |
87.228 ± 0.232 | 87.053 | 87.492 | 30.39 ± 0.65 |
crabz -p 4 -P 0 -l 9 -f gzip ./data.txt > ./data.out.txt.gz |
72.339 ± 0.166 | 72.187 | 72.517 | 25.21 ± 0.54 |
pigz -c -p 4 -9 ./data.txt > ./data.out.txt.gz |
106.579 ± 0.236 | 106.307 | 106.731 | 37.14 ± 0.79 |
crabz -p 8 -P 0 -l 9 -f bgzf ./data.txt > ./data.out.txt.gz |
42.988 ± 0.130 | 42.905 | 43.138 | 14.98 ± 0.32 |
bgzip -c -@ 8 -l 9 ./data.txt > ./data.out.txt.gz |
44.550 ± 0.097 | 44.449 | 44.642 | 15.52 ± 0.33 |
crabz -p 8 -P 0 -l 9 -f gzip ./data.txt > ./data.out.txt.gz |
36.555 ± 0.030 | 36.521 | 36.579 | 12.74 ± 0.27 |
pigz -c -p 8 -9 ./data.txt > ./data.out.txt.gz |
54.047 ± 0.016 | 54.030 | 54.062 | 18.83 ± 0.40 |
crabz -p 16 -P 0 -l 9 -f bgzf ./data.txt > ./data.out.txt.gz |
22.391 ± 0.234 | 22.154 | 22.623 | 7.80 ± 0.18 |
bgzip -c -@ 16 -l 9 ./data.txt > ./data.out.txt.gz |
24.041 ± 0.237 | 23.813 | 24.286 | 8.38 ± 0.20 |
crabz -p 16 -P 0 -l 9 -f gzip ./data.txt > ./data.out.txt.gz |
19.285 ± 0.125 | 19.141 | 19.363 | 6.72 ± 0.15 |
pigz -c -p 16 -9 ./data.txt > ./data.out.txt.gz |
27.645 ± 0.078 | 27.579 | 27.731 | 9.63 ± 0.21 |
crabz -p 32 -P 0 -l 9 -f bgzf ./data.txt > ./data.out.txt.gz |
15.148 ± 0.138 | 14.992 | 15.252 | 5.28 ± 0.12 |
bgzip -c -@ 32 -l 9 ./data.txt > ./data.out.txt.gz |
16.091 ± 0.193 | 15.874 | 16.243 | 5.61 ± 0.14 |
crabz -p 32 -P 0 -l 9 -f gzip ./data.txt > ./data.out.txt.gz |
11.832 ± 0.168 | 11.637 | 11.930 | 4.12 ± 0.11 |
pigz -c -p 32 -9 ./data.txt > ./data.out.txt.gz |
16.912 ± 0.095 | 16.804 | 16.982 | 5.89 ± 0.13 |
Command | Mean [s] | Min [s] | Max [s] | Relative |
---|---|---|---|---|
crabz -d -p 4 -f bgzf ./data.txt.gz > ./data.out.txt |
5.941 ± 0.172 | 5.745 | 6.070 | 1.11 ± 0.09 |
bgzip -d -c -@ 4 ./data.txt.gz > ./data.out.txt |
5.357 ± 0.407 | 4.925 | 5.734 | 1.00 |
crabz -d -p 8 -f bgzf ./data.txt.gz > ./data.out.txt |
5.569 ± 0.496 | 5.023 | 5.990 | 1.04 ± 0.12 |
bgzip -d -c -@ 8 ./data.txt.gz > ./data.out.txt |
5.867 ± 0.252 | 5.682 | 6.154 | 1.10 ± 0.10 |
crabz -d -p 16 -f bgzf ./data.txt.gz > ./data.out.txt |
5.663 ± 0.240 | 5.506 | 5.939 | 1.06 ± 0.09 |
bgzip -d -c -@ 16 ./data.txt.gz > ./data.out.txt |
5.534 ± 0.124 | 5.416 | 5.663 | 1.03 ± 0.08 |
- Add some form of auto format detection, even just by file extension