🎉 Zstd 1.5.0 Release 🎉 #2636

Memory constrained use cases that manage multiple archives benefit from retaining multiple archive seek tables without retaining a ZSTD_seekable instance for each. * New opaque type for seek table: ZSTD_seekTable. * ZSTD_seekable_copySeekTable() supports copying seek table out of a ZSTD_seekable. * ZSTD_seekTable_[eachSeekTableOp]() defines seek table API that mirrors existing seek table operations. * Existing ZSTD_seekable_[eachSeekTableOp]() retained; they delegate to ZSTD_seekTable the variant. These changes allow the above-mentioned use cases to initialize a ZSTD_seekable, extract its ZSTD_seekTable, then throw the ZSTD_seekable away to save memory. Standard ZSTD operations can then be used to decompress frames based on seek table offsets. The copy and delegate patterns are intended to minimize impact on existing code and clients. Using copy instead of move for the infrequent operation extracting a seek table ensures that the extraction does not render the ZSTD_seekable useless. Delegating to *new* seek table-oriented APIs ensures that this is not a breaking change for existing clients while supporting all meaningful operations that depend only on seek table data.

[contrib] Support seek table-only API

read-only objects are properly const-ified in parameters

Seekable hang fix

and simple roundtrip test

New direct seekTable access methods

It is a stack high-point for some compression strategies and has an easy fix. This moves the normalized count into the entropy workspace.

Reduce stack usage of ZSTD_buildCTable()

This saves ~700 bytes of stack space in HUF_writeCTable.

Add HUF_writeCTable_wksp() function

* Use `HUF_readStats_wksp()` * Use workspace in `HUF_fillDTableX2*()` * Clean up workspace usage to use a workspace struct

* Move `counting` into the workspace * Inrease `HUF_DECOMPRESS_WORKSPACE_SIZE` by 512 bytes

doc: ZSTD_free*() functions accept NULL pointer

Make the number of physical CPU cores detection more robust

This commit introduces a GitHub action that is triggered on release creation, which creates the release tarball, compresses it, hashes it, signs it, and attaches all of those files to the release.

When the tag is `v1.2.3`, name the artifacts `zstd-1.2.3.tar*` rather than `zstd-v1.2.3.tar*`. When the tag doesn't match, use the full tag.

Add GitHub Action to Automatically Publish Release Tarballs

Fix seek table descriptor check when loading

Fix the `unaligned.h` shim in the tests that was causing corruption in the tests. Note that this is a problem with the test shim, not the kernel code.

Update the test include stubs so they are able to run the current zstd version in the kernel, so I can compare stack usage.

Expose the zstd headers in `include/linux` to avoid struct duplication. This makes the member names not follow Kernel style guidelines, and exposes the zstd symbols. But, the LMKL reviewers are okay with that.

Remove tracing support from `freestanding.py` to keep things simple.

When defined, it disables tracing, and avoids including the header.

Refactor ZSTD_compressBlock_internal() to do the block header write within and add nbSeq argument to compressSequences()

good day. It seems to me that the developer intended to use a logical operator. so I suggest a simple fix.

* Move `counting` to a struct in `FSE_decompress_wksp_body()` * Fix error code in `FSE_decompress_wksp_body()` * Rename a variable in `HUF_ReadDTableX2_Workspace`

Fix dedicated dict search isSupported() requirements.

fixes #2519

meson: fix build by adding missing files

Following #2545, I noticed that one field in `seq_t` is optional, and only used in combination with prefetching. (This may have contributed to static analyzer failure to detect correct initialization). I then wondered if it would be possible to rewrite the code so that this optional part is handled directly by the prefetching code rather than delegated as an option into `ZSTD_decodeSequence()`. This resulted into this refactoring exercise where the prefetching responsibility is better isolated into its own function and `ZSTD_decodeSequence()` is streamlined to contain strictly Sequence decoding operations. Incidently, due to better code locality, it reduces the need to send information around, leading to simplified interface, and smaller state structures.

…ter sets as well

fix #2549

[huf] Reduce stack usage of HUF_readDTableX2 by ~972 bytes

Add memory monotonicity test over srcSize

@jwilk

fix #2553 detected by @jwilk

…perblocks

…l compression

…sizes

…w block size

fix man page typo

Recursive block splitting

Fixes for the next linux kernel patch version

Add ZSTD_defaultCLevel() function to public API

…piler warnings

Fix block splitter minor MSAN warning.

…ckup Add different dict modes to compression ratio regression test, update results.csv

Add enum for representing long length ID

… the linux-kernel files * Switch to yearless copyright per FB policy * Fix up SPDX-License-Identifier lines in `contrib/linux-kernel` sources * Add zstd copyright/license header to the `contrib/linux-kernel` sources * Update the `tests/test-license.py` to check for yearless copyright * Improvements to `tests/test-license.py` * Check `contrib/linux-kernel` in `tests/test-license.py`

* Do not emit last partitions of blocks as RLE/uncompressed * Fix repcode updates within block splitter * Add a entropytables confirm function, redo ZSTD_confirmRepcodesAndEntropyTables() for better function signature * Add a repcode updater to block splitter, no longer need to force emit compressed blocks

* Perform 64-byte alignment of wksp tables and aligneds internally * Clean up cwskp_finalize() function to only do two allocs * Refactor aligned/buffer reservation code, remove ASAN req for alignment reservations * Change from allocating 128 bytes always to allocating only buffer space as needed for tables/aligned * Back out aligned/table reservation order restriction * Add stricter bounds for new/resized wksps, fix comment in zstd_cwksp.h

If build_dir is set the zstd build complains about md5sum not being found. Fix this by checking if build_dir is set before checking and using the hash tool just like in lib/Makefile .

simple fix for using bit operator.

* Move cdict clevel override to before localdict init * Update results.csv after dict load changes

…epcodes (#2569)

SIMD Row Based Matchfinder 🚀

…rror_fix Adjust nb elements to prefetch in ZSTD_row_fillHashCache()

Correct the block splitter mismatched repcodes detection.

[1.5.0] Remove ZBUFF

Add DDS to oss fuzzer

[1.5.0] Promote ZSTD_getDictID_fromCDict() into stable API

Instead of providing a default no-op implementation, check the symbols for `NULL` before accessing them. Providing a default implementation doesn't reliably work with dynamic linking. Depending on link order the default implementations may not be overridden. By skipping the default implementation, all link order issues are resolved. If the symbols aren't provided the weak function will be `NULL`.

[regressiontest] Update results.csv

[trace] Remove default definitions of weak symbols

Replace kernel-style comments with regular comments. E.g. ``` /** Before */ /* After */ /** * Before */ /* * After */ /*********************************** * Before ***********************************/ /* ********************************* * After ***********************************/ ```

[linux-kernel] Replace kernel-style comments

`zstd_errors.h` and `zdict.h` are public headers, so they deserve to be in the root `lib/` directory with `zstd.h`, not mixed in with our private headers.

[1.5.0] Promote ZSTD_defaultCLevel() into stable API

[1.5.0] Promote ZSTD_c_literalCompressionMode to stable params

Fix chaintable check to include rowhash in ZSTD_reduceIndex()

* Fix overflow correction when `windowLog < cycleLog`. Previously, we got the correction wrong in this case, and our chain tables and binary trees would be corrupted. Now, we work as long as `maxDist` is a power of two, by adding `MAX(maxDist, cycleSize)` to our indices. * When `ZSTD_WINDOW_OVERFLOW_CORRECT_FREQUENTLY` is defined to non-zero run overflow correction as frequently as allowed without impacting compression ratio. * Enable `ZSTD_WINDOW_OVERFLOW_CORRECT_FREQUENTLY` in `fuzzer` and `zstreamtest` as well as all the OSS-Fuzz fuzzers. This has a 5-10% speed penalty at most, which seems reasonable.

Switch from `-T0` to the default `-T1` which significantly reduces memory usage for level 19 when there are many cores. This fixes 32-bit issues of running out of address space. Fixes #2603.

Bug fix & run overflow correction much more frequently in tests

[tests] Reduce memory usage of MT CLI tests

Any stable API entry point introduced after v1.0 should be documented with its minimum version number. Since PR fixes this requirement updating mostly new entry points since v1.4.0 and newly introduced ones for future v1.5.0.

This fixes #2568.

LDM does especially poorly on repetitive data when that data's hash happens to have `(hash & stopMask) == 0`. Either because the `stopMask == 0` or random chance. Optimize this case by skipping over repetitive patterns. The detection is very simplistic, but should catch most of the offending cases. ``` head -c 1G /dev/zero | perf stat -- ./zstd -1 -o /dev/null -v --zstd=ldmHashRateLog=1 --long 21.187881087 seconds time elapsed head -c 1G /dev/zero | perf stat -- ./zstd -1 -o /dev/null -v --zstd=ldmHashRateLog=1 --long 1.149707921 seconds time elapsed ```

previous lower limit was 1 MB. Note : by default, the lowest job size is 2 MB, achieved at level 1. Even lower job sizes can be achieved by manipulating this value directly, or manually modifying window sizes to lower amounts. Updated unit test to ensure that this new limit works fine (test would fail with previous 1 MB limit).

Detect Presence of `md5` on Darwin

[1.5.0] Move `zstd_errors.h` and `zdict.h` to `lib/` root

`ZSTD_highbit32(1u << x) == x` when it isn't undefined behavior.

Documented minimum version numbers

[easy] Rewrite rowHashLog computation

Dictionaries larger than `ZSTD_CHUNKSIZE_MAX` used to have to be loaded in multiple segments. Instead, when we detect large dictionaries, ensure that we reset the context's indicies. Then, for dictionaries larger than `ZSTD_CURRENT_MAX - 1`, only load the suffix of the dictionary. Finally, enable DDS for large dictionaries, since we no longer load in multiple segments. This simplifes the dictionary loading code, and reduces opportunities for non-determinism to slip in.

Dictionary size must be > `ZSTD_CHUNKSIZE_MAX`.

[lib] Always load the dictionary in one go

Fix bad integer wraparound in repcode index for fast, dfast, lazy

[LDM] Speed optimization on repetitive data

allow jobSize to be as low as 512 KB

* seekable_format: fix from-file reading (not in-memory) It tries to check the buffer boundary, but there is no buffer for from-file reading. * seekable_decompression: break when ZSTD_seekable_decompress() returns zero * seekable_decompression_mem: break when ZSTD_seekable_decompress() returns zero * seekable_format: cap the offset+len up to the last dOffset This will allow to read the whole file w/o gotting corruption error if the offset is more then the data left in file, i.e.: $ ./seekable_compression seekable_compression.c 8192 | head $ zstd -cdq seekable_compression.c.zst | wc -c 4737 Before this patch: $ ./seekable_decompression seekable_compression.c.zst 0 10000000 | wc -c ZSTD_seekable_decompress() error : Corrupted block detected 0 After: $ ./seekable_decompression seekable_compression.c.zst 0 10000000 | wc -c 4737

pipeline increased from 4 to 8 slots. This change substantially improves decompression speed when there are long distance offsets. example with enwik9 compressed at level 22 : gcc-9 : 947 -> 1039 MB/s clang-10: 884 -> 946 MB/s I also checked the "cold dictionary" scenario, and found a smaller benefit, around ~2% (measurements are more noisy for this scenario).

Allow Reading from Block Devices with `--force`

…od()

`open()`'s mode bits are only applied to files that are created by the call. If the output file already exists, but is not readable, the `fopen()` would fail, preventing us from removing it, which would mean that the file would not end up with the correct permission bits. It's not clear to me why the `fopen()` is there at all. `UTIL_isRegularFile()` should be sufficient, AFAICT.

I think in some unix emulation environments on Windows, (cygwin?) mode bits are somehow respected. So we might as well pass them in. Can't hurt.

* Take `params` by const reference in `ZSTD_resetCCtx_internal()`. * Add `simpleApiParams` to the CCtx and use them in the simple API functions, instead of creating those parameters on the stack. I think this is a good direction to move in, because we shouldn't need to worry about adding parameters to `ZSTD_CCtx_params`, since it should always be on the heap (unless they become absoultely gigantic). Some `ZSTD_CCtx_params` are still on the stack in the CDict functions, but I've left them for now, because it was a little more complex, and we don't use those functions in stack-constrained currently.

This allows us to quickly check for compile errors in debug log messages, which are compiled out when `DEBUGLEVEL < 2`.

Improve Setting Permissions of Created Files

benchmark results are not progressively displayed on Windows terminal. For long benchmark sessions, nothing is displayed, until the end, where everything is flushed. Force display to be flushed after each update. Updates happen roughtly every second, or even less, so it's not a substantial workload.

faster speed for decompressSequencesLong

This flag forces zstd to always load the prefix in ext-dict mode, even if it happens to be contiguous, to force determinism. It also applies to dictionaries that are re-processed. A determinism test case is also added, which fails without `ZSTD_c_deterministicRefPrefix` and passes with it set. Question: Should this be the default behavior? It isn't in this PR.

The LDM change in PR #2602 changed the algorithm slightly. The compressed size is generally positive, and when it is worse, it is only a few bytes.

[lib] Move some ZSTD_CCtx_params off the stack

improved benchmark experience on Windows

Cygwin is the longest Appveyor test Appveyor is typically the CI which finish last

[test][regression] Update results.csv

[lib] Add ZSTD_c_deterministicRefPrefix

Move Single-File Build Script from `contrib/` to `build/`

fix --filelist compatibility with Windows cr+lf line ending

Fix for excessive compiler warnings when building with clang-cl

The FAQ covers the questions asked in Issue #2566. It first covers why you would want to use a dictionary, then what a dictionary is, and finally it tells you how to train a dictionary, and clarifies some of the parameters. There is definitely more that could be said about some of the advanced trainers, but this should be a good start.

[zdict] Add a FAQ to the top of zdict.h

* Add deprecated macro to zstd.h, mark certain functions as deprecated * Remove ZSTD_compress.c dependencies on deprecated functions

[lib] Fix UBSAN warning in ZSTD_decompressSequences()

added a paragraph on coding style

Linearly back off the frequency of overflow correction based on the number of times the `ZSTD_window_t` has been overflow corrected. This will still allow the fuzzer to quickly find overflow correction bugs, while also keeping good speed for larger inputs. Additionally, the `nbOverflowCorrections` variable can be useful for debugging coredumps, since we can inspect the `ZSTD_CCtx` to see if overflow correction has happened yet. I've verified this fixes the timeouts in OSS-Fuzz (176 seconds -> 6 seconds). I've also verified that fuzzers and `fuzzer` and `zstreamtest` still catch the row-hash overflow correction bug.

... for consistency (this doesn't actually change the build flags used in practice, currently). Signed-off-by: Stephen Kitt <steve@sk2.org>

* Update lib Makefile to have new targets * Update lib/README.md for mt

attempt to make Appveyor's Cygwin test faster

[lib] Fix fuzzer timeouts by backing off overflow correction

changed strategy, now unconditionally prefetch the first 2 cache lines, instead of cache lines corresponding to the first and last bytes of the match. This better corresponds to cpu expectation, which should auto-prefetch following cachelines on detecting the sequential nature of the read. This is globally positive, by +5%, though exact gains depend on compiler (from -2% to +15%). The only negative counter-example is gcc-9.

…_prefetch_refactor

This seems to bring an additional ~+1.2% decompression speed on average across 10 compilers x 6 scenarios.

Refactor prefetching for the decoding loop

the new alignment setting is better for gcc-9 and gcc-10 by about ~+5%. Unfortunately, it's worse for essentially all other compilers. Make the new alignment setting conditional to gcc-9+.

Apply flags to libzstd-nomt in libzstd style

improved gcc-9 and gcc-10 decoding speed

When running armv6 userspace on armv8 hardware with a 64 bit Linux kernel, the mode 2 caused SIGBUS (unaligned memory access). Running all our arm builds in the build farm only on armv8 simplifies administration a lot. Depending on compiler and environment, this change might slow down memory accesses (did not benchmark it). The original analysis is 6 years old. Fixes #2632

Avoid SIGBUS on armv6

and restored limit to 256 when in 64-bit mode (it was reduced to 200 to give more room for 32-bit). This should fix test instability issues using lot of threads in 32-bit environments.

With small enough input files, the inferred value of fileWindowLog could be smaller than ZSTD_WINDOWLOG_MIN. This can be reproduced like so: $ echo abc > small $ echo abcdef > small2 $ zstd --patch-from small small2 -o patch previously, this would fail with the error "zstd: error 11 : Parameter is out of bound"

reduce ZSTDMT_NBWORKERS_MAX in 32-bit mode

hopefully, bionic will have a more recent version of python required to install meson.

Fixed meson test on travisCI

…#2645)

Commits on Apr 22, 2021

Add DDS to oss fuzzer

senhuang42 committed Apr 22, 2021

Configuration menu

View commit details

Copy full SHA for f80dec6

Browse repository at this point

Copy the full SHA

f80dec6 View commit details

Browse the repository at this point in the history

🎉 Zstd 1.5.0 Release 🎉 #2636

🎉 Zstd 1.5.0 Release 🎉 #2636

Commits on May 7, 2020

Commits on Mar 2, 2021

Commits on Mar 3, 2021

Commits on Mar 4, 2021

Commits on Mar 5, 2021

Commits on Mar 7, 2021

Commits on Mar 8, 2021

Commits on Mar 12, 2021

Commits on Mar 13, 2021

Commits on Mar 15, 2021

Commits on Mar 16, 2021

Commits on Mar 17, 2021

Commits on Mar 18, 2021

Commits on Mar 19, 2021

Commits on Mar 21, 2021

Commits on Mar 22, 2021

Commits on Mar 24, 2021

Commits on Mar 25, 2021

Commits on Mar 26, 2021

Commits on Mar 30, 2021

Commits on Mar 31, 2021

Commits on Apr 2, 2021

Commits on Apr 6, 2021

Commits on Apr 7, 2021

Commits on Apr 8, 2021

Commits on Apr 9, 2021

Commits on Apr 12, 2021

Commits on Apr 13, 2021

Commits on Apr 16, 2021

Commits on Apr 19, 2021

Commits on Apr 20, 2021

Commits on Apr 22, 2021

Commits on Apr 23, 2021

Commits on Apr 24, 2021

Commits on Apr 26, 2021

Commits on Apr 27, 2021

Commits on Apr 29, 2021

Commits on Apr 30, 2021

Commits on May 2, 2021

Commits on May 3, 2021

Commits on May 4, 2021

Commits on May 5, 2021

Commits on May 6, 2021

Commits on May 7, 2021

Commits on May 8, 2021

Commits on May 10, 2021

Commits on May 11, 2021

Commits on May 12, 2021

Commits on May 13, 2021