making better usage of default build rules

to be compatible with scenarios such as
`make -j allmost`

for better diagnosis

streamline zlibwrapper makefile

for some reasons, this test fails at _installing_ 32-bit dependencies
using the exact same command that actually works in other tests !!?

It's unclear why it fails repeateadly for this test only.
Try another way to install dependencies to fix that.

try to fix 32-bit test on github actions

try to keep libzstd.a "as is" once created

This ensures the symbols aren't redefined, which would result in a compiler
error.

I was getting redefined symbols for _LARGEFILE64_SOURCE when building for
32-bit x86 Linux on an older CentOS release in a CI environment. With this
change, I'm able to compile the single file library in this environment.

Closes #2443.

while waiting for it to be fixed

Add ifndef guards for _LARGEFILE_SOURCE and _LARGEFILE64_SOURCE

Generate skippable frame API

can't use address calculation with `void*`

remove flackey msan ossfuzz test

[license] Update year to 2021

* Add a test that runs without a pledgedSrcSize and with a dictionary.
* Add github.tar data with uses the github dictionary while compressing
  github.tar, instead of each file individually.

Fixes #2442.

1. When creating a dictionary keep the same behavior as before.
   Assume the source size is 513 bytes when adjusting parameters.
2. When calling ZSTD_getCParams() or ZSTD_adjustCParams() keep
   the same behavior as before.
3. When attaching a dictionary keep the same behavior of ignoring
   the dictionary size. When streaming this will select the
   largest parameters and not adjust them down. But, the CDict
   will use the correctly sized parameters, which seems like the
   right tradeoff.
4. When not attaching a dictionary (either forced not to, or
   using a prefix dictionary) we select parameters based on the
   dictionary size + source size, and assume the source size is
   small, which is the same behavior as before. But, now we don't
   adjust the window log (and hash and chain log) down when the
   source size is unknown.

When the source size is unknown all cdicts should attach, except
when the user disables attaching, or `forceWindow` is used. This
means that when streaming with a CDict we end up in the good case
where we get small CDict parameters, and large source parameters.

TODO: Add a streaming + dictionary regression test case.

Treat ZSTD_getCParams() and ZSTD_adjustCParams() in the same way
we treat streaming compression. Choose parameters based on the
dictionary size + source size, and assume the source size is small
if unkown. But, don't shrink the window log down in
ZSTD_adjustCParams_internal().

Don't shrink window log when streaming with a dictionary

ZSTD_cycleLog() is a very short function,
creating a rather large dependency onto libzstd's internal just for it is overkill.
Prefer duplicating this 2-lines function.

This PR makes the zstd CLI one step closer to being linkable to the dynamic library (see #2450)
More steps are still needed to reach this goal.

removed internal library dependency from CLI

clang v3.8 tests are either flacky or redundant,
prefer using clang-latest.

makint the CLI ons step closer to being linkable to the dynamic library

invoking target zstd-dll

removed redundant tests

fix typos (work done by Andrea Gelmini)

zstd CLI can now be linked to libzstd dynamic library

[RFC] Support references to multiple DDicts

Fix visibility of symbols in .so and add an alias for renamed API

Escaping in add_custom_target() seems to depend on the shell used in the cmake
generator and using Ninja on Windows, which uses cmd.exe, results in stray backslashes
in the .pc file.

Instead of going through escaping hell just use configure_file() with the existing
libzstd.pc.in file already used by the simple Makefile based build system.

This fixes the .pc file syntax when building zstd with CMake+Ninja+gcc on Windows.

cmake: use configure_file() for creating the .pc file

[license] Change year to 2021 compression_format.md

Also update option flag documentation.

This program takes a file with concatenated zstd frames and splits the
file up by frame. E.g. if `dir.zst` has 4 frames:

```
> ./recover_directory dir.zst recovery/file
Recovering 4 files...
Recovered recovery/file0
Recovered recovery/file1
Recovered recovery/file2
Recovered recovery/file3
Complete
```

[contrib][recovery] Add recovery_directory program

Allow Input From Console When `-f`/`--force` is Passed

fix #2474

parallel make build on linux

Simple performance improvements for ldm

The fuzzer CI found this bug.

from github actions.
replaced by equivalent "current" gcc tests when it makes sense

fixed minor cast warning

minor: removed flackey gcc6 tests from github actions

Don't start a new section when `dstSize == 0` to avoid NULL
pointer addition.

[zstd] Fix NULL pointer addition in ZSTD_checkContinuity()

Add (de)compression tracing functionality

The most common information that you want to track between begin() and
end() is the timestamp of the begin function, so you can measure the
duration of the (de)compression call. Allow the tracing library to put
this information inside the `ZSTD_TraceCtx`, so it doesn't need to keep
a global map in this case. If a single uint64_t is not enough, the
tracing library can return a unique identifier (like the context
pointer) instead, and use it as a key in a map.

This keeps the simple case simple.

[trace] Keep track of a uint64_t tracing context

This avoids a GNU coreutils dependency.

-n is used to match the output format of coreutils:
http://man.netbsd.org/md5.1

New algorithms for the long distance matcher

* Mark `ZSTD_CCtx_getParameter()` as const
* Add `extern "C"` guards to `zstd_trace.h`

[trace] Minor fixes found during integration

Forward the correct compressionLevel to the appliedParams in all cases.
It was already correct for the advanced API, so only the old single-pass
functions needed to be fixed.

This compression level is unused by the library, but is set so that the
tracing framework can consume it.

[lib] Set appliedParams.compressionLevel correctly

The simple compression functions are intended to ignore the advanced
parameters, but they were accidentally using them. All the
`ZSTD_parameters` were set correctly, but any extra parameters were
used as-is. E.g. `ZSTD_c_format`.

This PR makes all the simple single-pass functions listed below ignore
the advanced parameters, as intended.

* `ZSTD_compressCCtx()`
* `ZSTD_compress_usingDict()`
* `ZSTD_compress_usingCDict()`
* `ZSTD_compress_advanced()`
* `ZSTD_compress_usingCDict_advanced()`

It also adds a test case that ensures that each of these functions
ignore the advanced parameters.

[bug-fix] Make simple single-pass functions ignore advanced parameters

[easy][cli] Add newline to boundary of --help message of --trace and --[no-]check

This commit addresses #2491.

Note that a downside of this solution is that it is global: `umask()` affects
all file creation calls in the process. I believe this is safe since
`fileio.c` functions should only ever be used in the zstd binary, and these
are (almost) the only files ever created by zstd, and AIUI they're only
created in a single thread. So we can get away with messing with global state.

Note that this doesn't change the permissions of files created by `dibio.c`.
I'm not sure what those should be...

Use umask() to Constrain Created File Permissions