Add (de)compression tracing functionality

[terrelln]

This PR adds tracing hooks for zstd compression and decompression calls. Users can inject tracing by overriding the weakly linked tracing functions.

• Add lib/common/trace.{h,c} - see this file for what is traced
• Hook up tracing in lib/
• Provide tracing for the zstd CLI in programs/zstdcli_tracing.{h,c} enabled by --trace FILE.

The tracing functionality in the CLI is optional, if we don't want it I can remove it. I figured it would both provide an example of how to hook up the tracing, and potentially be useful to measure (de)compression speed and output in a machine readable format in some scenarios.

E.g.

zstd --trace trace.csv -r path/to/dir
zstd --trace trace.csv -t path/to/dir
zstd --trace trace.csv -b1e19

[Cyan4973]

So, is zstd_trace.h a new "public" API ?

[terrelln]

Yeah, this API would eventually become public. I was planning on leaving it out of the default installation for now, but publishing it internally.

[Cyan4973]

The tracing functionality in the CLI is optional, if we don't want it I can remove it. I figured it would both provide an example of how to hook up the tracing, and potentially be useful to measure (de)compression speed

Agreed, the CLI is a great place to use, show and experiment with this new trace interface

[Cyan4973]

So, by default, these functions exist but don't do anything,
they need to be replaced in target binary by real functions that are actually doing something
which is possible thanks to weak linking.

I wonder if there is a (future) opportunity here to have some "good enough" default functions, instead of empty ones,
although to be fair I don't know the topic nearly enough to tell if it makes sense.

[terrelln]

We (zstd) don't really know what to do with the data that is provided. So by default we just disable the tracing, so the cost is only one function call + one branch per (de)compression.

We could also introduce other methods of tracing in the future, like attaching the tracing functions to the cctx/dctx itself.

[Cyan4973]

Does this design mandates the use of global variables ?
Could it be mitigated by allowing passing some void* state parameter ?

[terrelln]

There are two options:

1. Global state inside the tracing library.
2. return void* from the begin() function, and consume it in the end() function.

I chose the former, because the void* would have to be malloc'd. That would mean we have to be certain never to leak the void*. Currently end() is only called on success, so we would leak it on errors. We could fix that by calling free() whenever we overwrite it with a new void* or free the context. But, that introduces extra complexity to both zstd and the tracing library. Because now, what if the tracing library is in C++. It couldn't use new to allocate the object, it would have to use malloc. And generally having one library allocate and another free seems complex.

So I decided to keep it simpler and have the tracing library keep track of the state. A more complex implementation would have a map<uintptr_t, State> to keep track of a separate state per-context, using the context pointer as a key.

[terrelln]

We could also change the zstd implementation to make it compulsory to call end(). Basically, if zstd detected that the void* tracingContext is not NULL when calling begin() or during ZSTD_freeCCtx() it would call end() and set the error member to some generic error (and we would have the flexibility to set a more specific error code later).

[Cyan4973]

I understand the argument about simplicity.

It's just that, as it's a user API (even a hidden one), I'm easily worried that it could be more difficult to update it afterwards.

But overall, I'm fine with the idea of starting simple, experiment, and introduce complexity only when it feels more palatable.

[terrelln]

It's just that, as it's a user API (even a hidden one), I'm easily worried that it could be more difficult to update it afterwards.

At this point I expect this API to change between versions. Once we have enough experience with it, we can think about what stability guarantees we want to provide.