materialize: extended logging #2019

Narrows the materialize "stream" interface to be focused on sending and receiving protocol messages. This is mostly removing baggage from when materializations only used gRPC and used the gRPC server interface more directly. Now they also support a JSON protocol so the prior implementation doesn't make much sense anymore. The lifecycle tests were also almost entirely redundant so rather than refactoring them I have deleted them.

Small refactoring to fix a linter complaint about a context leak of the loadCtx.

This logging will be implemented elsewhere in the coming commits, so the transactor is being cleaned up of most of its logging duties.

Creates a basic logger that will log out materialization status on a set frequency. Generally it will just log the current round and various counters, but if a commit has been running for a long time or the connector has been waiting for documents for a long time it will log information about that.

Adds an "extended" logger that will log materialization transaction progress in more detail, including the start & stop of most events of interest and how long they take to complete. This logging can be enabled via the new `MaterializeOptions` struct returned by `NewTransactor`. It will also be enabled if the log level is set to debug.

…ream Lifts the acknowledgement scheduling mechanism out of `RunTransactions` and into the new transactor message stream. This allows the new transactor message stream layer to provide useful logging information about what the materialization is doing in terms of an artificially applied delay. This also finishes up extracting meta operations out of the `RunTransactions` routine. It will no longer be responsible for computing acknowledgement delays or logging anything - that is all handled by the transactor message stream.

The `MaterializeOptions` configuration struct was added to the boilerplate in a previous commit to allow for materializations to tune specific high-level behaviors of transaction processing. This adapts all the materializations to use it.

The preceding commits added high-level logging capabilities that are applicable to all materializations, enabled via on configuration. But there are still some details that only the materialization implementation itself has access to. For these events, a BindingEvents construct is provided. It can provide start/stop and progress timings for evaluating load requests (load queries), and start/stop timings for committing stores to specific bindings. Materializations don't have to use this, but they can opt-in if it makes sense for their implementation. I have updated all of the materializations as it makes sense, also removing the now redundant logs that the boilerplate takes care of.

Getting the Open response for a materialization can take a long time, especially if it has to do a lot of work to setup the transactor or if querying a checkpoint from the store is not fast. This adds some progress logging for that case so it's more clear what the materialization is doing. Also a final round of refactoring: Split out the stream/logging logic into separate files, and added a general `repeatAsync` helper that several of these new logs make use of.

The 5 second and 15 second frequencies were just for testing, and the real values should be in minutes.

None of these needed to be exported anymore, so I unexported them. Also renamed some of them to be more clear, for example `ReadFlush` didn't actually read anything, but rather just validated that the last read message by the Load iterator was a "flush".

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

materialize: extended logging #2019

materialize: extended logging #2019

Commits on Oct 8, 2024