-
Notifications
You must be signed in to change notification settings - Fork 15
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
materialize: extended logging #2019
Commits on Oct 8, 2024
-
materialize(refactor): simplify stream interface
Narrows the materialize "stream" interface to be focused on sending and receiving protocol messages. This is mostly removing baggage from when materializations only used gRPC and used the gRPC server interface more directly. Now they also support a JSON protocol so the prior implementation doesn't make much sense anymore. The lifecycle tests were also almost entirely redundant so rather than refactoring them I have deleted them.
Configuration menu - View commit details
-
Copy full SHA for 3e068cf - Browse repository at this point
Copy the full SHA 3e068cfView commit details -
materialize: fix linter complaint about context leak
Small refactoring to fix a linter complaint about a context leak of the loadCtx.
Configuration menu - View commit details
-
Copy full SHA for 9702cd9 - Browse repository at this point
Copy the full SHA 9702cd9View commit details -
materialize: remove most logging from transactor
This logging will be implemented elsewhere in the coming commits, so the transactor is being cleaned up of most of its logging duties.
Configuration menu - View commit details
-
Copy full SHA for 2db1e2e - Browse repository at this point
Copy the full SHA 2db1e2eView commit details -
materialize: basic periodic logger
Creates a basic logger that will log out materialization status on a set frequency. Generally it will just log the current round and various counters, but if a commit has been running for a long time or the connector has been waiting for documents for a long time it will log information about that.
Configuration menu - View commit details
-
Copy full SHA for d36e27e - Browse repository at this point
Copy the full SHA d36e27eView commit details -
materialize: extended logger for more detail logging
Adds an "extended" logger that will log materialization transaction progress in more detail, including the start & stop of most events of interest and how long they take to complete. This logging can be enabled via the new `MaterializeOptions` struct returned by `NewTransactor`. It will also be enabled if the log level is set to debug.
Configuration menu - View commit details
-
Copy full SHA for e954d8e - Browse repository at this point
Copy the full SHA e954d8eView commit details -
materialize: move acknowledgement scheduling to transactor message st…
…ream Lifts the acknowledgement scheduling mechanism out of `RunTransactions` and into the new transactor message stream. This allows the new transactor message stream layer to provide useful logging information about what the materialization is doing in terms of an artificially applied delay. This also finishes up extracting meta operations out of the `RunTransactions` routine. It will no longer be responsible for computing acknowledgement delays or logging anything - that is all handled by the transactor message stream.
Configuration menu - View commit details
-
Copy full SHA for 46fc673 - Browse repository at this point
Copy the full SHA 46fc673View commit details -
materialize: adapt all materializations to use
MaterializeOptions
The `MaterializeOptions` configuration struct was added to the boilerplate in a previous commit to allow for materializations to tune specific high-level behaviors of transaction processing. This adapts all the materializations to use it.
Configuration menu - View commit details
-
Copy full SHA for d4176d0 - Browse repository at this point
Copy the full SHA d4176d0View commit details -
Configuration menu - View commit details
-
Copy full SHA for aff8cf7 - Browse repository at this point
Copy the full SHA aff8cf7View commit details -
materialize: add BindingEvents for logging events specific to bindings
The preceding commits added high-level logging capabilities that are applicable to all materializations, enabled via on configuration. But there are still some details that only the materialization implementation itself has access to. For these events, a BindingEvents construct is provided. It can provide start/stop and progress timings for evaluating load requests (load queries), and start/stop timings for committing stores to specific bindings. Materializations don't have to use this, but they can opt-in if it makes sense for their implementation. I have updated all of the materializations as it makes sense, also removing the now redundant logs that the boilerplate takes care of.
Configuration menu - View commit details
-
Copy full SHA for 5bba8da - Browse repository at this point
Copy the full SHA 5bba8daView commit details -
materialize: log progress of requesting Open
Getting the Open response for a materialization can take a long time, especially if it has to do a lot of work to setup the transactor or if querying a checkpoint from the store is not fast. This adds some progress logging for that case so it's more clear what the materialization is doing. Also a final round of refactoring: Split out the stream/logging logic into separate files, and added a general `repeatAsync` helper that several of these new logs make use of.
Configuration menu - View commit details
-
Copy full SHA for 5859d5b - Browse repository at this point
Copy the full SHA 5859d5bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 275dd80 - Browse repository at this point
Copy the full SHA 275dd80View commit details -
materialize: adjust loggers to use reasonable frequencies
The 5 second and 15 second frequencies were just for testing, and the real values should be in minutes.
Configuration menu - View commit details
-
Copy full SHA for 1f7866d - Browse repository at this point
Copy the full SHA 1f7866dView commit details -
materialize: unexport & rename some message handling routines
None of these needed to be exported anymore, so I unexported them. Also renamed some of them to be more clear, for example `ReadFlush` didn't actually read anything, but rather just validated that the last read message by the Load iterator was a "flush".
Configuration menu - View commit details
-
Copy full SHA for ede4d19 - Browse repository at this point
Copy the full SHA ede4d19View commit details