Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces the ability to easy add new logging targets while adding two of them: Loki and local files.
It is possible to use any number of logging targets concurrently. By default no logging targets, they need to be specified using env variable LOGWATCH_LOG_TARGETS, which currently accepts two values: file and loki (case insensitive).
LogWatch's API has not changed, but now the constructor accepts variadic options that can be used to override default implementations for log target handlers (useful for example for unit testing if we don't want to create any files or send anything to Loki but still make sure that correct handler was called expected number of times).
The constructor also executes some log target validations and operations:
is there a known handler for every requested log target?
deactivate known handlers that weren't requested/set via environment variables
Also, Loki client is now lazy initalised the first time it is requested instead of being always initialised in the constructor (even if streaming logs to Loki was not requested).
Last, but not least... file logs are also streamed instead of being saved at the end of test, which means that in case of container failure we will still have some logs which might help to debug failure cause.
When a test ends each active handler prints information about log location.
It is recommended to connect/disconnect LogWatch to containers using PostStarts/PostStops hooks instead of doing it manually.
Known issues existing in previous versions:
slow containers might result in panic when log producer cannot connect to a container within 5 seconds (context timeout hardcoded in testcontainers-go, there's a testcontainers/testcontainers-go#1278 that adds retries (although it seems from my experiments that if log producer fails to start in 5 seconds there's something off with the container and retrying won't help anyway)
if the test is too short logs might not be sent to Loki before test finishes (we should consider adding a way to flush them on shutdown)
Breaking changes:
env variable TEST_LOG_COLLECT is no longer used. To indicate that you want to collect logs simply add targets to LOGWATCH_LOG_TARGETS or leave it empty to skip log collection
New env variables:
GRAFANA_URL -- URL to Grafana instance that is connected to Loki instance to which logs where streamed
GRAFANA_DATASOURCE -- Loki datasource id that will be used when building the URL
What's missing:
tests for new functionality
validating whether Grafana-related env vars were set
creating/reading a global execution id that will be send together with logs and allow us to filter them in Grafana
building of Grafana Dashboard URL instead of Explore URL (as is currently implemented)
Example of changes to chainlink repo that uses testcontainers-go hooks to start and stop listening to logs can be found smartcontractkit/chainlink#11007.