[rsyslog] Disable rsyslog rate limit in pre_test and do a recover in post_test #2378
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
Summary:
This is a workaround for sonic-net/sonic-buildimage#5667
Syslog messages will sometimes be rate-limited for no apparent reason. The rate-limiting threshhold is set at 20k messages sent in 5 minutes, however syslog will sometimes start rate-limiting messages from orchagent when fewer than 20k messages have been logged in the past 5 minutes.
Some ERROR or WARNING log are supressed if rsyslog begin limiting rate, and errors might not be detected. This PR add a workaround for this issue.
Type of change
Approach
What is the motivation for this PR?
This PR is to disable the rate limit of rsyslog to ensure that all logs are recorded in syslog.
How did you do it?
test_disable_rsyslog_rate_limit
intest_pretest.py
to update the configuration of rsyslog to disable rate limit, and then reload thersyslogd
service in each container;test_recover_rsyslog_rate_limit
intest_posttest.py
to do a recover after all tests finish.How did you verify/test it?
Verified on dx010 with a sample script to generate logs running in one of containers:
Before disable rate limit, we can see a rate-limiting in syslog
After disable rate limit, all logs are recorded in syslog
And after recover, the rate-limiting begins to work again.
Any platform specific information?
No.
Supported testbed topology if it's a new test case?
No.
Documentation
No.