Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[System logs]: Eliminate duplicate log messages and attempt rotation more frequently #520

Merged
merged 14 commits into from
Apr 21, 2017
Merged

[System logs]: Eliminate duplicate log messages and attempt rotation more frequently #520

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
27e610b
Rename 'ACSFileFormat' -> 'SONiCFileFormat'
jleveque Apr 19, 2017
364c945
Rename '00-acs.conf' -> '00-sonic.conf'
jleveque Apr 19, 2017
c8e435d
Add logrotate.d and systemd-journald config files to image
jleveque Apr 19, 2017
3817db5
Log all SONiC process messages to /var/log/syslog; prevent duplicate …
jleveque Apr 19, 2017
21b08c2
Do not redirect cron and daemon logs to their own files, let them log…
jleveque Apr 19, 2017
6304f9d
Log all teamd messages to /var/log/teamd.log; Add more SONiC program …
jleveque Apr 19, 2017
7b0dd39
Remove duplicate code by condensing quagga programs into a list; Fix …
jleveque Apr 19, 2017
9bfaf4f
Kernel and LPR messages no longer getting duplicated to their own log…
jleveque Apr 19, 2017
88e58d6
Now calling logrotate every minute via cron job
jleveque Apr 19, 2017
c7e075c
Need full path to logrotate in cron job
jleveque Apr 20, 2017
7c02b86
Add '.log' suffix to wildcards, otherwise logrotate will rotate alrea…
jleveque Apr 20, 2017
6ccd0d6
Add microsecond granularity to syslog messages
jleveque Apr 20, 2017
de4552b
Don't overwrite system crontab, instead, install additional logrotate…
jleveque Apr 21, 2017
ea8e998
Removed incomplete concept of per-process SONiC logs. We can revisit …
jleveque Apr 21, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions dockers/docker-base/rsyslog.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ $ModLoad imuxsock # provides support for local system logging
#$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Define a custom template
$template ACSFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogseverity-text:::uppercase% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
$ActionFileDefaultTemplate ACSFileFormat
$template SONiCFileFormat,"%TIMESTAMP%.%timestamp:::date-subseconds% %HOSTNAME% %syslogseverity-text:::uppercase% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
$ActionFileDefaultTemplate SONiCFileFormat

#
# Set the default permissions for all log files.
Expand Down
9 changes: 9 additions & 0 deletions files/build_templates/sonic_debian_extension.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,9 @@ sudo cp -f $IMAGE_CONFIGS/bash/bash.bashrc $FILESYSTEM_ROOT/etc/
sudo dpkg --root=$FILESYSTEM_ROOT -i target/debs/sonic-device-data_*.deb || \
sudo LANG=C DEBIAN_FRONTEND=noninteractive chroot $FILESYSTEM_ROOT apt-get -y install -f

# Copy crontabs
sudo cp -f $IMAGE_CONFIGS/cron.d/* $FILESYSTEM_ROOT/etc/cron.d/

# Copy NTP configuration files and templates
sudo cp $IMAGE_CONFIGS/ntp/ntp-config.service $FILESYSTEM_ROOT/etc/systemd/system/
sudo LANG=C chroot $FILESYSTEM_ROOT systemctl enable ntp-config.service
Expand All @@ -101,6 +104,12 @@ sudo cp $IMAGE_CONFIGS/rsyslog/rsyslog-config.sh $FILESYSTEM_ROOT/usr/bin/
sudo cp $IMAGE_CONFIGS/rsyslog/rsyslog.conf.j2 $FILESYSTEM_ROOT/usr/share/sonic/templates/
sudo cp $IMAGE_CONFIGS/rsyslog/rsyslog.d/* $FILESYSTEM_ROOT/etc/rsyslog.d/

# Copy logrotate.d configuration files
sudo cp -f $IMAGE_CONFIGS/logrotate.d/rsyslog $FILESYSTEM_ROOT/etc/logrotate.d/

# Copy systemd-journald configuration files
sudo cp -f $IMAGE_CONFIGS/systemd/journald.conf $FILESYSTEM_ROOT/etc/systemd/

# Copy interfaces configuration files and templates
sudo cp $IMAGE_CONFIGS/interfaces/interfaces-config.service $FILESYSTEM_ROOT/etc/systemd/system/
sudo LANG=C chroot $FILESYSTEM_ROOT systemctl enable interfaces-config.service
Expand Down
3 changes: 3 additions & 0 deletions files/image_config/cron.d/logrotate
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Attempt to rotate system logs once per minute
* * * * * root /usr/sbin/logrotate -f /etc/logrotate.d/rsyslog

43 changes: 43 additions & 0 deletions files/image_config/logrotate.d/rsyslog
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
/var/log/syslog
/var/log/quagga/*.log
/var/log/teamd.log
{
rotate 7
# Removed 'daily' interval, as we now call logrotate more frequently via cron
# and we want to check these logs every time
# daily
size 100M
missingok
notifempty
compress
delaycompress
postrotate
invoke-rc.d rsyslog rotate > /dev/null
endscript
}
/var/log/mail.info
/var/log/mail.warn
/var/log/mail.err
/var/log/mail.log
/var/log/daemon.log
/var/log/kern.log
/var/log/auth.log
/var/log/user.log
/var/log/lpr.log
/var/log/cron.log
/var/log/debug
/var/log/messages
{
rotate 4
weekly
size 100M
missingok
notifempty
compress
delaycompress
sharedscripts
postrotate
invoke-rc.d rsyslog rotate > /dev/null
endscript
}

4 changes: 2 additions & 2 deletions files/image_config/rsyslog/rsyslog.conf.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,8 @@ $UDPServerRun 514
#$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Define a custom template
$template ACSFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogseverity-text:::uppercase% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
$ActionFileDefaultTemplate ACSFileFormat
$template SONiCFileFormat,"%TIMESTAMP%.%timestamp:::date-subseconds% %HOSTNAME% %syslogseverity-text:::uppercase% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
$ActionFileDefaultTemplate SONiCFileFormat

#
# Set the default permissions for all log files.
Expand Down
46 changes: 0 additions & 46 deletions files/image_config/rsyslog/rsyslog.d/00-acs.conf
View file
Open in desktop

This file was deleted.

22 changes: 22 additions & 0 deletions files/image_config/rsyslog/rsyslog.d/00-sonic.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
## Quagga rules

if $programname == ["quagga",
"watchquagga",
"zebra"]
then {
/var/log/quagga/zebra.log
stop
}

if $programname == "bgpd" then {
/var/log/quagga/bgpd.log
stop
}

## Teamd rules

if $programname contains "teamd_" then {
/var/log/teamd.log
stop
}

23 changes: 14 additions & 9 deletions files/image_config/rsyslog/rsyslog.d/99-default.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,13 @@
#
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
cron.* /var/log/cron.log
daemon.* -/var/log/daemon.log
kern.* -/var/log/kern.log
kern.* -/var/persist/log/kern.log
lpr.* -/var/log/lpr.log
# Do not redirect cron, daemon, kernel or lpr logs to
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need this?
Does SONiC emits all these types?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The *.* on line 5 catches all messages and logs them to /var/log/syslog. Then, all messages that match cron.*, daemon.* etc. are logged again to their respective files. I commented out the individual files so that all messages are only logged once, to /var/log/syslog.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see

# their own files. Let them log to /var/log/syslog
#cron.* /var/log/cron.log
#daemon.* -/var/log/daemon.log
#kern.* -/var/log/kern.log
#kern.* -/var/persist/log/kern.log
#lpr.* -/var/log/lpr.log
mail.* -/var/log/mail.log
user.* -/var/log/user.log

Expand All @@ -32,10 +34,13 @@ news.notice -/var/log/news/news.notice
*.=debug;\
auth,authpriv.none;\
news.none;mail.none -/var/log/debug
*.=info;*.=notice;*.=warn;\
auth,authpriv.none;\
cron,daemon.none;\
mail,news.none -/var/log/messages
#
# Removed as duplicates:
#*.=info;*.=notice;*.=warn;\
# auth,authpriv.none;\
# cron,daemon.none;\
# mail,news.none -/var/log/messages
#
*.=crit;*.=alert;*.=emerg -/var/persist/log/alarms
#
# Emergencies are sent to everybody logged in.
Expand Down
36 changes: 36 additions & 0 deletions files/image_config/systemd/journald.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# See journald.conf(5) for details

[Journal]
#Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitInterval=30s
#RateLimitBurst=1000
SystemMaxUse=50MB
#SystemKeepFree=
#SystemMaxFileSize=
RuntimeMaxUse=50MB
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to comment all entries out of this file?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is how John originally added the file. He only intended to take advantage of SystemMaxUse and RuntimeMaxUse. We can define other variables if we'd like.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm good with this. Thanks


4 changes: 2 additions & 2 deletions platform/p4/docker-sonic-p4/rsyslog.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ $ModLoad imuxsock # provides support for local system logging
#$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Define a custom template
$template ACSFileFormat,"%TIMESTAMP% %HOSTNAME% %syslogseverity-text:::uppercase% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
$ActionFileDefaultTemplate ACSFileFormat
$template SONiCFileFormat,"%TIMESTAMP%.%timestamp:::date-subseconds% %HOSTNAME% %syslogseverity-text:::uppercase% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
$ActionFileDefaultTemplate SONiCFileFormat

#
# Set the default permissions for all log files.
Expand Down