[S6100] Improve S6100 serial-getty monitor, wait and re-check when getty not running to avoid false alert. #14402
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
why s6100 only fix? |
prgeor
reviewed
Mar 27, 2023
Comment on lines
+13
to
+14
| # when serial-getty not running, recheck later, beause systemd will restart serial-getty automatically. | ||
| sleep 1 |
There was a problem hiding this comment.
we have seen cases where systemd is not restarting getty and that was the reason in S6100 we had to do it outside systemd like in monit service. Btw, this issue is not specific to S6100
prgeor
reviewed
Mar 27, 2023
| @@ -1,4 +1,5 @@ | |||
| #Dell S6100 serial getty monitor | |||
| check process serial-getty matching "ttyS" | |||
| check program serial-getty with path /usr/local/bin/check-getty.sh | |||
There was a problem hiding this comment.
This issue is not specific to S6100, we see in Arista as well
There was a problem hiding this comment.
Maybe we can fix S6100 script first then porting this to Arista?
I will check arista issue later.
There was a problem hiding this comment.
Update: I check the Arista issue, it's because some user operation stuck on console, so others can't login via console. it's not a serial-ketty down issue.
There was a problem hiding this comment.
I guess there is some typo in "some using stuck". @liuh-80 Could you check?
There was a problem hiding this comment.
Fixed, 'some user operation stuck on console'
The issue I'm trying to fix is: s6100_serial_getty_monitor report lots of false alert to syslog. For the serial-getty not start issue, I will check if the Arista issue caused by same reason. If it's same erason, I will move this monitor to a common place. |
liuh-80
changed the title
aravindmani-1
approved these changes
Mar 31, 2023
|
@liuh-80 can we make this generic fix instead of only fixing S6100? |
|
Hi @prgeor, it is not possible because s6100_serial_getty_monitor is a S6100 platform specific monit config. Other platforms have no "false alert" issue, since there is no alert at all. In reply to: 1492164681 |
qiluo-msft
approved these changes
Apr 2, 2023
prgeor
approved these changes
Apr 3, 2023
|
/azp run Azure.sonic-buildimage |
|
Commenter does not have sufficient privileges for PR 14402 in repo sonic-net/sonic-buildimage |
|
Close and reopen to trigger validation again. |
|
vstest skipped because the change not a VS image change, so following ruls skip vstest: The change already manually validated on starlab s6100 device. |
|
@liuh-80 PR conflicts with 202012 branch |
|
PR description updated, will create manually cherry-pick PR for 20201231 first. |
liuh-80
added a commit
to liuh-80/sonic-buildimage
that referenced
this pull request
Apr 20, 2023
…tty not running to avoid false alert. (sonic-net#14402) [S6100] Improve S6100 serial-getty monitor, wait and re-check when getty not running to avoid false alert. On S6100, the serial-getty service some time can't auto-restart by systemd. So there is a monit unit to check serial-getty service status and restart it. However, this monit will report false alert, because in most case when serial-getty not running, systemd can restart it successfully. To avoid the false alert, improve the monitor to wait and re-check. Steps to reproduce this issue: 1. User login to device via console, and keep the connection. 2. User login to device via SSH, check the serial-getty@ttyS1.service service, it's running. 3. Run 'monit reload' from SSH connection. 4. Check syslog 1 minutes later, there will be false alert: ' 'serial-getty' process is not running' Add check-getty.sh script to recheck again later when getty service not running. And update monit unit to check serial-getty service status with this script to avoid false alert. Pass all UT. Manually check fixed code work correctly: ``` admin@***:~$ sudo systemctl stop serial-getty@ttyS1.service admin@***:~$ sudo /usr/local/bin/check-getty.sh admin@***:~$ echo $? 1 admin@***:~$ sudo systemctl status serial-getty@ttyS1.service ● serial-getty@ttyS1.service - Serial Getty on ttyS1 Loaded: loaded (/lib/systemd/system/serial-getty@.service; enabled-runtime; vendor preset: enabled) Active: inactive (dead) since Tue 2023-03-28 07:15:21 UTC; 1min 13s ago admin@***:~$ sudo /usr/local/bin/check-getty.sh admin@***:~$ echo $? 0 admin@***:~$ sudo systemctl status serial-getty@ttyS1.service ● serial-getty@ttyS1.service - Serial Getty on ttyS1 Loaded: loaded (/lib/systemd/system/serial-getty@.service; enabled-runtime; vendor preset: enabled) ``` syslog: ``` Mar 28 07:10:37.597458 *** INFO systemd[1]: serial-getty@ttyS1.service: Succeeded. Mar 28 07:12:43.010550 *** ERR monit[593]: 'serial-getty' status failed (1) -- no output Mar 28 07:12:43.010744 *** INFO monit[593]: 'serial-getty' trying to restart Mar 28 07:12:43.010846 *** INFO monit[593]: 'serial-getty' stop: '/bin/systemctl stop serial-getty@ttyS1.service' Mar 28 07:12:43.132172 *** INFO monit[593]: 'serial-getty' start: '/bin/systemctl start serial-getty@ttyS1.service' Mar 28 07:13:43.286276 *** INFO monit[593]: 'serial-getty' status succeeded (0) -- no output ``` [S6100] Improve S6100 serial-getty monitor.
9 tasks
|
Cherry-pick PR for 202012 branch created: #14755 |
mssonicbld
pushed a commit
to mssonicbld/sonic-buildimage
that referenced
this pull request
Apr 20, 2023
…tty not running to avoid false alert. (sonic-net#14402) [S6100] Improve S6100 serial-getty monitor, wait and re-check when getty not running to avoid false alert. #### Why I did it On S6100, the serial-getty service some time can't auto-restart by systemd. So there is a monit unit to check serial-getty service status and restart it. However, this monit will report false alert, because in most case when serial-getty not running, systemd can restart it successfully. To avoid the false alert, improve the monitor to wait and re-check. Steps to reproduce this issue: 1. User login to device via console, and keep the connection. 2. User login to device via SSH, check the serial-getty@ttyS1.service service, it's running. 3. Run 'monit reload' from SSH connection. 4. Check syslog 1 minutes later, there will be false alert: ' 'serial-getty' process is not running' #### How I did it Add check-getty.sh script to recheck again later when getty service not running. And update monit unit to check serial-getty service status with this script to avoid false alert. #### How to verify it Pass all UT. Manually check fixed code work correctly: ``` admin@***:~$ sudo systemctl stop serial-getty@ttyS1.service admin@***:~$ sudo /usr/local/bin/check-getty.sh admin@***:~$ echo $? 1 admin@***:~$ sudo systemctl status serial-getty@ttyS1.service ● serial-getty@ttyS1.service - Serial Getty on ttyS1 Loaded: loaded (/lib/systemd/system/serial-getty@.service; enabled-runtime; vendor preset: enabled) Active: inactive (dead) since Tue 2023-03-28 07:15:21 UTC; 1min 13s ago admin@***:~$ sudo /usr/local/bin/check-getty.sh admin@***:~$ echo $? 0 admin@***:~$ sudo systemctl status serial-getty@ttyS1.service ● serial-getty@ttyS1.service - Serial Getty on ttyS1 Loaded: loaded (/lib/systemd/system/serial-getty@.service; enabled-runtime; vendor preset: enabled) ``` syslog: ``` Mar 28 07:10:37.597458 *** INFO systemd[1]: serial-getty@ttyS1.service: Succeeded. Mar 28 07:12:43.010550 *** ERR monit[593]: 'serial-getty' status failed (1) -- no output Mar 28 07:12:43.010744 *** INFO monit[593]: 'serial-getty' trying to restart Mar 28 07:12:43.010846 *** INFO monit[593]: 'serial-getty' stop: '/bin/systemctl stop serial-getty@ttyS1.service' Mar 28 07:12:43.132172 *** INFO monit[593]: 'serial-getty' start: '/bin/systemctl start serial-getty@ttyS1.service' Mar 28 07:13:43.286276 *** INFO monit[593]: 'serial-getty' status succeeded (0) -- no output ``` #### Description for the changelog [S6100] Improve S6100 serial-getty monitor. #### Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
|
Cherry-pick PR to 202211: #14767 |
mssonicbld
pushed a commit
that referenced
this pull request
Apr 20, 2023
…tty not running to avoid false alert. (#14402) [S6100] Improve S6100 serial-getty monitor, wait and re-check when getty not running to avoid false alert. #### Why I did it On S6100, the serial-getty service some time can't auto-restart by systemd. So there is a monit unit to check serial-getty service status and restart it. However, this monit will report false alert, because in most case when serial-getty not running, systemd can restart it successfully. To avoid the false alert, improve the monitor to wait and re-check. Steps to reproduce this issue: 1. User login to device via console, and keep the connection. 2. User login to device via SSH, check the serial-getty@ttyS1.service service, it's running. 3. Run 'monit reload' from SSH connection. 4. Check syslog 1 minutes later, there will be false alert: ' 'serial-getty' process is not running' #### How I did it Add check-getty.sh script to recheck again later when getty service not running. And update monit unit to check serial-getty service status with this script to avoid false alert. #### How to verify it Pass all UT. Manually check fixed code work correctly: ``` admin@***:~$ sudo systemctl stop serial-getty@ttyS1.service admin@***:~$ sudo /usr/local/bin/check-getty.sh admin@***:~$ echo $? 1 admin@***:~$ sudo systemctl status serial-getty@ttyS1.service ● serial-getty@ttyS1.service - Serial Getty on ttyS1 Loaded: loaded (/lib/systemd/system/serial-getty@.service; enabled-runtime; vendor preset: enabled) Active: inactive (dead) since Tue 2023-03-28 07:15:21 UTC; 1min 13s ago admin@***:~$ sudo /usr/local/bin/check-getty.sh admin@***:~$ echo $? 0 admin@***:~$ sudo systemctl status serial-getty@ttyS1.service ● serial-getty@ttyS1.service - Serial Getty on ttyS1 Loaded: loaded (/lib/systemd/system/serial-getty@.service; enabled-runtime; vendor preset: enabled) ``` syslog: ``` Mar 28 07:10:37.597458 *** INFO systemd[1]: serial-getty@ttyS1.service: Succeeded. Mar 28 07:12:43.010550 *** ERR monit[593]: 'serial-getty' status failed (1) -- no output Mar 28 07:12:43.010744 *** INFO monit[593]: 'serial-getty' trying to restart Mar 28 07:12:43.010846 *** INFO monit[593]: 'serial-getty' stop: '/bin/systemctl stop serial-getty@ttyS1.service' Mar 28 07:12:43.132172 *** INFO monit[593]: 'serial-getty' start: '/bin/systemctl start serial-getty@ttyS1.service' Mar 28 07:13:43.286276 *** INFO monit[593]: 'serial-getty' status succeeded (0) -- no output ``` #### Description for the changelog [S6100] Improve S6100 serial-getty monitor. #### Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
mssonicbld
added
Included in 202211 Branch
and removed
Approved for 202211 Branch
Created PR to 202211 Branch
labels
qiluo-msft
pushed a commit
that referenced
this pull request
Apr 21, 2023
…tty not running to avoid false alert. (#14402) (#14755) [S6100] Improve S6100 serial-getty monitor, wait and re-check when getty not running to avoid false alert. This is cherry-pick PR for: #14402 #### Why I did it On S6100, the serial-getty service some time can't auto-restart by systemd. So there is a monit unit to check serial-getty service status and restart it. However, this monit will report false alert, because in most case when serial-getty not running, systemd can restart it successfully. To avoid the false alert, improve the monitor to wait and re-check. Steps to reproduce this issue: 1. User login to device via console, and keep the connection. 2. User login to device via SSH, check the serial-getty@ttyS1.service service, it's running. 3. Run 'monit reload' from SSH connection. 4. Check syslog 1 minutes later, there will be false alert: ' 'serial-getty' process is not running' ##### Work item tracking - Microsoft ADO :17424426 #### How I did it Add check-getty.sh script to recheck again later when getty service not running. And update monit unit to check serial-getty service status with this script to avoid false alert. #### How to verify it Pass all UT. Manually check fixed code work correctly: ``` admin@***:~$ sudo systemctl stop serial-getty@ttyS1.service admin@***:~$ sudo /usr/local/bin/check-getty.sh admin@***:~$ echo $? 1 admin@***:~$ sudo systemctl status serial-getty@ttyS1.service ● serial-getty@ttyS1.service - Serial Getty on ttyS1 Loaded: loaded (/lib/systemd/system/serial-getty@.service; enabled-runtime; vendor preset: enabled) Active: inactive (dead) since Tue 2023-03-28 07:15:21 UTC; 1min 13s ago admin@***:~$ sudo /usr/local/bin/check-getty.sh admin@***:~$ echo $? 0 admin@***:~$ sudo systemctl status serial-getty@ttyS1.service ● serial-getty@ttyS1.service - Serial Getty on ttyS1 Loaded: loaded (/lib/systemd/system/serial-getty@.service; enabled-runtime; vendor preset: enabled) ``` syslog: ``` Mar 28 07:10:37.597458 *** INFO systemd[1]: serial-getty@ttyS1.service: Succeeded. Mar 28 07:12:43.010550 *** ERR monit[593]: 'serial-getty' status failed (1) -- no output Mar 28 07:12:43.010744 *** INFO monit[593]: 'serial-getty' trying to restart Mar 28 07:12:43.010846 *** INFO monit[593]: 'serial-getty' stop: '/bin/systemctl stop serial-getty@ttyS1.service' Mar 28 07:12:43.132172 *** INFO monit[593]: 'serial-getty' start: '/bin/systemctl start serial-getty@ttyS1.service' Mar 28 07:13:43.286276 *** INFO monit[593]: 'serial-getty' status succeeded (0) -- no output ``` #### Tested branch (Please provide the tested image version) - [x] 20201231.77 #### Description for the changelog [S6100] Improve S6100 serial-getty monitor.
|
@liuh-80 can you create separate PR for 202205 branch? |
liuh-80
added a commit
to liuh-80/sonic-buildimage
that referenced
this pull request
May 12, 2023
…tty not running to avoid false alert. (sonic-net#14402) [S6100] Improve S6100 serial-getty monitor, wait and re-check when getty not running to avoid false alert. #### Why I did it On S6100, the serial-getty service some time can't auto-restart by systemd. So there is a monit unit to check serial-getty service status and restart it. However, this monit will report false alert, because in most case when serial-getty not running, systemd can restart it successfully. To avoid the false alert, improve the monitor to wait and re-check. Steps to reproduce this issue: 1. User login to device via console, and keep the connection. 2. User login to device via SSH, check the serial-getty@ttyS1.service service, it's running. 3. Run 'monit reload' from SSH connection. 4. Check syslog 1 minutes later, there will be false alert: ' 'serial-getty' process is not running' #### How I did it Add check-getty.sh script to recheck again later when getty service not running. And update monit unit to check serial-getty service status with this script to avoid false alert. #### How to verify it Pass all UT. Manually check fixed code work correctly: ``` admin@***:~$ sudo systemctl stop serial-getty@ttyS1.service admin@***:~$ sudo /usr/local/bin/check-getty.sh admin@***:~$ echo $? 1 admin@***:~$ sudo systemctl status serial-getty@ttyS1.service ● serial-getty@ttyS1.service - Serial Getty on ttyS1 Loaded: loaded (/lib/systemd/system/serial-getty@.service; enabled-runtime; vendor preset: enabled) Active: inactive (dead) since Tue 2023-03-28 07:15:21 UTC; 1min 13s ago admin@***:~$ sudo /usr/local/bin/check-getty.sh admin@***:~$ echo $? 0 admin@***:~$ sudo systemctl status serial-getty@ttyS1.service ● serial-getty@ttyS1.service - Serial Getty on ttyS1 Loaded: loaded (/lib/systemd/system/serial-getty@.service; enabled-runtime; vendor preset: enabled) ``` syslog: ``` Mar 28 07:10:37.597458 *** INFO systemd[1]: serial-getty@ttyS1.service: Succeeded. Mar 28 07:12:43.010550 *** ERR monit[593]: 'serial-getty' status failed (1) -- no output Mar 28 07:12:43.010744 *** INFO monit[593]: 'serial-getty' trying to restart Mar 28 07:12:43.010846 *** INFO monit[593]: 'serial-getty' stop: '/bin/systemctl stop serial-getty@ttyS1.service' Mar 28 07:12:43.132172 *** INFO monit[593]: 'serial-getty' start: '/bin/systemctl start serial-getty@ttyS1.service' Mar 28 07:13:43.286276 *** INFO monit[593]: 'serial-getty' status succeeded (0) -- no output ``` #### Description for the changelog [S6100] Improve S6100 serial-getty monitor. #### Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
10 tasks
yxieca
pushed a commit
that referenced
this pull request
May 17, 2023
…tty not running to avoid false alert. (#14402) (#15032) [S6100] Improve S6100 serial-getty monitor, wait and re-check when getty not running to avoid false alert. #### Why I did it On S6100, the serial-getty service some time can't auto-restart by systemd. So there is a monit unit to check serial-getty service status and restart it. However, this monit will report false alert, because in most case when serial-getty not running, systemd can restart it successfully. To avoid the false alert, improve the monitor to wait and re-check. Steps to reproduce this issue: 1. User login to device via console, and keep the connection. 2. User login to device via SSH, check the serial-getty@ttyS1.service service, it's running. 3. Run 'monit reload' from SSH connection. 4. Check syslog 1 minutes later, there will be false alert: ' 'serial-getty' process is not running' #### How I did it Add check-getty.sh script to recheck again later when getty service not running. And update monit unit to check serial-getty service status with this script to avoid false alert. #### How to verify it Pass all UT. Manually check fixed code work correctly: ``` admin@***:~$ sudo systemctl stop serial-getty@ttyS1.service admin@***:~$ sudo /usr/local/bin/check-getty.sh admin@***:~$ echo $? 1 admin@***:~$ sudo systemctl status serial-getty@ttyS1.service ● serial-getty@ttyS1.service - Serial Getty on ttyS1 Loaded: loaded (/lib/systemd/system/serial-getty@.service; enabled-runtime; vendor preset: enabled) Active: inactive (dead) since Tue 2023-03-28 07:15:21 UTC; 1min 13s ago admin@***:~$ sudo /usr/local/bin/check-getty.sh admin@***:~$ echo $? 0 admin@***:~$ sudo systemctl status serial-getty@ttyS1.service ● serial-getty@ttyS1.service - Serial Getty on ttyS1 Loaded: loaded (/lib/systemd/system/serial-getty@.service; enabled-runtime; vendor preset: enabled) ``` syslog: ``` Mar 28 07:10:37.597458 *** INFO systemd[1]: serial-getty@ttyS1.service: Succeeded. Mar 28 07:12:43.010550 *** ERR monit[593]: 'serial-getty' status failed (1) -- no output Mar 28 07:12:43.010744 *** INFO monit[593]: 'serial-getty' trying to restart Mar 28 07:12:43.010846 *** INFO monit[593]: 'serial-getty' stop: '/bin/systemctl stop serial-getty@ttyS1.service' Mar 28 07:12:43.132172 *** INFO monit[593]: 'serial-getty' start: '/bin/systemctl start serial-getty@ttyS1.service' Mar 28 07:13:43.286276 *** INFO monit[593]: 'serial-getty' status succeeded (0) -- no output ``` #### Description for the changelog [S6100] Improve S6100 serial-getty monitor. #### Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Approved for 202012 Branch
Cherry Pick Conflict_202012
Created PR to 202012 Branch
Created PR to 202205 Branch
Enhancement ➕
Included in 202012 Branch
Included in 202205 Branch
Included in 202211 Branch
Request for 201811 Branch 
Request for 201911 Branch
Request for 202012 Branch
Request for 202205 Branch
Request for 202211 Branch
Request for 201911 Branch
Request for 202012 Branch
Request for 202205 Branch
Request for 202211 Branch
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
[S6100] Improve S6100 serial-getty monitor, wait and re-check when getty not running to avoid false alert.
Why I did it
On S6100, the serial-getty service some time can't auto-restart by systemd. So there is a monit unit to check serial-getty service status and restart it.
However, this monit will report false alert, because in most case when serial-getty not running, systemd can restart it successfully.
To avoid the false alert, improve the monitor to wait and re-check.
Steps to reproduce this issue:
Work item tracking
How I did it
Add check-getty.sh script to recheck again later when getty service not running.
And update monit unit to check serial-getty service status with this script to avoid false alert.
How to verify it
Pass all UT.
Manually check fixed code work correctly:
syslog:
Which release branch to backport (provide reason below if selected)
Tested branch (Please provide the tested image version)
Description for the changelog
[S6100] Improve S6100 serial-getty monitor.
Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU.
Link to config_db schema for YANG module changes
A picture of a cute animal (not mandatory but encouraged)