[techsupport] Handle minor fixes of TS Lock and update auto-TS #2114

vivekrnv · 2022-03-25T03:35:33Z

Signed-off-by: Vivek Reddy Karri vkarri@nvidia.com

What I did

Print the last statement as the techsupport dump name, as some automation processes might depend of parsing the last line to infer the dump path.

Previously:
handle_exit
Removing lock. Exit: 0
removed '/tmp/techsupport-lock/PID'
removed directory '/tmp/techsupport-lock'

Updated:
handle_exit
Removing lock. Exit: 0
removed '/tmp/techsupport-lock/PID'
removed directory '/tmp/techsupport-lock'
/var/dump/sonic_dump_r-bulldog-03_20220324_195553.tar.gz

Don't acquire the lock when running in NOOP mode
Set the set -v option just before running main so that it won't print the generate_dump code to stdout
Update the auto-techsupport script to handle EXT_RETRY and EXT_LOCKFAIL exit codes returned by show techsupport command.
Update the minor error in since argument for auto-techsupport

How I did it

How to verify it

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

dgsudharsan · 2022-03-25T17:22:37Z

@qiluo-msft Can you please help to review?

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

…ities into ts_error

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

dgsudharsan · 2022-03-28T04:50:33Z

scripts/coredump_gen_handler.py

+            return ""
+        elif rc == EXT_RETRY:
+            if num_retry <= MAX_RETRY_LIMIT:
+                print(num_retry)


Should there be a gap of few seconds before next retry?

Not required, EXT_RETRY happening is less likely and response should be quick in order to grab the lock. so i don't think a gap is required.
I'll remove the print statement though

dgsudharsan · 2022-03-28T04:52:06Z

utilities_common/auto_techsupport_helper.py

+EXT_LOCKFAIL = 2
+EXT_RETRY = 4
+EXT_SUCCESS = 0
+MAX_RETRY_LIMIT = 2


Is MAX_RETRY_LIMIT configurable?

EXT_RETRY happening more than one time for a single process is even more unlikely and thus a MAX_RETRY_LIMIT need not be configurable.

dgsudharsan · 2022-03-31T18:49:56Z

@qiluo-msft Can you please help to review this PR?

qiluo-msft · 2022-03-31T20:55:14Z

scripts/coredump_gen_handler.py

+                return self.invoke_ts_cmd(since_cfg, num_retry+1)
+            else:
+                syslog.syslog(syslog.LOG_ERR, "MAX_RETRY_LIMIT for show techsupport invocation exceeded, stderr: {}".format(stderr))
+        elif rc != EXT_SUCCESS:
            syslog.syslog(syslog.LOG_ERR, "show techsupport failed with exit code {}, stderr: {}".format(rc, stderr))


syslog

Do you want to exit immediately after the error logging? #Closed

yeah, i'll add it

qiluo-msft · 2022-03-31T21:08:08Z

scripts/generate_dump

@@ -63,6 +63,10 @@ handle_exit()
    ECODE=$?
    echo "Removing lock. Exit: $ECODE" >&2
    $RM $V -rf ${LOCKDIR}
+    # Echo the filename as the last statement if the generation suceeds


suceeds

typo #Closed

qiluo-msft · 2022-03-31T21:08:37Z

scripts/generate_dump

@@ -63,6 +63,10 @@ handle_exit()
    ECODE=$?
    echo "Removing lock. Exit: $ECODE" >&2
    $RM $V -rf ${LOCKDIR}
+    # Echo the filename as the last statement if the generation suceeds


Echo

Isn't it already supported? I am seeing

mkdir: created directory '/var/dump/sonic_dump_vlab-01_20220331_204737/log' sonic_dump_vlab-01_20220331_204737/log/techsupport_time_info.Gg18UE4AVH removed '/var/dump/sonic_dump_vlab-01_20220331_204737/log/techsupport_time_info.Gg18UE4AVH' removed directory '/var/dump/sonic_dump_vlab-01_20220331_204737/core' removed directory '/var/dump/sonic_dump_vlab-01_20220331_204737/log' removed directory '/var/dump/sonic_dump_vlab-01_20220331_204737' /var/dump/sonic_dump_vlab-01_20220331_204737.tar: 5.5% -- replaced with /var/dump/sonic_dump_vlab-01_20220331_204737.tar.gz /var/dump/sonic_dump_vlab-01_20220331_204737.tar.gz ``` #Closed

After the addition of lock code, no. handle_exit is the last trap that runs before exiting, and it does print a few statements and thus the issue

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

qiluo-msft

LGTM

liat-grozovik · 2022-04-02T06:59:04Z

@ganglyu could yuo please review recent changes following feedback provided?

1. Print the last statement as the techsupport dump name, as some automation processes might depend of parsing the last line to infer the dump path. Previously: handle_exit Removing lock. Exit: 0 removed '/tmp/techsupport-lock/PID' removed directory '/tmp/techsupport-lock' Updated: handle_exit Removing lock. Exit: 0 removed '/tmp/techsupport-lock/PID' removed directory '/tmp/techsupport-lock' /var/dump/sonic_dump_r-bulldog-03_20220324_195553.tar.gz 2. Don't acquire the lock when running in NOOP mode 3. Set the set -v option just before running main so that it won't print the generate_dump code to stdout 4. Update the auto-techsupport script to handle EXT_RETRY and EXT_LOCKFAIL exit codes returned by show techsupport command. 5. Update the minor error in since argument for auto-techsupport Signed-off-by: Vivek Keddy Karri <vkarri@nvidia.com>

includes: 320591a [DualToR] Handle race condition between tunnel_decap and mux orchestrator (sonic-net#2114) 5027a8f Handling Invalid CRM configuration gracefully (sonic-net#2109) 0b120fa [ci]: use native arm64 and armhf pool (sonic-net#2013) 394e88a Don't handle buffer pool watermark during warm reboot reconciling (sonic-net#1987) 9008a01 patch for issue sonic-net#1971 - enable Rx Drop handling for cisco-8000 (sonic-net#2041) 2723ee3 create debug_shell_enable config to enable debug shell (sonic-net#2060) d7be0b9 [request parser] Add unit tests for request parser for multiple values (sonic-net#1766)

vivekrnv and others added 2 commits March 24, 2022 19:51

Minor Setbacks

d73bceb

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Update generate_dump

2f1d545

dgsudharsan requested a review from qiluo-msft March 25, 2022 17:22

dgsudharsan previously approved these changes Mar 25, 2022

View reviewed changes

dgsudharsan added the Request for 202111 Branch label Mar 25, 2022

vivekrnv added 2 commits March 25, 2022 23:18

auto-ts updated

8ba42b1

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

Merge branch 'ts_error' of https://github.com/vivekreddynv/sonic-util…

27002ac

…ities into ts_error

vivekrnv dismissed dgsudharsan’s stale review via 27002ac March 25, 2022 23:19

vivekrnv changed the title ~~[techsupport] Handle Minor Issues because of TS Locking~~ [techsupport] Handle minor fixes of TS Lock and update auto-TS Mar 25, 2022

vivekrnv requested a review from dgsudharsan March 25, 2022 23:24

Disable signal after the test

1901275

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

dgsudharsan reviewed Mar 28, 2022

View reviewed changes

Update coredump_gen_handler.py

fdd57fb

dgsudharsan previously approved these changes Mar 31, 2022

View reviewed changes

qiluo-msft requested a review from ganglyu March 31, 2022 20:45

qiluo-msft reviewed Mar 31, 2022

View reviewed changes

ganglyu previously approved these changes Apr 1, 2022

View reviewed changes

Comments handled

230c8df

Signed-off-by: Vivek Reddy Karri <vkarri@nvidia.com>

vivekrnv dismissed stale reviews from ganglyu and dgsudharsan via 230c8df April 1, 2022 02:18

qiluo-msft approved these changes Apr 1, 2022

View reviewed changes

ganglyu approved these changes Apr 2, 2022

View reviewed changes

liat-grozovik merged commit f70dc27 into sonic-net:master Apr 3, 2022

judyjoseph added the Included in 202111 Branch label Apr 4, 2022

vivekrnv mentioned this pull request Apr 4, 2022

[auto-ts] add memory check #2116

Merged

vivekrnv deleted the ts_error branch June 10, 2022 20:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[techsupport] Handle minor fixes of TS Lock and update auto-TS #2114

[techsupport] Handle minor fixes of TS Lock and update auto-TS #2114

vivekrnv commented Mar 25, 2022 •

edited

Loading

dgsudharsan commented Mar 25, 2022

dgsudharsan Mar 28, 2022

vivekrnv Mar 29, 2022

dgsudharsan Mar 28, 2022 •

edited

Loading

vivekrnv Mar 29, 2022

dgsudharsan commented Mar 31, 2022

qiluo-msft Mar 31, 2022 •

edited

Loading

vivekrnv Apr 1, 2022

qiluo-msft Mar 31, 2022 •

edited

Loading

qiluo-msft Mar 31, 2022 •

edited

Loading

vivekrnv Apr 1, 2022

qiluo-msft left a comment

liat-grozovik commented Apr 2, 2022

[techsupport] Handle minor fixes of TS Lock and update auto-TS #2114

[techsupport] Handle minor fixes of TS Lock and update auto-TS #2114

Conversation

vivekrnv commented Mar 25, 2022 • edited Loading

What I did

How I did it

How to verify it

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

dgsudharsan commented Mar 25, 2022

dgsudharsan Mar 28, 2022

Choose a reason for hiding this comment

vivekrnv Mar 29, 2022

Choose a reason for hiding this comment

dgsudharsan Mar 28, 2022 • edited Loading

Choose a reason for hiding this comment

vivekrnv Mar 29, 2022

Choose a reason for hiding this comment

dgsudharsan commented Mar 31, 2022

qiluo-msft Mar 31, 2022 • edited Loading

Choose a reason for hiding this comment

vivekrnv Apr 1, 2022

Choose a reason for hiding this comment

qiluo-msft Mar 31, 2022 • edited Loading

Choose a reason for hiding this comment

qiluo-msft Mar 31, 2022 • edited Loading

Choose a reason for hiding this comment

vivekrnv Apr 1, 2022

Choose a reason for hiding this comment

qiluo-msft left a comment

Choose a reason for hiding this comment

liat-grozovik commented Apr 2, 2022

vivekrnv commented Mar 25, 2022 •

edited

Loading

dgsudharsan Mar 28, 2022 •

edited

Loading

qiluo-msft Mar 31, 2022 •

edited

Loading

qiluo-msft Mar 31, 2022 •

edited

Loading

qiluo-msft Mar 31, 2022 •

edited

Loading