Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

There is only IPv6 address(es) in lldp_loc_man_addr in Redis #5603

Closed
gechiang opened this issue Oct 7, 2020 · 8 comments
Closed

There is only IPv6 address(es) in lldp_loc_man_addr in Redis #5603

gechiang opened this issue Oct 7, 2020 · 8 comments
Assignees
@shlomibitton

Comments

@gechiang
Copy link
Collaborator

gechiang commented Oct 7, 2020

Seeing SNMP subagent crash during boot up with master branch based image.
This issue is very easy to reproduce.
With any recent daily build master image should be able to hit this.

image version:

SONiC Software Version: SONiC.master.434-f61ff95e
Distribution: Debian 10.6
Kernel: 4.19.0-9-2-amd64
Build commit: f61ff95e
Build date: Tue Oct  6 08:20:44 UTC 2020
Built by: johnar@jenkins-worker-8

Syslog that shows events leading to the crash as following:

...
[snmpCrash_master1.txt](https://github.com/Azure/sonic-buildimage/files/5337992/snmpCrash_master1.txt)

Oct  7 01:51:32.883957 SONIC-02 ERR snmp#snmp-subagent [ax_interface] ERROR: MIBUpdater.start() caught an unexpected exception during update_data()#012Traceback (most recent call last):#012  File "/usr/local/lib/python3.7/dist-packages/ax_interface/mib.py", line 37, in start#012    self.reinit_data()#012  File "/usr/local/lib/python3.7/dist-packages/sonic_ax_impl/mibs/ieee802_1ab.py", line 325, in reinit_data#012    *mgmt_ip_sub_oid)#012TypeError: 'NoneType' object is not iterable
Oct  7 01:51:32.898820 SONIC-02 WARNING snmp#snmp-subagent [sonic_ax_impl] WARNING: Invalid interface name in *' in APP_DB, skipping
Oct  7 01:51:33.055819 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: AgentX socket connection established. Initiating opening handshake...
Oct  7 01:51:34.057430 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Sending open...
Oct  7 01:51:34.059318 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: AgentX session starting with ID: 8
Oct  7 01:51:34.059698 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.4.1.9.9.117.1.1.2.1.2]
Oct  7 01:51:34.060254 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.4.1.9.9.580.1.5.5.1.4]
Oct  7 01:51:34.060796 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.4.1.9.9.813.1.2.1.2]
Oct  7 01:51:34.061313 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.4.1.9.9.813.1.2.1.3]
Oct  7 01:51:34.061873 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.4.1.9.9.813.1.1.1.1]
Oct  7 01:51:34.062436 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.4.1.9.9.813.1.1.1.2]
Oct  7 01:51:34.063244 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.4.1.9.9.187.1.2.5.1.3]
Oct  7 01:51:34.063768 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.4.1.6027.3.10.1.2.9.1.2.1]
Oct  7 01:51:34.064286 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.4.1.6027.3.10.1.2.9.1.3.1]
Oct  7 01:51:34.064798 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.4.1.6027.3.10.1.2.9.1.4.1]
Oct  7 01:51:34.065309 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.4.1.6027.3.10.1.2.9.1.5.1]
Oct  7 01:51:34.065857 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.4.2.1.3]
Oct  7 01:51:34.066099 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.4.2.1.4]
Oct  7 01:51:34.066314 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.4.2.1.5]
Oct  7 01:51:34.066528 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.4.1.1.4]
Oct  7 01:51:34.066744 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.4.1.1.5]
Oct  7 01:51:34.066956 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.4.1.1.6]
Oct  7 01:51:34.067172 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.4.1.1.7]
Oct  7 01:51:34.067382 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.4.1.1.8]
Oct  7 01:51:34.067595 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.4.1.1.9]
Oct  7 01:51:34.067804 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.4.1.1.10]
Oct  7 01:51:34.068019 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.4.1.1.11]
Oct  7 01:51:34.068232 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.4.1.1.12]
Oct  7 01:51:34.068444 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.3.8.1.3]
Oct  7 01:51:34.068656 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.3.8.1.4]
Oct  7 01:51:34.068867 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.3.8.1.5]
Oct  7 01:51:34.069120 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.3.8.1.6]
Oct  7 01:51:34.069319 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.3.7.1.2]
Oct  7 01:51:34.069535 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.3.7.1.3]
Oct  7 01:51:34.069759 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.3.7.1.4]
Oct  7 01:51:34.069969 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.3.1]
Oct  7 01:51:34.070183 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.3.2]
Oct  7 01:51:34.070393 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.3.3]
Oct  7 01:51:34.070607 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.3.4]
Oct  7 01:51:34.070818 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.3.5]
Oct  7 01:51:34.071027 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.0.8802.1.1.2.1.3.6]
Oct  7 01:51:34.071245 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.4.24.4.1.1]
Oct  7 01:51:34.071458 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.4.24.4.1.16]
Oct  7 01:51:34.071673 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.17.7.1.2.2.1.2]
Oct  7 01:51:34.071894 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.31.1.1.1.1]
Oct  7 01:51:34.072105 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.31.1.1.1.2]
Oct  7 01:51:34.072320 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.31.1.1.1.3]
Oct  7 01:51:34.072532 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.31.1.1.1.4]
Oct  7 01:51:34.072742 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.31.1.1.1.5]
Oct  7 01:51:34.072956 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.31.1.1.1.6]
Oct  7 01:51:34.073170 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.31.1.1.1.7]
Oct  7 01:51:34.073382 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.31.1.1.1.8]
Oct  7 01:51:34.073596 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.31.1.1.1.9]
Oct  7 01:51:34.073827 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.31.1.1.1.10]
Oct  7 01:51:34.074045 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.31.1.1.1.11]
Oct  7 01:51:34.074250 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.31.1.1.1.12]
Oct  7 01:51:34.074465 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.31.1.1.1.13]
Oct  7 01:51:34.074681 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.31.1.1.1.14]
Oct  7 01:51:34.074894 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.31.1.1.1.15]
Oct  7 01:51:34.075106 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.31.1.1.1.16]
Oct  7 01:51:34.075316 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.31.1.1.1.17]
Oct  7 01:51:34.075527 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.31.1.1.1.18]
Oct  7 01:51:34.075737 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.31.1.1.1.19]
Oct  7 01:51:34.075950 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.99.1.1.1.1]
Oct  7 01:51:34.076165 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.99.1.1.1.2]
Oct  7 01:51:34.076377 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.99.1.1.1.3]
Oct  7 01:51:34.076590 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.99.1.1.1.4]
Oct  7 01:51:34.076802 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.99.1.1.1.5]
Oct  7 01:51:34.077021 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.47.1.1.1.1.2]
Oct  7 01:51:34.077235 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.47.1.1.1.1.5]
Oct  7 01:51:34.077453 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.47.1.1.1.1.7]
Oct  7 01:51:34.077671 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.47.1.1.1.1.8]
Oct  7 01:51:34.077997 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.47.1.1.1.1.9]
Oct  7 01:51:34.078260 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.47.1.1.1.1.10]
Oct  7 01:51:34.078469 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.47.1.1.1.1.11]
Oct  7 01:51:34.078687 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.47.1.1.1.1.12]
Oct  7 01:51:34.078902 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.47.1.1.1.1.13]
Oct  7 01:51:34.079120 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.4.21.1.7]
Oct  7 01:51:34.079336 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.4.22.1.2]
Oct  7 01:51:34.079547 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.1]
Oct  7 01:51:34.079759 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.1]
Oct  7 01:51:34.079971 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.2]
Oct  7 01:51:34.080182 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.3]
Oct  7 01:51:34.080392 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.4]
Oct  7 01:51:34.080601 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.5]
Oct  7 01:51:34.080813 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.6]
Oct  7 01:51:34.081022 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.7]
Oct  7 01:51:34.081232 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.8]
Oct  7 01:51:34.081442 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.9]
Oct  7 01:51:34.081653 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.10]
Oct  7 01:51:34.081897 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.11]
Oct  7 01:51:34.082102 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.12]
Oct  7 01:51:34.082314 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.13]
Oct  7 01:51:34.082523 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.14]
Oct  7 01:51:34.082735 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.15]
Oct  7 01:51:34.082947 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.16]
Oct  7 01:51:34.083156 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.17]
Oct  7 01:51:34.083368 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.18]
Oct  7 01:51:34.083577 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.19]
Oct  7 01:51:34.083789 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.20]
Oct  7 01:51:34.084001 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.21]
Oct  7 01:51:34.084211 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: Registering subID: [.1.3.6.1.2.1.2.2.1.22]
Oct  7 01:51:34.084344 SONIC-02 INFO snmp#snmp-subagent [ax_interface] INFO: OID registration complete. Waiting to receive PDUs...
Oct  7 01:51:35.883018 SONIC-02 INFO swss#supervisord 2020-10-07 01:51:29,440 INFO exited: enable_counters (exit status 0; expected)
Oct  7 01:51:36.886956 SONIC-02 ERR snmp#snmp-subagent [ax_interface] ERROR: MIBUpdater.start() caught an unexpected exception during update_data()#012Traceback (most recent call last):#012  File "/usr/local/lib/python3.7/dist-packages/ax_interface/mib.py", line 37, in start#012    self.reinit_data()#012  File "/usr/local/lib/python3.7/dist-packages/sonic_ax_impl/mibs/ieee802_1ab.py", line 325, in reinit_data#012    *mgmt_ip_sub_oid)#012TypeError: 'NoneType' object is not iterable
Oct  7 01:51:48.047393 SONIC-02 INFO systemd[1]: Stopped Telemetry container.
Oct  7 01:51:48.049699 SONIC-02 INFO systemd[1]: Starting Telemetry container...

Steps to reproduce the issue:

  1. Load master image 434
  2. sudo reboot
  3. Wait for system to complete boot up, issue "show logging" and search for "snmp-subagent [ax_interface] ERROR:"
@lguohan
Copy link
Collaborator

lguohan commented Oct 7, 2020

@qiluo-msft , can you take a look at this issue?

@qiluo-msft qiluo-msft self-assigned this Oct 7, 2020
@qiluo-msft qiluo-msft transferred this issue from sonic-net/sonic-buildimage Oct 7, 2020
@qiluo-msft
Copy link
Collaborator

qiluo-msft commented Oct 12, 2020 •
edited
Loading

I raised a PR sonic-net/sonic-snmpagent#164 to correctly handle that lldp_loc_man_addr contains only IPv6 address without IPv4 address.

However the root issue is outside snmp container. Why there is only IPv6 address(es) in lldp_loc_man_addr in Redis. @gechiang Please help further troubleshoot.

@qiluo-msft qiluo-msft reopened this Oct 12, 2020
@qiluo-msft qiluo-msft changed the title snmp#snmp-subagent [ax_interface] ERROR: MIBUpdater.start() caught an unexpected exception during update_data()... There is only IPv6 address(es) in lldp_loc_man_addr in Redis Oct 13, 2020
@qiluo-msft qiluo-msft transferred this issue from sonic-net/sonic-snmpagent Oct 13, 2020
@liat-grozovik
Copy link
Collaborator

@shlomibitton i am unable to assign you to this issue. can you please assign yourself?

@shlomibitton
Copy link
Contributor

@liat-grozovik I can't also, only the owner can or when given write permission I think

@gechiang
Copy link
Collaborator Author

@shlomibitton Assigned...
cc @liat-grozovik
Thanks!

@bratashX
Copy link
Contributor

This PR can fix the problem with IPv6 address: #5699

@akokhan akokhan mentioned this issue Oct 23, 2020
Merged
3 tasks
@shlomibitton
Copy link
Contributor

@gechiang this issue resolved by PR #5699
thanks @bratashX

@liat-grozovik
Copy link
Collaborator

as mentioned, resolved

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shlomibitton shlomibitton

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@lguohan @qiluo-msft @liat-grozovik @shlomibitton @gechiang @bratashX