Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SNMP missing expected LLDP data in master image #5812

Closed
daall opened this issue Nov 4, 2020 · 6 comments · Fixed by #5699
Closed

SNMP missing expected LLDP data in master image #5812

daall opened this issue Nov 4, 2020 · 6 comments · Fixed by #5699
Assignees
Labels
Master Branch Quality P1 Priority of the issue, lower than P0

Comments

@daall
Copy link
Contributor

daall commented Nov 4, 2020

Description
https://github.com/Azure/sonic-mgmt/blob/master/tests/snmp/test_snmp_lldp.py is failing consistently in the master image, going back at least as far as image #447.

Steps to reproduce the issue:

  1. Run snmp/test_snmp_lldp.py

Describe the results you received:

# Check if lldpLocPortTable is present for all ports
        for k, v in snmp_facts['snmp_interfaces'].items():
            if "Ethernet" in v['name'] or "eth" in v['name']:
                for oid in ['lldpLocPortIdSubtype', 'lldpLocPortId', 'lldpLocPortDesc']:
                    assert v.has_key(oid)
                    assert "No Such Object currently exists" not in v[oid]
    
        # Check if lldpLocManAddrTable is present
        for k in ['lldpLocManAddrLen', \
                   'lldpLocManAddrIfSubtype', \
                   'lldpLocManAddrIfId', \
                   'lldpLocManAddrOID']:
>           assert snmp_facts['snmp_lldp'][k]
E           KeyError: 'lldpLocManAddrLen'

Describe the results you expected:
The test should pass.

Additional info
Looking at the data, it doesn't look like any of these keys are present. I'm not sure if this is an issue in lldp or snmp, or if this is some sort of platform issue.

Output of show version:

admin@sonic:~$ show ver

SONiC Software Version: SONiC.master.469-84d3a260
Distribution: Debian 10.6
Kernel: 4.19.0-9-2-amd64
Build commit: 84d3a260
Build date: Wed Nov  4 00:55:31 UTC 2020
Built by: johnar@jenkins-worker-8

Docker images:
REPOSITORY                    TAG                   IMAGE ID            SIZE
docker-snmp                   latest                e21c75cf3e75        427MB
docker-snmp                   master.469-84d3a260   e21c75cf3e75        427MB
docker-teamd                  latest                93c913c5a017        425MB
docker-teamd                  master.469-84d3a260   93c913c5a017        425MB
docker-sonic-mgmt-framework   latest                244b796dd024        551MB
docker-sonic-mgmt-framework   master.469-84d3a260   244b796dd024        551MB
docker-router-advertiser      latest                c31265cbdcd2        390MB
docker-router-advertiser      master.469-84d3a260   c31265cbdcd2        390MB
docker-platform-monitor       latest                106e63b358ad        503MB
docker-platform-monitor       master.469-84d3a260   106e63b358ad        503MB
docker-lldp                   latest                9ba8b14bb844        456MB
docker-lldp                   master.469-84d3a260   9ba8b14bb844        456MB
docker-dhcp-relay             latest                d48a5f726cd7        397MB
docker-dhcp-relay             master.469-84d3a260   d48a5f726cd7        397MB
docker-database               latest                9af7507f9821        390MB
docker-database               master.469-84d3a260   9af7507f9821        390MB
docker-orchagent              latest                6276adebac5b        439MB
docker-orchagent              master.469-84d3a260   6276adebac5b        439MB
docker-nat                    latest                0a1ee7e92192        428MB
docker-nat                    master.469-84d3a260   0a1ee7e92192        428MB
docker-sonic-telemetry        latest                e387512c848f        459MB
docker-sonic-telemetry        master.469-84d3a260   e387512c848f        459MB
docker-fpm-frr                latest                0a87f992c6a5        441MB
docker-fpm-frr                master.469-84d3a260   0a87f992c6a5        441MB
docker-sflow                  latest                c5edda64f062        429MB
docker-sflow                  master.469-84d3a260   c5edda64f062        429MB
docker-syncd-brcm             latest                bac1974ef6a7        527MB
docker-syncd-brcm             master.469-84d3a260   bac1974ef6a7        527MB
@daall daall added P1 Priority of the issue, lower than P0 Master Branch Quality labels Nov 4, 2020
@lguohan
Copy link
Collaborator

lguohan commented Nov 4, 2020

https://sonic-jenkins.westus2.cloudapp.azure.com/job/vs/job/buildimage-vs-image/test_results_analyzer/

the test passing for kvm image consistently. can you check if it is testbed issue?

@daall
Copy link
Contributor Author

daall commented Nov 4, 2020

Nope, it looks like the snmp data is missing:

admin@sonic:~$ docker exec -it snmp snmpwalk -v2c 1.0.8802.1.1.2.1.3.8.1.3
iso.0.8802.1.1.2.1.3.8.1.3 = No Such Instance currently exists at this OID
admin@sonic:~$ docker exec -it snmp snmpwalk -v2c 1.0.8802.1.1.2.1.3.8.1.4
iso.0.8802.1.1.2.1.3.8.1.4 = No Such Instance currently exists at this OID
admin@sonic:~$ docker exec -it snmp snmpwalk -v2c 1.0.8802.1.1.2.1.3.8.1.5
iso.0.8802.1.1.2.1.3.8.1.5 = No Such Instance currently exists at this OID
admin@sonic:~$ docker exec -it snmp snmpwalk -v2c 1.0.8802.1.1.2.1.3.8.1.6
iso.0.8802.1.1.2.1.3.8.1.6 = No Such Instance currently exists at this OID

This is noted on 4 different physical testbeds, so it seems like there is some sort of consistent issue here.

@daall
Copy link
Contributor Author

daall commented Nov 4, 2020

The data is present in APP DB so it seems like the subagent is failing to fetch it:

admin@sonic:~$ redis-cli -n 0 hgetall "LLDP_LOC_CHASSIS"
 1) "lldp_loc_man_addr"
 2) "fc00:2::32"
 3) "lldp_loc_sys_name"
 4) "sonic"
 5) "lldp_loc_sys_desc"
 6) "Debian GNU/Linux 10 (buster) Linux 4.19.0-9-2-amd64 #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07) x86_64"
 7) "lldp_loc_sys_cap_supported"
 8) "28 00"
 9) "lldp_loc_chassis_id_subtype"
10) "4"
11) "lldp_loc_chassis_id"
12) "00:e0:ec:c2:ae:74"
13) "lldp_loc_sys_cap_enabled"
14) "28 00"

I also noticed this in the logs:

Nov  4 22:43:32.312211 sonic WARNING snmp#snmp-subagent [sonic_ax_impl] WARNING: Exception when updating lldpRemTable: b'lldp_rem_sys_cap_enabled'
Nov  4 22:43:38.388668 sonic ERR snmp#snmp-subagent [sonic_ax_impl] ERROR: Could not find IPv4 address in lldp_loc_man_addr
Nov  4 23:08:46.262865 sonic ERR snmp#snmp-subagent message repeated 21 times: [ [sonic_ax_impl] ERROR: Could not find IPv4 address in lldp_loc_man_addr]
Nov  4 23:08:46.262865 sonic WARNING snmp#snmp-subagent [sonic_ax_impl] WARNING: Exception when updating lldpRemTable: b'lldp_rem_sys_cap_enabled'
Nov  4 23:09:04.348220 sonic ERR snmp#snmp-subagent [sonic_ax_impl] ERROR: Could not find IPv4 address in lldp_loc_man_addr
Nov  4 23:10:30.809883 sonic ERR snmp#snmp-subagent [sonic_ax_impl] ERROR: Could not find IPv4 address in lldp_loc_man_addr
Nov  4 23:10:30.809971 sonic WARNING snmp#snmp-subagent [sonic_ax_impl] WARNING: Exception when updating lldpRemTable: b'lldp_rem_sys_cap_enabled'
Nov  4 23:10:30.824227 sonic WARNING snmp#snmp-subagent [sonic_ax_impl] WARNING: Exception when updating lldpRemTable: b'lldp_rem_index'
Nov  4 23:11:29.879079 sonic ERR snmp#snmp-subagent [sonic_ax_impl] ERROR: Could not find IPv4 address in lldp_loc_man_addr
Nov  4 23:12:44.567951 sonic ERR snmp#snmp-subagent [sonic_ax_impl] ERROR: Could not find IPv4 address in lldp_loc_man_addr
Nov  4 23:12:44.568060 sonic WARNING snmp#snmp-subagent [sonic_ax_impl] WARNING: Exception when updating lldpRemTable: b'lldp_rem_sys_cap_enabled'
Nov  4 23:13:45.990756 sonic ERR snmp#snmp-subagent [sonic_ax_impl] ERROR: Could not find IPv4 address in lldp_loc_man_addr
Nov  4 23:22:42.126567 sonic ERR snmp#snmp-subagent message repeated 7 times: [ [sonic_ax_impl] ERROR: Could not find IPv4 address in lldp_loc_man_addr]
Nov  4 23:22:42.126567 sonic WARNING snmp#snmp-subagent [sonic_ax_impl] WARNING: Exception when updating lldpRemTable: b'lldp_rem_index'
Nov  4 23:23:00.223435 sonic ERR snmp#snmp-subagent [sonic_ax_impl] ERROR: Could not find IPv4 address in lldp_loc_man_addr
Nov  4 23:25:06.761558 sonic ERR snmp#snmp-subagent [sonic_ax_impl] ERROR: Could not find IPv4 address in lldp_loc_man_addr
Nov  4 23:25:06.761558 sonic WARNING snmp#snmp-subagent [sonic_ax_impl] WARNING: Exception when updating lldpRemTable: b'lldp_rem_index'
Nov  4 23:25:15.371230 sonic ERR snmp#snmp-subagent [sonic_ax_impl] ERROR: Could not find IPv4 address in lldp_loc_man_addr
Nov  4 23:30:36.179492 sonic ERR snmp#snmp-subagent message repeated 4 times: [ [sonic_ax_impl] ERROR: Could not find IPv4 address in lldp_loc_man_addr]
Nov  4 23:30:36.179492 sonic WARNING snmp#snmp-subagent [sonic_ax_impl] WARNING: Exception when updating lldpRemTable: b'lldp_rem_index'
Nov  4 23:30:36.185234 sonic WARNING snmp#snmp-subagent [sonic_ax_impl] WARNING: Exception when updating lldpRemTable: b'lldp_rem_sys_cap_enabled'
Nov  4 23:31:07.458639 sonic ERR snmp#snmp-subagent [sonic_ax_impl] ERROR: Could not find IPv4 address in lldp_loc_man_addr

@daall
Copy link
Contributor Author

daall commented Nov 4, 2020

@lguohan I am pretty sure it is because the physical DUTs have an IPv6 address for lldp_loc_man_addr. I see the subagent is explicitly looking for a v4 address here:

https://github.com/Azure/sonic-snmpagent/blob/57e54d9561934b853a44185bbb401207f4503292/src/sonic_ax_impl/mibs/ieee802_1ab.py#L321

Is this a bug in snmp or somewhere else?

Btw the VS testbed has an IPv4 address in that field, which is why it is passing consistently.

@lguohan
Copy link
Collaborator

lguohan commented Nov 5, 2020

since @qiluo-msft has recently fixed a similiar issue, let me assign to @qiluo-msft . @qiluo-msft , can you take a look at this issue?

@qiluo-msft
Copy link
Collaborator

There is a fix PR #5699
lldp_loc_man_addr should be an IPv4 address.

@daall daall linked a pull request Nov 5, 2020 that will close this issue
3 tasks
lguohan pushed a commit that referenced this issue Nov 7, 2020
…5699)

Fix #5812

LLDP conf Jinja2 Template does not verify IPv4 address and can use IPv6 version. This issue does not effect control LLDP daemon. Issue can be reproduced via `test_snmp_lldp` test. LLDP conf Jinja2 Template selects first item from the list of mgmt interfaces.

TESTBED_1 LLDP conf

```
# cat /etc/lldpd.conf 
configure ports eth0 lldp portidsubtype local eth0
configure system ip management pattern FC00:3::32
configure system hostname dut-1
```
TESTBED_2  LLDP conf

```
# cat /etc/lldpd.conf
configure ports eth0 lldp portidsubtype local eth0
configure system ip management pattern 10.22.24.61
configure system hostname dut-2
```
TESTBED_1  MGMT_INTERFACE

```
$ redis-cli -n 4 keys "*" | grep MGMT_INTERFACE
MGMT_INTERFACE|eth0|10.22.24.53/23
MGMT_INTERFACE|eth0|FC00:3::32/64
```
TESTBED_2  MGMT_INTERFACE

```
$ redis-cli -n 4 keys "*" | grep MGMT_INTERFACE
MGMT_INTERFACE|eth0|FC00:3::32/64
MGMT_INTERFACE|eth0|10.22.24.61/23

```

Signed-off-by: Petro Bratash <petrox.bratash@intel.com>
abdosi pushed a commit that referenced this issue Feb 11, 2021
…5699)

Fix #5812

LLDP conf Jinja2 Template does not verify IPv4 address and can use IPv6 version. This issue does not effect control LLDP daemon. Issue can be reproduced via `test_snmp_lldp` test. LLDP conf Jinja2 Template selects first item from the list of mgmt interfaces.

TESTBED_1 LLDP conf

```
configure ports eth0 lldp portidsubtype local eth0
configure system ip management pattern FC00:3::32
configure system hostname dut-1
```
TESTBED_2  LLDP conf

```
configure ports eth0 lldp portidsubtype local eth0
configure system ip management pattern 10.22.24.61
configure system hostname dut-2
```
TESTBED_1  MGMT_INTERFACE

```
$ redis-cli -n 4 keys "*" | grep MGMT_INTERFACE
MGMT_INTERFACE|eth0|10.22.24.53/23
MGMT_INTERFACE|eth0|FC00:3::32/64
```
TESTBED_2  MGMT_INTERFACE

```
$ redis-cli -n 4 keys "*" | grep MGMT_INTERFACE
MGMT_INTERFACE|eth0|FC00:3::32/64
MGMT_INTERFACE|eth0|10.22.24.61/23

```

Signed-off-by: Petro Bratash <petrox.bratash@intel.com>
santhosh-kt pushed a commit to santhosh-kt/sonic-buildimage that referenced this issue Feb 25, 2021
…onic-net#5699)

Fix sonic-net#5812

LLDP conf Jinja2 Template does not verify IPv4 address and can use IPv6 version. This issue does not effect control LLDP daemon. Issue can be reproduced via `test_snmp_lldp` test. LLDP conf Jinja2 Template selects first item from the list of mgmt interfaces.

TESTBED_1 LLDP conf

```
# cat /etc/lldpd.conf 
configure ports eth0 lldp portidsubtype local eth0
configure system ip management pattern FC00:3::32
configure system hostname dut-1
```
TESTBED_2  LLDP conf

```
# cat /etc/lldpd.conf
configure ports eth0 lldp portidsubtype local eth0
configure system ip management pattern 10.22.24.61
configure system hostname dut-2
```
TESTBED_1  MGMT_INTERFACE

```
$ redis-cli -n 4 keys "*" | grep MGMT_INTERFACE
MGMT_INTERFACE|eth0|10.22.24.53/23
MGMT_INTERFACE|eth0|FC00:3::32/64
```
TESTBED_2  MGMT_INTERFACE

```
$ redis-cli -n 4 keys "*" | grep MGMT_INTERFACE
MGMT_INTERFACE|eth0|FC00:3::32/64
MGMT_INTERFACE|eth0|10.22.24.61/23

```

Signed-off-by: Petro Bratash <petrox.bratash@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Master Branch Quality P1 Priority of the issue, lower than P0
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants