Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

show interface transceiver broken in recent builds #5001

Closed
davidfordouce opened this issue Jul 18, 2020 · 7 comments
Closed

show interface transceiver broken in recent builds #5001

davidfordouce opened this issue Jul 18, 2020 · 7 comments
Assignees
@jleveque
Labels
Bug 🐛 Triaged this issue has been triaged

Comments

@davidfordouce
Copy link

davidfordouce commented Jul 18, 2020 •
edited
Loading

Description

I have recently upgraded a number of Dell z9100 (Broadcom) switchs we use here to the latest build, and whilst the interfaces come up, the data presented by the show interface transceiver series of commands seems to now be broken.

Steps to reproduce the issue:

  1. install a recent sonic build eg. 1ca47da
  2. run show int tran pres
  3. it shows no transceivers even though the interfaces are up.

Describe the results you received:

$ sudo show int tran pres
Port         Presence
-----------  -----------
Ethernet0    Not present
Ethernet4    Not present

Describe the results you expected:

$ sudo show int tran pres
Port         Presence
-----------  -----------
Ethernet0    Present
Ethernet4    Present

Output of show version:

SONiC Software Version: SONiC.master.350-1ca47da4
Distribution: Debian 10.4
Kernel: 4.19.0-9-2-amd64
Build commit: 1ca47da4
Build date: Thu Jul 16 20:16:33 UTC 2020
Built by: johnar@jenkins-worker-11

Platform: x86_64-dell_z9100_c2538-r0
HwSKU: Force10-Z9100-C32
ASIC: broadcom
Serial Number: CN098HTFCES0094J0038
Uptime: 12:36:40 up 55 min,  1 user,  load average: 0.66, 0.42, 0.39

Docker images:
REPOSITORY                    TAG                   IMAGE ID            SIZE
docker-teamd                  latest                0beefd3dd4f3        380MB
docker-teamd                  master.350-1ca47da4   0beefd3dd4f3        380MB
docker-router-advertiser      latest                7bf201c77382        350MB
docker-router-advertiser      master.350-1ca47da4   7bf201c77382        350MB
docker-database               latest                a6e51f985e04        350MB
docker-database               master.350-1ca47da4   a6e51f985e04        350MB
docker-lldp                   latest                c5510cddfb61        377MB
docker-lldp                   master.350-1ca47da4   c5510cddfb61        377MB
docker-orchagent              latest                35b37c5ec89d        393MB
docker-orchagent              master.350-1ca47da4   35b37c5ec89d        393MB
docker-dhcp-relay             latest                6c16937acc47        357MB
docker-dhcp-relay             master.350-1ca47da4   6c16937acc47        357MB
docker-sonic-telemetry        latest                9ac7ecf61b12        414MB
docker-sonic-telemetry        master.350-1ca47da4   9ac7ecf61b12        414MB
docker-sonic-mgmt-framework   latest                afa98179e020        473MB
docker-sonic-mgmt-framework   master.350-1ca47da4   afa98179e020        473MB
docker-sflow                  latest                f2ad6f747351        383MB
docker-sflow                  master.350-1ca47da4   f2ad6f747351        383MB
docker-snmp                   latest                4e9b52a8ffa4        390MB
docker-snmp                   master.350-1ca47da4   4e9b52a8ffa4        390MB
docker-syncd-brcm             latest                292cec320095        442MB
docker-syncd-brcm             master.350-1ca47da4   292cec320095        442MB
docker-platform-monitor       latest                f7a53bf77033        358MB
docker-platform-monitor       master.350-1ca47da4   f7a53bf77033        358MB
docker-nat                    latest                46635e57e505        317MB
docker-nat                    master.350-1ca47da4   46635e57e505        317MB
docker-fpm-frr                latest                bff226a863c8        335MB
docker-fpm-frr                master.350-1ca47da4   bff226a863c8        335MB
@jleveque
Copy link
Contributor

Are you noticing xcvrd crashes in the syslog? If so, I believe this should be fixed by #4975.

Please try installing the latest master branch image and report your results.

@jleveque jleveque self-assigned this Jul 18, 2020
@davidfordouce
Copy link
Author

I'm afraid this all seems even more broken in the latest build, so I can't get as far as seeing a status:

admin@sonic:~$ sudo show int tran pres
Traceback (most recent call last):
  File "/usr/bin/sfpshow", line 361, in <module>
    cli()
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/bin/sfpshow", line 358, in presence
    sfp.display_presence(port)
  File "/usr/bin/sfpshow", line 327, in display_presence
    for i in port_table_keys:
TypeError: 'NoneType' object is not iterable

Here are the version details:

admin@sonic:~$ sudo show version

SONiC Software Version: SONiC.master.353-b3ae7858
Distribution: Debian 10.4
Kernel: 4.19.0-9-2-amd64
Build commit: b3ae7858
Build date: Sun Jul 19 13:27:43 UTC 2020
Built by: johnar@jenkins-worker-7

Platform: x86_64-dell_z9100_c2538-r0
HwSKU: Force10-Z9100-C32
ASIC: broadcom
Serial Number: CN098HTFCES0094J0038
Uptime: 19:38:20 up  1:12,  1 user,  load average: 0.01, 0.05, 0.07

Docker images:
REPOSITORY                    TAG                   IMAGE ID            SIZE
docker-teamd                  latest                be843e740dd2        380MB
docker-teamd                  master.353-b3ae7858   be843e740dd2        380MB
docker-nat                    latest                d7faaa0c4d23        382MB
docker-nat                    master.353-b3ae7858   d7faaa0c4d23        382MB
docker-router-advertiser      latest                f5d8a5da5e15        350MB
docker-router-advertiser      master.353-b3ae7858   f5d8a5da5e15        350MB
docker-platform-monitor       latest                844697e3e64a        422MB
docker-platform-monitor       master.353-b3ae7858   844697e3e64a        422MB
docker-database               latest                f63204dd0f61        351MB
docker-database               master.353-b3ae7858   f63204dd0f61        351MB
docker-lldp                   latest                ebfc33f71809        377MB
docker-lldp                   master.353-b3ae7858   ebfc33f71809        377MB
docker-orchagent              latest                d8c14690e66c        393MB
docker-orchagent              master.353-b3ae7858   d8c14690e66c        393MB
docker-sonic-telemetry        latest                8764f9533b8d        414MB
docker-sonic-telemetry        master.353-b3ae7858   8764f9533b8d        414MB
docker-snmp                   latest                8605e02d8f8b        390MB
docker-snmp                   master.353-b3ae7858   8605e02d8f8b        390MB
docker-dhcp-relay             latest                870bed6def46        357MB
docker-dhcp-relay             master.353-b3ae7858   870bed6def46        357MB
docker-sonic-mgmt-framework   latest                a1cf99e915fe        473MB
docker-sonic-mgmt-framework   master.353-b3ae7858   a1cf99e915fe        473MB
docker-sflow                  latest                577f5c5eed53        383MB
docker-sflow                  master.353-b3ae7858   577f5c5eed53        383MB
docker-syncd-brcm             latest                21dd40c5d954        447MB
docker-syncd-brcm             master.353-b3ae7858   21dd40c5d954        447MB
docker-fpm-frr                latest                4e1bfd2f9ab7        340MB
docker-fpm-frr                master.353-b3ae7858   4e1bfd2f9ab7        340MB

@jleveque
Copy link
Contributor

jleveque commented Jul 19, 2020 •
edited
Loading

I still believe the original problem you reported is fixed. However, there appears to be a new, unrelated issue. I believe I have reproduced it. Can you please provide the output of the following commands:

show interfaces status 
docker ps

@davidfordouce
Copy link
Author

Thanks, that seems plausible.

Here is the output you requested.

root@sonicshow interfaces status
root@sonic:~# docker ps
CONTAINER ID        IMAGE                                COMMAND                  CREATED             STATUS              PORTS               NAMES
21cda31e5df1        docker-sonic-telemetry:latest        "/usr/bin/supervisord"   3 hours ago         Up 3 hours                              telemetry
a1256eb33e3d        docker-snmp:latest                   "/usr/bin/supervisord"   3 hours ago         Up 3 hours                              snmp
882634acfad5        docker-nat:latest                    "/usr/bin/supervisord"   3 hours ago         Up 3 hours                              nat
1e1c93e25ff9        docker-dhcp-relay:latest             "/usr/bin/docker_ini…"   3 hours ago         Up 3 hours                              dhcp_relay
b76f63dc9146        docker-lldp:latest                   "/usr/bin/docker-lld…"   3 hours ago         Up 3 hours                              lldp
930b619ac4bc        docker-router-advertiser:latest      "/usr/bin/docker-ini…"   3 hours ago         Up 3 hours                              radv
7aa17a5f4a3f        docker-sonic-mgmt-framework:latest   "/usr/bin/supervisord"   3 hours ago         Up 3 hours                              mgmt-framework
176ed28cd234        docker-syncd-brcm:latest             "/usr/bin/supervisord"   3 hours ago         Up 3 hours                              syncd
d49ad8b258d0        docker-teamd:latest                  "/usr/bin/supervisord"   3 hours ago         Up 3 hours                              teamd
56dd06787aa6        docker-orchagent:latest              "/usr/bin/docker-ini…"   3 hours ago         Up 3 hours                              swss
4bcf5f04d217        docker-platform-monitor:latest       "/usr/bin/docker_ini…"   3 hours ago         Up 3 hours                              pmon
905a0bbd560c        docker-fpm-frr:latest                "/usr/bin/docker_ini…"   3 hours ago         Up 3 hours                              bgp
14f888e2897e        docker-database:latest               "/usr/local/bin/dock…"   3 hours ago         Up 3 hours                              database

@jleveque
Copy link
Contributor

That's the same symptom I saw. show interfaces status displays no output.

It appears this new bug was introduced in a commit between 18bfa6d and b3ae785, inclusive. Unfortunately, there were no other nightly builds on Jenkins between these commits, so we don't have images readily available to test.

@yxieca yxieca added the Triaged this issue has been triaged label Aug 5, 2020
@jleveque
Copy link
Contributor

@davidfordouce: This issue appears to be resolved in the latest master branch. Can you please test an image from the current head?

@davidfordouce
Copy link
Author

I agree, this has been resolved in the current head.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jleveque jleveque

Labels
Bug 🐛 Triaged this issue has been triaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jleveque @yxieca @davidfordouce