Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GNMI CLI Subscribes With No Namespace Results GNMI Crashes on Multi-Asic Chassis #20510

Open
wumiaont opened this issue Oct 15, 2024 · 2 comments
Assignees

Comments

@wumiaont
Copy link
Contributor

Description

We are using multi-asic chassis (Nokia-7250) to test telemetry. It is found if I provide the following CLI from ptf server without providing namespace information, GNMI container crashes and restarts.

python /root/gnxi/gnmi_cli_py/py_gnmicli.py -g -t 10.250.6.233 -p 50052 -m subscribe -x NEIGH_STATE_TABLE -xt STATE_DB -o ndastreamingservertest --subscribe_mode 0 --submode 1 --interval 0 --update_count 2 --create_connections 1

Sending SubscribeRequest
subscribe {
prefix {
target: "STATE_DB"
}
subscription {
path {
elem {
name: "NEIGH_STATE_TABLE"
}
}
mode: ON_CHANGE
}
}

Received an exception from server side and error message is: '<_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Connection reset by peer"
debug_error_string = "{"created":"@1729002509.589264167","description":"Error received from peer ipv4:10.250.6.233:50052","file":"src/core/lib/surface/call.cc","file_line":1070,"grpc_message":"Connection reset by peer","grpc_status":14}"

'.
Client receives an exception 'Connection reset by peer' indicating gNMI server is shut down and Exiting ...

Look at chassis: GNMI container restarts by that command.

root@ixre-egl-board30:/# admin@ixre-egl-board30:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c93e2246ff1f docker-snmp:latest "/usr/local/bin/supe?" 2 weeks ago Up 26 minutes snmp
c4c5e555d8d9 docker-sonic-mgmt-framework:latest "/usr/local/bin/supe?" 2 weeks ago Up 27 minutes mgmt-framework
f2784babf897 docker-lldp:latest "/usr/bin/docker-lld?" 2 weeks ago Up 27 minutes lldp1
ec0b77ec803f docker-lldp:latest "/usr/bin/docker-lld?" 2 weeks ago Up 27 minutes lldp0
fbec90b49288 docker-sonic-gnmi:latest "/usr/local/bin/supe?" 2 weeks ago Exited (0) 4 seconds ago gnmi

Steps to reproduce the issue:

  1. Using multi-asic chassis. Config GNMI to have certs and correct ports and no client_auth.
  2. Restart GNMI service.
  3. Run "python /root/gnxi/gnmi_cli_py/py_gnmicli.py -g -t 10.250.6.233 -p 50052 -m subscribe -x NEIGH_STATE_TABLE -xt STATE_DB -o ndastreamingservertest --subscribe_mode 0 --submode 1 --interval 0 --update_count 2 --create_connections 1" from PRF container
  4. Watch chassis GNMI container crashes and restarts.
  5. If you run with namespace associated then it will be fine. Such as "python /root/gnxi/gnmi_cli_py/py_gnmicli.py -g -t 10.250.6.233 -p 50052 -m subscribe -x NEIGH_STATE_TABLE -xt STATE_DB/asic0 -o ndastreamingservertest --subscribe_mode 0 --submode 1 --interval 0 --update_count 2 --create_connections 1

Describe the results you received:

GNMI crashes.

Describe the results you expected:

No crash no matter what wrong CLI input provided.

Output of show version:

2405

(paste your output here)
@wumiaont
Copy link
Contributor Author

Log from gnmi.log when that's happening.

2024 Oct 16 15:52:03.976726 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native I1016 15:52:03.976540 21 db_client.go:675] Invalid db table Path STATE_DB NEIGH_STATE_TABLE
2024 Oct 16 15:52:03.976726 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native I1016 15:52:03.976566 21 connection_manager.go:81] Closing connection: 10.250.6.253:56678|STATE_DB|2024-10-16T1
5:52:03Z
2024 Oct 16 15:52:03.976938 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native I1016 15:52:03.976680 21 panic.go:890] Client 10.250.6.253:56678 shutdown
2024 Oct 16 15:52:03.979007 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native panic: runtime error: invalid memory address or nil pointer dereference
2024 Oct 16 15:52:03.979007 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native [signal SIGSEGV: segmentation violation code=0x1 addr=0x0
2024 Oct 16 15:52:03.979201 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native pc=0xe54b89]
2024 Oct 16 15:52:03.979222 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native
2024 Oct 16 15:52:03.979222 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native goroutine 117 [running]:
2024 Oct 16 15:52:03.979317 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native github.com/sonic-net/sonic-gnmi/gnmi_server.(*Client).Run(0xc00019e870, {0x12f8958?, 0xc0004d4160})
2024 Oct 16 15:52:03.979317 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native #11/sonic/src/sonic-gnmi/gnmi_server/client_subscribe.go:182 +0xac9
2024 Oct 16 15:52:03.979317 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native github.com/sonic-net/sonic-gnmi/gnmi_server.(*Server).Subscribe(0xc00094c550, {0x12f8958, 0xc0004d4160})
2024 Oct 16 15:52:03.979356 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native #11/sonic/src/sonic-gnmi/gnmi_server/server.go:320 +0x3ab
2024 Oct 16 15:52:03.979391 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native github.com/openconfig/gnmi/proto/gnmi._GNMI_Subscribe_Handler({0x10f0fe0
2024 Oct 16 15:52:03.979517 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native ?, 0xc00094c550}, {0x12f4c68?, 0xc000552600})
2024 Oct 16 15:52:03.979627 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native #11/sonic/src/sonic-gnmi/vendor/github.com/openconfig/gnmi/proto/gnmi/gnmi.pb.go:3412 +0x9f
2024 Oct 16 15:52:03.979669 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native google.golang.org/grpc.(*Server).processStreamingRPC(0xc0004ee000, {0x12f9378, 0xc000758480}, 0xc000206000, 0xc0000f
ae40, 0x19a4200, 0x0)
2024 Oct 16 15:52:03.979726 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native #11/sonic/src/sonic-gnmi/vendor/google.golang.org/grpc/server.go:1457 +0xd4a
2024 Oct 16 15:52:03.979726 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native google.golang.org/grpc.(*Server).handleStream(0xc0004ee000, {0x12f9378, 0xc000758480}, 0xc000206000, 0x0)
2024 Oct 16 15:52:03.979726 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native #11/sonic/src/sonic-gnmi/vendor/google.golang.org/grpc/server.go:1537 +0x9ea
2024 Oct 16 15:52:03.979821 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native google.golang.org/grpc.(*Server).serveStreams.func1.2()
2024 Oct 16 15:52:03.979821 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native #11/sonic/src/sonic-gnmi/vendor/google.golang.org/grpc/server.go:871 +0x98
2024 Oct 16 15:52:03.979821 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native created by google.golang.org/grpc.(*Server).serveStreams.func1
2024 Oct 16 15:52:03.979847 ixre-egl-board30 INFO gnmi#supervisord: gnmi-native #11/sonic/src/sonic-gnmi/vendor/google.golang.org/grpc/server.go:869 +0x28a

@wumiaont
Copy link
Contributor Author

sonic-net/sonic-gnmi#309

@arlakshm arlakshm added Issue for 202405 Chassis 🤖 Modular chassis support labels Oct 25, 2024
@zbud-msft zbud-msft self-assigned this Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: No status
Development

No branches or pull requests

4 participants