[SNMP][IPv6]: Fix SNMP IPv6 reachability issue in certain scenarios #15487
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Modify snmpd.conf to start snmpd to listen on specific management and loopback ips instead of listening on any ip. Signed-off-by: Suvarna Meenakshi <sumeenak@microsoft.com>
Loopback0 will not exist on multi-asic platform on host namespace. Signed-off-by: Suvarna Meenakshi <sumeenak@microsoft.com>
qiluo-msft
reviewed
Jun 19, 2023
dockers/docker-snmp/snmpd.conf.j2
Outdated
| # use management and loopback ip addresses | ||
| # use docker0 ipv4 and ipv6 address | ||
| # else if none of the above config is present | ||
| # listen on any IP |
There was a problem hiding this comment.
The logic is complex.
how about minigraph.py just generated correct SNMP_AGENT_ADDRESS_CONFIG?
- single-asic: bind to mgmt and lo
- multi-asic: bind to mgmt and docker0 #Closed
There was a problem hiding this comment.
I think docker0 IP is not part of user config, you can use template here.
However, mgmt IP, lo IP are from user config, it is natural to add them into ConfigDB SNMP_AGENT_ADDRESS_CONFIG table.
There was a problem hiding this comment.
I see the challenge of determine single/multi-asic in minigraph.py. So it is okay to keep complex code here.
There was a problem hiding this comment.
I have modified the snmpd.conf.j2 template to use management and loopback IP for single asic platform.
With this, we could backport to previous branches to ensure that we could use ipv6 .
Next step is to:
- If the change to modify to use bridge network for snmp docker is merged in, we could add NAT rule for single asic platform similar to multi-asic platform.
- After that change, will update minigraph parser to update SNMP_AGENT_ADDRESS_CONFIG table with management IP and docker0 IP for both single and multi-asic platform that way minigraph parser need not have the logic to differentiate between single and multi-asic platform.
- Remove the changes in snmpd.conf.j2 and directly use SNMP_AGENT_ADDRESS_CONFIG which will be updated by minigraph parser.
table with Management IP and Loopback0 IP address so that snmpd will listen on management IP and Loopback0Ip instead of listening on any IP. Update snmpd.conf to listen on Management IP and Loopback0 IP for single-asic platform and management IP and docker0 Ip for multi-asic platform. Signed-off-by: Suvarna Meenakshi <sumeenak@microsoft.com>
qiluo-msft
reviewed
Jun 23, 2023
dockers/docker-snmp/start.sh
Outdated
| @@ -16,11 +16,16 @@ mkdir -p /etc/ssw /etc/snmp | |||
| # Parse snmp.yml and insert the data in Config DB | |||
| /usr/bin/snmp_yml_to_configdb.py | |||
|
|
|||
| DOCKER0_V4=$(ip -4 addr show docker0 | grep -oP '(?<=inet\s)\d+(\.\d+){3}') | |||
11 tasks
ganglyu
approved these changes
Jun 27, 2023
qiluo-msft
reviewed
Jun 27, 2023
src/sonic-config-engine/minigraph.py
Outdated
| if asic_name is None: | ||
| results['SNMP_AGENT_ADDRESS_CONFIG'] = {} | ||
| for mgmt_if in results['MGMT_INTERFACE'].keys(): | ||
| snmp_key = mgmt_if[1].split('/')[0] + '|161|' |
There was a problem hiding this comment.
The MGMT_INTERFACE keys contains interface name and ip address.
For example: ('eth0', '10.250.0.101/24')
So used index 1 to get the IP address.
There was a problem hiding this comment.
oh, you mean 1 is the index of tuple. Could you prevent using magic number? how about tuple assignment or define a constant for 1.
qiluo-msft
reviewed
Jun 27, 2023
src/sonic-config-engine/minigraph.py
Outdated
| results['SNMP_AGENT_ADDRESS_CONFIG'][snmp_key] = {} | ||
| for loip in results['LOOPBACK_INTERFACE']: | ||
| if len(loip) == 2 and loip[0] == 'Loopback0': | ||
| snmp_key = loip[1].split('/')[0] + '|161|' |
Signed-off-by: Suvarna Meenakshi <sumeenak@microsoft.com> (cherry picked from commit d033b032e9632587abec05f7000278593888efcd)
Signed-off-by: Suvarna Meenakshi <sumeenak@microsoft.com>
Update snmpd.conf template to support ipv6 address. Signed-off-by: Suvarna Meenakshi <sumeenak@microsoft.com>
Signed-off-by: Suvarna Meenakshi <sumeenak@microsoft.com>
Signed-off-by: Suvarna Meenakshi <sumeenak@microsoft.com>
other branches. Signed-off-by: Suvarna Meenakshi <sumeenak@microsoft.com>
arlakshm
reviewed
Jun 29, 2023
| def testsnmp_agent_address_config(self): | ||
| argument = ['-m', self.sample_graph, '-p', self.port_config, '-v', 'SNMP_AGENT_ADDRESS_CONFIG.keys()|list'] | ||
| output = self.run_script(argument) | ||
| import pdb; pdb.set_trace() |
| def testsnmp_agent_address_config(self): | ||
| argument = ['-m', self.sample_graph, '-p', self.sample_port_config, '-v', 'SNMP_AGENT_ADDRESS_CONFIG.keys()|list'] | ||
| output = self.run_script(argument) | ||
| import pdb; pdb.set_trace() |
arlakshm
previously approved these changes
Jun 29, 2023
This was referenced Jul 3, 2023
SuvarnaMeenakshi
added a commit
to sonic-net/sonic-mgmt
that referenced
this pull request
Jul 10, 2023
…os or vsonic (#8802) What is the motivation for this PR? sonic-net/sonic-buildimage#15487 modifies snmpd in SONiC to listen on management and loopback ip by default instead of listening on any IP. test_interop_protocol.py::test_snmp sends a query to the neighbor device using the link ip address. After the above change to listen on management and loopback ip, the snmp query will fail. Hence, modifying the test to send SNMP query to SONiC DUT from neighboring vsonic/eos using Loopback IP of SONiC DUT. How did you do it? Get loopback IPv4 address DUT and use that to query from neighbor device. Added a generic helped function to send SNMP query to DUT from a neighbor.
|
/azp run |
|
Commenter does not have sufficient privileges for PR 15487 in repo sonic-net/sonic-buildimage |
6 tasks
SuvarnaMeenakshi
added a commit
to SuvarnaMeenakshi/sonic-buildimage
that referenced
this pull request
Aug 24, 2023
…narios (sonic-net#15487)" This reverts commit 9864dfe.
11 tasks
|
@SuvarnaMeenakshi cherry pick PR didn't pass PR checker. Please check!!! Auto cherry pick PR will be closed in 2 days. |
SuvarnaMeenakshi
added a commit
to sonic-net/sonic-mgmt
that referenced
this pull request
Aug 25, 2023
What is the motivation for this PR? Issue sonic-net/sonic-buildimage#16165 Changes added in the PR sonic-net/sonic-buildimage#15487 to support SNMP over IPv6 does not handle if mgmt IP address is obtained via DHCP. How did you do it? Skip the sonic-mgmt SNMP IPv6 related tests in 202211 so that the sonic-net/sonic-buildimage#15487 can be reverted.
|
@SuvarnaMeenakshi cherry pick PR didn't pass PR checker. Please check!!! Auto cherry pick PR will be closed in 2 days. |
SuvarnaMeenakshi
added a commit
to SuvarnaMeenakshi/sonic-buildimage
that referenced
this pull request
Aug 29, 2023
…onic-net#15487) Modify snmpd.conf to start snmpd to listen on specific management and loopback ips instead of listening on any ip. #### Why I did it SNMP over IPv6 is not working for all scenarios for a single asic platforms. The expectation is that SNMP query over IPv6 should work over Management or Loopback0 addresses. **Specific scenario where this issue is seen** In case of Lab T0 device, when SNMP request is sent from a directly connected T1 neighbor over Loopback IP, SNMP response was not received. This was because the SRC IP address in SNMP response was not Loopback IP, it was the PortChannel IP connected to the neighboring device. ``` 23:18:51.620897 In 22:26:27:e6:e0:07 ethertype IPv6 (0x86dd), length 105: fc00::72.41725 > **fc00:1::32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:18:51.621441 Out 28:99:3a:a0:97:30 ethertype IPv6 (0x86dd), length 241: **fc00::71**.161 > fc00::72.41725: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` In case of IPv4, the SRC IP in SNMP response was correctly set to Loopback IP. ``` 23:25:32.769712 In 22:26:27:e6:e0:07 ethertype IPv4 (0x0800), length 85: 10.0.0.57.56701 > **10.1.0.32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:25:32.975967 Out 28:99:3a:a0:97:30 ethertype IPv4 (0x0800), length 221: **10.1.0.32**.161 > 10.0.0.57.56701: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` **Sequence of SNMP request and response** 1. SNMP request will be sent with SRC IP fc00::72 DST IP fc00:1::32 2. SNMP request is received at SONiC device is sent to snmpd which is listening on port 161 :::161/ 3. snmpd process will parse the request create a response and sent to DST IP fc00::72. snmpd process does not track the DST IP on which the SNMP request was received, which in this case is Loopback IP. snmpd process will only keep track what is tht IP to which the response should be sent to. 4. snmpd process will send the response packet. 5. Kernel will do a route look up on destination IP and find the best path. ip -6 route get fc00::72 fc00::72 from :: dev PortChannel101 proto kernel src fc00::71 metric 256 pref medium 5. Using the "src" ip from about, the response is sent out. This SRC ip is that of the PortChannel and not the device Loopback IP. The same issue is seen when SNMP query is sent from a remote server over Management IP. SONiC device eth0 --------- Remote server SNMP request comes with SRC IP <Remote_server> DST IP <Mgmt IP> If kernel finds best route to Remote_server_IP is via BGP neighbors, then it will send the response via front-panel interface with SRC IP as Loopback IP instead of Management IP. Main issue is that in case of IPv6, snmpd ignores the IP address to which SNMP request was sent, in case of IPv6. In case of IPv4, snmpd keeps track of DST IP of SNMP request, it will keep track if the SNMP request was sent to mgmt IP or Loopback IP. Later, this IP is used in ipi_spec_dst as SRC IP which helps kernel to find the route based on DST IP using the right SRC IP. https://github.com/net-snmp/net-snmp/blob/master/snmplib/transports/snmpUDPBaseDomain.c#L300 ipi.ipi_spec_dst.s_addr = srcip->s_addr Reference: https://man7.org/linux/man-pages/man7/ip.7.html ``` If IP_PKTINFO is passed to sendmsg(2) and ipi_spec_dst is not zero, then it is used as the local source address for the routing table lookup and for setting up IP source route options. When ipi_ifindex is not zero, the primary local address of the interface specified by the index overwrites ipi_spec_dst for the routing table lookup. ``` **This issue is not seen on multi-asic platform, why?** on multi-asic platform, there exists different network namespaces. SNMP docker with snmpd process runs on host namespace. Management interface belongs to host namespace. Loopback0 is configured on asic namespaces. Additional inforamtion on how the packet coming over Loopback IP reaches snmpd process running on host namespace: sonic-net#5420 Because of this separation of network namespaces, the route lookup of destination IP is confined to routing table of specific namespace where packet is received. if packet is received over management interface, SNMP response also is sent out of management interface. Same goes with packet received over Loopback Ip. ##### Work item tracking - Microsoft ADO **17537063**: #### How I did it Have snmpd listen on specific Management and Loopback IPs specifically instead of listening on any IP for single-asic platform. Before Fix ``` admin@xx:~$ sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 15631/snmpd udp6 0 0 :::161 :::* 15631/snmpd ``` After fix ``` admin@device:~$ sudo netstat -tulnp | grep 161 udp 0 0 10.1.0.32:161 0.0.0.0:* 215899/snmpd udp 0 0 10.3.1.1:161 0.0.0.0:* 215899/snmpd udp6 0 0 fc00:1::32:161 :::* 215899/snmpd udp6 0 0 fc00:2::32:161 :::* 215899/snmpd ``` **How this change helps with the issue?** To see snmpd trace logs, modify snmpd to start using the below parameters, in supervisord.conf file ``` /usr/sbin/snmpd -f -LS0-7i -Lf /var/log/snmpd.log ``` When snmpd listens on any IP, snmpd binds to IPv4 and IPv6 sockets as below: ``` netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[0.0.0.0]:161 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 8 to UDP/IPv6: [::]:161 ``` When IPv4 response is sent, it goes out of fd 7 and IPv6 response goes out of fd 8. When IPv6 response is sent, it does not have the right SRC IP and it can lead to the issue described. When snmpd listens on specific Loopback/Management IPs, snmpd binds to different sockets: ``` trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[10.250.0.101]:161 trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 8 to UDP: [0.0.0.0]:0->[10.1.0.32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 8 netsnmp_udpbase: binding socket: 10 to UDP/IPv6: [fc00:1::32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 10 netsnmp_ipv6: fmtaddr: t = (nil), data = 0x7fffed4c85d0, len = 28 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 9 to UDP/IPv6: [fc00:2::32]:161 ``` When SNMP request comes in via Loopback IPv4, SNMP response is sent out of fd 8 ``` trace: netsnmp_udpbase_send(): transports/snmpUDPBaseDomain.c, 511: netsnmp_udp: send 170 bytes from 0x5581f2fbe30a to UDP: [10.0.0.33]:46089->[10.1.0.32]:161 on fd 8 ``` When SNMP request comes in via Loopback IPv6, SNMP response is sent out of fd 10 ``` netsnmp_ipv6: fmtaddr: t = (nil), data = 0x5581f2fc2ff0, len = 28 trace: netsnmp_udp6_send(): transports/snmpUDPIPv6Domain.c, 164: netsnmp_udp6: send 170 bytes from 0x5581f2fbe30a to UDP/IPv6: [fc00::42]:43750 on fd 10 ``` #### How to verify it Verified on single asic and multi-asic devices. Single asic SNMP query with Loopback ``` ARISTA01T1#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ARISTA01T1#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xxx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ``` On multi-asic -- no change. ``` sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 17978/snmpd udp6 0 0 :::161 :::* 17978/snmpd ``` Query result using Loopback IP from a directly connected BGP neighbor ``` ARISTA01T2#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ARISTA01T2#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ``` <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. --> (cherry picked from commit 9864dfe)
SuvarnaMeenakshi
added a commit
to SuvarnaMeenakshi/sonic-buildimage
that referenced
this pull request
Aug 29, 2023
…sonic-net#16013) <!-- Please make sure you've read and understood our contributing guidelines: https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md ** Make sure all your commits include a signature generated with `git commit -s` ** If this is a bug fix, make sure your description includes "fixes #xxxx", or "closes #xxxx" or "resolves #xxxx" Please provide the following information: --> #### Why I did it fixes: sonic-net#16001 Caused by: sonic-net#15487 The above PR introduced change to use Management and Loopback Ipv4 and ipv6 addresses as snmpagent address in snmpd.conf file. With this change, if Link local IP address is configured as management or Loopback IPv6 address, then snmpd tries to open socket on that ipv6 address and fails with the below error: ``` Error opening specified endpoint "udp6:[fe80::5054:ff:fe6f:16f0]:161" Server Exiting with code 1 ``` From RFC4007, if we need to specify non-global ipv6 address without ambiguity, we need to use zone id along with the ipv6 address: <address>%<zone_id> Reference: https://datatracker.ietf.org/doc/html/rfc4007 ##### Work item tracking - Microsoft ADO **(number only)**: #### How I did it Modify snmpd.conf file to use the %zone_id representation for ipv6 address. #### How to verify it In VS testbed, modify config_db to use link local ipv6 address as management address: "MGMT_INTERFACE": { "eth0|10.250.0.101/24": { "forced_mgmt_routes": [ "172.17.0.1/24" ], "gwaddr": "10.250.0.1" }, "eth0|fe80::5054:ff:fe6f:16f0/64": { "gwaddr": "fe80::1" } }, Execute config_reload after the above change. snmpd comes up and check if snmpd is listening on ipv4 and ipv6 addresses: ``` admin@vlab-01:~$ sudo netstat -tulnp | grep 161 tcp 0 0 127.0.0.1:3161 0.0.0.0:* LISTEN 274060/snmpd udp 0 0 10.1.0.32:161 0.0.0.0:* 274060/snmpd udp 0 0 10.250.0.101:161 0.0.0.0:* 274060/snmpd udp6 0 0 fc00:1::32:161 :::* 274060/snmpd udp6 0 0 fe80::5054:ff:fe6f::161 :::* 274060/snmpd -- Link local admin@vlab-01:~$ sudo ifconfig eth0 eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.250.0.101 netmask 255.255.255.0 broadcast 10.250.0.255 inet6 fe80::5054:ff:fe6f:16f0 prefixlen 64 scopeid 0x20<link> ether 52:54:00:6f:16:f0 txqueuelen 1000 (Ethernet) RX packets 36384 bytes 22878123 (21.8 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 261265 bytes 46585948 (44.4 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 admin@vlab-01:~$ docker exec -it snmp snmpget -v2c -c public fe80::5054:ff:fe6f:16f0 1.3.6.1.2.1.1.1.0 iso.3.6.1.2.1.1.1.0 = STRING: "SONiC Software Version: SONiC.master.327516-04a6031b2 - HwSku: Force10-S6000 - Distribution: Debian 11.7 - Kernel: 5.10.0-18-2-amd64" ``` Logs from snmpd: ``` Turning on AgentX master support. NET-SNMP version 5.9 Connection from UDP/IPv6: [fe80::5054:ff:fe6f:16f0%eth0]:44308 ``` Ran test_snmp_loopback test to check if loopback ipv4 and ipv6 works: ``` ./run_tests.sh -n vms-kvm-t0 -d vlab-01 -c snmp/test_snmp_loopback.py -f vtestbed.yaml -i ../ansible/veos_vtb -e "--skip_sanity --disable_loganalyzer" -u === Running tests in groups === Running: pytest snmp/test_snmp_loopback.py --inventory ../ansible/veos_vtb --host-pattern vlab-01 --testbed vms-kvm-t0 --testbed_file vtestbed.yaml --log-cli-level warning --log-file-level debug --kube_master unset --showlocals --assert plain --show-capture no -rav --allow_recover --ignore=ptftests --ignore=acstests --ignore=saitests --ignore=scripts --ignore=k8s --ignore=sai_qualify --junit-xml=logs/tr.xml --log-file=logs/test.log --skip_sanity --disable_loganalyzer .. snmp/test_snmp_loopback.py::test_snmp_loopback[vlab-01] PASSED ``` <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. --> #### Which release branch to backport (provide reason below if selected) <!-- - Note we only backport fixes to a release branch, *not* features! - Please also provide a reason for the backporting below. - e.g. - [x] 202006 --> - [ ] 201811 - [ ] 201911 - [ ] 202006 - [x] 202012 - [x] 202106 - [x] 202111 - [x] 202205 - [x] 202211 - [x] 202305 #### Tested branch (Please provide the tested image version) <!-- - Please provide tested image version - e.g. - [x] 20201231.100 --> - [ ] <!-- image version 1 --> - [ ] <!-- image version 2 --> #### Description for the changelog <!-- Write a short (one line) summary that describes the changes in this pull request for inclusion in the changelog: --> <!-- Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU. --> #### Link to config_db schema for YANG module changes <!-- Provide a link to config_db schema for the table for which YANG model is defined Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md --> #### A picture of a cute animal (not mandatory but encouraged) (cherry picked from commit 803c71c)
11 tasks
SuvarnaMeenakshi
added a commit
to SuvarnaMeenakshi/sonic-mgmt
that referenced
this pull request
Sep 14, 2023
What is the motivation for this PR? SNMP over Loopback ipv6 is currently being skipped. sonic-net/sonic-buildimage#15487 -fixes snmp over ipv6. After the above fix, SNMP over Loopback and management should work. How did you do it? Remove skip of ipv6. How did you verify/test it? Test on vs testbed with single-asic image with sonic-buildimage 15487 fix. Prerequisite PRs: sonic-net/sonic-buildimage#15688 sonic-net/sonic-buildimage#15487 (cherry picked from commit bcf44a1)
6 tasks
wangxin
pushed a commit
to sonic-net/sonic-mgmt
that referenced
this pull request
Sep 15, 2023
Cherry-pick with conflict resolution of #8824 What is the motivation for this PR? SNMP over Loopback ipv6 is currently being skipped. sonic-net/sonic-buildimage#15487 -fixes snmp over ipv6. After the above fix, SNMP over Loopback and management should work. How did you do it? Remove skip of ipv6. How did you verify/test it? Test on vs testbed with single-asic image with sonic-buildimage 15487 fix. Prerequisite PRs: sonic-net/sonic-buildimage#15688 sonic-net/sonic-buildimage#15487 (cherry picked from commit bcf44a1)
sonic-otn
pushed a commit
to sonic-otn/sonic-buildimage
that referenced
this pull request
Sep 20, 2023
…onic-net#15487) Modify snmpd.conf to start snmpd to listen on specific management and loopback ips instead of listening on any ip. #### Why I did it SNMP over IPv6 is not working for all scenarios for a single asic platforms. The expectation is that SNMP query over IPv6 should work over Management or Loopback0 addresses. **Specific scenario where this issue is seen** In case of Lab T0 device, when SNMP request is sent from a directly connected T1 neighbor over Loopback IP, SNMP response was not received. This was because the SRC IP address in SNMP response was not Loopback IP, it was the PortChannel IP connected to the neighboring device. ``` 23:18:51.620897 In 22:26:27:e6:e0:07 ethertype IPv6 (0x86dd), length 105: fc00::72.41725 > **fc00:1::32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:18:51.621441 Out 28:99:3a:a0:97:30 ethertype IPv6 (0x86dd), length 241: **fc00::71**.161 > fc00::72.41725: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` In case of IPv4, the SRC IP in SNMP response was correctly set to Loopback IP. ``` 23:25:32.769712 In 22:26:27:e6:e0:07 ethertype IPv4 (0x0800), length 85: 10.0.0.57.56701 > **10.1.0.32**.161: C="msft" **GetRequest**(28) .1.3.6.1.2.1.1.1.0 23:25:32.975967 Out 28:99:3a:a0:97:30 ethertype IPv4 (0x0800), length 221: **10.1.0.32**.161 > 10.0.0.57.56701: C="msft" **GetResponse**(162) .1.3.6.1.2.1.1.1.0="SONiC Software Version: SONiC.xxx - HwSku: xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64" ``` **Sequence of SNMP request and response** 1. SNMP request will be sent with SRC IP fc00::72 DST IP fc00:1::32 2. SNMP request is received at SONiC device is sent to snmpd which is listening on port 161 :::161/ 3. snmpd process will parse the request create a response and sent to DST IP fc00::72. snmpd process does not track the DST IP on which the SNMP request was received, which in this case is Loopback IP. snmpd process will only keep track what is tht IP to which the response should be sent to. 4. snmpd process will send the response packet. 5. Kernel will do a route look up on destination IP and find the best path. ip -6 route get fc00::72 fc00::72 from :: dev PortChannel101 proto kernel src fc00::71 metric 256 pref medium 5. Using the "src" ip from about, the response is sent out. This SRC ip is that of the PortChannel and not the device Loopback IP. The same issue is seen when SNMP query is sent from a remote server over Management IP. SONiC device eth0 --------- Remote server SNMP request comes with SRC IP <Remote_server> DST IP <Mgmt IP> If kernel finds best route to Remote_server_IP is via BGP neighbors, then it will send the response via front-panel interface with SRC IP as Loopback IP instead of Management IP. Main issue is that in case of IPv6, snmpd ignores the IP address to which SNMP request was sent, in case of IPv6. In case of IPv4, snmpd keeps track of DST IP of SNMP request, it will keep track if the SNMP request was sent to mgmt IP or Loopback IP. Later, this IP is used in ipi_spec_dst as SRC IP which helps kernel to find the route based on DST IP using the right SRC IP. https://github.com/net-snmp/net-snmp/blob/master/snmplib/transports/snmpUDPBaseDomain.c#L300 ipi.ipi_spec_dst.s_addr = srcip->s_addr Reference: https://man7.org/linux/man-pages/man7/ip.7.html ``` If IP_PKTINFO is passed to sendmsg(2) and ipi_spec_dst is not zero, then it is used as the local source address for the routing table lookup and for setting up IP source route options. When ipi_ifindex is not zero, the primary local address of the interface specified by the index overwrites ipi_spec_dst for the routing table lookup. ``` **This issue is not seen on multi-asic platform, why?** on multi-asic platform, there exists different network namespaces. SNMP docker with snmpd process runs on host namespace. Management interface belongs to host namespace. Loopback0 is configured on asic namespaces. Additional inforamtion on how the packet coming over Loopback IP reaches snmpd process running on host namespace: sonic-net#5420 Because of this separation of network namespaces, the route lookup of destination IP is confined to routing table of specific namespace where packet is received. if packet is received over management interface, SNMP response also is sent out of management interface. Same goes with packet received over Loopback Ip. ##### Work item tracking - Microsoft ADO **17537063**: #### How I did it Have snmpd listen on specific Management and Loopback IPs specifically instead of listening on any IP for single-asic platform. Before Fix ``` admin@xx:~$ sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 15631/snmpd udp6 0 0 :::161 :::* 15631/snmpd ``` After fix ``` admin@device:~$ sudo netstat -tulnp | grep 161 udp 0 0 10.1.0.32:161 0.0.0.0:* 215899/snmpd udp 0 0 10.3.1.1:161 0.0.0.0:* 215899/snmpd udp6 0 0 fc00:1::32:161 :::* 215899/snmpd udp6 0 0 fc00:2::32:161 :::* 215899/snmpd ``` **How this change helps with the issue?** To see snmpd trace logs, modify snmpd to start using the below parameters, in supervisord.conf file ``` /usr/sbin/snmpd -f -LS0-7i -Lf /var/log/snmpd.log ``` When snmpd listens on any IP, snmpd binds to IPv4 and IPv6 sockets as below: ``` netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[0.0.0.0]:161 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 8 to UDP/IPv6: [::]:161 ``` When IPv4 response is sent, it goes out of fd 7 and IPv6 response goes out of fd 8. When IPv6 response is sent, it does not have the right SRC IP and it can lead to the issue described. When snmpd listens on specific Loopback/Management IPs, snmpd binds to different sockets: ``` trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 7 to UDP: [0.0.0.0]:0->[10.250.0.101]:161 trace: netsnmp_udpipv4base_transport_bind(): transports/snmpUDPIPv4BaseDomain.c, 207: netsnmp_udpbase: binding socket: 8 to UDP: [0.0.0.0]:0->[10.1.0.32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 8 netsnmp_udpbase: binding socket: 10 to UDP/IPv6: [fc00:1::32]:161 trace: netsnmp_register_agent_nsap(): snmp_agent.c, 1261: netsnmp_register_agent_nsap: fd 10 netsnmp_ipv6: fmtaddr: t = (nil), data = 0x7fffed4c85d0, len = 28 trace: netsnmp_udp6_transport_bind(): transports/snmpUDPIPv6Domain.c, 303: netsnmp_udpbase: binding socket: 9 to UDP/IPv6: [fc00:2::32]:161 ``` When SNMP request comes in via Loopback IPv4, SNMP response is sent out of fd 8 ``` trace: netsnmp_udpbase_send(): transports/snmpUDPBaseDomain.c, 511: netsnmp_udp: send 170 bytes from 0x5581f2fbe30a to UDP: [10.0.0.33]:46089->[10.1.0.32]:161 on fd 8 ``` When SNMP request comes in via Loopback IPv6, SNMP response is sent out of fd 10 ``` netsnmp_ipv6: fmtaddr: t = (nil), data = 0x5581f2fc2ff0, len = 28 trace: netsnmp_udp6_send(): transports/snmpUDPIPv6Domain.c, 164: netsnmp_udp6: send 170 bytes from 0x5581f2fbe30a to UDP/IPv6: [fc00::42]:43750 on fd 10 ``` #### How to verify it Verified on single asic and multi-asic devices. Single asic SNMP query with Loopback ``` ARISTA01T1#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ARISTA01T1#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: Arista-7260xxx - Distribution: Debian 10.13 - Kernel: 4.19.0-12-2-amd64 ``` On multi-asic -- no change. ``` sudo netstat -tulnp | grep 161 udp 0 0 0.0.0.0:161 0.0.0.0:* 17978/snmpd udp6 0 0 :::161 :::* 17978/snmpd ``` Query result using Loopback IP from a directly connected BGP neighbor ``` ARISTA01T2#bash snmpget -v2c -c xxx 10.1.0.32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ARISTA01T2#bash snmpget -v2c -c xxx fc00:1::32 1.3.6.1.2.1.1.1.0 SNMPv2-MIB::sysDescr.0 = STRING: SONiC Software Version: SONiC.xx - HwSku: xx - Distribution: Debian 9.13 - Kernel: 4.9.0-14-2-amd64 ``` <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. -->
sonic-otn
pushed a commit
to sonic-otn/sonic-buildimage
that referenced
this pull request
Sep 20, 2023
…sonic-net#16013) <!-- Please make sure you've read and understood our contributing guidelines: https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md ** Make sure all your commits include a signature generated with `git commit -s` ** If this is a bug fix, make sure your description includes "fixes #xxxx", or "closes #xxxx" or "resolves #xxxx" Please provide the following information: --> #### Why I did it fixes: sonic-net#16001 Caused by: sonic-net#15487 The above PR introduced change to use Management and Loopback Ipv4 and ipv6 addresses as snmpagent address in snmpd.conf file. With this change, if Link local IP address is configured as management or Loopback IPv6 address, then snmpd tries to open socket on that ipv6 address and fails with the below error: ``` Error opening specified endpoint "udp6:[fe80::5054:ff:fe6f:16f0]:161" Server Exiting with code 1 ``` From RFC4007, if we need to specify non-global ipv6 address without ambiguity, we need to use zone id along with the ipv6 address: <address>%<zone_id> Reference: https://datatracker.ietf.org/doc/html/rfc4007 ##### Work item tracking - Microsoft ADO **(number only)**: #### How I did it Modify snmpd.conf file to use the %zone_id representation for ipv6 address. #### How to verify it In VS testbed, modify config_db to use link local ipv6 address as management address: "MGMT_INTERFACE": { "eth0|10.250.0.101/24": { "forced_mgmt_routes": [ "172.17.0.1/24" ], "gwaddr": "10.250.0.1" }, "eth0|fe80::5054:ff:fe6f:16f0/64": { "gwaddr": "fe80::1" } }, Execute config_reload after the above change. snmpd comes up and check if snmpd is listening on ipv4 and ipv6 addresses: ``` admin@vlab-01:~$ sudo netstat -tulnp | grep 161 tcp 0 0 127.0.0.1:3161 0.0.0.0:* LISTEN 274060/snmpd udp 0 0 10.1.0.32:161 0.0.0.0:* 274060/snmpd udp 0 0 10.250.0.101:161 0.0.0.0:* 274060/snmpd udp6 0 0 fc00:1::32:161 :::* 274060/snmpd udp6 0 0 fe80::5054:ff:fe6f::161 :::* 274060/snmpd -- Link local admin@vlab-01:~$ sudo ifconfig eth0 eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.250.0.101 netmask 255.255.255.0 broadcast 10.250.0.255 inet6 fe80::5054:ff:fe6f:16f0 prefixlen 64 scopeid 0x20<link> ether 52:54:00:6f:16:f0 txqueuelen 1000 (Ethernet) RX packets 36384 bytes 22878123 (21.8 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 261265 bytes 46585948 (44.4 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 admin@vlab-01:~$ docker exec -it snmp snmpget -v2c -c public fe80::5054:ff:fe6f:16f0 1.3.6.1.2.1.1.1.0 iso.3.6.1.2.1.1.1.0 = STRING: "SONiC Software Version: SONiC.master.327516-04a6031b2 - HwSku: Force10-S6000 - Distribution: Debian 11.7 - Kernel: 5.10.0-18-2-amd64" ``` Logs from snmpd: ``` Turning on AgentX master support. NET-SNMP version 5.9 Connection from UDP/IPv6: [fe80::5054:ff:fe6f:16f0%eth0]:44308 ``` Ran test_snmp_loopback test to check if loopback ipv4 and ipv6 works: ``` ./run_tests.sh -n vms-kvm-t0 -d vlab-01 -c snmp/test_snmp_loopback.py -f vtestbed.yaml -i ../ansible/veos_vtb -e "--skip_sanity --disable_loganalyzer" -u === Running tests in groups === Running: pytest snmp/test_snmp_loopback.py --inventory ../ansible/veos_vtb --host-pattern vlab-01 --testbed vms-kvm-t0 --testbed_file vtestbed.yaml --log-cli-level warning --log-file-level debug --kube_master unset --showlocals --assert plain --show-capture no -rav --allow_recover --ignore=ptftests --ignore=acstests --ignore=saitests --ignore=scripts --ignore=k8s --ignore=sai_qualify --junit-xml=logs/tr.xml --log-file=logs/test.log --skip_sanity --disable_loganalyzer .. snmp/test_snmp_loopback.py::test_snmp_loopback[vlab-01] PASSED ``` <!-- If PR needs to be backported, then the PR must be tested against the base branch and the earliest backport release branch and provide tested image version on these two branches. For example, if the PR is requested for master, 202211 and 202012, then the requester needs to provide test results on master and 202012. --> #### Which release branch to backport (provide reason below if selected) <!-- - Note we only backport fixes to a release branch, *not* features! - Please also provide a reason for the backporting below. - e.g. - [x] 202006 --> - [ ] 201811 - [ ] 201911 - [ ] 202006 - [x] 202012 - [x] 202106 - [x] 202111 - [x] 202205 - [x] 202211 - [x] 202305 #### Tested branch (Please provide the tested image version) <!-- - Please provide tested image version - e.g. - [x] 20201231.100 --> - [ ] <!-- image version 1 --> - [ ] <!-- image version 2 --> #### Description for the changelog <!-- Write a short (one line) summary that describes the changes in this pull request for inclusion in the changelog: --> <!-- Ensure to add label/tag for the feature raised. example - PR#2174 under sonic-utilities repo. where, Generic Config and Update feature has been labelled as GCU. --> #### Link to config_db schema for YANG module changes <!-- Provide a link to config_db schema for the table for which YANG model is defined Link should point to correct section on https://github.com/Azure/sonic-buildimage/blob/master/src/sonic-yang-models/doc/Configuration.md --> #### A picture of a cute animal (not mandatory but encouraged)
SuvarnaMeenakshi
added a commit
to SuvarnaMeenakshi/sonic-buildimage
that referenced
this pull request
Sep 21, 2023
…narios (sonic-net#15487) (sonic-net#15826)" This reverts commit 7cfb71b.
SuvarnaMeenakshi
added a commit
to SuvarnaMeenakshi/sonic-buildimage
that referenced
this pull request
Sep 21, 2023
…narios (sonic-net#15487) (sonic-net#15874)" This reverts commit 83aa8b8.
This was referenced Sep 21, 2023
AharonMalkin
pushed a commit
to AharonMalkin/sonic-mgmt
that referenced
this pull request
Jan 25, 2024
…os or vsonic (sonic-net#8802) What is the motivation for this PR? sonic-net/sonic-buildimage#15487 modifies snmpd in SONiC to listen on management and loopback ip by default instead of listening on any IP. test_interop_protocol.py::test_snmp sends a query to the neighbor device using the link ip address. After the above change to listen on management and loopback ip, the snmp query will fail. Hence, modifying the test to send SNMP query to SONiC DUT from neighboring vsonic/eos using Loopback IP of SONiC DUT. How did you do it? Get loopback IPv4 address DUT and use that to query from neighbor device. Added a generic helped function to send SNMP query to DUT from a neighbor.
AharonMalkin
pushed a commit
to AharonMalkin/sonic-mgmt
that referenced
this pull request
Jan 25, 2024
What is the motivation for this PR? SNMP over Loopback ipv6 is currently being skipped. sonic-net/sonic-buildimage#15487 -fixes snmp over ipv6. After the above fix, SNMP over Loopback and management should work. How did you do it? Remove skip of ipv6. How did you verify/test it? Test on vs testbed with single-asic image with sonic-buildimage 15487 fix. Prerequisite PRs: sonic-net/sonic-buildimage#15688 sonic-net/sonic-buildimage#15487
AharonMalkin
pushed a commit
to AharonMalkin/sonic-mgmt
that referenced
this pull request
Jan 25, 2024
…et#9630) What is the motivation for this PR? Issue sonic-net/sonic-buildimage#16165 Changes added in the PR sonic-net/sonic-buildimage#15487 to support SNMP over IPv6 does not handle if mgmt IP address is obtained via DHCP. How did you do it? Skip the sonic-mgmt SNMP IPv6 related tests in 202211 so that the sonic-net/sonic-buildimage#15487 can be reverted.
lixiaoyuner
pushed a commit
to lixiaoyuner/sonic-buildimage
that referenced
this pull request
Feb 6, 2024
…sonic-buildimage into internal Fix conflict for rsyslog. Skip partial DNS unit test in internal branch after confirmed with Gang. Related work items: sonic-net#113, sonic-net#131, sonic-net#132, sonic-net#134, sonic-net#321, sonic-net#331, sonic-net#381, sonic-net#382, sonic-net#2525, sonic-net#2676, sonic-net#2698, sonic-net#2737, sonic-net#2789, sonic-net#2839, sonic-net#2845, sonic-net#2850, sonic-net#2882, sonic-net#2885, sonic-net#2887, sonic-net#2890, sonic-net#2895, sonic-net#13338, sonic-net#14105, sonic-net#15142, sonic-net#15223, sonic-net#15456, sonic-net#15487, sonic-net#15520, sonic-net#15726, sonic-net#15727, sonic-net#15758, sonic-net#15764, sonic-net#15765, sonic-net#15772, sonic-net#15779, sonic-net#15782, sonic-net#15785, sonic-net#15797, sonic-net#15798, sonic-net#15810, sonic-net#15811, sonic-net#15821
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Modify snmpd.conf to start snmpd to listen on specific management and loopback ips instead of listening on any ip.
Why I did it
SNMP over IPv6 is not working for all scenarios for a single asic platforms.
The expectation is that SNMP query over IPv6 should work over Management or Loopback0 addresses.
Specific scenario where this issue is seen
In case of Lab T0 device, when SNMP request is sent from a directly connected T1 neighbor over Loopback IP, SNMP response was not received.
This was because the SRC IP address in SNMP response was not Loopback IP, it was the PortChannel IP connected to the neighboring device.
In case of IPv4, the SRC IP in SNMP response was correctly set to Loopback IP.
Sequence of SNMP request and response
snmpd process does not track the DST IP on which the SNMP request was received, which in this case is Loopback IP.
snmpd process will only keep track what is tht IP to which the response should be sent to.
ip -6 route get fc00::72
fc00::72 from :: dev PortChannel101 proto kernel src fc00::71 metric 256 pref medium
The same issue is seen when SNMP query is sent from a remote server over Management IP.
SONiC device eth0 --------- Remote server
SNMP request comes with SRC IP <Remote_server> DST IP
If kernel finds best route to Remote_server_IP is via BGP neighbors, then it will send the response via front-panel interface with SRC IP as Loopback IP instead of Management IP.
Main issue is that in case of IPv6, snmpd ignores the IP address to which SNMP request was sent, in case of IPv6.
In case of IPv4, snmpd keeps track of DST IP of SNMP request, it will keep track if the SNMP request was sent to mgmt IP or Loopback IP.
Later, this IP is used in ipi_spec_dst as SRC IP which helps kernel to find the route based on DST IP using the right SRC IP.
https://github.com/net-snmp/net-snmp/blob/master/snmplib/transports/snmpUDPBaseDomain.c#L300
ipi.ipi_spec_dst.s_addr = srcip->s_addr
Reference: https://man7.org/linux/man-pages/man7/ip.7.html
This issue is not seen on multi-asic platform, why?
on multi-asic platform, there exists different network namespaces.
SNMP docker with snmpd process runs on host namespace.
Management interface belongs to host namespace.
Loopback0 is configured on asic namespaces.
Additional inforamtion on how the packet coming over Loopback IP reaches snmpd process running on host namespace: #5420
Because of this separation of network namespaces, the route lookup of destination IP is confined to routing table of specific namespace where packet is received.
if packet is received over management interface, SNMP response also is sent out of management interface. Same goes with packet received over Loopback Ip.
Work item tracking
How I did it
Have snmpd listen on specific Management and Loopback IPs specifically instead of listening on any IP for single-asic platform.
Before Fix
After fix
How this change helps with the issue?
To see snmpd trace logs, modify snmpd to start using the below parameters, in supervisord.conf file
When snmpd listens on any IP, snmpd binds to IPv4 and IPv6 sockets as below:
When IPv4 response is sent, it goes out of fd 7 and IPv6 response goes out of fd 8.
When IPv6 response is sent, it does not have the right SRC IP and it can lead to the issue described.
When snmpd listens on specific Loopback/Management IPs, snmpd binds to different sockets:
When SNMP request comes in via Loopback IPv4, SNMP response is sent out of fd 8
When SNMP request comes in via Loopback IPv6, SNMP response is sent out of fd 10
How to verify it
Verified on single asic and multi-asic devices.
Single asic SNMP query with Loopback
On multi-asic -- no change.
Query result using Loopback IP from a directly connected BGP neighbor
Which release branch to backport (provide reason below if selected)
Tested branch (Please provide the tested image version)
Description for the changelog
Link to config_db schema for YANG module changes
A picture of a cute animal (not mandatory but encouraged)