Fix the Orchagent crash seen during Port channel OC test cases (Issue#17665) #3042
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The function addNeighbor adds the remote system neighbor against the remote system port and increment the reference count for remote system port's RIF. However when it adds the nextHop in addNextHop function , it adds it against Inband port with RIF-ID of remote system port, but increases the RIF reference count of Inband port instead of remote system port.When the neighbor is removed in removeNeighbor, it decreases the ref count of remote system port for RIF. But when it removes the nexthop in removeNextHop, it decreases the ref count for remote system port. So if the remote system port has both ipv4 and ipv6 configured, then the ref count is incremented by 2 for remote system port's RIF (ipv4 and ipv4 nbr) and incremented by 2 (ipv4 and ipv6 nexthop) for Inband Port's RIF. But the ref count is decremented 4 times for remote system port's RIF. So sometimes, as soon as the ipv4 or ipv6 is delted, the orchagent tries to delete the remote system port's RIF, but since SAI meta layer has different ref count, it returns failure and orchagent crashes. Signed-off-by: saksarav <sakthivadivu.saravanaraj@nokia.com>
|
This needs to be cherry-picked to 202205 |
|
@prsunny please review this |
|
@judyjoseph for viz |
prsunny
approved these changes
Feb 6, 2024
|
MSFT ADO: 26718860 @yxieca , @StormLiangMS , This is a bug fix that is needed to be back ported to 202205 and 202305. |
mssonicbld
pushed a commit
to mssonicbld/sonic-swss
that referenced
this pull request
Feb 7, 2024
…c-net#3042) The function addNeighbor adds the remote system neighbor against the remote system port and increment the reference count for remote system port's RIF. However when it adds the nextHop in addNextHop function , it adds it against Inband port with RIF-ID of remote system port, but increases the RIF reference count of Inband port instead of remote system port.When the neighbor is removed in removeNeighbor, it decreases the ref count of remote system port for RIF. But when it removes the nexthop in removeNextHop, it decreases the ref count for remote system port. So if the remote system port has both ipv4 and ipv6 configured, then the ref count is incremented by 2 for remote system port's RIF (ipv4 and ipv4 nbr) and incremented by 2 (ipv4 and ipv6 nexthop) for Inband Port's RIF. But the ref count is decremented 4 times for remote system port's RIF. So sometimes, as soon as the ipv4 or ipv6 is delted, the orchagent tries to delete the remote system port's RIF, but since SAI meta layer has different ref count, it returns failure and orchagent crashes. Signed-off-by: saksarav <sakthivadivu.saravanaraj@nokia.com>
|
Cherry-pick PR to 202311: #3043 |
mssonicbld
pushed a commit
to mssonicbld/sonic-swss
that referenced
this pull request
Feb 7, 2024
…c-net#3042) The function addNeighbor adds the remote system neighbor against the remote system port and increment the reference count for remote system port's RIF. However when it adds the nextHop in addNextHop function , it adds it against Inband port with RIF-ID of remote system port, but increases the RIF reference count of Inband port instead of remote system port.When the neighbor is removed in removeNeighbor, it decreases the ref count of remote system port for RIF. But when it removes the nexthop in removeNextHop, it decreases the ref count for remote system port. So if the remote system port has both ipv4 and ipv6 configured, then the ref count is incremented by 2 for remote system port's RIF (ipv4 and ipv4 nbr) and incremented by 2 (ipv4 and ipv6 nexthop) for Inband Port's RIF. But the ref count is decremented 4 times for remote system port's RIF. So sometimes, as soon as the ipv4 or ipv6 is delted, the orchagent tries to delete the remote system port's RIF, but since SAI meta layer has different ref count, it returns failure and orchagent crashes. Signed-off-by: saksarav <sakthivadivu.saravanaraj@nokia.com>
|
Cherry-pick PR to 202205: #3044 |
mssonicbld
added
Included in 202205 Branch
and removed
Created PR to 202205 Branch
labels
mssonicbld
pushed a commit
that referenced
this pull request
Feb 8, 2024
The function addNeighbor adds the remote system neighbor against the remote system port and increment the reference count for remote system port's RIF. However when it adds the nextHop in addNextHop function , it adds it against Inband port with RIF-ID of remote system port, but increases the RIF reference count of Inband port instead of remote system port.When the neighbor is removed in removeNeighbor, it decreases the ref count of remote system port for RIF. But when it removes the nexthop in removeNextHop, it decreases the ref count for remote system port. So if the remote system port has both ipv4 and ipv6 configured, then the ref count is incremented by 2 for remote system port's RIF (ipv4 and ipv4 nbr) and incremented by 2 (ipv4 and ipv6 nexthop) for Inband Port's RIF. But the ref count is decremented 4 times for remote system port's RIF. So sometimes, as soon as the ipv4 or ipv6 is delted, the orchagent tries to delete the remote system port's RIF, but since SAI meta layer has different ref count, it returns failure and orchagent crashes. Signed-off-by: saksarav <sakthivadivu.saravanaraj@nokia.com>
mssonicbld
added
Included in 202311 Branch
and removed
Created PR to 202311 Branch
labels
prsunny
pushed a commit
that referenced
this pull request
Feb 9, 2024
… (#3044) The function addNeighbor adds the remote system neighbor against the remote system port and increment the reference count for remote system port's RIF. However when it adds the nextHop in addNextHop function , it adds it against Inband port with RIF-ID of remote system port, but increases the RIF reference count of Inband port instead of remote system port.When the neighbor is removed in removeNeighbor, it decreases the ref count of remote system port for RIF. But when it removes the nexthop in removeNextHop, it decreases the ref count for remote system port. So if the remote system port has both ipv4 and ipv6 configured, then the ref count is incremented by 2 for remote system port's RIF (ipv4 and ipv4 nbr) and incremented by 2 (ipv4 and ipv6 nexthop) for Inband Port's RIF. But the ref count is decremented 4 times for remote system port's RIF. So sometimes, as soon as the ipv4 or ipv6 is delted, the orchagent tries to delete the remote system port's RIF, but since SAI meta layer has different ref count, it returns failure and orchagent crashes.
|
@StormLiangMS I see this got approved for 202305 but I don't see it ever get picked into 202305. Can you help take a look what happened? perhaps auto cherry-pick got disabled in 202305? |
StormLiangMS
added
Approved for 202305 Branch
and removed
Approved for 202305 Branch
labels
mssonicbld
pushed a commit
to mssonicbld/sonic-swss
that referenced
this pull request
Mar 8, 2024
…c-net#3042) The function addNeighbor adds the remote system neighbor against the remote system port and increment the reference count for remote system port's RIF. However when it adds the nextHop in addNextHop function , it adds it against Inband port with RIF-ID of remote system port, but increases the RIF reference count of Inband port instead of remote system port.When the neighbor is removed in removeNeighbor, it decreases the ref count of remote system port for RIF. But when it removes the nexthop in removeNextHop, it decreases the ref count for remote system port. So if the remote system port has both ipv4 and ipv6 configured, then the ref count is incremented by 2 for remote system port's RIF (ipv4 and ipv4 nbr) and incremented by 2 (ipv4 and ipv6 nexthop) for Inband Port's RIF. But the ref count is decremented 4 times for remote system port's RIF. So sometimes, as soon as the ipv4 or ipv6 is delted, the orchagent tries to delete the remote system port's RIF, but since SAI meta layer has different ref count, it returns failure and orchagent crashes. Signed-off-by: saksarav <sakthivadivu.saravanaraj@nokia.com>
|
Cherry-pick PR to 202305: #3072 |
mssonicbld
pushed a commit
that referenced
this pull request
Mar 8, 2024
The function addNeighbor adds the remote system neighbor against the remote system port and increment the reference count for remote system port's RIF. However when it adds the nextHop in addNextHop function , it adds it against Inband port with RIF-ID of remote system port, but increases the RIF reference count of Inband port instead of remote system port.When the neighbor is removed in removeNeighbor, it decreases the ref count of remote system port for RIF. But when it removes the nexthop in removeNextHop, it decreases the ref count for remote system port. So if the remote system port has both ipv4 and ipv6 configured, then the ref count is incremented by 2 for remote system port's RIF (ipv4 and ipv4 nbr) and incremented by 2 (ipv4 and ipv6 nexthop) for Inband Port's RIF. But the ref count is decremented 4 times for remote system port's RIF. So sometimes, as soon as the ipv4 or ipv6 is delted, the orchagent tries to delete the remote system port's RIF, but since SAI meta layer has different ref count, it returns failure and orchagent crashes. Signed-off-by: saksarav <sakthivadivu.saravanaraj@nokia.com>
mssonicbld
added
Included in 202305 Branch
and removed
Created PR to 202305 Branch
labels
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What I did
Modified addNextHop function to pass the alias of remote system port instead of Inband port if the neighbor is remote system neighbor.
Why I did it
Fix for sonic-net/sonic-buildimage#17665
The function addNeighbor adds the remote system neighbor against the remote system port and increment the reference count for remote system port's RIF.
However when it adds the nextHop in addNextHop function , it adds it against Inband port with RIF-ID of remote system port, but increases the RIF reference
count of Inband port instead of remote system port.When the neighbor is removed in removeNeighbor, it decreases the ref count of remote system port for RIF.
But when it removes the nexthop in removeNextHop, it decreases the ref count for remote system port. So if the remote system port has both ipv4 and ipv6 configured,
then the ref count is incremented by 2 for remote system port's RIF (ipv4 and ipv4 nbr) and incremented by 2 (ipv4 and ipv6 nexthop) for Inband Port's RIF.
But the ref count is decremented 4 times for remote system port's RIF. So sometimes, as soon as the ipv4 or ipv6 is delted, the orchagent tries to delete the
remote system port's RIF, but since SAI meta layer has different ref count, it returns failure and orchagent crashes.
How I verified it
Ran the Port channel OC suites and db consistency OC suites multiple times and verified that the orchagent crash is not seen anymore.
Details if related