Fpmsyncd Next Hop Table Enhancement HLD #1425

ntt-omw · 2023-07-13T06:51:43Z

Fpmsyncd Next Hop Table Enhancement HLD

repo	PR title	status
sonic-buildimage	sonic-net/sonic-buildimage#16762
sonic-swss	sonic-net/sonic-swss#2919
sonic-mgmt-common	sonic-net/sonic-mgmt-common#107
sonic-mgmt-framework	sonic-net/sonic-mgmt-framework#122

linux-foundation-easycla · 2023-07-13T06:51:47Z

The committers listed above are authorized under a signed CLA.

✅ login: nakano-omw / name: Kanji Nakano (a14e3af, 0a50e20)
✅ login: ebiken-ntt / name: Kentaro Ebisawa (cb49f1b, 1199962, 76cf405, 583a4b1, f7faddb, c658cc4)

lguohan · 2023-08-01T15:34:56Z

doc/fpmsyncd/hld_fpmsyncd-NTT.md

+-->
+
+There should be no impact to Warmboot and Fastboot Design Impact.
+The change is how fpmsyncd handle message from zebra.


is the nexthop group id stable? this could have impact to warm reboot, can you confirm?

Thank you for the comment. I will check and update.

Based on the discussion during the last Routing WG meeting, I have added description that Warmboot and Fastboot is NOT supported when this feature is enabled. (It works same as today if disabled which is default config)

lguohan · 2023-08-01T15:35:29Z

doc/fpmsyncd/hld_fpmsyncd-NTT.md

+We can disable `net.ipv4.nexthop_compat_mode` (set to 0) for performance optimization.
+
+For backward comptibility, we can keep `fpm use-next-hop-groups` disabled by default in FRR configuration.
+This way, zebra would not send `RTM_NEWNEXTHOP` and `RTM_DELNEXTHOP` to fpmsyncd and entry created in `APPL_DB` should be identical to when this NHG feature did not exist.


we need sonic config db to enable and disable this feature, and this new feature should be disable by default.

Thanks for the comment. Yes, we intend to make this disabled by default.
We are also studying how to make this configurable using config db.
I will update the HLD and make it clear in the doc.

Based on the discussion during the last Routing WG meeting, I have added requirement to disable/enable this via CONFIG_DB in the Configuration and management section.
Will update more details like data models after further investigating how this should be implemented.

Updated Configuration and management section with description of how this feature will be disabled(default) and enabled using CONFIG_DB and config_db.json. One can directly edit config_db.json file or use Klish CLI to enable/disable this feature. Also, corresponding test case are added.

eddyk-nvidia · 2023-08-08T03:37:57Z

doc/fpmsyncd/hld_fpmsyncd-NTT.md

+-->
+
+This design modifies `fpmsyncd` to use the new `APPL_DB` tables.
+


As discussed during the review, it is more about move of logic of NHG creation from SWSS (Route/NHG OA) to Zebra. What is the main motivation for this move ? How such features as Dynamically Ordered ECMP (used by MSFT) with per NHGm's Sequence number work with this move ?

What is the main motivation for this move ?

Motivation is to reduce NHG/ROUTE translation in zebra.
Also, supporting NHG in fpmsyncd as a first step to improve update performance (reducing DB change) by zebra to update a NHG shared among routes instead of updating all ECMP routes in ROUTE entry. (e.g., when NHG member was reduced from n to n-1)

How such features as Dynamically Ordered ECMP (used by MSFT) with per NHGm's Sequence number work with this move ?

My understanding is this would not impact people using SWSS to create NHG if the feature is disabled (by default) since fpm will not send RTM_NEWNEXTHOP and thus fpmsyncd will set entry in CONFIG_DB using ROUTE entry as today.
Do you expect anyone to use Dynamically Ordered ECMP / per NHGm's Sequence number while enabling NHG in fpmsyncd in near future? (While I agree that it would be nice to have both available in long term but could be a different PR in future)

The use case would be BGP PIC, and recursive routes handling. BGP PIC has started in design discussion in routing WG. Recursive routes support would be discussed after.

https://lists.sonicfoundation.dev/g/sonic-wg-routing/wiki/34786

I have updated "Scope" for clarification and added PIC example in "Overview" section.

eddyk-nvidia · 2023-08-08T03:41:54Z

doc/fpmsyncd/hld_fpmsyncd-NTT.md

+- When `fpmsyncd` receive the `RTM_NEWROUTE` on sequence 4, the process will write the NextHop Group with ID 118 to the `NEXTHOP_GROUP_TABLE` using the information of gateway and interface from the NextHop Group events with IDs 116 and 117.
+- Then `fpmsyncd` will create a new route entry to `ROUTE_TABLE` with a `nexthop_group` field with value `ID118`.
+- When `fpmsyncd` receives the last `RTM_NEWROUTE` on sequence 5, the process will create a new route entry (but no NextHop Group entry) in `ROUTE_TABLE` with `nexthop_group` field with value `ID118`. (Note: This NextHop Group entry was created when the `fpmsyncd` received the event sequence 4.)
+


Will Zebra/FRR re-create the NHG (delete old one + create new one) and update all routes with new ID on change of NHG (e.g. due to failed link to NHGm) ?

Yes, if there are one or more route referring to the NHG with members before the change.
(In this case we will have an old NHG with n members referenced from some routes and a newly created NHG n-x members referenced from other routes)

I think it's No if there are no routes referencing to the NHG with members before the change.
But I'm not 100% confident so let me check again and update the result here.

That would depend on NHG type (BGP vs IGP), the type of triggered events and how the trigger events get handled in zebra.

I have added explanation that NHG ID and members are managed by FRR in "Overview" section for clarification.

eddyk-nvidia · 2023-09-21T06:59:47Z

doc/fpmsyncd/hld_fpmsyncd-NTT.md

+
+- FRR configuration
+ - (1) config zebra to use `dplane_fpm_nl` instead of `fpm` module (this is default since 202305 release)
+ - (2) set `fpm use-nexthop-groups` option (this is disabled by default and enabled via `CONFIG_DB`)


does it assume the use of Unified Management Framework ?

Is this SONiC_Design_Doc_Unified_FRR_Mgmt_Interface.md the Unified Management Framework you have asked?
If yes, we are using sonic-cfggen and zebra.conf.j2 template to generate zebra config file based on CONFIG_DB entry.

…test case

nakano-omw · 2023-10-02T11:01:10Z

I have PR the source code

lguohan · 2023-10-02T16:29:46Z

doc/fpmsyncd/hld_fpmsyncd-NTT.md

+
+### Configuration and management 
+
+This NextHop Group feature is enabled/disabled by config option of zebra (BGP container): `[no] fpm use-next-hop-groups`


can you follow example here?

https://github.com/sonic-net/SONiC/blob/master/doc/BGP/BGP-supress-fib-pending.md#632--device_metadata

Yes. @nakano-omw is working on moving enabled/disabled config under DEVICE_METADATA.

Updated both HLD and Source Code to config use DEVICE_METADATA.

doc/fpmsyncd/diagram/fig2.svg

eddieruan-alibaba · 2023-10-05T19:39:34Z

@ebiken-ntt do you plan to add NHG update handling when local link is down? i.e. take some benefits from using NHG. What's your plan on that?

ebiken-ntt · 2023-10-13T04:48:28Z

@ebiken-ntt do you plan to add NHG update handling when local link is down? i.e. take some benefits from using NHG. What's your plan on that?

No, at least not for the 202311. We expect FRR to notify any NHG update via dplane_fpm_nl including ones due to local link down. But we can discuss for future release if there are any idea.

Signed-off-by: Kanji Nakano <kanji.nakano@ntt.com>

nakano-omw · 2023-11-16T00:51:01Z

Hi @zhangyanzhao @eddieruan-alibaba
All the comments of this PR has been addressed. It has been reviewed in Routing WG. Could we merge the PR ?

eddieruan-alibaba · 2023-11-16T00:58:10Z

I have approved this HLD. You could merge it. We will continue this path further to PIC and recursive routes convergence support.

nakano-omw · 2023-11-16T01:05:41Z

@eddieruan-alibaba I can't merge because I don't have enough permissions. Would you be able to grant me the permission?

nakano-omw · 2023-11-16T01:08:39Z

@eddieruan-alibaba can you run merge ?

dgsudharsan · 2024-05-21T04:08:44Z

@eddieruan-alibaba @lguohan @StormLiangMS @ntt-omw Due to recent performance issues found in fib suppression feature we plan to disable the feature in 202405. The dplane_fpm_nl is tied to fib suppression feature and would not be available if fib suppression feature is removed.
As this feature is dependent on dplane_fpm_nl module, unless we have fib suppression in 202405 this feature will not work.

nakano-omw · 2024-05-21T04:30:57Z

@dgsudharsan Could you please elaborate on the performance issue? Is it a FRR issue?

dgsudharsan · 2024-05-21T04:33:44Z

@dgsudharsan Could you please elaborate on the performance issue? Is it a FRR issue?

Yes. Its an FRR issue. If there are large number of routes and if link flap happens resulting in multiple withdraw and advertisement of the routes, zebra's processing queue will be built up resulting in zebra consuming memory in GBs. This memory will not get freed to the OS (due to low level libc implementation) although its freed by zebra at the end. So systems with lower memory might get impacted and would eventually run out of memory.

eddieruan-alibaba · 2024-05-21T05:04:56Z

@dgsudharsan do we have an issue opened for this issue? We could discuss this issue in routing WG. Alibaba saw this issue as well. I could ask someone from my team to discuss our workaround in FRR.

eddieruan-alibaba · 2024-05-21T05:06:15Z

@nakano-omw where are you on the work which @hasan-brcm and NTT worked on to further enhancing NHG handling?

eddieruan-alibaba · 2024-05-21T05:10:28Z

@dgsudharsan But why fib suppression feature is removed. This issue is independent to this feature, right?

dgsudharsan · 2024-05-21T05:13:54Z

@eddieruan-alibaba We removed it in 202305 and 202311 sonic-net/sonic-buildimage#17660 . With this feature since there is a delay zebra queue builds up.
However for 202405 I am not still sure if we do full revert or just disable fib suppression and keep dplane_fpm_nl. @stepanblyschak can you please clarify?

nakano-omw · 2024-05-21T05:24:03Z

@nakano-omw where are you on the work which @hasan-brcm and NTT worked on to further enhancing NHG handling?

The NHG support we are developing will not work if dplane_fpm_nl is disabled because it is required.

stepanblyschak · 2024-05-21T08:30:31Z

@nakano-omw @dgsudharsan I plan to keep dplane_fpm_nl for 202405

zice312963205 · 2024-05-22T09:23:31Z

@dgsudharsan Could you please elaborate on the performance issue? Is it a FRR issue?

Yes. Its an FRR issue. If there are large number of routes and if link flap happens resulting in multiple withdraw and advertisement of the routes, zebra's processing queue will be built up resulting in zebra consuming memory in GBs. This memory will not get freed to the OS (due to low level libc implementation) although its freed by zebra at the end. So systems with lower memory might get impacted and would eventually run out of memory.

I have submitted a description of the issue to the FRR community: FRRouting/frr#15016
the system runs out of memory due to excessive temporary memory consumption by the dplane. And I have attempted to fix this issue in the PR below: FRRouting/frr#16067

zhangyanzhao · 2024-05-22T16:32:14Z

review is on-going, keep it in 202405 release for now.

dgsudharsan · 2024-05-22T17:24:52Z

@nakano-omw @lguohan @eddieruan-alibaba Stepan did some experiments and confirmed that there is no memory issue with dplane_fpm_nl without fib suppression and the reason for memory increase is due to fib suppression feature itself. We will not revert dplane_fpm_nl but disable fib suppression feature alone.

zhangyanzhao · 2024-06-04T17:16:07Z

code PRs are not approved yet, move to backlog

nakano-omw and others added 2 commits July 13, 2023 15:43

init fpmsyncd HLD

a14e3af

fpmsyncd: hld: update diagram and explanation

cb49f1b

lguohan reviewed Aug 1, 2023

View reviewed changes

eddyk-nvidia reviewed Aug 8, 2023

View reviewed changes

ebiken-ntt added 4 commits September 18, 2023 14:51

Clarify scope and limitation based on Routing WG discussion on 14th Sep.

1199962

Add Warmboot/Fastboot limitation

76cf405

WIP: add Configuration requirement

583a4b1

Add description of who manages NHG ID and members

f7faddb

eddyk-nvidia reviewed Sep 21, 2023

View reviewed changes

fpmsyncd: hld: add feature enable/disable config design with CLI and …

c658cc4

…test case

lguohan reviewed Oct 2, 2023

View reviewed changes

eddieruan-alibaba reviewed Oct 5, 2023

View reviewed changes

doc/fpmsyncd/diagram/fig2.svg Outdated Show resolved Hide resolved

Update feature enable/disable design and CLI. Update test plan.

0a50e20

Signed-off-by: Kanji Nakano <kanji.nakano@ntt.com>

eddieruan-alibaba approved these changes Nov 16, 2023

View reviewed changes

eddieruan-alibaba merged commit 1d57663 into sonic-net:master Nov 16, 2023
1 check passed

nakano-omw mentioned this pull request Jan 25, 2024

Fpmsyncd Next Hop Table Enhancement #1410

Closed

		-->

		This design modifies `fpmsyncd` to use the new `APPL_DB` tables.


		### Configuration and management

		This NextHop Group feature is enabled/disabled by config option of zebra (BGP container): `[no] fpm use-next-hop-groups`

Fpmsyncd Next Hop Table Enhancement HLD #1425

Fpmsyncd Next Hop Table Enhancement HLD #1425

Conversation

ntt-omw commented Jul 13, 2023 • edited Loading

linux-foundation-easycla bot commented Jul 13, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nakano-omw commented Oct 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eddieruan-alibaba commented Oct 5, 2023

ebiken-ntt commented Oct 13, 2023

nakano-omw commented Nov 16, 2023

eddieruan-alibaba commented Nov 16, 2023 • edited Loading

nakano-omw commented Nov 16, 2023

nakano-omw commented Nov 16, 2023

dgsudharsan commented May 21, 2024 • edited Loading

nakano-omw commented May 21, 2024

dgsudharsan commented May 21, 2024

eddieruan-alibaba commented May 21, 2024

eddieruan-alibaba commented May 21, 2024

eddieruan-alibaba commented May 21, 2024

dgsudharsan commented May 21, 2024

nakano-omw commented May 21, 2024 • edited Loading

stepanblyschak commented May 21, 2024 • edited Loading

zice312963205 commented May 22, 2024

zhangyanzhao commented May 22, 2024

dgsudharsan commented May 22, 2024

zhangyanzhao commented Jun 4, 2024

ntt-omw commented Jul 13, 2023 •

edited

Loading

linux-foundation-easycla bot commented Jul 13, 2023 •

edited

Loading

eddieruan-alibaba commented Nov 16, 2023 •

edited

Loading

dgsudharsan commented May 21, 2024 •

edited

Loading

nakano-omw commented May 21, 2024 •

edited

Loading

stepanblyschak commented May 21, 2024 •

edited

Loading