Restore neighbor table to kernel during system warm-reboot

[zhenggen-xu]

Restore neighbor table to kernel during system warm-reboot

Added a service: "restore_neighbors" to restore neighbor table into
kernel during system warm reboot. The service is started by supervisord
in swss docker when the docker is started.

In case system warm reboot is enabled, it will try to restore the neighbor
table from appDB into kernel through netlink API calls and update the neighbor
table by sending arp/ns requests to all neighbor entries, then it sets the
stateDB flag for neighsyncd to continue the reconciliation process.

-- Added tcpdump python-scapy debian package into orchagent and vs dockers.
-- Added python module: pyroute2 netifaces into orchagent and vc dockers.
-- Workarounded tcpdump issue in the vs docker

Signed-off-by: Zhenggen Xu zxu@linkedin.com

- What I did
Restore neighbor table to kernel during system warm-reboot

- How I did it
Added a service: "restore_neighbors" to restore neighbor table into
kernel during system warm reboot. The service is started by supervisord
in swss docker when the docker is started.

In case system warm reboot is enabled, it will try to restore the neighbor
table from appDB into kernel through netlink API calls and update the neighbor
table by sending arp/ns requests to all neighbor entries, then it sets the
stateDB flag for neighsyncd to continue the reconciliation process.

-- Added tcpdump python-scapy debian package into orchagent and vs dockers.
-- Added python module: pyroute2 netifaces into orchagent and vc dockers.
-- Workarounded tcpdump issue in the vs docker

- How to verify it
1, vs test cases in sonic-swss
2, load the image and tried the restore_neighbors service manually, and it worked.

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

[nikos-github]

Can you explain the reason why we would want to restore neighbor table to the kernel in a warm-restart?

[zhenggen-xu]

@nikos-github This is to support whole system warm reboot case, where we will reboot the kernel, and then the neighbor entries would be gone in kernel. We need restore them into kernel before neighsyncd kick off the reconciliation logic.
In case we don't do this, we may lose the active neighbor entries in appDB and ASICDB thus impact the traffic.

[lguohan]

I think we should put restore_neighbors.py in the sonic-swss repo and let debian to install it. in terms of function, it is part of neigborsyncd.

[lguohan]

as comments

[jipanyang]

If the arp request/NS packet is being built for each neighbor and the neighbor is supposed to reply, in that case kernel will have up to date neighbor data.

Injecting old data into kernel seems unnecessary and causing problem here(stale entries) ?

[nikos-github]

@zhenggen-xu How are you guaranteeing that the entry you are pushing in, is not stale and not overriding a valid/newer one?

[nikos-github]

I think a more correct approach to consider is to arp for the neighbors in the DB which are not in the kernel already. You can compare with what you have with what arp puts into the kernel and for anything that you have in the DB but not in the kernel, you can insert/complement and let it age out. Any entry in the kernel that's also in the DB, is guaranteed to be valid and newer against what the DB has.

[zhenggen-xu]

If the arp request/NS packet is being built for each neighbor and the neighbor is supposed to reply, in that case kernel will have up to date neighbor data.

Injecting old data into kernel seems unnecessary and causing problem here(stale entries) ?

There were performance issues where we need wait for the neighbors to reply. So the approach here is that we will keep all updated neighbor entry, and still insert the entries that is not updated to make sure we don't remove them from HW due to the un-replied messages and cause traffic lose, those entries will be expired based on Linux timer.

[zhenggen-xu]

I think a more correct approach to consider is to arp for the neighbors in the DB which are not in the kernel already. You can compare with what you have with what arp puts into the kernel and for anything that you have in the DB but not in the kernel, you can insert/complement and let it age out. Any entry in the kernel that's also in the DB, is guaranteed to be valid and newer against what the DB has.

@nikos-github To arp for all neighbors in DB should not cause problems as long as we don't overwrite the new entries with old state. The code is updated so we would not write the old entry to kernel if exists in kernel. Code is moving to sonic-net/sonic-swss#661

[jipanyang]

There were performance issues where we need wait for the neighbors to reply. So the approach here is that we will keep all updated neighbor entry, and still insert the entries that is not updated to make sure we don't remove them from HW due to the un-replied messages and cause traffic lose, those entries will be expired based on Linux timer.

Could you share the performance data? I thought with scapy replacing arping the performance is not problem any more.

[zhenggen-xu]

I think we should put restore_neighbors.py in the sonic-swss repo and let debian to install it. in terms of function, it is part of neigborsyncd.

Done

[zhenggen-xu]

There were performance issues where we need wait for the neighbors to reply. So the approach here is that we will keep all updated neighbor entry, and still insert the entries that is not updated to make sure we don't remove them from HW due to the un-replied messages and cause traffic lose, those entries will be expired based on Linux timer.

Could you share the performance data? I thought with scapy replacing arping the performance is not problem any more.

The performance is related to receive side after using scapy. If arp/nd entries didn't get back in time, it is not safe to assume it is gone, we may have to repeat to send arp/ns packets and check responses. Anyway, I think we can document this as an improvement bug, we can tackle it later.