Increased NSE expiration time might cause traffic disturbance

[zolug]

Describe the bug
The NSE expiration time calculation has changed in NSM:
With the changes basically it's NSM MaxTokenLifetime that determines the lifetime of an NSE.
ExpirationTime time parameter in the NetworkServiceEndpoint structure that has been used during the registration procedure is ignored right after the first refresh (falls back using token lifetime).

This might cause traffic disturbances if for example a node hosting a LB-FE gets rebooted.
That's because the related NSE Custom Resource could remain in etcd much longer. NSM_MAX_TOKEN_LIFETIME defaults to 10 minutes, while default expiration time was 1 minute before.
Until the NSE lifetime of an unavailable LB expires, proxies consider it a valid egress next hop (assuming datapath monitoring is off).

To Reproduce
Steps to reproduce the behavior:

1. Use a recent NSM version that contains the NSM changes mentioned above. (e.g. NSM 1.9)
2. Check the expiration time of the related LB NSE in etcd.
3. Reboot a node with an LB-FE.
4. The proxy will get informed about the loss of LB-FE with severe delay (when the CR expires). (Next hop route for the rebooted LB will be maintained by the proxy until that.)

Expected behavior
We should be able to control NSE expiration time independent from the MaxTokenLifetime.

A possible way forward could be to introduce a custom chain element that sets the expiration time to 1 minute, thus ensuring backward compatibility.

Check if datapath monitoring should be enabled (e.g. through env variable) between Proxy and LB.

Context

• Network Service Mesh: v1.9
• Meridio: 1.0.7

Logs
NA