Destructive Chaos Testing: Sandbox healing simple cases

[Bolodya1997]

Overview

Cover the following heal scenarios with sandbox tests:

1 death cases:

1. Local NSMgr:
   • restored:
2. Local Forwarder:
   • restored
   • died
3. Local Endpoint:
   • restored
   • died:
     1. A new local Endpoint selected.
     2. A new remote Endpoint selected.
4. Remote NSMgr:
   • restored:
   • died:
     1. A new local Endpoint selected.
     2. A new remote Endpoint selected.
5. Remote Forwarder:
   • restored
   • died
6. Remote Endpoint:
   • restored
   • died:
     1. A new local Endpoint selected.
     2. A new remote Endpoint selected.

2 death cases:

1. Local NSMgr(r) +
   1. Local Endpoint(r/d)
   2. Remote NSMgr(r/d)
   3. Remote Forwarder(r/d)
   4. Remote Endpoint(d)
2. Local Forwarder(r) +
   1. Local Endpoint(r/d)
   2. Remote NSMgr(r/d)
3. Remote NSMgr(r) +
   1. Remote Endpoint(r/d)
4. Remote Forwarder(r) +
   1. Remote Endpoint(r/d)

3 death cases:

1. Local NSMgr(r) +
   1. Remote NSMgr(r) + Remote Endpoint(r)
   2. Remote Forwarder(r) + Remote Endpoint(r)
2. Local Forwarder(r) +
   1. Remote NSMgr(r) + Remote Endpoint(r)
   2. Remote Forwarder(r) + Remote Endpoint(r)

Should be tested:

1. Shutdown becomes when hop:
   1. doesn't pass any Request/Close.
2. If case includes selecting another Forwarder/Endpoint, during the healing should be selected:
   1. the same mechanism.

References

• Existing heal tests - https://github.com/networkservicemesh/sdk/blob/main/pkg/networkservice/chains/nsmgr/heal_test.go
• To disable refresh Requests / timeout Closes should be used clockmock - https://github.com/networkservicemesh/sdk/blob/main/pkg/tools/clockmock/mock.go

Blockers

1. clock in sandbox - [sdk#764] Rework sandbox to use clock.Clock #785
2. registry chain healing - Endpoint registry chain doesn't handle GRPC connection failure #713
3. issue with endpoint restore healing - Heal doesn't update restored endpoint name #918
4. issue in connect client / heal client - Connect client connection and heal client monitor stream fail in different times #925
5. client healing - Support heal for NSC #938

[Bolodya1997]

After the restore was disabled, all single death tests were implemented.
Multi death cases + tests with additional parameters are all moved to #899.