Look at bloomberg/PowerfulSeal for testing

[lanzafame]

@hsanjuan @ZenGround0 Do either of you see any value in using a tool like PowerfulSeal? It is sort-of ChaosMonkey for Kubernetes.

[ZenGround0]

Yes! I definitely see value in using something like this. In the past I hacked together some very rough tooling to accomplish a simple version of this, see the process running in the cluster testing container and the k8s test making use of it.

Doing something more principled would be very good and at a glance PowerfulSeal would be a good candidate for this. When it comes to tests that bring processes up and down one of the trickier things to keep in mind is persisting state between periods of unavailability. I built out some hacky things to accomplish a version of this that gets the job done but they're inelegant and difficult to use and extend. It would be great to re-examine this part of k8s-ipfs at the moment the random failures behavior is re-evaluated.