Resiliency - failure frequency x consequences

[iawells]

This is from a review comment (on pull #60). I think it's got potential to be its own document, because it's useful to discuss how cloud native benefits us differently from the way it does others.

@jeffsaelens:

This LGTM, but as a side question, is there a possible future where we say BGP isn't a good candidate for this style of packaging and hosting? I'm picking here solely because its the first use-case, but I'm curious... do we have a threshold for how much we try and get a use case to work before we say "maybe this is a bad idea?", or do we keep engineering until we beat BGP into submission?

@iawells:

Interesting starting point. In addition to BGPs allergy to NAT, there is also the convergence issue, and the time it takes for BGP to get to a "happy" status upon standup, or recovery.

It's an interesting point. We discuss recovery from failure, but the consequences of failure have to be considered in this. What we're really saying with more conventional apps (e.g. web services) is that no meaningful consequences if a component fails as long as we're ready to accept another request. Here, we have different consequences for failure and we have to see if it matters.

is there a possible future where we say BGP isn't a good candidate for this style of packaging and hosting ?[...] do we have a threshold for how much we try and get a use case to work before we say "maybe this is a bad idea?"

I think we can keep judgement out of this and say with an even hand 'this is the best that this allows us to do, take it or leave it'. It may not suit the use cases it used to; it might be better for other ones (e.g. much bigger RIBs).

Bear in mind, with BGP, that it does fail when whatever it runs on dies, and it always has - by 'fail', I mean 'withdraw all routes'. There's nothing new about that.

The point about clouds that we perhaps forget, is that failures are more likely - because we (theoretically) buy cheaper servers and use more fragile equipment, there are more points of failure, and we also expect ops activities like upgrades to be more frequent and more disruptive (killing containers during an upgrade is fine, e.g.). We are supposed to use the new tools to minimise the consequences of them. But failures are still monthly, not hourly, so they might be acceptable without doing more.

The overall resilience would depend several factors. We can get a replacement BGP server running very quickly, even if the original dies - that's new. We can make the RIB live in a distributed database, so a new process can use GR as it subs in for an old one - that's new too. And no-one builds their network to rely on one BGP server always running, so BGP never had to be 100% even in the before times.

We can use this sort of judgement method with whatever we're doing. This is how it was before, this is how it is now - better in some ways, worse in others - and now you-the-user decide if that's worth having. Using cloud native does have benefits; they're just not as straightforward, so we have to consider this a bit more closely.

[I'm wondering whether this discussion wants putting somewhere where people will find it.]

[electrocucaracha]

This seems related with the openness offered by open source given that we don't denied the possibility of failures in the implementation but at least offers a partial solution that iteratively will provide a better solution. I like the idea to provide a judgment as long as it includes some possible alternatives/solutions, like failing fast, recovering quickly, etc.

[taylor]

Recommendation: K8s Workload resource (Pod, Deployment, ReplicaSet) work as expected after being rescheduled to a new node

[taylor]

Recommendation: Container is responsive after a reset

Testing Implementation History

• Resilience test: Does CNF pass the Container Kill Chaos Mesh Experiment 🗡️💀♻️? #158

Tentative/maybe for a different best practice

• Statelessness test: Is data lost or not accessible by the CNF when its Container is killed/rescheduled? #160