Initial docs for coordinator recovery and e2e test

[johscheuer]

Description

Adding documentation and an e2e test on how to recover a multi-region cluster if the majority of coordinators are lost.

Type of change

Please select one of the options below.

• Documentation

Discussion

Testing

Added an e2e test for the documented steps.

Documentation

Added.

Follow-up

Those steps will be added into the plugin to automate this recovery setup.

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 7f29195
• Duration 2:55:00
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[nicmorales9]

Looks good to me and seems like a great addition!

[johscheuer]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 7f29195
• Duration 2:55:00
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

The failed test was not related to those changes.

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 04979b2
• Duration 3:05:55
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)