Proposal to support multi-node etcd clusters

docs/proposals/multi-node/etcd-backup-restore-work-flows-life-cycle.txt

@@ -0,0 +1,32 @@
+// build: https://sequencediagram.org
+
+title etcd-backup-restore work-flow life-cycle
+
+
+participant etcd-backup-restore http server (initialize)
+participant health checker and staus updater
+participant etcd-backup-restore leader elector/detector
+participant snapshotter
+participant client request disconnector
+participant defragmenter
+
+activate etcd-backup-restore http server (initialize)
+activate health checker and staus updater
+activate etcd-backup-restore leader elector/detector
+
+etcd-backup-restore leader elector/detector->etcd-backup-restore leader elector/detector: OnLeaderStart
+etcd-backup-restore leader elector/detector->snapshotter: Start
+activate snapshotter
+etcd-backup-restore leader elector/detector->client request disconnector: Start
+activate client request disconnector
+etcd-backup-restore leader elector/detector->defragmenter: Start
+activate defragmenter
+
+etcd-backup-restore leader elector/detector->etcd-backup-restore leader elector/detector: OnLeaderStop
+OnLeaderStart
+etcd-backup-restore leader elector/detector->snapshotter: Stop
+deactivate snapshotter
+etcd-backup-restore leader elector/detector->client request disconnector: Stop
+deactivate client request disconnector
+etcd-backup-restore leader elector/detector->defragmenter: Stop
+deactivate defragmenter

docs/proposals/multi-node/etcd-member-initialization-sequence.txt

@@ -0,0 +1,38 @@
+// build: https://sequencediagram.org
+
+title Etcd Member Initialization
+
+participant kubelet
+participant etcd container bootstrap script
+participant etcd-backup-restore sidecar container
+database etcd resource
+database etcd configuration template
+database backup
+database etcd client service
+participant etcd container etcd
+
+activate kubelet
+activate etcd resource
+activate etcd configuration template
+activate backup
+activate etcd client service
+kubelet->etcd container bootstrap script: (re)start
+activate etcd container bootstrap script
+kubelet->etcd-backup-restore sidecar container: (re)start
+activate etcd-backup-restore sidecar container
+etcd container bootstrap script->etcd-backup-restore sidecar container:initialize
+etcd-backup-restore sidecar container->etcd configuration template: Load
+etcd-backup-restore sidecar container->etcd resource: Get Status
+etcd-backup-restore sidecar container->etcd client service: List members
+etcd-backup-restore sidecar container->backup: Get latest backed up revision
+etcd-backup-restore sidecar container->etcd-backup-restore sidecar container: Check if WAL directory contains valid metadata
+etcd-backup-restore sidecar container->etcd client service: If necessary, remove old member and add new member
+etcd-backup-restore sidecar container->etcd resource: If necessary, remove old member and add new member
+etcd-backup-restore sidecar container->etcd-backup-restore sidecar container: Check if data directory contains valid data
+etcd-backup-restore sidecar container->etcd-backup-restore sidecar container: If necessary restore latest full snapshot from backup
+etcd-backup-restore sidecar container->backup: If necessary, fetch latest full snapshot and restore the database from it
+etcd-backup-restore sidecar container->backup: If necessary, fetch and apply incremental snapshots on the restored database
+etcd-backup-restore sidecar container->backup: If necessary, take a fresh full snapshot and upload
+etcd container bootstrap script<--etcd-backup-restore sidecar container: If necessary, generate appropriate etcd configuration (based on the loaded template) and return success.
+etcd container bootstrap script->etcd container etcd: start using the generated configuration
+activate etcd container etcd

docs/proposals/multi-node/multinodeetcd.drawio

@@ -0,0 +1 @@
+<mxfile host="app.diagrams.net" modified="2020-11-10T19:30:41.514Z" agent="5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.193 Safari/537.36" etag="ZGOACWqTFRD_QPWywvvx" version="13.8.9-1" type="device"><diagram id="C5RBs43oDa-KdzZeNtuy" name="Page-1">7Vxbe5s4EP01fnQ/c7cfYyfttpvdZJN0kz5ykW0abFEQjd1fXwkkDBI24ADGTfLgwCDrMjpzNDMSHiiz1eZTYPrLf6ADvIE8cjYD5XIgy5PxGH8SwTYRaMokESwC10lE0k5w7/4CVDii0sh1QJgriCD0kOvnhTZcr4GNcjIzCOBLvtgcevlWfXMBBMG9bXqi9NF10DKRjmVjJ/8LuIsla1nS6fhWJitMRxIuTQe+ZETK1UCZBRCi5Gq1mQGP6I7p5fHz9tG7ftY/ffkv/GF+nf798O//w6Syj3W+kg4hAGt0dNUX1vXH7ewFBsHTYnbzdH334/NiqCZV/zS9iOprIOsebmTqkxGjLVWj/iMiw5yuzGDhrgfKBX468jf4EwtjZRH5EEE/eaZnniGwQUPTcxf0ezYeBQh2deKrBf0ft2wxwdXDDDc9uoUOe4RHZ/HFscznZcuAlxw1HA/MUfJwTB7yHb5CNu7ZaAbXyHTX8ZD29qiN9lOFCaPtqgdT036OSMV3IEQwAOXakHO9kCksLpN2pvAnCOZebGRzF6MXzyRa4X+XEr50Aug/4G4BYgOjtLZSs6Dmg6vGSMyQAjWTTwCuAAq2uAh9KjNyoJSnMgZ42RGINKayZYY8JlRmUs5apFXv7BJfUNOsYaZagZlymsTs5JNLG658uI41Mv0OENoy1hszQUp3aqEOD/JEZR1KgsbkAoUxWQA8E7k/85RdpEXa3i10cXfTxoaymp+wfAVwPg8xaPhJSDt9/Lzo7/SZH072szPOLG20HaIsbfbPZEeFMR9zCPVTs6NxanbU6+rwbbDj+J0d39nxjbGj0TdynJyaHMd1Vfg2yFGS+8yOs7t+htwDmZRNuFtkLfJMpJWug/Jp3LjvuTZG0dkSGe/mjU9NZJJyaiZLTfadyvJ6qZCeAGvngmRzyeR4Zhi6NtZ7iMwAieIM3PGEBNsnAvUP2kRlgm+xYCSlJS431BqSu2327hYELh4hpj0q3LjoKXP9jTWFr3f1kBtWTT0TC2EU2KA8l4KYFZcsnMDJJbhFrGXQpRWgS2sGXYqc5wOVN/Rk3PRb2Tw0XxHnIqkqV1GiGKGixtBalLRpGK1SHqlaCU7rAezcgMMtJPpIOw44kjbBNj/Z/Y1z9Wq6/mGkdgulCpmHRqFUSniM2z4oCVtSfhtitiQTdIDkyA1PlQ0Tn14Rv2qv8JvyE8MZz1eV8asfxq/ROX7lIvz2JgYByHaGThC5JCViQwx06Hm53EDvApPyLp9ZFNC3IEAW0xlfQ6LgPYFAtPIubBwGYhWRQeOQzLs2LeDdwtBFLiT6tiBCcJUpcEEnIkZzVuttKdngGGaiCVpWCpSst6VkRRKUXG1Z69wVKl1yGF5K1xwW2/Vk0VEmeUikTlTdRYdfvRSlW29bqRC1BzBaO8ChCHhZugjc+2Y8ry+ByRkhIcMZ9IhJX65xjN9mCmSSV50hsl+a381iIA2VmrdM8bwO3ceYeVGICoiQrL55BYZ4UXoGnAoLtMrWI7pei/y4ch2HNFM4Y/k5bWuGZA7detECZRRNUQMzNHRU9PHLw1Ng3liKpeP5sbf0DFZPPSqKlXtkIjCPvDgd1F9niiH7dMerivPJ3W9TCQZR1ZYqZ3exfrrz7AoNp4LP8arsbokO9xvz20nuFupAFublDuDIxnbxrNR2Cml6gl1nXML9eYk0A3yMI8m5c016lkYtz1JE0Ws9RJ2zYD4cqOwhllXUnIdYCDDRQcwALF6rolBEGp7XOJAjrgYI3V+mFT8a5QFX3YdJWqCH0Gllg9R4jyGP1077q9hU9BObtlrjOLOV+2i3dcj/lWbLHxCRlCOz4XyEKFS0x2zxVJvbTDGfFAgP8Ay37yOp48P94ssbo1rlsYRDf9LjRjlH3LDMcY4P7HNkHPWUjNPro9As4rknPlLsv/c/l1yhz2cWdPQu5hC3P96cb3vIlt/qEtnUKiOef/4zPFvjlOuMuAH0epvd2Wn5YaDsiaSc1fbPZvdvrrdvs8fuV/A2K2vtuLUS12HW7t5+Scah8u24qewd6j/MT52ckj8kMdF4MgKRz5JBxq0wiMR5h8roWAbh396tuOPZNoNI7JhspwxyMH17tgwi7Qk0OqKQgynLuhRypD9xDPX0hUH0TnwQIW6sfFCPf4nL6AeDyHq98urh+ltinIOp5/NlHOWkjFP0hoVurshJi7UVkn831nfy8zkkrkx2qecwGOy2rzmNp1upW89dOyBQBqUHb6zkRMe1lQpw1Yv4nMdNhHA1gMrD5DeAJK3FVJLBv41giGfmio7m8GFJczMkBvvXwHRIug7z4TMISaNrB39GvgdNh9xbxXPTp0xT4Rs0DS4jVV+QOWAVIka6OaKnSRwEj92A1ThaV6V2lhuhHV2r1S9NypWvu3zg291vZSXFdz84plz9Bg==</diagram></mxfile>

hack/print-content.sh

@@ -0,0 +1,3 @@
+#!/bin/bash
+
+cat docs/proposals/multi-node/README.md | gh-md-toc - | sed 's/\*/-/g' | sed 's/   /  /g' | sed 's/^  //'