From e255ba8a7d104027eb816709c434b75647d9e675 Mon Sep 17 00:00:00 2001 From: Nir Date: Sun, 15 Nov 2020 17:13:20 +0200 Subject: [PATCH] baremetal: add poison pill remediation enhancement Suggesting an opt-in remediation feature, which doesn't rely on BMC credentials. Each node will take care of rebooting itself in case of a failure. Signed-off-by: Nir --- .../baremetal-poison-pill-health-flow.png | Bin 0 -> 39338 bytes .../baremetal-poison-pill-remediation.md | 189 ++++++++++++++++++ 2 files changed, 189 insertions(+) create mode 100644 enhancements/baremetal/baremetal-poison-pill-health-flow.png create mode 100644 enhancements/baremetal/baremetal-poison-pill-remediation.md diff --git a/enhancements/baremetal/baremetal-poison-pill-health-flow.png b/enhancements/baremetal/baremetal-poison-pill-health-flow.png new file mode 100644 index 0000000000000000000000000000000000000000..9235e4b3dd0f39166f0c8c1738b8d2aa5cc366c7 GIT binary patch literal 39338 zcmbTdc_5Tu`#)|A*~yaZTOnD-GDe9RGt3z4*v67&?7Nw<&B(q}NR}*Zh%DJkD2nV! zMY2~y5-A}{=zHe$I@k5OUa#wL@UK|CuK60N?*ijDaQ&LDoqOl)B-BHb3^B4$RkH|(*q}qTOoQxUn$}SFFk`haEeEf_U~f2u zh`?j~{T=+|AYcL!{{G(DRt{thA0MbXK19pN*V5C+%m7AJhlB;7pzgj99L!Q1MFevY z3_+n#+U{7jNDFn-Fk=mvucwA8)gMm@H>GGo9kkr2WM2bstgS85L))I#wWXPD2+>2s z)W^X<4esaf6Nojl4WpWRyIBS!)x(030mfK$k5FF&4GRZ?Eych9NA^I%O^{&G5gI-Q z=nzksJH*2d27?(Gn&P#n1n`5QU8tt9H(b*WuWjs$0{7!WJfLQHR5;eaNIM)wvNMcC znnZe7khH_S1H80xUIE}9sF?|veFW7ITqA?Pl^eM5f`)5oVJ$tO>S1;`Eg#chYl}b) zOG;=2J_2p7h9-s(KtnrCygvd<_On8{`yfafAyBl1DcV=n#|^JeK?j9dV%!~!DHs!f zV@+!VtQ9<*V(V{ghqco*upv?rgwQ~gTabT%H_nUf;15AKcm$~Wk>KujCO8`~07Htp z5!N=q4q_SvHN+t(mOkFvmg*7a-hs9@))rO<_9zoWl7p4M4P3+4zz-Wt(DD!RP{XSS zkziDUwXGW5Ufny?k7k^H#uniOFN6gKk3m{lVl=5XHgF%DPXx>?I1p=#aQD+9YG{QT zLcPh>+GtdGK!~-cS3od$yKhpjHkly;ZQ$mCq!^hX zG)y$z!$^4TFbyv#6ry40V@5z4P$9-DK@j{MtP$b(AX|!Ngr|=m zUc*ew2S?IGP>iu8QzIJ(9F}N`(WF?1*n3+0*m+=~L<>R)1Z8659i)a2_kx@GB2d;) zD~cJ$-5@du?t#^`i`2xR+%(MsLoKb1O@gdZRGRG?s2QP=T4YNTLkvM1O9@3Al0twL zAW#}U!TtuOp*CR{yoR9-3FGMpH!vXM;P!A_NH|3cW2=QxMfrklZ%7DIB|*^EzEDdF z)d3E3CurLm8bH-Zq4w6EYB;92V*j2*+92`bQdtxLJ4xd!i!^EyGda#(~zxzW%DF1S>Ot zZ)?i{yt!|Lm$`$NH*m;4UL+%gwr7}$otLd9k!T#`5r$Ov!dXI)0aQ3O!rRLdJlM$F zG|)>mEKH5;hP5?`G>WjY^|4mwCuhC{qmP24dSMzk&D0kJc}d72rU`Dt2sc$rbu z36YwCZlQK+w9!YHyBQJP%`jF(1lkPDg0(65))yCQZEqgt12d%c7EUFY6E(p_WKaOo zlB#a)7j9}|u`atl(7B*H`Z)&J7 z)!awb7Xti>8WwGA6y!}ZCQ}faFdqw~CCbe^5@(Ml+lNyjFxu+ja8@WQ6NC+L1csBncQX!qUT<0M*b4^l|gl)QUvHLkX7lNGmMKD?k-y zjH23jSy2o%LVVRDKo7>Arfz6e4SRcAco4=P4p9pXrR_+(Elk_iHWD5l;14Fm5btGT zLEG|L{(;sSaH?^znwGVjIotuPyhWsqDKf$Y6KJMs>S5?_<6vQ>h9%mYg<3}X2ixEf zwyF>{J2L`pTxM2Qk?LBJ>L_g^9F>f4GeZQ25$*h;z5yC)*50s8Dgs$iZZpaQMaX1P!T4k zWP}FZ10IPt_15+=3P*uvQTN8VA50gMb02m7b1JiKuHHX<)n0aYgSyQy# zG&MqDb~qn`yO}|VX(T?7011S8g+!o3N#P{Fa1Rqyq@@wUjiBYNMo=f3+Xhq2LLIzd zelR;fKdcAT6K3PBszqB@EQl@$w3-)0Eds=GLk$E7cH%zL#pF zqm!h=AmP@L&R?$|oilyA`!_=RxoG}vX@j9W4$M^~0)xURCgUvSM-(-LGo%F+A?$Me zIH?nwJ~+#xmMLSBoF@(&cK4!RZRyoVeItMC{QO>huEn|GbKlk+dB^$B{oVFIXrce zId0{t>=N_alRSnwC3o-h|xx$YSQ07ru%0u>Ye!*<69AuIK3Pd_LPaA4kf9>`l48K z_}oSpqu1`YcV4p(1D9888?;*AQ)|CW+$*81R#&|s4v2XRD&bwZK(h|`>=~-W#f|tc zv#kxkKNY&XIREFX%x?I~Yj}Ohjz+BsCDP_6HYj68YRNQ{QNboh0ptkP)o!MZ7J`3jw*-1DX zb$i9lZ=z1Y*05h+ud0-k@%!`5*caC$-YeK*mR8EM1y5H*EA3zZ@pS9u*|Nnfv9lEi zn5!p?G6YVasETP19?X$iPCs=+^^S2qq{V+y%QYVLD)8x#khcBJ;gMH*f0f+#mP>+P z?5xk4PB(iV;Z`DtXWp>-+f@4|Xw?71lTN&eQMUM*q++$d8$I01^NHLA?QT!NE31>= zm_mn6G_vE#dQ%P_7hhacTYm7y46E%`rRl}n36?50yZ!rQgUeh@v8KP5-}qfAa^&8Z zs!YFkk0YRAtK({&QM*|$4XR~+h=J93VjizZvu0;Wy7r!AmUQifG?^s4b*-%_dN$_I zdSL@9c;)r&=AoC@&rXil*kI9^#s165csJe-V2MkAli7s?C#5RpN1z`xE<6fVyL>{E z&+hZy&$Ye`mL^j33jAW0&Z@jRdD(=#Idm#>ji=K$^HrAKVdU120SWdiCECQY?XA7F z2KJ9p-#d4c8ovS4WR8}yb8;6-6W_hKJ!<6j_h1)#;c>X;r{O}O=Ed>vDV!oN1niuo z)(y6gAKNw9p4(YV>iP9!@y^Px)NLB5~2;aH7&w(3ZgCEfAnpQ2VIUhntP)hvIG1n>4B z_cK(pgLB)j4G>RmQkI3c)*7A`*Vxp8xt$mky$dEz>K5|VY1tMrls@l<7`+yDaB8N- ze`NJ-?Z^Y)H%X28O7;+!vWQWX&`=7uaPv~t%@{tc-9VNYr$-uR>95m+bOkc3)1@$S zuS0L=zN_3bzfZ_*=jfj{UA`%O3B zxs=%?nPZ+d_dNgft~B*#%wBc8t5l9;Xyt{fm@nLA0aw)eC+~ZkOJRckggv7dJAT?$ zecLsr(G@I{V)hE0>H1y-cKcc|T9tv=pBw%X_IYq$g1T~>uzsI?$~mO_$TE1_0zLQc zr3h6s4PapXqA*_Fsu;XWN1pz-Mt(@)#PN%<*R10QfwNPASXDDxGLb~$!7j8Uw1l)Y zk51lmv{Rxwwgw5XV`9{(Ij}+TZ?@Pcf4-fysx%Rvj*~KO3(JNrA8&dR@|q#@L_(&( z@au)X%#kuwW)dADccE{E$f@j>mLiA0sqGZ}q0 zG(H|t%!n&wl^@;WGY!P@S}knaQP-zaDRmxV_Y+F#?LZ8cKE_(MJ0i*My_n0T-$*+Y>-IJcbH$N>Heo|%LoddpS zEJjJ8@vqs_l|Jd=R;d{@UY|HlbwIv4zYEN6Ip9)BnatiT8=M;_bQ%Wd?lr99nY6bt?vrvQ7AQHBH{Du9t#NQL3vR<{t-B z^#}nbv5U-fmld9_oV)FnxzhA;Or<2a&Z4J)jZ;?tA-imvX|YFNjy|7+75;{zaNbB6 zfr;`O?T3A5ZHCU>xb~>2niaM>_|_r*DlxDiUlT* zkAek*Cxf+liP#HN7{y8$1j2b)bs6wQjSq9u3KjWe8}3<_r{tXEdzb5TEf<$Zcc0z% z0f<<}D3%z*EK!9?D`Pf){qOHPccoF`nd12;8%y03%2Q73mxtbI>p`Et}uMZoxv;qT$F?LUtb^Fgz^U*#WdH8bTHalId;oZ4ij=icGB?<&mR!3tGR z&<&NxD*4jF;|QaW!c&6;zsdSDOGj`M6_{hwNm4oi<9Tm%7B8vyCJVkQ8_tk^Kfr|hQ*@{HOr`ZdFP5UDkCc%j)$&kjqfAjZesl+?Cbr1&b~^2BdX&z zQ2dr)hC!#CD&lJZw$~oG@`FuH5;AJkr;A|gjEp8_~ zz&eieB=?3yKMr5-%}sccqoE~U=Yp1=ymukuxud`t2kX#&LidzXS!zp`whsEzh1ZO0 zauet4;P|k{*L?+b2_ddx4n&1YS~%*py;#A* zB>i6ds}9_!>u7U8A+pVYuqi{G@6zs0D&%4!7>~H5iHBXJ(8J}iM#36tb#lMkwe7j_ zX5E}5)7k4fVHsxytJF4ud6(-!uj#Z5&$KuTM$&hUVN1d{NA9mAUFXyN@ML0l1)uJ$ zx(sijfrU8E-vIpcXBNt}-DnKr%`|**B6zKdp~;=G_h-^$fniQtwmPYxG;!2=6W+8~ zL!rTUka>bbDZTBJj|4lvt%y=!0roiL*Ynd2X+N)U9&27jZckm4Viqh)T(R_HtCe&g zFiz~_?k1%Rc)#zUL~#h=(oww9Xva2dDW3hT+F3p&`8MR2H+*SAa`mr2q_4p7O&J@B zu6+iL@(zN!T5}^M(2_(h9W+|>m#a0KjJh`=BZhIp=G^_qf<<$9uy-$T80N%cB|knGaq92s zgeNlN5`~&G8iTF{KVF-VrsAR9$K1@?xN(g113H<#6Ee%e>1pp9-FiH(BTf+i5?bE1 zldEz%BHiP);j-=ngHGi}%~PLJ6$%hlEpYkN>nm!z=k8Ao3%_C@Rz)%_xK$xvwKn|J z)v^7bOPSwR5MeKocbUpzVii|I@hG1vGur1Xk?2i}Jf~zkTyW+XmgT0KSbD_sv1XGd zjO$QBlB;jAntMubbyIiKZT)kNxTdB?djl#!{U>d;>qK%i36p|n9AtYrAaAS8Z+4nt zAj*U#NsSz|+4MqCLIqKxKu5KU5bs|wQg~houA+1?=jNKxkb)JCAS)uXn_%}ulp^Ojx!SHuUGL?355QFxRrN4( zu@SB9-ZL_&jVI&7n{we5XIlNv%}M3dN%Wgs6s@_#|w(g z{u%5R$cyqfZB(WVb=3?G^BnL5#U0Uok56O&L`f5qkz(D~eGeQ63uc(xDwO0PaxQj< z*raU#s$;OJKC)YeBSEMsa!O%zYu*0>u>D(hXZYjiCFzK9y*$HxUg$=c%L2Z+G)?@6 zL)*gCQe&;KI_v++}4_^rpaK6cbl!1?QX|v_o=M|aQ*xBO`@ufQ;xg`qo za`!SPeqy!@`mxaWNqjYlpm1A^(DVBG$eeNxQVWe{A^JSvBqAhZDx5#6J6(RjINh9f zq$#28ij;2~X8M5?H-3BJx?(>f;g8@j$NDW5Z7DM(_rB!ufT{U(f9~jX-}`Ce#8{`E zYttc3=lTAiBt)g-ToVO|HkY_Bc1fI;5VSx`Rit0OPU&Arkz;AjV;hxWe(n3t^^42| z=bA8i5YC}=lP7jBvx+4ZS=#S*6ZNO#1ApwlVK?F(`lnx8ZZkg?{>XD4zUO%_{3&MHsv zvqXfr-%q*5G86I449xG7#kc>pcK~1igJL&C0T{bsP{XPCU))M#nK%GeJ>o8I&HlfJ z7EI@i^wV#A&hbn9htu*9qJQ4Xdw&3af~%Fs2(47Ct3VFd9DA=HCmJJqTskY~*ivsd z@wp?4ckm>jiJH?N?)*z=rLu+$CKxd9hcxFOYvDPU?<&>R&BZmD2nmvqlcR4R3eXVx1}i<^VXjv4UQ0!@RI&*mk97Smlnk9@SlX7UlKd@zIm++I zD_8Q!$7K|}n)CK&ywP1ia!47*8M7@(KiGVCW$m&PMe(CiTt_p7MktWHf~V@rS(~JDoQF|(tz79T>(@V7ECB5z4GBd zQ=or>HQ>XO3pI%rEY~1j0N9$88gKJ#Z11|!M87Z>4l7HhF&c6Bh;{YgpY^FoqigXSYBF56 zVe8pleT#h)RFUSH7v%Uj{9YZ)+)lYscEQYJ#SHx#kk?=ZV4eTuHSzqsk-P6AMhv)h zJ>(-lJdOw^zYCiz;*&G3m@0lRXv3skbk=WNp7^OX_`~^{o^hfl4a*BN<4f#lNl_Of zu128O<&5J)lR0wq-nLpYn*c73pWcWmQg~&wJhj3Vpr$B(1(c*Ocsn`HqWAw$r1~V* zj=zV0Qm*D>3zan|U1B@;eahyRR9}3m#aw#V85Lo#zGO+`0Lj8!Vo|xY0rRPLI{jn9 zt*mvgMV_d{mWeY{V&t^tLoH1cf7#( zL!0ZYp;>(!QRhGKpj+CmK!dAOqH;`Mw_&} zl`jkbY*R^2rnJIGyYHh+K|4-0>j6=+$%%Vk?q$S9F@=ABe?~U||L8 zPhZSog97&?WYV#mVdgyHaTi7Xd_BTXY;4}l=A~c%Ued})k@|pjy%Wrv_&WZ3dK_T0 zdH=D}rNl0o>~qqn<@D5qC9#Ve-5EvFJZL6uekATwJNa`lAjnv7MmCC%Y44=`ex?L4 zmZN7x0g(K9k%1Jua#r==m*mP`p*bYXcC(c6~l!SBzJ$1RPHT)6O7rH6K#YS&B$ z?KbZCh9`T*N)g|m{He!pDtBx=+u%fKzVu~6w?nAgKGn~0AZ0D>m!4#4K&njI#bvpG zciPd9OBBpG$147}MxMb9<=UIZ>owO#WF8I$l>uoY7^TA*xv>c<`P5if6KK@yqtGveVh!H~ z5yiZPlJn7z>>v1C=`YyRVm}P8L%wp^Z;7CJGU0f@4i_ID{3!vr&}po)Gi9=OR8t=YGA`JFs8E>HD@OANIAeM;7lqbU}+}x0M_HL37f{T`OqL z2i?40I2XBidHJgO`71&k`Ev;1Zh}TyvuwdMUAi*DMM#Y**z}w>Hi(z|elzy)N}B>` zGoNv~UiHtn#|;eyx}j45rWI)gIt0bvTr9UKD_msPPiitJSCIjBnXbi`M)VQwImORSLH*fSl)YZv2n3np^*38rh7(U2a%G-i z(cI6YL0~I$-M4cWw?--fcNtSt7VS+BX4an0{xNOW#_~uWW*tlA=>Y8LdkMM2$Tej{&DVo?}$!_qOIxp0E~nGkQl25ZEqHs6ZmY5?FQtXV9a&KYrKJpq~h+{8JrI z@lAx>lPN@R48<5kay)=ePX~p zN8>xDMUXz=fe_Bh)@7+Q4>VGph2|Z^Frkaa8eH_=v=QBPYUDtt_<$lPmF9&`zMS4fhI8U|UNR7u(?lLfV%?}s1F5{WZ&;97_uqrS= z|GUD;<(58ubzavOY#j{00Qh~etpSPHYhq`^(=sVz6_`pwUq;mYwVN>w9h({cnc^); zQj37W=lA~%ECtyLTpRihPZbd|$wLi2clc{@1kih1Ei16gXD#&vZ}8j#cprVv5l}^+ zmUi}kZ{7L3`Du7KGug5B_H04 zxXVHbleZ00Ikt0Su7a1vODBhYt**L}yBV-H*#I3^N!}nfG7gI=TMn{K;(I*+v|UaX za#nrNchL4Z%`WNJg^NG>me*T9Xr+ph(u&Sen)9L@JE<)^b!zVY*Rs#=4Ss$4_ps7` zvK~Qbd!_vQi;=WXYrSKdc(q}p^F)o!u#)4$7op$ZzjG~;Edrbmm&ExwQDQIeG6;8q zh_uXWKj3E9omy8GHG-TL(l?wQg;u+EpGZ8{;Mi&wdI;jnk=VM1oWNFM5HO*=)5`6F zcV9o#PV==%7X(qTO!KDB36x|N&$Y7=_3a;1L}bG`Gx_7c!3A1H&LL~u2eK+7e=I=w zzn$G%DVLKoc~faB7k5LEC2;m({wtWxrIw@q*HGJ%Pn|OF)B`nUxnt|)RfvDYdrH-B z@X9X|xzAr?IaXjk7tYgTu6HngNkY&0JmV|iRGy#PnR1Wf?L_X$9V=%UO{|FC=w_ir z>rZIQ>z^o%Ttw>SUU1yq2dXW?MEL2kQ4tuaVzDi1d*up`xze4s+oJ-fvAGhRY-jBqPYRa;d(|0@-r zqlN{7rhejEHapd!(tD!8PVRr-Jvy&#d`*I zOSA$>vQ#|$dVloKw|1v`ul_=ROT}N!VuG93m9Na^aJY{}ZN9d(hS^wuZC!6yIBTMcwKN5YqocYHF; zmfe9c_x?-g%8Z_KC^;8|aYbl-Gjt*sI+4=nveyJRjp{^{FF3UVYMr21eS$Pc97uoi zLc!L!ipZBe5P!_+=cRO_6;bxen&85UMv&5--yeU21;~uyAt+IMS37_7ixA*3OIM7R zk`u$e7|zu*LIXV)PB?w<6-9WA+Q{t#cf|fzbd?pjBYird3m=|Ddm8jT8>_$Y*y8f+ zI`d75(H_VVn2aIk*eH-xU>5P7FG=qff_i@r&@=P=w-(^{wD-+)$E({AQNu1V4$B4V z#u6QU(1?2$%&_hv82U0VLq^H*$?P9@cv-#m{%wnYGjkMC@ud}qd+ws-UC z$^50SIfayomtP!#skZ$!}rrF5M^Kngwt1zp%#s;susmD79$Vfr27spj=Sx^XHh9?Y^zclM~VWvOIE zGSWXUG%eM(WED?O>QQu?p1+^oz}Fd2cHPOPH-=PGl&I4-Y4a=-*x|;@>wfjD$P@Go z&xY)60Szfd3EIj#BET<4gWGbe>x|U@b0Ew^!%#y86YLXf`nSSbB|48s>u}b^#%i(|)f>Tz@mc zqI{zEN=jhjuHW3u%Nwjofnp&(pQqfj^RIq`D=OSwd#(%oz?}OXI~t5;=AE~em;GM< z_*<^k9m}hYfg6@Rdh&}$7j+^Jeoo54G!&u$nIgz9Z^KQK(#S{ioGZc3DuTOyetEsp zuE80XA0H`c_6KFTKQ}+5Ns}8@2nJ)TDJ9X1fpDh%;@ahyEkTU8W$_}l1t zioKxp>hCeF^qUw%6Nb+~$S(n|6Em9^eLw9z*jvE>OncLioTS91Q_gxL#hO=_h96*c zJ(6<$f#QGV*t?iCm$k0UCIae?#)&&0%KRw13{|&UZX;EE)XVnd8F-E2mfCo-#_nL* zhsycwl+$jCoC_mjOiZd(GbRik$=|kZbcIWKT2D#l8-mglnn(7j@7Oe!ir>jQk)9^b zbj453&WV_8|1_o(wtBbx1(*=Ye3)cr&hzDwlHnoK)o*-TeQqLS;jdMqImL9Wotk_7 zZiJ3uXayL{X);+n3uvb)fCveakp*j76Ex8knMd+`b#Cj@@1K({pWi}cIBSt z%4eMo!ljN0l(g4eBolH zl2p}FQ%((&2*BOlHzeMg@v>?%h%&VOUM@+Vr%6(`#j%f`|7q`@O@B{ASQNH&V$X#c z5D=ytrQpCMCmw^674{~b{f!>Eoh(LyYCc^qhq>_CK(E97-+b(7Cy5^}IZv?4?Z}0n zT-x(=WDt?q)LQ2536A#UXNK_=K|C|cKfNx7)+OaQEU^AFuF1uexW#9mKEdVMXlwgz z?ygscz7KELk%5#Bpn@FjYOufW0| z?!83fV*e0kb<#O%B-p1VmW9S3jL~wDS3Gf3oTAzmq&q~2`KP}!JT08gpS3iZ%G$Wl zgQn7+X5?BXZrU7{9D9rAWffy+^Mg(QdKdBG=#Wgkv8Op`*u5rtDqEj_lHNBMWFT@? zWk09QV@y5vxa6oshWJ@KCh06XWfAm^M}Ei5L;0b<(p1(jaj4X+xo*sNlYCP0PAhiP z62``5gBD9BL^9o5&cPu=4*pXRn^NAiRptir4N(S5XO^W?|C9_dJunWTF@UA*7N)?| zB=sFQ$E-s#uDAy1+LZhqM^O@A#cmradd`JSUhvvn8ak@d>S7dg?)k$jt}sQuPht9; zPGj;vI`#99<-5`(_TyW(9OM~p!{eysIp@X{@HTvi@{a5bLrP)*oLw}z4mTe#?MZ0rJm`G9l#>=|O zY_6cbc?(|it6S{DqcH!iE=F$j3xU{j!Vwz{bh#z)rdBo93YhaI*XM$t{5Bl>vJKd{ zu3PiEm^@4ZU_4l`5U(@`nx?yRrm6Hzm3bkbU(*3ja3to!)0l%zC8})1@@}Cs6{F~- z;Wv)nQ?M*UBBJMBS%lYftm_e<4HW0SYQ+B6U3!KZeM=gVe z4KwrWzj5S`cS6veHfardX9D(A($W#)7DvxTrm(~+>d~c)DecuM#8l%rd|T3EaUfeL z^7nQBsa`4Di?-D`x!;#Q{N}Jb1O15^U-_PfQnm$ zJV3DBTl-o+O9S*XePP%R$wz!=wcYzU0Gn&y^aVOoZp58_QxxKX-n;C}t^f#c70;Dz zLzLtr*7=}r1mh4WCUpp9Dek)hYQ@)QA69;Bg1ex<$p!WsI{WSSZnwAP7e$F7itWjC zFc!>qX{{7*;LL+7$KFgjkxhe4FVY#-{>wYh>HPmnUGYm@N3pW zmc!2Bs=N1vlqfkOo0hnvvUW!@wSJ_x)o1c6V}vjBw^DMSe0>Q&8I?FkoKMJ*DRr)_aH@CM{Hkqr{>rLi#dFIc zae%R!Q(JZ_SeY2;jqNPWR)M1_cvP-fUwU`Mk@Iv$1=F5}~5k{|8wF7X;E#)PFg zWe$;xQJV@0!;iw&Hfw5-?){0nZ2yhs8~pT2Xiyh%gNLII4G3tEnR*T;7%V=>9Y50Q;lfT$``GZ)KjmR12x~uW9Q+iM$}JJ}%UOCo!+05h zor`H2Nd_J|;z51B?bCsl@+#ID?L=$1We!QI!0Cg&D zK<#(|=u+R4pp?l>=Zoxz?blfX^z0zgJ`z-7`V~YIklR9RAhe9_#X{}9(+p1}U*B_t z!lt!A^|sY*#1WU!*Jc9ocv>VcZj^a_3(iOe@TwT>IR#Z=n%c$O{kBpdK0v0CvgJj5 zYg_MRUq=D5P7W=4R!ncW6nfvF^q&LxW*H#lU6&$Of3TOIU9B}3KEuQsM60Ni1#%)i{u z9DGUo#Zrk>*(b03WL+bk${ z&B-==n+>7yLFXPFkb{0Z0KtTo zjQC_Nw71+s^vrR&ty>Se%| zT9nd&2e?hcDnAS*q~aU^vJm(42itGzcEx^j<*%TkQ^GD@LCn9ka}mgm7(UZFSXh=J z&2tj0RkH8H<_3R&txZ$8#5gGWHXWck@u0uJ>FJe6`z~ha zpQM@cQS0(ICv7{QME@xUY^AJDB3l<4=&-JQ)<;m1dQ2%C!US=e!Vx&5K?YT>OA+lc zAW-pvC5taD&{qiS7xqS*>2Ai?L?(in7t%fi4w(7srjbOnM+&#Zw^@dY@Gqe9J6N2u z#M?vMuC0G${9?|T{Q!uO?`XBlzR>6id+#E4l@1HO-;V@x4{iWfdU^YeIdtw-=YuP? z37I>9XcNpe>SG7ORviu8uD$Ry|4vh59D_UG(iq^V>xLbtV;RmCP1w|AbF4FK8c?x!c!#Af~xIC zVNJihYRmF|aR(YzjC+~1*2D%x6K~*sVI-ZL%e84uK6eXuvWkaWq|gR`)aTcNPi#F`f33ck#ztn zuk~;izrm@Bygz_x^=T>)i=2&~v39T!zk?aHN+|?H}RnIf}b`P9I%;Ws8dB*#o7_n*7^5C{> zcfuxg0YsK0kEGjeI`ryUWIPr7>f3CO?@ zkg0i$TD<=nkP7lSBY?2C479489okU@DSW{4k3C_)1Iv~qYd1D=12O{ru{ZtLu0GAD zqI4w7X?)+_)sPfQkR&5_@2?%v~x4 zTgkiusGP!|bXO9-ssOE9`ytLT-&)Y2E*=^zwc0R|RT6%HlLH|GXkG z>TU)&Z?~BJA4a1rxa)ol?n;9>=O#2>B>}G{0K6K?^Q5{S*OL+o+?yMR;f`-ZW*>{T zRp3iwoH$CFb!{RT+-G8+1xH(wR;JR$G2GCADaqJ`e1j*Aukp=cYqnAsbCh1(2b@LW zO#uHgQE8k)P=#G7e9+7De*&5y2(~GcOd%EJn{dQvyu(eNquX`X)wvh@zt1s#$(S?; zHCTPPBlM`Laj6FYJk2q_jRc2`*oi%xXv5%s!qLUJ<8eZhaDE}7F z;1hAt?;tKbx_=xz&$*6240LSgXP4MYlT*_x8=$sK@ai->TVZ(hFS5yUUzC~zAv%v1 zqVG(<_=;Bz-@rW${WA69dWjJh7W0)Y)0o3Id{-w%rCKE3`(V{Ztl znF5oFgulAZdOQrITze6SEHk^9vhCwu8R`m)0xH-k#mW6>uq6Hj5mU~cDG=W^U%+H_ zs*EMQZ!_uBrL}IsvLmocF{of?I$n?4rQ4O+aL#!^!2vg^`-!xQ%cY3u-I$R zWp-ed0;i>p-&RjR6;V81;I*6P9ehn2-!Ig+I z`~cShk^lfgT&53M4CI_L4&*1Smf&w-Z;Qsl`q~N3UM)P5D-=@mY7=z7(7H2n zl~Zt7*cMW!5OG3hV9#k58iIOZId$D6#s=ya;{7$3o&(ywDOB<#M!k1a7G-@D} zcB0<9&S!<;c6{C?5va0fS?b#njCm-F|6y_|p(t@%|3T9@{JB(7>G}TEWPiX9r6inH zHXfAR^6XQp#BM7^oRd}DQB6T@Hx@TJH|p>;vcFtO`t&l$%_{~Ufk%sI_(-ZS;hpST zd3vS_`6i&+`d;={bu1ICPio{ z`}Qm?naa`tvXX!4x7?3x6I46wO{lHNit{f^J?r~*stNHhE&rKff+lBwU?DzJ{H9xV zmrkz%mxAQHc&)(1**aMsR;~*narEqx^^PZnGIS;dWwY_BnVII7>~?xOS!sDbGzS)l?GiNi%lEmhMruXXYP|5)CTg_3e)Wr`8a_k zj?MT#LVVY1P}#r9W5=+8Kv?aAq@p+qHEd1O#0zxOAWf+gZkM_fS%#e;W?WgSDl2Tl zb@um{#n+?+GR6ImEcEV`ML$j+1Lu7%w=lOSY}`Ltgk`m_+gRwj%~|hu2L2jv)b?{7 z$t@a5`Pju7bTLh4?fOrBG~G9Mf?QMcCX6K&(5W9q>)j4Rq8A^9o$mOh3e7oZdUHWg zvs3vJ4FE0H{jOF2!~0>nH>tKFMPKqwZ7*NNcuHl20)ru1GBqVqxNF_KqbiIw#dLr@ zJ-QoRX-aT|srJH;4>kk0>Pd5}lyG%*tq!eTYl9tY9PJp5U)z@pQiyq=jPMOhe#6K7 zTId@2mS2I`ZagIX>?1vQI}W0!5DhZ2-~YOP{Kp(`#9zQlf9EY&3{)8nI+KG`naNpO zA&Jp1r~vuyEZ2XM5nO2 zz0RO_Mwq!Aj5e6mLg#y|vX|+q<0ILVm-=eOb3+rUiPAhc+E_vt7|I^ngt4vN$cO@; z+XsvrteJ8R<2L=r`BIiFOYA_cA#FlGZGNrN3n!K4p8eUwt!MJINz@PikAL9Oe9W4I zEMZfq2**?!8-}-xzvqwt3XbNfMB$BEPSx=Jc;}b@_0p<#r)VLcuFB~6L;m=`Qu;;O zs~-@X=<&KTAU_!@u3G#+qj=E*^yg&rnD1n@XPl)CrU%o2V{($|8gf|G8<;Q&e;4O` zmleqmz&Y>rDJAYq+?5WbRV7aDfBAkJB2K4GOea*8#%#>IpryBsr@*v?q9WtznqJYW z7lhA-*9ncFo^XPE^~S zI~n>Zv}u=eCD+-nCQ5|9hUe=4&N1K`Y8pMI=OjlRaxd5xqaAgm0608S68uErhXW{4 z(f~NCAun_q)aXB2`ret&{NGG0)f@6V|v40Px#%3gJ@>e{0cEM@nx7%N*-;CGUzX2Y#uQYtdw-VH) zL~XNv{PXqxD~>x&H0tnvz%=B3^q+7t$YHK&1sN8S`xDmc!s*L2!a_*wwF_lM%VDYG2M-NRYoDedC-Dtj*D**7 zxZoCB(F7y@VY{-%Zkbc4RV6G9*NQ5#aNvEHRxJ7e?&dAq7QZt`?Lv*WEngs0`B41r zuva?R&j>K4K0Laa#Qku3_7rOcj8Kmsi!aJ&GP-zx6Z*ltamEWe&;`}|iyyf`PMSVM zO9%vVR5wkPLLz1C9yxiqf3cTO2*tiOl3vY46w?m0B9n6?pSX>fHhh!`R z9fXQQ7w?l;@L?L-%f3^YF>`*r#{8$4)pCKMOVA4MKdwAA#QFb(sB6 z%~w$c??0b6h-u}H5WSTay>Aj*yei%m2C>mA^df_Z@7j+j1GLFiP4Ho^_~eYmJ$T=9 zH9%L0@4%;>|Dvb#{yT9P#zA(@06YTe)i8>|AaqQrCpRv*#I9b4Y&LQ_m;hC4RTXQ{ChA(BzR(ex#9q&%Pb~rkh~Jd^`U2~6!Vq3! z!Q&mayGs)7SJ=$zkueCB*>av>=tea-zfG6+^CW&p`fPirNzGE(`86Tj^lM8EX?PnffWq;qFyqUg4 z!N%#!d*>Du2U8TfLHw&$iT%uRtsUo^tYF)@MbUsko4?O9N_^*X4mm;1+%Uit$OYBv zOi>0qvj#niOH+HAw4=P>JY zmZVodjX?f^CGr-sQ0b7) zd>z5$%cEyL>T(T(RzApTP3KDPsc1TZ4rTBImeCd;H)WxF0p2P9kR6}k8u|=VOd&{k zc^4U53?Y?vEwvTu^)c_uAqCf=EgVYc0FrT~Wso=UbEiP2Qn6hUODXk1Upo{8ROEd4 zaDN!fcViG+7lJ;W?K)_eecrwT7jxRs^Ckfp*Hr zTr5)f7h$7R`t`SBvBInaLp&uW2D?-5-MyC;g$F7lAkkAAb%fH<$@S=KT_$ zo$@QBN=y+MnH=eMOFzr>ARcpFIQr8T_{TR~-tt7t5VNc1$8(kRN&+grylEw+1GYKg zgGJ;lcyy>Jd*@=EqH>)~NWs|bzX8EPFL7^p3Sx|mliQPPe1+GyG?Ab(}GvHPqB|Za91Zg6!ZEXqh zTgDfBkiNr1?_EM_ao{`48u!Bb;V+fcH6Zs3CW$*K%O4oc<3Uh{q^aLbXwZu?p_}NX zq5EM!**FxQ^YDY6%(>Dhj8#IzEO1fkMT;}*odEM7p?h!_q~UC4tTG3}CEotH`F@Gh z5^q(XU^_7S?Nt)gAi-F(*P#e5lXnh7<^#$*@mg!F&B{?GuPM3>G4p~sa?I=88^c#2 zS11aC^P7{Stq>c?Cs}Mla7dZh)Aw~i=;PwP^G^wwD@s9VWx6HK@xnrGYmYw|XALgG zE9$fTHbSMvM=*w5n#B4|?wPdTY4R&dp^tk~8UmJTEdaNWJ1@*4TqQiBpMW(od>v2& zD#YV3ccW!|@vJ@_-Ft=or5HA>q*>{q#%j4Z zc=``;W=bTzr56@W_reQ!8u_5Qaf$c?d&_xhQk+3od<%#l!@7UHx@LF3hgKb{WYHTE zlh(Qx-%2w5P(*GWcI3m4$c#$ol9a>V4A|X|%8uAg`a~f5#lQLjKoealhM;O8F(h%+ z?rKY@FN7dQ?tHHZ8bX346x7yXXI2YC!Hj|1|+2!{n>mcB$eU$=brB^4BxM))S(` z#l#YG*r1FNzdoL+UIy-hX-d-;?@HT+y3$oNWx<*6Lv$&qBhrzq;VE~EmGhE3~)NIQ`B9Ts&M~79@H9+ z$W@{OMh&BxKwHqHbL^}Hx$=t6GYA!{+M%eN>DK6f&eokZF#GR(jRY~k%IDf9qO7&N zgNlm90annL1_u%zj3V}SReJ6{?V*=xz9UhGlbL?$aw2s5j*u%PCa|!Y1;?`$37>vd z-UV!L8yX}yQ83T%TUH3(d#f;lt@O?42(rd1Qte^*6oc!1Nkg^;S*hx7ZyVC8Zmut| z>vMiX@fYcpXRmS?=Z9I9qK?F{hd%yymvk6X1$p`3*+fmA(yc$!6H))WSsrderh^rk z;{D*ky-@^WN8cA@brJ8kH8G`A@U6%1nH}ao&WQoVd@RH~bJo{19ni zDkz{6TkkD4Y7c%h!7t|ARA35-FBSKTt+1sO#7UEe1d$8z)WhpRdDM=q{!PM_j)62m- z81bGdPgqH7wM%3rIT=c)$PdPDIV~HVE`mNK+>b^1DHtr2+gn;PlVsL^rb~EyK%L<+ z4CTw5xUUU!eO*0Jol$Mr64Jfo*?J#yG%MUl5ap2{DET}RV$H`@*Bm4<@feU(O>;fm zQ@&b;h>tYPKeX9Ck&8vQk*I5FQL8NSbqb;f_NBz*TGVDYi3l0-g{l>Q^^EI0knoe}&|IwatYd^K~^&?~fUkDzd!c8h;<@N>yx zo^q1TYvmoLrG})CYg|4EI_jl{rNKxOOV*rS&P6-cw1@f6FBDwz^%xHfjW*if_Jp@uY~}>z>(5`*%$yoszY$?GFBJs=Lhk z5qSaXsPlQ`t`=ynyA0*lQ?{|aO?cZL>@7o`;EChMKb|AEr(Q0&DX}l*l%;)FD>tY* z&91(#^QU)HMe=+lbO`tag*SZWu%uN3$f_gWDn*&3qCKq57`b!#pv2Fx&i)4J-JLTZ z&L$VXa4W}}ce|WM$0e1vQCrsvJ^ZU%`%fB&MZV7M$6bKG!+22p;AQUDt^=@cm7i4^@NR{|Lt!R4@J2Aev$J3ROAcs~_=59NO7HN3@tS zT%DzK2(M$$XI&F;?#BM`#f`jkHTTjOiUp?1i9sU9QDm8Z$QsFnD zxOfu^Zx0IPbpBI!3e~!c!cmuik|&;TFZRqw`K0$PmNKMdQ*3;j{`=`fH<-q;8Q~lQ zUkU-%!U+_^HE56+kZKn@uRQ~=889t;{yI;UU(Q4H_;`^BK4(R_k zLkIGI31~(9I6{uj(j9ySmj{p(7HxiPe*^lQ~@tF?SGZ}aA+1FA8@VKAF_Wmx19ch zMoO59PiGc02zI+Z-ZePVX28-yN3N_*Pcy(8C1v{&igVU*q={7sCL*#!qnGGBwBJTy z&n{Lvf5%oHWH;TZp}B^*O#E@u*)@E?ha*Q^u{lPJ`Uv5UUjy-l#U|lZJ8lViITjYB zmQz4(nIqNevlcig<_g`l??2gBAaDPNRHN>^`bhCmO7o zY~MF{yVCrI;AV7fCt>hL)8|q=^h{~LxoeQJRt+l_k1 zlC96*F3^RtneRlH#W9F$m2^veiTB)$@agsy15>EfKr+SL#-$ywI-V2A{vKr-L6s=R zHEvvDbPf_?>?Z_9+a%S7T^B0hZo9c>@sc?!>!jZFp%MEChmbq_y%*noUz69_kV$t3 z_h6Nyr`myjS8il}a|1)lyUyjCaD7qq!j{C;>CZdQkXa%0H~WOxwo|M9Q{arT7w42U z%n6PVq6;YE-6p-+mRW)^M#?LqjW-N1vT!VQKq&z53T^h>)O!`%Z6z4@;O#Zj3j6b# zM@{aw7k5n$Ad!ytRrs^Sr|z#ctlVtA_Prp4N_O^ckUfv3m z_STd5V3eECn^l5nDiKN%91)i8Ve7we+{80?drC4u5#dnLC9|R~wuDo|R?dAf^+j?05PBD-V1x){6Dx35z=dxjkRRm6xHu^o`tg&?Ej3m20Da zQZb!C5-pX4IOtk9z6)yt!;IB1{tnjB75$*zF_HrHN;EkfA0L@l*Gj75nXxRqA@!%a zpzbSs;$3O}lKpoq6)78CIrC-|N=~wf{f-O7(N#WvoELD)i5!M^!MHR-VT9zk$H@F` z7SPxjh}pAWZ2bVIDs|P1g#?AtmI~5`dkUVc2QbQ6q+cKp74tLt)LT4~(>GqSZ2@E^ zgtjChpZ=w5FP3c&GK?}SEDL((Zsk4OGdlOsnq%{Y!+*-{VO7FoRqrcJrfx&GtCoGl zS2u81ABP_Dx9Mq9!P5R=e>dPSSpC2iDZR^BPvz8u%BXl{F7C`;B(-8cA5U1~fugvu zFa--P>UKrD`|h-wh*;n(R;5-YS;dd41d5zzaNgg0zf2^$Xa<4I-Twe^4u46}W42{$ zW{u(%6j;aZg)jTN7=(mxLTd3QWM+ph7Z94?I$D6h>UA{J{NK`KW9-kXASU;*!VqOX zUY%sh3m(GH4VLMxNL~?fWy2?;7oqG7QH-~%c=EZLbS^5}&?4o_YnNsd1YLqI7ml6m zB4^JkiDLB;59Vhj^)7piS*^N zsV5%Z@xW%>`?!nyEYIVGh`q(Wg_jMsK5ni1$5hT-B8XygAu{Yd?liE-@?&*$?bKJ; zf>V%tts}YgT~cFyPdp)CQezVx@3wTz0k#1`l8=@{58;{FqPl(5SzfeU1$IUhlLKSm zIgpsvGrv^oR>xNCxqgp0Bxu9k7jiHHb+`I2$)Rw1|JGOci-{!1)|^PCoLTpRDlKfo zw#f>f0*xYvRQ|G~z{xn+tR3txlFY-DBckIzF35~ddZF0!O^vw+p}e}1RImTa-6#*R!{#CkDUi7mIC z>mdW-=d4P)ilV9u`H;#L^A!2n%!_fWi9)a?Gd&6`3vlWRZYv z>*Yd`^oe;J|JRt$1WT;FQA!w&g3IR@9Tw382}D!KY)nOM)RQii(MXQHQ(Lr+FM;wt z-dHo3SBE4ITiDA)He-lh3|*V%Ba7RvBxY$ud3WFy6ARI!>csgmi^50cBYnQL_hZpo z?N=~u4P|g=+|5f$Ba9?S)tROriAyq3S11RWn9HTPdd01;fxB>eEV=96X%Vi-t} z)#~nx73(Dq-@rXP=3JXUO!V*WZ=-u3v8~m_>XjMsQfn<%wBVVIbwBM1h$6O>8E(WO z6uDY$&EhL{_=9+BHYz*mZ`5YUA>lN4z{=2yB)c}b>Ioj1nZprd{~4PJeYRW4mzMY- zS|Q33H}?mR7Vpdy)s>%JfbdNE*^d#W!xm1<+Lai}L>JK6h@U3yVGgH>(4z|qsdOhmne-3n!+ZT`*=v=2jxoT^9aOI7%5zEMk)rhfak8{=M5TZ9`%sL>K)n7VrA zrbY6sjDHfAe?epw)WD#GMK6%5E;0|t$YI2oCt}{Fuph#N;v2=B5y-%&#|yi8cg8^~ zH^OeVNSfc9;d{_`88+6jc;W6X1#*^A6z3RoM{DS9s=a-S&j&JWOXS7Bsq+?o!aY6# z9*T=0sdS{!D~qAv7xF=<;>z{AOk?i^rZLOp7QIks-ahSFj*Q(aZtmeV)jk$~g7#!_ zqHb6o_XtN~rw%3iw0mSIl`BvKu@Fzl%w#s+<)-|O&-3vFE4MX$1>Bgp43DJk3+2>p z^Eo$z9;ttWw00qBf{_HPl%!7e3UM~wt_#NjHlH`pRFQx!kmT8}a0^oJAB< zfZ6*J(~oUZpGc+oyO1Qvp<;I&r!N#lZ5fH(E{$v~2IsB+M_w4P8~kRG-3|2W#Z8>`pH)c;FtIgd$kDJqoI4yFb0=)+c zghbX*xoJ=2v&G%=L-4HahwYgf?(+vE z)@~38b%Sj`VlZk^&p#m2RL%MZbJWIF6}a0#(^-ygc-D8441z-JY-Z%kudV6* zOKI?==vf`}a#Uv{O*D>LupLZX%-25j^moTti*WF}p(kvHXWCE~3x~}{eaE(C?(JsS zG+>5mq*b@_k9PWP5Q$tfjn-uwW|3YY#kQEPSb0m+F@x%i#66LN%yOJX%g@f1RSa$} zN^Jgp6!!+R-#s$GT_MVlOasl)a$U&POj=!F#GG~yvfkEFUv(@3#;?8^MT zstFf&rFX^b^P??M>pKrcZ0xw)Wag8m{g&eB`Ah7e0vf7|RmNFQ$9u(@h@5AC1Ct5e z+a2$M#mrQQy5m#na0iGHT4YFhoU){JWhUg?`(H*Bi=0-d`3meVnXs5`EV2hCrDd^1 zDV{_M+{61B^nI1p+;~pF-{b((ME6W7eB0%FY?@kqiVm3CPS3S2gT=}(m90ym9)wdG zX*AG*M{GxW9{WD-DlITbh%o%kN!fezt+~CYTfw-O^XlJiT}LVP6YF1jHh<^0siH_L z)MebBU^wBMB(W&*k5#*eeE;#=g0~Kj`A3GyKBnUbKWB2LUA9@5hB&GBU2g2_=^S-Q zs0T(PQiP}zu_vF~J_lb7OWMuMi<54D405zbC6yx&^AiKywg<9nvTt;$@(r)v&2|~z zW(mU~eU2u3#z&My-pV4CaOYc4@k`u}V%)_7^C$KC3PD0w)X!Zz6DF!E<-Qsczbed& zK--uST96ADJ@LwO;*78P^pP{j*SBSLb-Jw_o(=wBbenV&~-aP z-rwx`MvOgPJVTW*X*od3cDrOHu5W12KHCbyXTs9bT$QeHEjF;MiDbeh%i>$>B5!@b zc^O-~{dqUysxPs4Vrz)P-*|mu?ndRfP0v4IT8zR6# zT5>|rc^{ESNvHEWUKvkzW_EdTDylHQ8uN*;vd0;V6E(Bn00h+^o65JbU&8Orz1;1* z`Yz&eq_>=reV(%GHC?r*{Ni0479FgL6dx(H9~k82D9>7)SQyt$^BM`A1;ZVf)w0dq z{4$oRu@uqrW?hA|E%xGG?_-rF!AK^%XolqJqQFQp@mtFGsp7U*|9X6QLE+c>=W=Pl z>PwWxNlD;n(qd$Z)9F=W7J~!Z^66%B>GJq1wvr}yR=&xt;Eu7U-)P*OwTjec%_6L% zifY3fXS5*=^;nAS8!9qfZu)usxuL@GM?w1n$I2Q~FGay2ok%9-ioR!2pL;$824mxq zxBnuH4*ND)O3_hWapKC{Ti(T&RI1yHRsH>bvD9dj1)K3u1WSptv;QxUAzY_4>esIVP{P;@DVNO1ENC>H&mF zR@!mT7XS7oxpsPb#eVVAk>@f5Xo(cZ&nbjZDd*Tm`W(C)ENBwP_mmXzuy*MyN#bww z>1))63R|Qq#?6xaeORj9hT5~xKRF*~u@0Z*VRC-(H+5aEqHLitMukbbevPQ#Lw0a! z^W;HQBl-Q$n?pBao1<;gRK&5;jx&!Lj%bxzE{Ivq8B}q8SY^JpYOznmiy&aZk-F>( z`XwNn8RPxxJmvm~ng_|zd%5Kw7-qGfJ#0DAY=l-KXq=`=aYb2V8r(}iQ9)IhnS*!C z{7OOeZO6!Ln!dJuUJEpozn`{!_oxcTk{jdCWd zjjpj?n@kc>^dG=rzn{l;3QBhT+C|L~muML;`E>6HM|s7mWy0BWrc|5UIMSrb*L*4` zjGoI|g!?2YUQU8S|KvBpv`4f5AC!=z$3Xb5R?&+Dd?V)!&86o<_<9T(S7rZ{c=6Su zsHJ~xDiReF-ay&UCl`O(P`*OW`JE25Q1l|Q=cp&nQwB$H!lNyrZ`gCTyc95?{d7nh zY5i}VIedK;|Ltb;47?oZsxhIs-(d3v`%Dnrz)UnY$%VRw0oQqhH)HPnMSN_nDws}Pn2GE zxDDTYiZ3MDKvgtp`A|^{4wpWj0=Z{!{OfO(x*14dl{`wk*rPdw{PZ{1_>TjwjSWb3 z2Oo6A;byj*8Pzug%u?dDaOra$_JxMY$l%=>N&Ws65Iwh*TL@{_8XaaU-iC3-|BJiI zZm}QPw&Nnl0`b|pRfc^w$<_foy#;eVPPHiXT~m?SIoaYx+S6A zi}TJHq*kwRztoK4E{bj^?TAgl_oR&d{7PN4XL|n?8h1~vprh|B1!h#rt;nBt6ZGlF zOTUb42uzi{S;?+0t+9|}!qHY!#m3?>F6Wj>Xg0l&w)Z!; zEMBY7u`=>jv%yd&gKf+M`jfYNs2SDo1p@p&%OPP_WtRCP9`U;G&YHk1&orS?TS($v zm_Ok4*rWCQ4t5z;@1xyhyP&^bZy#fS{;AfVW<(iK_D&j>hg|887qRNh-~Cy~3GMN= zeeq{pD_jp;K4g7T=ksGD-B|SQ8?3!E)0aclW|y)Z?ldelg<^Joxcb8160+K&pHDOW z^30?3Sh@V)sXVm&sLoeV;IVnYb(_kEa3D?SO{03vV^dO=kDQH0XBTk{G<@PwL5_>! z;`_WL1~aPtqZfgeW$R%BBu_q@JB1a}tFv7K9byyrO(2snmXhebRWwdt)(kPBDM*K_ z-g@^XwJgnf#6{4s%2hzrMEDvNn~bCO$KrU7chx4l#KAy0px+s|Hs%(g$5B51QZjMk zb*1b#-@dCg`VL|<789;FL`|v)p|(&ajNok%%u!YS&QEm39E9tIAZZ}2rk6`ID&8M$ z(Bxu+z1i!YyUQu8ZR@&GQ7mp(p?Qz|nDNEmC@eB?{aG`#y>t_Sn!Kpk0Y%&n7 z$!1bbefu{!bh&4FqT%gKjTyeVP;1A1`;lSoQ!0kfI!QHTktz&^WuKYzrGL!Js3S%h z?tJmgP@PkS)3p>|W7u`L+N z8=INPg5|z9tJ!OEyg3wFl=n0R{`9Sae~k2oNbeDwVnVoX z6W~N*^#hQBYAtQ65Foxf|)mwS*L$WaK9R>4qbD-G%<#*|<1jDA>sXFKYR<4!MalMooRR@+{ zvhz4at|$+NG6I}ZZa+EtAMs$p#Kg(B38Fq-@t1yIqqy@Yl9%a$hZdhEh$n&N?yDo9 z$ij*!`+~8k;r9n;y>6wnlO*YslqptAP0O6yn(Qg$<^kv3w+!9(RzP1de)Eu5`fI5qS!8j&; z;LvXf3L&4E0@Kn;9}-$&gfjN;N29^F{>7kbeG55)OiQzd_g;t*?V!$6-Xaeo^FX(RfBgQ+q zQ(*M>qS<};;1k}CsN%~OLESiTh53$D`5XcV8+>!u)NRcNyQc0P9)}DJk6_&Y3{O&( z1!*hzw{+Ckbc=NbHat(dzq6QScQLO+*ob6rJjMTTOwOVuU1PIW>IsPm$LrV6b{_4WPV(;FAg8 zL(+(Idj^+B_`!g^`vOw(Pn5h`UED-9hKej_$m{Rt#`KyDUbi8g_a)ma9ijhq!q{Vk z${BC{JJIKu!BDzMhvd@}dlRvI)!G+$FtBXG37QPg5xM%#vSkxEh6V@IZ-(zZ0@#Eo zPS(43{Et2)BX|*B{VJ z$NNFbV^~*m0n|LidrrxM3<){%II(IxlceB}#2)@ZuPb!7dIvK^A0F*7Lh;$0DSGE1 z2#o_t=ifpM@fJv1AALjuofY(dec*Q&)u*Wp;TiH=6JbNl{CSu3>F3VNq~69>1h=pnCevbQ&T?0!Fv1Z0U&=LGlq##6N6L5G#oBk)n z-&9riA7iI{XVa*6G)82yun}f`CziXdhb8wMxk3#kSOjrOnziH))aB1z8&HXIT9rGm zb`m^LA}fTv86tfhf_x19I9cI*w9GlO08XZqYq?0MDJ$o|ybwZuL`rjNQ8oJw|o|aOW-V zk%=K4AG7&L$x zg4yi^+jHJ`p3>KR7_7*H@8!PUzXNaI(O$l2UADB}Yw%rVTpVu|%)~EeJ;Fl7+VJuU zx&hI}y4H06!T%>)$MxTXRsN3bhitZZ4h+z3G&MqXPz?b%v}g+fV>MU@dDHk@eNDTO zKU^YfF%?;g@oF5G=1Fbl0GCv9TAam;(TIqajk5m)#Q|8kom**28!J&^rKEaT7}UA7 z;>UwGlm5@o>wT1~2U4eFxFLixxoG4u^(aw`gF^1^LcNnH$Xis3eG+M*G8VoG`mYDY zjE-%f`n1}MgAOF)4sg}8e8944G1n_4uql5zbPo-uSaYM}_6^+ta^d>yA@YjDh5UED z7F{9`z0;Ml!HTGiN{N@>GTnx6KwY@e7T5=~&r5&tVZ23r)K3Zoe?`Lzfjf2yB7s4W zaZB^+ydfUD1VIMp$7K-zWRvCM`nb~-&>tNGA0`7-i=PLxB=yv@P7lA_O%{9r4LE;! z?u@<-{We*Y1QAKV$;Wp0uxJ?>WGcl+Y1}@&$I`EG2$FFTAkbX@j~E!zpDJLN|L~0? zpI({$QT4XD?AkIsvgn_75LH>P_a<1dhU{r5enkwAQf?gipZj%k74WM;Jt8lDqjwa- zJSR|TWY~3B#B4g<^}j06N3=z16}_{c9mn3i1czXc%wNyFS=InB z`1R?Opj_hoo{&h7McKU;ad-+__Tvg)eQIe!+zMI#0lKxOlW|pS$?0Ci?04q5J3S-? z&AXoGwbTq0Eh6?|5Th!VHG}V!NFj8-5m*5)S1?Zj7&*t(HHy#Otmp*e{51r~m7qImcU773*c`xF>j|-{qu#PO0{W-vy2#ErRNx zg+l}l!Qcz)9XUq7*prX29r{p9EAjr+smum>qAH#-RKP!%4!lxcjES8p*8B1yb%W+pE6eEO+2l+XUTC%Bk}w0|gSVb3~>F4qinPysRa)U8^z6gT1CwlU9jujltW} zAVL;ygVJ9+B6)}IW18et%pQ}P)_|LP?(Z3flY?uOLPJt;JLO49Umzj0DbtVSbc0x) z2V)LVym&fKn`p_vd?pcF_B)hjeT)iZVIf7w3#QNYf|?B>nS4dJ5p;!;ld9NR0+MS! z=&6oqtw>|Yx{hFwM3KH}zZu!E$QG&|$N0cA33r|^bwv6ybw4|)3qEhMOdUX15V)Vym`q3?(7oM16>%XclU_KzJUyl0>jd4 z3EI?M9??Fl$08TC>Lt|%c#B&HHq9%jnC{i`rQ>pE?kSKIv(D&OT9c~r7gLW}2C|Ic zQQz4F&zg$D$zBs|DgnwPM6aCYYj_ViL&Wcbg=7f&+6dT%ir0o+SOv&zW=-qlR&rgU zLh+9~7Xb;5;F@8j1WZI4C26Ga7!aRdq5PG8#}HBvOoMHAgEWLb7gx5E&5}%d*K6@r z0}m?=o-SkJ#P|$?kA^)dMU(^^>w8#FBg0U3}!soUtu~p<7r&{C&x} z7*1bOHhvZAvFd-09?Q@$EYu@{8Ouu`)5;EU>Ecf2B} zoHa&!(^~1DKaP1AZi1`f;=j=rr`@#5P&>E$~LTa-3F%_(8&YbcLE~Vu6 zpMS~WEv46WbbA`&DmguBT#WzTHyMvZXb&=8GyHL|Oi4}R`G(?b6 zqVK^j82&8`d5eg{uZP?`uvFs0>J}KY{Cj!wE$Kd`0DtB8d5^U-t1U!~dej*d_+B+F zwGm>gSLO~3KI;j&H_Hrrr8#0}WjCjzwmIqSs^V_D_0F>%Xd~3ZDgY!v627E0FGwai ziYfJmh_8sn*9Nik_Eijv?QvFL5TX?@G~_Urn8nlcdm=(w;Fra#Sk$t(p?WJY z$Uc3x>GD_|E`>k^bvTMnbAFOgu(a&Q}ej$MGj(N2lBOeJRfE%QtciGp)~ zGOxv31rp-0l@Hy2jB0oO*&RN}as04+w@oj!Xq@HNuiJ4#wz|rpTtq6Nxmw9w+LU}m zTCA)?Yzm=VVH#bX#kZb}7QRT)$sNz(`DyxQ>*H{9*>3jbvft9T%;P0HeGgDK{PBMi zFnp0|Lo$->2r7=c-e5c`%{#d$8CF6PG~4GNqEyH>Vj_yp%f1z8^KZYsN&l}ZI+cgj z8D7+v$+9G+=seV8*Ga}nPR=jg13rxk5Q>eaG)67sV*R>KlC)|_ZJERVY2*skn*7~S zRcpJZ`rBmon-8k4-cuw{3jWQ*GCB}9{DBh_yYq@KZSPy{JQ)=NG~cM_1KJYeJ-D?_ zUzVxcD|`+AxBragOB5PFnC2iT6g>sJ!-Ea@~!pt8$#r2r4dI0fXRh>_A=aO>bNMhom@o?uwsfRn7Vc z`4k6s2=a5p+Mv-930orI2qw*0(a?AB}lYO!$GNUoV3TblgGrnOKB8H668@rDovLy)aqo|y2~r)vHLAu7b#O!y1b_P772&BrKo8EA zkQ?J;8NFsO6IN#q0xRKJNeIvFJAza^sjiAS=OF2&(k2J!N3zk9GCVhh(o4el0RuG2 z9j%KVY%{^dbA;Zt%Izlv67m*!J8IN%RgO|pf^+**5ZcCU{t-#7S7{Pn( z-FBFtT!O$24!OyKu-ghr`~{yLp{B55ol0;BGH8XrKS-rxU`YRrv(QAE4&wL$als2R z1_G(p*kz2+_@Pev7$K)O)D?GVu@oW`Yo`&f#xRElog25e3nA5;mBWr#=C(DA{?zBk zEe`!OIZpdnCqdIM_>13j4iaaxV&xNV{i7xxIv!XZwQYN z>fIs`&r=VIBurpYxdK-`=E+naXoc$*?Mmo`wE;?kjpGL(EEjME_3p8eN!jg87L|cm z^#D$UaXCMQ^$95N!B2#~*rmL%D5=VFFlt@LZu@+SEBjR)$z7rS;1M8pTsOyrlNXt9WW! zXecyFSnDVMcoBkJ`wHlRDQcMaHW3vp?zVaL%17AM6P4iZrFdaUO&a~c1t~z^Z+7`N z2@2Xldr5TqDs&q1Wu3S4BJKXd(Z?7x!Al5>q^neCB2T6PR?P5$H*c%%<IRJc(khfoS5jww#?RZ|) zC1~7Ziu9kxP~z3X2xv5m&UWLUgzSIhsm~UF(ryCBEP=@TT zZ>@M5ZjV{8iL?LvR4!dJ=#o@7!gOH_a$`qaBt|T=8Oc6?SR^*g9_eQbC$N8kaO&)7 z1hRAY4LyW8EQ|-Blb+jeZjejM-lJB5$Nn~1uXO`?gY?)KNzt%pIj+uPJ`rr%X7d6n zl0HEN*moM=#Y*s@QQI3)1UEslVAi2=BlQ$v=O6?=Hw6)JC?5bSzRrbe+a4x-B7B-Q zF0TPRI>lYO+x%+SDyphPr#Qq6+k(w|!l9W+_Al2+lgGYD*mYg09_~@t4cI2oE*ad^ zd<5jZ_jom~!`0YHCRAw!K{Dl1;U|Z(T4wM<9zaa(1~HX0e5WVq)2a(Xu-gxy;hg_fCjHT0ZY?|!6pn)b-Tgfv{|}zL!aRT z#1mO;30dDd*2Pc6lKi_K)N9z~jhl35>q!p{bjnpy_>i}(43K)CXLV|o3Br(pD_o=% zw8ZjVJ+jd$odc%I5(J{woA8#^J$MdAizbM&7jnaRlHs4H_rba=a@Q%tp)P9BBAMr0 zSA@DA=Fr#Ehk1tFqcJs?@{gr1dtQPsnuu1o3tb8<%yF`&0JB+y%mo#nVG`}97KKgT zQ!o;Hd>iK!_8idQ)lh;V3fD39;D!agfSCLqb}zvVKqHH~Z)-p|n=j(J8{lvieUa~I zHUS1AV>reyL6m6(DGfIB>KDlv;kq(8dQV6S15M$Bb{Nd{f-=h?r!08W?Q2&QPS8!8 zVfVX#&n;(%NU{R>uiK911n(2}^4U@GYFvS#jP4+ODldmzc~w&@2;h^}L$2Waa0O4B zE4WQ9lLtw{OBJ4QtG!+aQs&{#Ou5siNj|NdQRFe~qfeDbX8)2|r$Re|v&Lh;p`tMP zV3-JV{ZBhg#D3FSUHfl|?OiB~gcXWfT}k1^k>R?tOi&~H_tR!YH7R%$u0HemM~k3B7s5i!&hXNIlUu|4VQ7qmj#f!Rg}&Hi>Vcs;nb8VcYy3`MNle$+2hPeA;)u5MBIB6ykx=%HqwN>oAEaoEsbWYHnw1)wAI@ zz2iTe4vAnB(iX}wksWke&7S*%_CJrEC5`?P)jtb0%aReoGl_me|1;=C;TpSj-7lX8 zM!|p27e2~XsOQjN2`SdR^p&|c`Gf4H;UB3*;H1*i{=UQ zV;=2%e18j0EY6TO@B%)#Gi+u&+IB1L!3XgFJ%DrVYCzyvH)kDd=U6=?!kZ;(Gt7Xr zh;D>@a-O??J}$ES==Q*IH2xdsv?PfnEJA*IP$8cIz_BP;fc4;jowo1$l|G~FS+zxp zARD#?Jo_R^*n)j!s7vi8D=4YZPE=G>7GVuIsGGGQJcGoaJ(~ZwQ#X>654xp0Kuh~Z zEV0dEr`9HO2$6v+Y%^|PBE28YXXS~iLm$MH4)TOFk{5Ddk`cN&vt1GdI~L)qPq~we z?xsPSB>$-cdL*;_Y4d=jgabU0%*aEL+j6J>f0wW z<925D3Woq(bkQ^RNI_a%WlfYBOU?k6^QV0}Q~;Zy88SnU*7*|1=MnjWf$?F>aV#bi z{CXR^325iC;koQ%6H~KJHz0?GMoOTpAgg2SvypNDUJgYnrITtEACf%`@P zXK4PlQ@jLP*ui{Ad@aW=LCPQ#*B9ao5BUgUkNXTRVYZsE<%a!4GPF=P`m_YTBHb-W zS?ogywgFkKq^fc;jgiWl_3;jICv~xgLJWO_%=~svt`i~4VE4ZNXNNDFO|6* z&-OLg4!wZfHH>$HRt%rC^yl!|zni@=FSoP6nh|@s1&+KMN)nJvT7qckkWSQb&+|)w z*BMw1n;{cBDGjJR;vDoUY9W(7WE+C`_GvtlC2I!q-{@Z^lroktLmW{My@ns`1ayU; zQ;%d0A*<;H?UfQbp2F=cLXQ23Gxj#R7m~`ogecSqHj<%KNdI2c9Ye@nW48p)WjYM; z375^l4kbLQN;#&gL*kKph`89cM4pVcLwBMf#Mol{6P6hR*KaEf$ni0_uqh7$U@bIk zSK2Xltuv^vGyn%q%q;4{W^l|8{9g92&LK0-mqBt4gfD##qU)Ht2(r$Nv;xn}B@4(pNQ)4NF9y|FJL{cS<<6i> z0qr&d2&8s-hxX<-;(H(WBwzK1A=r}`{DkRSd3SEqA=h_w^_698<@VN{%e}Q>iCuot zXMcbH%n?m75gb)j}@|=d} z#Ev_sm(&gfLW}6Q3@au+8&nz07chyqv0G;qnN6gd_o`z2;Lg<^%EVz`moaVPH2#a0Qb=q3zRG&r7(wYKW{+;L}fGkfEzw_x?(Hru6k?mTBX~GV8aT=3{+F; zT_0yJnfUE*Aoj*johIo0=R^EWl~HSBoOcY;Rq9E%-{&UA_PvL&!stV>ohmvnsSkElYn+L{0!eT> zrZ*tm=W{V-4uO1ASx6m^Q*wQ#%!J}^-K$Yu-UCOhYsp{O;$`G%mhD|<(&yB}O=V2l3Rj6Ib zWdDcLqi60=BqLOkonovdLx3d*-^sF=?83hB<;DAnD19whnmAcKX#blCTm*Pn!kPkN zTfHAUemp`MzP>$b>89d&-G=1dd1sY4DqW0;LrtoT9HWR3yFYOC6{@CGR zRh8S`FxLDlz~!V+EufDnom!ss_c>$VF}CmI{lxE3&ZB8PtNN4pG9bNYCw|{I+GJMh z_VvOXy^nzb%|dggY60vIvp9#IVqw@4P**^l5+u$aHl#*>g7~Zb%q#AbN#-VrWf%`b z<-pEQv@(udInPqnULkH$k|UqXhE)`8OpGm{FHtjTJi;j4-avVL7b)#%ojEgQJg?G! z@;i0|*zWnYF&jTJCjgyoXIRBCKPx|cZ+x*L^bgSY_K|R%NNc(4KF`IF4+6&AR}AOc z6Z-sDO+olVbm#Gz_i2sA2;XUl9JgoILtQ7f)-l*y*<>pvD$*Xos#BDg*X6Ik#~N78 z%jfsxGnWAFr2gaQ(|a>NL20?rW)wx9#eGB=)fuHx8AnB*T19rv&*SH}iJsHBqhmrp zw}cm&OM6i|x%kZ&6fcj8xgR}SCH&P3Sr_EAWYae|*Ao*^`}?=zKLn^GK*Q6M!x$hc z@(!(s!_KR!>!|V!*SB)_&Wz1R!CWT}5A{F# zCVtbkA8`qmsr2D!-cNW|U0hI>+J<2p%;bDO3{8ghhKLY`ClT(vcW@fb7`yDPs3O{@ z2w`a7e1z*w3y;_rA==x@&5)lk6EYm;oO6e+GNo+_WYRffp|33Cq^CXo3zj%YY-T(^ za?IQ3D$^@dgq5OER`iq|2Cnpf=?P&H5XktjHf(WB5J^;);?PDPIbV+ECV`BJn*D{l(?$5*JS%EtLTe><@%Xr=EvUTpW`j zX#H+uUn6=aH?7d=`bNkdxpRQ8<@{r;6Q|Dp!vH1eo#E|g?WVbYsh77jr5BF_o5ub{ zm*QbX53$cV#Qn}NmiDMUp;96|X^_&SGZ9p1^~O6E#+^M65hOVQCK$Ks5XEK1AK)VM zyLPJ1!lN0J@Z{uX>lvs9fzYYzF{ZQ$Du`v~n4qr?U6{>3D)C7*E7$)Meg=X0IUl`r zH=IWVhzJQU$8jNhgK!W*;>Gmu5n>APS-Em$gH0l_+-nRO^YL`tamSV24?vxF4z_YX^HId{=3sfk1BcE%epp-$Hvp^``}Te1Ve=he#KZf;x5T7#J;IzT94b z<$)ew>f~Bv$i^!SrSU)qru>pob1W|uhMmTY8DmioV9pQp02+ojfd#J+{W<>l<1M0- zD7PY@N6H^wWP9$pr=5nq0EL$r+`AUAN!lM?@Dw5l3=P~dWljH{Q&g* z>IF3X^d{ot2YQL>VF;4pT86?HrXzX+R8^`}X}!Au&?8bp+bEOoL$Poeu&cbNDS8==t>x z?nz*qegy=3!?CdqC89@^j^SAfFh%>wp-KT1Ng&U#D}9aV90}~{`$Vvb6rwlRAWBN% zLyC_M{7V`yK+`J>0h6wa``#}fYNC3`>*}Xl>`-*EJpR#({=yKqO1Ko@{l@*tJ5ogo z6iR`#K#zwmFEzft8LAKhUs8I%CfC}@nhz>BP$?v5>^*fgU+5JssjF+#@ZT^oVx(-7iE1 zse@@0zz)|W3demI3wnIqa9`3l%j+9Wnlv%{xaaU;e2E7=hH7aC&jmh)m=XZk-gx7U z27g4>fN{W(ECb`mk2k%!Z}~8yt=!A_hNvVFT!5Mn!e^g-)*`t~523#%x@WmJ6V=1H ze!4L&px6pf&odx?ib%1wQ|(D#0Y-56XyRF@A_WSiKw6+jgo%eS4`v?VytH{R)0ugR z@+J^$*k`Cx4ODvxX=mEZd_X{6e^k*QC z`x2=1;2(hZ`Ud~cy#u^4#UfeXVE!u$hlup@Ay*#gQLnccf_sd<2S5{v<9_4i1_1J` z;NywKA?OWIau0Dnb5t3+r4UI(Z=MVEQQAy2B-uUdSC5}=g?i8?s}~ReYP~3WF}6`{ z%1i;~1M@)y$g4c|5yLQc)GVx%Yvx)+CVOHn!QN$>TN;ni?x&m`i`N&a`0)cx*|+ z>Q08Xa#aL+xl%Sa*Co1Mt5z+G$P+y!>Y1w|(DPRo*kgK0r%s)0?xN=G<*FCv?z+s$ z<0G2sNz7vxnode communication is partially failed +* To avoid false positives caused by failure of the control plane + +### Non-Goals + +* Self healing in combination with existing reprovisioning or reboot based remediation mechanisms in the Machine Health Check controller +* Recovering from all types of failure +* Having a distributed mesh of health checks where each node is checking health for its peers +* Creation of fake Machine objects for UPI clusters + +## Proposal + +Utilizing a DaemonSet to have a privileged critical pod running on each node. This pod will periodically check the +health status of the node it’s running on, and tickle a [watchdog device](https://en.wikipedia.org/wiki/Watchdog_timer). + +This will be performed by looking for the `host.metal3.io/external-remediation` annotation added by MHC to the unhealthy +Machine CR. + +If the unhealthy annotation was found, the node will remediate itself by using the watchdog device to +trigger a reset of the node. Watchdog devices, especially the hardware kind, are considered more reliable and predictable than the +reboot command. + +If Machine CR is not accessible in 3 consecutive attempts, it needs external assistance to detect its +health status. + +Failure to read Machine CR could stem from one of the following: +1. The node is unhealthy and therefore can’t read the Machine CR (e.g. local network failure) +1. There’s an etcd failure +1. Connectivity issues between the node and the control plane nodes (which doesn’t stem from local problem in the node) +1. Resource starvation, preventing our DaemonSet from initiating the check +1. Authentication failure to API (ServiceAccount deleted?) +1. RBAC disallows read operation on Machine (Role deleted or changed + +We only want to remediate if the problem is local (#1 and #4). Otherwise, we could create a "remediation storm" where +all nodes are trying to remediate themselves, even if the problem is not theirs. + +The semantics of the watchdog device automatically handle #4 and to distinguish between the remaining three situations, +the node will ask its peers whether they think it is healthy. + +The possible responses from a peer are: +1. You're unhealthy (the annotation exists) +1. You're healthy (the annotation does not exist) +1. The peer also cannot contact the api-server +1. No response (timeout or any TCP/HTTP error) + +To avoid saturating the network, the node will pick *min(cluster_size/10, cluster_size-1)* random nodes, and ask these +nodes if they think it’s healthy using HTTP. +Each pod will listen on a specific node-port, and will run an HTTP web server that will be able to tell other peers if +the machine is healthy or not. +If two or more peers responded with a different result, we take an optimistic approach - one result of +*"you’re healthy"* is enough to determine that the node is healthy. +If all randomized peers returned *"you’re unhealthy"* - the node considers itself unhealthy. +If all randomized peers returned an error (#3 or #4), another set of random nodes will be used to determine health +status. +The node considers itself healthy if more than 50% of the nodes responded with #3. +The node considers itself unhealthy if we do not receive any peer responses. +The above algorithm to determine health state of a node is demonstrated in the following flow chart: + +![Machine health check](./baremetal-poison-pill-health-flow.png) + +The state required to run this algorithm will be stored in-memory, including nodes addresses, retries count and so on. + +Remediation will be done by using a watchdog device to trigger a reset of the node. + +In the absence of a watchdog device, including softdog, a software reboot will be issued. + +While reboot might not fix the health issue of the node, it will allow the cluster to assume that the workload on the failing node is no longer running, such that stateful applications scheduled on that node can be rescheduled to other healthy nodes. + +Steps taken by the poison pill pod (on all nodes in the cluster) for any unhealthy node: +1. Mark the node as unschedulable (this prevents the node to run new workload after reboot) +1. Add current time to annotation +1. Unhealthy node reboot itself +1. After *timeout-to-assume-node-rebooted* (either configurable or calculated by software) - delete the unhealthy node (to signal the cluster that the workload is no longer running there, and it is safe to run it elsewhere) +1. Recreate the node object, without the unschedulable taint + +The first step of marking the node as unschedulable is important, as the reboot might fix the health issue, and the scheduler could assign workload to that node after the it rebooted, which could result in other nodes deleteing the rebooted node while it is running some workload. To avoid that, the others will delete it and report it as unhealthy only after marking it as unschedulable which ensures it will not get any workloads after reboot. + +### User Stories [optional] + +#### Story 1 +As a non-BMC baremetal cluster admin, I would like to have automated self-healing nodes. This will make my cluster more robust and immune to failures + +#### Story 2 +As a non-BMC baremetal cluster admin, I would like to minimze downtime for my applications, even if there's a node failure. + +### Implementation Details/Notes/Constraints [optional] + +This enhancement relies on the existence of MHC, which is currently working only in IPI and assisted insaller. +We might want to enable MHC for UPI clusters as well, maybe by faking Machine CRs for the nodes. + +### Risks and Mitigations + +If the unhealthy node doesn't reboot in time, then there's a risk that a node will be deleted while the workload is still running. +To mitigate this we prefer a hardware watchdog which ensures that reboot will take place if the poison pill is not running. + +There could be cases where this algorithm will declare a node as unhealthy while it’s healthy (false negative) +For example, if someone made some network configuration mistake which blocks communication between the nodes - all nodes will consider themselves unhealthy and remediation will be triggered. +In this case, the cluster is mostly in an unusable state anyway, so no significant harm is made. + + +## Design Details + +### Upgrade / Downgrade Strategy + +N/A. This is a new component. + +### Version Skew Strategy + +Downgrade from a version with poison pill to a version without it during remediation process can result in a node that doesn't exist in api-server. +As part of the remediation process, the poison pill agents keep a backup for the node yaml. +They are expected to use that backup after the node has been deleted, to re-create the node. +If a downgrade happens during that process, there will be no poison pill agents to re-create the node. + +In addition, a node might consult with its peers for its health status. If they are downgraded to a version without poison piil, +they won't respond, and we might get false positive. + +## Drawbacks + +TBD + +## Alternatives + +Instead of contacting other peers, we could use a shared storage to signal the health status for a peer. + +## Infrastructure Needed [optional] + +A new repo might be needed to host the new poison pill codebase.