Timeout idle node-to-node channels

[eddyashton]

Resolves #2672.

I believe all of the groundwork to do this safely (decoupling host and enclave state for these channels, robustly recreating channels on-demand) has been done already, and all that was left (this PR) is adding idle time tracking? There's a non-zero risk of this introducing subtle new node-to-node failures, so I'll see what the CI thinks of a few runs.

[achamayou]

@eddyashton a good candidate for a canary poke?

[ghost]

timeout_idle_n2n_channels@70595 aka 20230519.10 vs main ewma over 20 builds from 70261 to 70589

Click to see table

main

build_id	build_number	Commit latency factor	tpcc_virtual_cft^	tpcc_sgx_cft^	tpcc_sgx_cft_mem	ls_virtual_cft^	pi_ls_virtual_cft^	ls_jwt_virtual_cft^	pi_ls_jwt_virtual_cft^	ls_sgx_cft^	ls_sgx_cft_mem	pi_ls_sgx_cft^	pi_ls_sgx_cft_mem	ls_js_virtual_cft^	ls_jwt_sgx_cft^	ls_jwt_sgx_cft_mem	ls_full_js_virtual_cft^	pi_ls_jwt_sgx_cft^	pi_ls_jwt_sgx_cft_mem	ls_js_jwt_virtual_cft^	ls_js_sgx_cft^	ls_js_sgx_cft_mem	hist_sgx_cft^	ls_full_js_sgx_cft^	ls_full_js_sgx_cft_mem	ls_js_jwt_sgx_cft^	ls_js_jwt_sgx_cft_mem	RB put (/s)^	CHAMP put (/s)^	RB get (/s)^	CHAMP get (/s)^
70261	20230512.8	0.793108	17352.1	7295.52	8.39025e+07	45648.5	48486.7	12479.4	12894.4	20392.7	1.88908e+07	20498.7	1.25993e+07	4453.58	6354.7	1.67936e+07	3737.85	6571.2	6.30784e+06	3257.06	1728.3	1.05021e+07	48508	1506.3	1.05021e+07	1401.04	1.05021e+07	836613	1.17938e+06	8.15589e+06	3.06904e+07
70275	20230512.12	0.785595	17156.8	7250.95	8.59996e+07	45864.2	48165.4	12454.4	12751.4	20302.1	1.88908e+07	20449.5	1.25993e+07	4437.32	6738.61	1.67936e+07	3755.97	6529	6.30784e+06	3297.44	1708.35	1.05021e+07	46881	1507.2	1.05021e+07	1396.42	1.05021e+07	837465	1.18068e+06	8.15342e+06	3.08759e+07
70289	20230515.3	0.799853	17312.2	7281.73	8.59996e+07	45808.3	48283	12548.4	13155.1	20441.2	1.88908e+07	20499.2	1.25993e+07	4396.68	6721.4	1.67936e+07	3712.14	6512.4	6.30784e+06	3327.2	1730.34	1.05021e+07	48800.8	1508.62	1.05021e+07	1419.76	1.05021e+07	828268	1.17905e+06	8.17464e+06	3.0822e+07
70303	20230515.9	0.819834	17679.1	7321.41	8.59996e+07	45567.7	48321.6	12478.4	13199.4	20309.2	1.88908e+07	20498.6	1.25993e+07	4465.73	6710.46	1.67936e+07	3710.61	6516.7	6.30784e+06	3264.87	1715.17	1.05021e+07	46887.6	1506.97	1.05021e+07	1402.36	1.05021e+07	831746	1.17442e+06	8.15254e+06	3.06977e+07
70335	20230516.1	0.80985	17287.5	7350.59	8.59996e+07	45570.7	48141.5	12530.6	12924	20498.8	1.88908e+07	20543.2	1.25993e+07	4457.15	6730.45	1.67936e+07	3723.38	6514.8	6.30784e+06	3156.67	1732.07	1.05021e+07	50998.3	1504.78	1.05021e+07	1418.89	1.05021e+07	837041	1.17884e+06	8.17415e+06	3.073e+07
70368	20230516.16	0.79649	17288.3	7268.6	8.59996e+07	45871.3	48381.4	12427.2	13019.6	20232.3	1.88908e+07	20403.2	1.25993e+07	4449.72	6710.13	1.88908e+07	3742.33	6555.1	6.30784e+06	3266.68	1715.97	1.05021e+07	48262.6	1509.48	1.05021e+07	1405.69	1.05021e+07	831415	1.1833e+06	8.1527e+06	3.07452e+07
70397	20230516.27	0.795206	17398.4	7323.93	8.59996e+07	45922.1	47878.5	12423.3	12995.1	20228.7	1.88908e+07	20455.5	1.25993e+07	4420.93	6642.82	1.67936e+07	3773.36	6505.7	6.30784e+06	3237.04	1708.5	1.05021e+07	48088.4	1495.5	1.05021e+07	1399.03	1.05021e+07	812481	1.17682e+06	8.16857e+06	3.08336e+07
70403	20230516.29	0.785613	17631.1	7258.96	8.59996e+07	45781.4	47933.1	12672.5	12975.9	20313.9	1.88908e+07	20474.4	1.25993e+07	4485.53	6693.74	1.67936e+07	3743.89	6502.3	6.30784e+06	3272.05	1711.51	1.05021e+07	50684.5	1505.04	1.05021e+07	1401.43	1.05021e+07	829271	1.17692e+06	8.15504e+06	3.06037e+07
70416	20230516.34	0.788347	17122.6	7262.97	8.39025e+07	45551.2	48150.8	12388	12908.5	20219.1	1.88908e+07	20440	1.25993e+07	4465.34	6699.56	1.67936e+07	3757.58	6529.3	6.30784e+06	3291.94	1715.02	1.05021e+07	46012.8	1504.07	1.05021e+07	1409.83	1.05021e+07	841376	1.18218e+06	8.15472e+06	3.07374e+07
70426	20230517.1	0.81237	17263.5	7305.04	8.59996e+07	45808.8	47715.8	12588.6	12853.1	20362.8	1.88908e+07	20507.9	1.25993e+07	4455.17	6722.38	1.67936e+07	3753.59	6564.1	6.30784e+06	3306.29	1729.35	1.05021e+07	45997.3	1507.22	1.05021e+07	1404.19	1.05021e+07	836591	1.17793e+06	8.15316e+06	3.08188e+07
70446	20230517.11	0.808969	17113.7	7276.85	8.59996e+07	45830.1	44462.2	12412.1	12991.1	20276.3	1.88908e+07	20491.8	1.25993e+07	4435.62	6688.21	1.67936e+07	3758.95	6553.9	6.30784e+06	3263.36	1736.1	1.05021e+07	42557.7	1493.11	1.05021e+07	1405.17	1.05021e+07	837823	1.17591e+06	8.16587e+06	3.14409e+07
70455	20230517.14	0.820054	17426.1	7328.46	8.59996e+07	45840.3	48262.3	12592.4	13017.5	20150.6	1.88908e+07	20383.9	1.25993e+07	4406.23	6345.12	1.67936e+07	3844.57	6552.2	6.30784e+06	3375.8	1709.45	1.05021e+07	48706.8	1507.48	1.05021e+07	1401.45	1.05021e+07	833673	1.18118e+06	8.15511e+06	3.10717e+07
70482	20230518.1	0.782545	17375.4	7283.5	8.59996e+07	45917.5	47944.8	12557.3	12755.6	20293.7	1.88908e+07	20493	1.25993e+07	4438.61	6376.04	1.67936e+07	3714.67	6549.8	6.30784e+06	3264.91	1733.99	1.05021e+07	43812.8	1505.44	1.05021e+07	1409.96	1.05021e+07	828045	1.17621e+06	8.15468e+06	3.09487e+07
70501	20230518.9	0.760928	17297.8	7365.56	8.59996e+07	45534.8	48083.4	12397	13013.7	20409.1	1.88908e+07	20456.5	1.25993e+07	4456.78	6729.09	1.67936e+07	3747.93	6574.6	6.30784e+06	3266.19	1713.67	1.05021e+07	50895.9	1508.72	1.05021e+07	1404.86	1.05021e+07	828628	1.18065e+06	8.15238e+06	3.06849e+07
70523	20230518.15	0.810363	17340.9	7223.27	8.39025e+07	45927	48404.1	12549	13164.5	20338.9	1.88908e+07	20460.4	1.25993e+07	4454.85	6403.9	1.67936e+07	3711.14	6556.9	6.30784e+06	3248.07	1729.7	1.05021e+07	48937.9	1508.06	1.05021e+07	1420.8	1.05021e+07	822475	1.17642e+06	8.15407e+06	3.08987e+07
70550	20230518.25	0.796913	17145.3	7334.91	8.39025e+07	45688.4	48136.9	12530.4	12899.8	20117.7	1.88908e+07	20437.4	1.25993e+07	4411.8	6355.43	1.67936e+07	3420.74	6555.7	6.30784e+06	3245.34	1712.95	1.05021e+07	50905.7	1424.1	1.05021e+07	1408.73	1.05021e+07	820462	1.18188e+06	8.17157e+06	3.08904e+07
70563	20230518.28	0.812575	17125.3	7252.69	8.59996e+07	45820.9	48650	12377.8	13067.5	20196.3	1.88908e+07	20351.4	1.25993e+07	4430.28	6733.62	1.67936e+07	3557.24	6545.4	6.30784e+06	3241.85	1712.56	1.05021e+07	48895.6	1424.67	1.05021e+07	1404.81	1.05021e+07	829922	1.18287e+06	8.17415e+06	3.08127e+07
70569	20230519.2	0.766523	17417.3	7335.72	8.59996e+07	45834.8	48591.3	12629.3	13189.5	20310.2	1.88908e+07	20413.2	1.25993e+07	4464.15	6381.15	1.67936e+07	3493.67	6557.9	6.30784e+06	3263.95	1734.35	1.05021e+07	45874.4	1420.15	1.05021e+07	1420.55	1.05021e+07	824598	1.18101e+06	8.02451e+06	3.14294e+07
70577	20230519.6	0.787617	17291.5	7305.86	8.59996e+07	47890.3	48853.4	12612.7	13378.8	20215.9	1.88908e+07	20424.4	1.25993e+07	4419.46	6727.28	1.67936e+07	3540.16	6555.7	6.30784e+06	3253.95	1731.41	1.05021e+07	49376.6	1422.39	1.05021e+07	1406.75	1.05021e+07	822570	1.18101e+06	8.14995e+06	3.08438e+07
70589	20230519.9	0.785013	17435	7255.07	8.59996e+07	45689.1	48452.2	12449	13186.2	20173	1.88908e+07	20404.8	1.25993e+07	4417.9	6710.23	1.67936e+07	3568.74	6510.5	6.30784e+06	3277.78	1730.48	1.05021e+07	51265.9	1423.21	1.05021e+07	1407.63	1.05021e+07	834496	1.17305e+06	8.17186e+06	3.11815e+07

timeout_idle_n2n_channels

build_id	build_number	Commit latency factor	tpcc_virtual_cft^	ls_virtual_cft^	pi_ls_virtual_cft^	ls_jwt_virtual_cft^	tpcc_sgx_cft^	tpcc_sgx_cft_mem	pi_ls_jwt_virtual_cft^	ls_js_virtual_cft^	ls_full_js_virtual_cft^	ls_sgx_cft^	ls_sgx_cft_mem	pi_ls_sgx_cft^	pi_ls_sgx_cft_mem	ls_js_jwt_virtual_cft^	ls_jwt_sgx_cft^	ls_jwt_sgx_cft_mem	pi_ls_jwt_sgx_cft^	pi_ls_jwt_sgx_cft_mem	ls_js_sgx_cft^	ls_js_sgx_cft_mem	hist_sgx_cft^	ls_full_js_sgx_cft^	ls_full_js_sgx_cft_mem	ls_js_jwt_sgx_cft^	ls_js_jwt_sgx_cft_mem	RB put (/s)^	CHAMP put (/s)^	RB get (/s)^	CHAMP get (/s)^
70352	20230516.9	0.824896	17434.6	45952.3	48569.1	12641.1	7326.94	8.59996e+07	13295.4	4459.45	3752.16	20429.2	1.88908e+07	20574.6	1.25993e+07	3273.59	6733.62	1.67936e+07	6563.4	6.30784e+06	1710.33	1.05021e+07	46596.6	1503.02	1.05021e+07	1405.46	1.05021e+07	835987	1.18233e+06	8.10729e+06	3.10106e+07
70356	20230516.11	0.800134	17407.1	45564.8	42269.7	12506.4	7285.78	8.39025e+07	13160.1	4478.87	3761.08	20217.2	1.88908e+07	20528.1	1.25993e+07	3277.39	6702.03	1.67936e+07	6515.4	6.30784e+06	1704.56	1.05021e+07	48875.1	1493.32	1.05021e+07	1407.73	1.05021e+07	824447	1.1809e+06	8.15563e+06	3.07595e+07
70407	20230516.30	0.80087	17170	45573.2	48671	12500.2	7267.41	8.39025e+07	13214.2	4430.08	3753.98	20365.6	1.88908e+07	20567	1.25993e+07	3253.79	6716.16	1.67936e+07	6550.5	6.30784e+06	1715.47	1.05021e+07	46942.1	1498.18	1.05021e+07	1406.5	1.05021e+07	833291	1.17618e+06	8.02508e+06	3.08114e+07
70465	20230517.17	0.794382	17056.9	45626.4	48598.5	12420.3	7289.44	8.59996e+07	13067.6	4460.93	3759.44	20230.6	1.88908e+07	20469.9	1.25993e+07	3259.93	6743.61	1.67936e+07	6504	6.30784e+06	1719.77	1.05021e+07	48917.1	1497.55	1.05021e+07	1407.67	1.05021e+07	827244	1.16253e+06	8.13325e+06	3.0724e+07
70595	20230519.10	0.806897	17312.7	47954.9	49484.9	12461.5	7354.83	8.59996e+07	12767.6	4418.96	3553.57	20455.2	1.88908e+07	20489	1.25993e+07	3275.95	6740.43	1.67936e+07	6552.6	6.30784e+06	1714.45	1.05021e+07	49661.2	1416.67	1.05021e+07	1418.39	1.05021e+07	834245	1.17893e+06	8.12785e+06	3.15402e+07