Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Re-enable joining from old snapshot test #3573

Merged
merged 47 commits into from
May 4, 2022
Merged

Re-enable joining from old snapshot test #3573

merged 47 commits into from
May 4, 2022

Conversation

jumaffre
Copy link
Contributor

@jumaffre jumaffre commented Feb 18, 2022 •
edited
Loading

The run_join_old_snapshot seems to have been disabled for a while now so this PR re-enables it.

This also contains a fix so that nodes started with no snapshots (i.e. need to replay the entire historical ledger on join) are also subject to the same check.

We now also try to join/recover from a snapshot as much as possible in the Python infra.

Note: the justification for this is outlined in the original PR #2422.

@jumaffre jumaffre requested a review from a team February 18, 2022 12:41
@ghost
Copy link

ghost commented Feb 18, 2022 •
edited by ghost
Loading

old_snapshot_test_enable@44998 aka 20220504.17 vs main ewma over 20 builds from 44708 to 44994

Click to see table

main

build_id build_number tpcc_sgx_cft^ tpcc_sgx_cft_mem ls_sgx_cft^ ls_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_v8_sgx_cft^ ls_v8_sgx_cft_mem ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_full_v8_sgx_cft^ ls_full_v8_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem hist_sgx_cft^ RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
44708 20220425.11 6391.91 8.42835e+07 19462.1 1.63882e+07 5462.51 1.56017e+07 2520.84 1.00967e+07 1647.28 1.62927e+08 2114.88 9.04813e+06 1446.35 9.87014e+07 1965.65 8.78598e+06 21546.8 903953 1.36788e+06 9.24601e+06 3.56174e+07
44730 20220426.1 6324.99 8.40213e+07 19615.1 1.58639e+07 5686.45 1.53396e+07 2541.06 1.00967e+07 1604.29 1.61092e+08 2115.95 9.04813e+06 1448.2 9.87014e+07 2010.07 8.78598e+06 19238.4 890896 1.37662e+06 9.20863e+06 3.56174e+07
44750 20220426.10 6679.87 8.32349e+07 18050.6 1.66503e+07 5351.12 1.56017e+07 2461.59 1.00967e+07 1584.2 1.63189e+08 2095.19 9.31027e+06 1417.67 9.84392e+07 2012.01 8.78598e+06 20992.7 856289 1.378e+06 9.00612e+06 3.47119e+07
44758 20220426.13 6325.56 8.50699e+07 19235.1 1.66503e+07 5582.14 1.53396e+07 2481.27 1.00967e+07 1626.05 1.63975e+08 2112.78 9.04813e+06 1431.71 9.81771e+07 1974.09 8.78598e+06 17194.4 905628 1.39414e+06 9.326e+06 3.54939e+07
44771 20220427.2 6152.09 8.32349e+07 18944.9 1.71746e+07 5540.4 1.53396e+07 2503.31 1.00967e+07 1618.73 1.62402e+08 2111.11 9.31027e+06 1453.71 9.66042e+07 1976.51 8.52384e+06 17705.4 891360 1.37837e+06 9.22103e+06 3.57411e+07
44793 20220427.10 6258.62 8.32349e+07 19506.3 1.6126e+07 5639.94 1.53396e+07 2485.66 1.00967e+07 1624.25 1.63189e+08 2120.68 9.04813e+06 1438.22 9.84392e+07 1937.69 8.78598e+06 20844.7 911032 1.37505e+06 9.18798e+06 3.57411e+07
44820 20220428.2 6353.67 8.32349e+07 18950.6 1.6126e+07 5509.19 1.53396e+07 2568.48 9.83456e+06 1640.2 1.62927e+08 2127.14 9.04813e+06 1451.18 9.84392e+07 1934.94 1.11453e+07 19101.8 901089 1.37182e+06 9.3303e+06 3.56168e+07
44850 20220429.2 6159.34 8.29727e+07 19417.6 1.6126e+07 5606.55 1.53396e+07 2537.78 1.03588e+07 1627.61 1.63451e+08 2118.07 9.04813e+06 1412.23 9.66042e+07 1889.46 8.78598e+06 16310.3 907878 1.36797e+06 9.29215e+06 3.58042e+07
44856 20220429.5 6154.87 8.50699e+07 18943.6 1.66503e+07 5548.47 1.56017e+07 2601.81 1.00967e+07 1619.14 1.62927e+08 2116.13 1.16696e+07 1369.4 9.79149e+07 1920.31 9.04813e+06 17473.7 887230 1.37579e+06 9.20859e+06 3.63766e+07
44865 20220429.8 6224.77 8.42835e+07 19357.5 1.58639e+07 5597.44 1.53396e+07 2556.88 1.03588e+07 1635.33 1.63189e+08 2119.86 9.04813e+06 1422.96 9.81771e+07 1931.65 1.08831e+07 18972.5 898713 1.37375e+06 9.29211e+06 3.58663e+07
44873 20220429.11 6195.75 8.3497e+07 19531.5 1.6126e+07 5556.65 1.56017e+07 2495.05 1.2456e+07 1606.73 1.63189e+08 2111.33 9.04813e+06 1426.43 9.84392e+07 2009.44 8.78598e+06 17485.6 907836 1.37708e+06 9.33876e+06 3.57411e+07
44882 20220429.14 6182.26 8.45456e+07 19367.9 1.56017e+07 5564.7 1.56017e+07 2550.65 1.00967e+07 1609.78 1.62927e+08 1998.43 9.04813e+06 1409.71 9.81771e+07 1925.78 8.78598e+06 19486.1 904868 1.38555e+06 9.30901e+06 3.57411e+07
44901 20220502.2 7131.84 8.37592e+07 19512.4 1.6126e+07 5673.05 1.58639e+07 2540.07 1.00967e+07 1636.06 1.63189e+08 2153.41 9.04813e+06 1426.28 9.66042e+07 1977.94 8.78598e+06 17673.7 909897 1.36134e+06 9.20036e+06 3.56794e+07
44907 20220503.2 7223.94 8.40213e+07 19308.2 1.66503e+07 5533 1.53396e+07 2546.55 1.00967e+07 1624.37 1.63189e+08 2116.27 9.04813e+06 1446.97 9.79149e+07 2011.52 1.0621e+07 19673.8 910059 1.39462e+06 9.53445e+06 3.5993e+07
44940 20220503.14 6252.48 8.3497e+07 19501.3 1.6126e+07 5585.6 1.50774e+07 2561.91 1.03588e+07 1620.62 1.62927e+08 2128.28 9.31027e+06 1427.69 9.84392e+07 2013.84 8.78598e+06 19185.7 888995 1.37373e+06 9.3472e+06 3.63114e+07
44955 20220504.2 5753.37 8.32349e+07 19355.1 1.63882e+07 5514.08 1.53396e+07 2554.57 1.00967e+07 1630.54 1.63189e+08 2118.92 9.31027e+06 1436.49 9.81771e+07 1974.67 8.78598e+06 18859.9 907313 1.36025e+06 9.36867e+06 3.58669e+07
44960 20220504.3 6173.56 8.42835e+07 19689.6 1.6126e+07 5573.58 1.53396e+07 2566.29 1.00967e+07 1649.15 1.61092e+08 2121 9.04813e+06 1460.13 9.76528e+07 1935.34 8.78598e+06 17656.9 902716 1.35557e+06 9.24601e+06 3.60563e+07
44970 20220504.7 6297.42 8.32349e+07 19618.1 1.6126e+07 5504.15 1.53396e+07 2558.26 1.03588e+07 1609.81 1.63189e+08 2119.57 9.04813e+06 1435.84 9.81771e+07 1974.97 8.78598e+06 19874.7 916286 1.40024e+06 9.3601e+06 3.65714e+07
44983 20220504.12 6262.62 8.3497e+07 19753.4 1.63882e+07 5543.09 1.56017e+07 2548.49 1.00967e+07 1617.86 1.62927e+08 2118.25 9.04813e+06 1429.31 9.84392e+07 1927.23 8.78598e+06 21288.3 901562 1.37099e+06 9.25436e+06 3.58669e+07
44994 20220504.16 6257.56 8.42835e+07 19605.7 1.6126e+07 5578.57 1.56017e+07 2544.33 1.00967e+07 1611.37 1.62927e+08 2120.31 9.04813e+06 1426.38 9.63421e+07 2011.52 8.78598e+06 17389.3 908808 1.38247e+06 9.21692e+06 3.65714e+07

old_snapshot_test_enable

build_id build_number tpcc_sgx_cft^ tpcc_sgx_cft_mem ls_sgx_cft^ ls_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_v8_sgx_cft^ ls_v8_sgx_cft_mem ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_full_v8_sgx_cft^ ls_full_v8_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem hist_sgx_cft^ RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
44947 20220503.16 6159.97 8.42835e+07 19434.6 1.66503e+07 5624.57 1.50774e+07 2558.04 1.00967e+07 1633.71 1.6083e+08 2103.54 9.04813e+06 1423.97 9.81771e+07 1975.77 8.78598e+06 17748.4 901483 1.37986e+06 9.22518e+06 3.58669e+07
44966 20220504.5 6346.68 8.40213e+07 19272.3 1.63882e+07 5635.75 1.53396e+07 2556.9 1.03588e+07 1628.13 1.63975e+08 2119.88 9.04813e+06 1443.61 9.76528e+07 1924.72 8.78598e+06 19441.9 879984 1.37283e+06 9.19615e+06 3.63759e+07
44981 20220504.11 6375.83 8.32349e+07 19640 1.56017e+07 5623.61 1.53396e+07 2551.81 1.00967e+07 1642.03 1.63451e+08 2126.29 9.04813e+06 1441.14 9.63421e+07 1918.28 8.52384e+06 20171 908115 1.39899e+06 9.20851e+06 3.5189e+07
44989 20220504.14 6344.89 8.3497e+07 19420.5 1.58639e+07 5687.45 1.53396e+07 2554.57 1.03588e+07 1609.79 1.62665e+08 2117.5 9.04813e+06 1424.97 9.87014e+07 1918.37 8.78598e+06 17414.7 881733 1.37542e+06 9.22523e+06 3.58669e+07
44998 20220504.17 6214.36 8.37592e+07 19752 1.58639e+07 5420.47 1.53396e+07 2581.57 1.00967e+07 1619.57 1.63451e+08 2111.35 9.31027e+06 1440.77 9.81771e+07 1930.55 9.04813e+06 21698.7 899028 1.37874e+06 9.23763e+06 3.56794e+07

images

@jumaffre jumaffre mentioned this pull request Feb 18, 2022
Closed
@jumaffre
Copy link
Contributor Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 3 pipeline(s).

achamayou
achamayou reviewed Feb 24, 2022
View reviewed changes
@jumaffre jumaffre marked this pull request as draft February 28, 2022 15:11
@jumaffre jumaffre mentioned this pull request Apr 27, 2022
Closed
@jumaffre
Copy link
Contributor Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 3 pipeline(s).

@jumaffre
Copy link
Contributor Author

/azp run

@jumaffre jumaffre marked this pull request as ready for review April 28, 2022 15:31
@azure-pipelines
Copy link

Azure Pipelines successfully started running 3 pipeline(s).

achamayou
achamayou reviewed Apr 29, 2022
View reviewed changes
@achamayou achamayou enabled auto-merge (squash) May 4, 2022 12:22
@achamayou achamayou merged commit e165709 into microsoft:main May 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@achamayou achamayou achamayou approved these changes

@wintersteiger wintersteiger wintersteiger approved these changes

@raghavthind2005 raghavthind2005 raghavthind2005 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@jumaffre @achamayou @wintersteiger @raghavthind2005