Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert test case for large historical queries #5040

Merged

Conversation

eddyashton
Copy link
Member

I think the large messages in the historical range query test, added in #5026, have introduced some new CI instability (possibly #5032, possibly because we weren't previously testing #5039?).

Experimentally reverting the e2e test for now - if that fixes the CI jobs I'll merge this, and then write an equivalent but more targeted test.

@eddyashton eddyashton requested a review from a team February 21, 2023 14:30
@ghost
Copy link

ghost commented Feb 21, 2023

retarget_historical_large_messages@65438 aka 20230221.16 vs main ewma over 20 builds from 64917 to 65405

Click to see table

main

build_id build_number Commit latency factor tpcc_virtual_cft^ ls_virtual_cft^ tpcc_sgx_cft^ tpcc_sgx_cft_mem pi_ls_virtual_cft^ ls_jwt_virtual_cft^ pi_ls_jwt_virtual_cft^ ls_js_virtual_cft^ ls_sgx_cft^ ls_sgx_cft_mem pi_ls_sgx_cft^ pi_ls_sgx_cft_mem ls_full_js_virtual_cft^ ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem ls_js_jwt_virtual_cft^ pi_ls_jwt_sgx_cft^ pi_ls_jwt_sgx_cft_mem ls_js_sgx_cft^ ls_js_sgx_cft_mem hist_sgx_cft^ ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
64917 20230215.36 0.796886 17145.5 45429.3 6278.39 8.19242e+07 47658.8 13972.5 14052.7 4553.6 15840.7 1.53396e+07 16206.8 7.99955e+06 3733.84 5810.05 1.48153e+07 3337.61 5812.2 4.59168e+06 1537.98 7.73741e+06 24779 1346.66 7.73741e+06 1273.01 6.95098e+06 821215 1.17669e+06 8.15556e+06 3.07928e+07
64948 20230216.3 0.783009 17178 45750 6287.21 8.21863e+07 49027.8 14009.7 13936.5 4503.11 15838.4 1.50774e+07 16212.3 7.99955e+06 3793.64 5818.07 1.4291e+07 3282.85 5844.2 4.59168e+06 1545.93 7.73741e+06 27272.6 1344.14 7.73741e+06 1272.54 6.95098e+06 839665 1.17924e+06 8.15611e+06 3.08894e+07
64952 20230216.5 0.768871 17116.4 43669.6 6286.94 8.24484e+07 47447.8 12312.6 13598.8 4334.54 15819.2 1.56017e+07 16253.2 7.99955e+06 3831.55 5825.43 1.45532e+07 3312.9 5818.4 4.32954e+06 1538.3 7.73741e+06 26809.9 1339.01 7.21312e+06 1265.91 6.95098e+06 834929 1.17923e+06 8.17408e+06 3.07364e+07
64973 20230216.11 0.77964 17120.9 43716.3 6291.12 8.21863e+07 47752.5 12735.6 13741.2 4479.68 15851.9 1.50774e+07 16226 7.99955e+06 3830.47 5801.55 1.48153e+07 3381.61 5781.5 4.32954e+06 1543.72 7.73741e+06 23430.1 1344.64 7.47526e+06 1270.49 6.95098e+06 838175 1.18161e+06 8.15231e+06 3.20411e+07
65000 20230216.21 0.791976 17402.4 45903.7 6244.39 8.21863e+07 48696.4 12287.2 13913.3 4518.9 15831.9 1.53396e+07 16219.6 7.99955e+06 3823.86 5807.03 1.4291e+07 3326.67 5741 4.59168e+06 1541.3 7.73741e+06 25013.4 1342.64 7.47526e+06 1252.44 6.95098e+06 826814 1.18231e+06 8.14768e+06 3.0816e+07
65057 20230217.2 0.792405 17143.9 43472.8 6298.6 8.1662e+07 47341.8 12267.5 13581.4 4570.79 15797.7 1.53396e+07 16214.4 7.99955e+06 3794.33 5848.79 1.48153e+07 3343.63 5771.2 4.59168e+06 1533.25 7.73741e+06 24269.4 1343.02 7.47526e+06 1271.08 6.95098e+06 834024 1.17337e+06 8.15102e+06 3.11379e+07
65074 20230217.9 0.81627 17115.8 43374.6 6302.69 8.24484e+07 46781.9 12504.5 13214.1 4459.26 15868.3 1.50774e+07 16189.8 7.99955e+06 3689.68 5848.61 1.48153e+07 3303.34 5777.8 4.59168e+06 1544.66 7.47526e+06 23233.3 1347.63 7.21312e+06 1270.05 6.95098e+06 831493 1.17746e+06 8.12637e+06 2.99999e+07
65094 20230217.15 0.799868 17107.3 43627.7 6279.58 8.21863e+07 47369.6 12531.7 12954.4 4443.61 15853.7 1.50774e+07 16109.3 7.99955e+06 3794.98 5824.51 1.4291e+07 3298.82 5771.8 4.59168e+06 1535.17 7.73741e+06 23284.4 1339.94 7.47526e+06 1262.97 6.95098e+06 820841 1.17776e+06 8.15576e+06 3.0816e+07
65127 20230217.23 0.772319 17159.6 45685.4 6304.64 8.21863e+07 48339.6 12414.1 13299.5 4474.02 15794 1.53396e+07 16205.2 7.99955e+06 3793.01 5823.32 1.48153e+07 3321.86 5762.6 4.32954e+06 1536.84 7.73741e+06 23236.5 1344.99 7.99955e+06 1257.52 6.95098e+06 829509 1.18355e+06 8.15563e+06 1.80705e+07
65176 20230220.3 0.797649 17153.3 45743.2 6334.96 8.21863e+07 47322.9 14029.3 13336.5 4508.1 15951.9 1.53396e+07 16177.7 7.99955e+06 3817.48 5847.47 1.48153e+07 3436.64 5777.7 4.32954e+06 1541.22 7.73741e+06 27095.5 1339.48 7.73741e+06 1266.79 6.95098e+06 829340 1.18169e+06 8.13528e+06 3.1386e+07
65208 20230220.10 0.813333 17090.1 45699.5 6265.97 8.1662e+07 46507.7 12734.8 13047.5 4468.94 15781.1 1.50774e+07 16117.6 7.99955e+06 3829.31 5828.24 1.45532e+07 3471.39 5744.7 4.32954e+06 1525.75 7.73741e+06 24377.9 1338.49 7.21312e+06 1258.13 6.95098e+06 838175 1.18477e+06 8.17043e+06 3.11502e+07
65264 20230220.23 0.832354 17117.1 45651.8 6328.02 8.19242e+07 46536.8 12556.4 13765.6 4614.39 15849.9 1.50774e+07 16163 7.99955e+06 3841.53 5803.89 1.4291e+07 3325.78 5772 4.59168e+06 1547.61 7.73741e+06 25293.7 1347.67 7.47526e+06 1267.89 6.95098e+06 831007 1.1823e+06 8.15264e+06 3.14574e+07
65297 20230220.36 0.776059 17137.5 43485.7 6214.08 8.21863e+07 48684.1 12609.5 12995.1 4601.13 15781 1.50774e+07 16120.1 7.99955e+06 3785.65 5806.92 1.4291e+07 3342.19 5774.8 4.59168e+06 1545.31 7.73741e+06 26840.3 1342.31 7.21312e+06 1269.29 6.95098e+06 837789 1.17789e+06 8.15511e+06 3.11663e+07
65333 20230220.49 0.78117 17354 46065.2 6337.38 8.21863e+07 48488.6 12595.2 13267.3 4608.39 15893.5 1.50774e+07 16090.4 7.99955e+06 3807.96 5816.8 1.4291e+07 3326.69 5783.2 4.59168e+06 1537.46 7.73741e+06 24615.3 1341.82 7.47526e+06 1263.85 6.95098e+06 826783 1.17968e+06 8.15546e+06 3.2511e+07
65344 20230220.53 0.814576 17204.5 43495.4 5668.99 8.24484e+07 48698 11587.4 13082.3 4350.65 15460.6 1.50774e+07 15994.7 7.99955e+06 3831.07 5771.4 1.4291e+07 3287.52 5731.8 4.32954e+06 1520.95 7.47526e+06 26614.8 1334.52 7.21312e+06 1248.25 6.95098e+06 831370 1.17115e+06 8.15238e+06 3.07595e+07
65360 20230220.56 0.823358 17109.3 43630.2 6300.6 8.21863e+07 48383.2 12866.6 12843.1 4385.46 15709 1.50774e+07 16076.9 7.99955e+06 3725.49 5790.45 1.4291e+07 3323.02 5738.8 4.59168e+06 1543.25 7.73741e+06 23152.6 1328.52 7.21312e+06 1266.23 6.95098e+06 832056 1.17678e+06 8.15076e+06 3.07933e+07
65372 20230220.62 0.788533 17256.8 45703.8 6305.11 8.21863e+07 47982.6 12452.2 13197.8 4368.56 15866.1 1.50774e+07 16153.8 7.99955e+06 3822.36 5847.76 1.4291e+07 3395.3 5734 4.59168e+06 1542.5 7.73741e+06 42873.7 1336.99 7.47526e+06 1264.98 6.95098e+06 846468 1.17611e+06 8.15267e+06 3.08466e+07
65388 20230220.67 0.792798 17112.4 44008.9 6329.01 8.21863e+07 48921 12452.3 13870.8 4488.31 15755.8 1.50774e+07 16141.1 7.99955e+06 3795.98 5793.8 1.4291e+07 3319.49 5728.7 4.59168e+06 1537.13 7.73741e+06 48473.8 1339.93 7.21312e+06 1261.9 6.95098e+06 825594 1.1741e+06 8.17395e+06 3.13155e+07
65395 20230220.69 0.787055 17201.2 43790.5 6303.67 8.1662e+07 49701.8 12379.1 13052.3 4630.77 15805.6 1.50774e+07 16136.2 7.99955e+06 3804.63 5503.63 1.45532e+07 3441.09 5671.3 4.06739e+06 1528.03 7.73741e+06 45845.4 1339.44 7.21312e+06 1247.67 6.95098e+06 831800 1.17776e+06 8.15576e+06 3.11488e+07
65405 20230221.3 0.817252 17147.1 43675.1 6348.16 8.1662e+07 48039.3 13048.9 13008.5 4413.92 15855.9 1.50774e+07 16148.2 7.99955e+06 3831.64 5511.03 1.45532e+07 3318.21 5672 4.06739e+06 1529.87 7.47526e+06 48277.8 1342.99 7.21312e+06 1265.42 6.95098e+06 835213 1.17709e+06 8.15397e+06 3.20631e+07

retarget_historical_large_messages

build_id build_number Commit latency factor tpcc_sgx_cft^ tpcc_sgx_cft_mem ls_sgx_cft^ ls_sgx_cft_mem tpcc_virtual_cft^ pi_ls_sgx_cft^ pi_ls_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem ls_virtual_cft^ pi_ls_jwt_sgx_cft^ pi_ls_jwt_sgx_cft_mem pi_ls_virtual_cft^ ls_jwt_virtual_cft^ pi_ls_jwt_virtual_cft^ ls_js_virtual_cft^ ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_full_js_virtual_cft^ ls_js_jwt_virtual_cft^ ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem hist_sgx_cft^ ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
65438 20230221.16 0.786975 6294.56 8.27106e+07 15766.1 1.50774e+07 17178.8 16097.5 7.99955e+06 5490.03 1.48153e+07 45590.7 5675.6 4.06739e+06 47549.1 12355.7 13628.1 4539.41 1527.15 7.73741e+06 3837.67 3389.35 1341.46 7.99955e+06 46448.8 1266.31 6.95098e+06 837685 1.18378e+06 8.14473e+06 3.08072e+07

images

@achamayou
Copy link
Member

@eddyashton that will potentially need backporting as well.

@eddyashton
Copy link
Member Author

@eddyashton that will potentially need backporting as well.

The backports of #5026 (#5033 and #5034) are currently parked as Draft, and not merged - I'll combine this revert and any additional fix before merging them to their respective target branches.

@eddyashton
Copy link
Member Author

Merging as it looks like an improvement in test stability, though I think there remains a new issue in test_recover_service_with_wrong_identity.

@eddyashton eddyashton merged commit b8df874 into microsoft:main Feb 21, 2023
eddyashton added a commit to eddyashton/CCF that referenced this pull request Feb 27, 2023
@eddyashton eddyashton added 2.x-todo PRs which should be backported to 2.x 3.x-todo PRs which should be backported to 3.x labels Feb 27, 2023
eddyashton added a commit to eddyashton/CCF that referenced this pull request Feb 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.x-todo PRs which should be backported to 2.x 3.x-todo PRs which should be backported to 3.x
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants