Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for host process stdio. #5056

Merged
merged 14 commits into from
Feb 28, 2023
Merged

Add support for host process stdio. #5056

merged 14 commits into from
Feb 28, 2023

Conversation

plietar
Copy link
Contributor

@plietar plietar commented Feb 23, 2023

When an application launches a process on the host, it now has the option to attach a bytestring which will be passed on the process' standard input.

Additionally, the standard output and error of the process are captured and printed in CCF's log. This makes it a lot easier to monitor and diagnose issues in production.

When an application launches a process on the host, it now has the
option to attach a bytestring which will be passed on the process'
standard input.

Additionally, the standard output and error of the process are captured
and printed in CCF's log. This makes it a lot easier to monitor and
diagnose issues in production.
@plietar plietar requested a review from a team February 23, 2023 16:09
@plietar plietar changed the title Add processing of host process stdio. Add support for host process stdio. Feb 23, 2023
@ghost
Copy link

ghost commented Feb 23, 2023

plietar/host-proc-stdio@66100 aka 20230228.39 vs main ewma over 20 builds from 65395 to 66092

Click to see table

main

build_id build_number Commit latency factor tpcc_virtual_cft^ tpcc_sgx_cft^ tpcc_sgx_cft_mem ls_virtual_cft^ pi_ls_virtual_cft^ ls_jwt_virtual_cft^ pi_ls_jwt_virtual_cft^ ls_js_virtual_cft^ ls_sgx_cft^ ls_sgx_cft_mem pi_ls_sgx_cft^ pi_ls_sgx_cft_mem ls_full_js_virtual_cft^ ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem pi_ls_jwt_sgx_cft^ pi_ls_jwt_sgx_cft_mem ls_js_jwt_virtual_cft^ ls_js_sgx_cft^ ls_js_sgx_cft_mem hist_sgx_cft^ ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
65395 20230220.69 0.787055 17201.2 6303.67 8.1662e+07 43790.5 49701.8 12379.1 13052.3 4630.77 15805.6 1.50774e+07 16136.2 7.99955e+06 3804.63 5503.63 1.45532e+07 5671.3 4.06739e+06 3441.09 1528.03 7.73741e+06 45845.4 1339.44 7.21312e+06 1247.67 6.95098e+06 831800 1.17776e+06 8.15576e+06 3.11488e+07
65405 20230221.3 0.817252 17147.1 6348.16 8.1662e+07 43675.1 48039.3 13048.9 13008.5 4413.92 15855.9 1.50774e+07 16148.2 7.99955e+06 3831.64 5511.03 1.45532e+07 5672 4.06739e+06 3318.21 1529.87 7.47526e+06 48277.8 1342.99 7.21312e+06 1265.42 6.95098e+06 835213 1.17709e+06 8.15397e+06 3.20631e+07
65441 20230221.17 0.80644 17170.1 5674.58 8.24484e+07 45602.1 49340.8 12330.1 13749.7 4662.92 15754.4 1.50774e+07 16062.6 7.99955e+06 3836.97 5542.41 1.48153e+07 5669.8 4.06739e+06 3325.78 1522.74 7.73741e+06 44387.9 1337.52 7.21312e+06 1241.45 6.95098e+06 836782 1.17219e+06 8.13651e+06 3.07961e+07
65456 20230222.2 0.787913 17248.7 6308.62 8.21863e+07 45771.5 48055.9 12465.5 13974.3 4548.91 15730.7 1.50774e+07 15444.5 7.99955e+06 3893.18 5480.3 1.48153e+07 5664.5 4.06739e+06 3348.55 1533.77 7.73741e+06 46506.4 1341.05 7.21312e+06 1254.32 6.95098e+06 825679 1.17963e+06 8.14933e+06 3.08109e+07
65467 20230222.8 0.814029 16922.3 6314.31 8.24484e+07 43920.7 49814.5 12640.3 13756.5 4550.95 15550.4 1.50774e+07 16096.8 7.99955e+06 3754.76 5596.93 1.48153e+07 5669.3 4.06739e+06 3419.78 1528.11 7.99955e+06 47792.7 1336.69 7.99955e+06 1249.71 6.95098e+06 817630 1.17699e+06 8.17395e+06 3.26432e+07
65492 20230222.16 0.792438 17326.4 5572.44 8.24484e+07 45738.7 48222.5 12275.5 13721.5 4400.76 15412.2 1.50774e+07 16032.5 7.99955e+06 3681.37 5361.95 1.50774e+07 5633.5 4.06739e+06 3290.5 1537.4 7.99955e+06 47353.8 1319.06 7.73741e+06 1253.28 6.95098e+06 832455 1.18002e+06 8.17297e+06 3.07291e+07
65534 20230223.2 0.795707 17129.5 6259.09 8.19242e+07 45774.3 46241.5 12759.4 12968 4238.13 15727.7 1.50774e+07 16064.3 7.99955e+06 3772.11 5552.7 1.48153e+07 5666.2 4.06739e+06 3431.46 1532.9 7.73741e+06 44436.5 1339.44 7.73741e+06 1248.39 6.95098e+06 834169 1.17443e+06 8.17164e+06 3.11948e+07
65696 20230224.3 0.800575 17207.2 6272.76 8.21863e+07 45541.5 47574.9 12405.1 13902 4397.4 15749.8 1.50774e+07 16111.6 7.99955e+06 3826.36 5534.75 1.48153e+07 5683 4.06739e+06 3400.41 1528.4 7.73741e+06 44456.5 1341.63 7.21312e+06 1252.52 6.95098e+06 834055 1.18116e+06 8.1567e+06 3.08643e+07
65705 20230224.6 0.78948 17268.8 6241.22 8.24484e+07 43549.7 47509.5 12856.2 13058.9 4410.34 15498.5 1.53396e+07 16064.6 7.99955e+06 3800.5 5503.39 1.48153e+07 5677.4 4.06739e+06 3306.36 1540.18 7.73741e+06 48308.6 1339.13 7.21312e+06 1263.03 6.95098e+06 833135 1.17628e+06 8.13554e+06 3.07438e+07
65748 20230224.16 0.783917 17175.6 6234 8.21863e+07 43876.7 48773.6 12408.3 13732.1 4638.57 15783 1.50774e+07 16132.6 7.99955e+06 3844.13 5525.08 1.45532e+07 5703.2 4.06739e+06 3343.87 1541.66 7.73741e+06 43140.2 1342.66 7.21312e+06 1261.05 6.95098e+06 838176 1.17692e+06 8.14914e+06 3.11625e+07
65835 20230227.2 0.803495 17122.6 6323.57 8.1662e+07 43856.6 47985.6 12298.6 12642.2 4505.39 15698.9 1.50774e+07 16124.5 7.99955e+06 3736.13 5482.34 1.45532e+07 5673.2 4.06739e+06 3320.5 1541.99 7.73741e+06 44280.5 1338.53 7.21312e+06 1254.22 6.95098e+06 841984 1.17778e+06 8.15397e+06 3.08049e+07
65837 20230227.4 0.80932 17317.2 6274.57 8.21863e+07 43644.7 48657.8 12696.7 13023.4 4487.02 15811.2 1.50774e+07 16164.9 7.99955e+06 3801.1 5506.78 1.48153e+07 5671.4 4.59168e+06 3329.78 1536.88 7.73741e+06 47709.5 1343.79 7.21312e+06 1249.28 7.21312e+06 830710 1.17551e+06 8.17418e+06 3.08127e+07
65847 20230227.7 0.798798 17186.7 6273.56 8.24484e+07 45859.3 47941.4 12897.6 13185.3 4348.5 15827.4 1.50774e+07 16108.4 7.99955e+06 3692.98 5478.28 1.48153e+07 5675.2 4.06739e+06 3437.49 1534.74 7.73741e+06 45788.5 1333.8 7.21312e+06 1251.3 6.95098e+06 832584 1.1833e+06 8.15436e+06 3.18919e+07
65877 20230227.15 0.79398 17349.5 5664.91 8.24484e+07 43746.6 47761.4 12366.4 13180.4 4366.45 15732.6 1.50774e+07 16079 7.99955e+06 3665.95 5530.31 1.45532e+07 5674.3 4.06739e+06 3325.45 1538.77 7.73741e+06 44624.8 1344.3 7.73741e+06 1250.19 6.95098e+06 836973 1.18009e+06 8.15556e+06 3.07069e+07
65885 20230227.17 0.797773 17242.7 5652.56 8.27106e+07 45737.2 47533.3 12865.5 13860.4 4380.68 15799.3 1.50774e+07 16094.7 7.99955e+06 3830.6 5507.5 1.48153e+07 5639.8 4.32954e+06 3324.1 1526.82 7.73741e+06 44211.7 1328.99 7.73741e+06 1242 6.95098e+06 835814 1.17601e+06 8.15193e+06 3.08035e+07
65937 20230227.30 0.784716 17181.1 6278.88 8.19242e+07 45718.6 48568 12842.8 13304.8 4331.77 15823.1 1.50774e+07 16139.9 7.99955e+06 3715.11 5499.37 1.4291e+07 5675.5 4.06739e+06 3294.52 1542.13 7.73741e+06 48387.5 1336.15 7.47526e+06 1257.2 6.95098e+06 836085 1.17634e+06 8.14794e+06 3.10614e+07
65959 20230227.36 0.802885 17164.4 6224.99 8.24484e+07 43586.7 49062.4 12863 13760.4 4338.4 15775 1.50774e+07 16045.3 7.99955e+06 3831.55 5492.28 1.45532e+07 5613 4.06739e+06 3344.51 1515.56 7.73741e+06 43900.1 1323.89 7.47526e+06 1237.88 7.21312e+06 833464 1.17411e+06 8.15468e+06 3.07364e+07
65995 20230228.2 0.784824 16945.8 6288 8.21863e+07 43743.6 46799.4 12848.9 13773.8 4343.54 15786.6 1.50774e+07 16143 7.99955e+06 3822.45 5521.67 1.50774e+07 5677.9 4.06739e+06 3311.98 1532.31 7.73741e+06 44237.9 1343.93 7.73741e+06 1256.92 6.95098e+06 839395 1.18029e+06 8.14849e+06 3.0649e+07
66071 20230228.30 0.802223 17135.2 5655.46 8.24484e+07 45675.4 39401.2 12916.7 12979.4 4361.08 15490.5 1.50774e+07 16017.8 7.99955e+06 3826.06 5499.37 1.48153e+07 5606 4.06739e+06 3321.86 1519.61 7.73741e+06 45149.3 1337.08 7.21312e+06 1239.01 6.95098e+06 840187 1.18113e+06 8.15472e+06 3.09249e+07
66092 20230228.37 0.785347 17133.6 6294.14 8.1662e+07 41993.5 45970.7 12273.4 13718.8 4308.59 15811.2 1.50774e+07 16157.3 7.99955e+06 3820.91 5509.63 1.45532e+07 5847.5 4.59168e+06 3298.95 1541.24 7.73741e+06 43848.3 1345.88 7.73741e+06 1262.56 6.95098e+06 601730 1.03303e+06 8.01726e+06 1.684e+07

plietar/host-proc-stdio

build_id build_number Commit latency factor tpcc_virtual_cft^ ls_virtual_cft^ pi_ls_virtual_cft^ ls_jwt_virtual_cft^ pi_ls_jwt_virtual_cft^ tpcc_sgx_cft^ tpcc_sgx_cft_mem ls_js_virtual_cft^ ls_full_js_virtual_cft^ ls_js_jwt_virtual_cft^ ls_sgx_cft^ ls_sgx_cft_mem pi_ls_sgx_cft^ pi_ls_sgx_cft_mem ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem pi_ls_jwt_sgx_cft^ pi_ls_jwt_sgx_cft_mem hist_sgx_cft^ ls_js_sgx_cft^ ls_js_sgx_cft_mem ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
65915 20230227.25 0.79221 17156.2 45643.9 48628.9 12601.6 12836.6 5664.17 8.29727e+07 4591.41 3664.58 3328.38 15725.4 1.50774e+07 16074.7 7.73741e+06 5502.59 1.45532e+07 5649.5 4.06739e+06 45839.6 1531.83 7.73741e+06 1336.78 7.47526e+06 1243.6 6.95098e+06 837844 1.18357e+06 8.17033e+06 3.07198e+07
65988 20230227.47 0.799104 17154.5 43912.6 47892.3 12758.2 12636.9 6322.34 8.19242e+07 4450.75 3786.44 3307.83 15804.8 1.50774e+07 16153 7.99955e+06 5519.53 1.48153e+07 5667.2 4.06739e+06 45726.3 1532.55 7.73741e+06 1341.11 7.47526e+06 1263.99 6.95098e+06 841004 1.1764e+06 8.15228e+06 3.08378e+07
66064 20230228.27 0.811008 17142.1 45701.8 47934.5 12435.3 13413.3 6309.59 8.1662e+07 4642.19 3834.08 3311.61 15884.6 1.53396e+07 16207.7 7.99955e+06 5487.31 1.45532e+07 5674.8 4.06739e+06 46664.3 1542.6 7.73741e+06 1349.6 7.47526e+06 1260.18 6.95098e+06 805732 1.17885e+06 8.15598e+06 3.0842e+07
66100 20230228.39 0.781359 17751.5 47809.5 47837.3 12844.5 12769.1 6308.58 8.21863e+07 4499.28 3653.19 3301 15963.8 1.53396e+07 16217.3 7.99955e+06 5767.4 1.45532e+07 5709.1 4.06739e+06 46375.8 1542.68 7.73741e+06 1339.12 7.21312e+06 1264.95 6.95098e+06 833795 1.17657e+06 8.13703e+06 3.076e+07

images

@plietar
Copy link
Contributor Author

plietar commented Feb 23, 2023

launch_host_process_test is marked as a LONG_TEST, so I've triggered a Daily CI.

I'm not sure if it really needs be marked as such, since it isn't particularly long to run from what I can tell, but I don't really know what the criteria is.

@plietar plietar added auto-backport Automatically backport this PR to LTS branch 3.x-todo PRs which should be backported to 3.x labels Feb 23, 2023
@achamayou
Copy link
Member

launch_host_process_test is marked as a LONG_TEST, so I've triggered a Daily CI.

I'm not sure if it really needs be marked as such, since it isn't particularly long to run from what I can tell, but I don't really know what the criteria is.

if it's less than 30 seconds, please remove that condition.

If writing as two separate commands, the Python script could read the
partial file, leading to a test failure.
Test was marked as LONG_TESTS, meaning it only ran on the Daily CI.
It seems to run pretty fast (<10 seconds), so this seems unnecessary.
@plietar
Copy link
Contributor Author

plietar commented Feb 24, 2023

It's about 10 seconds, so I've changed it

@plietar plietar force-pushed the plietar/host-proc-stdio branch from 613dfc4 to e0f5b1e Compare February 27, 2023 13:49
@plietar plietar requested a review from achamayou February 27, 2023 14:44
@achamayou
Copy link
Member

@plietar please add an entry to the changelog if you are going to backport this. Also note the auto-backporting is currently broken due to a restriction put in place on the org that forbids PR creation from GH actions unfortunately. You will need to run the backport CLI.

@achamayou achamayou enabled auto-merge (squash) February 28, 2023 21:23
@achamayou achamayou merged commit 9461128 into main Feb 28, 2023
@achamayou achamayou deleted the plietar/host-proc-stdio branch February 28, 2023 22:12
@github-actions
Copy link

💔 All backports failed

Status Branch Result
release/3.x Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

backport --pr 5056

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

@plietar
Copy link
Contributor Author

plietar commented Mar 1, 2023

💔 All backports failed

Status Branch Result
release/3.x An unhandled error occurred. Please see the logs for details

Manual backport

To create the backport manually run:

backport --pr 5056

Questions ?

Please refer to the Backport tool documentation

1 similar comment
@plietar
Copy link
Contributor Author

plietar commented Mar 1, 2023

💔 All backports failed

Status Branch Result
release/3.x An unhandled error occurred. Please see the logs for details

Manual backport

To create the backport manually run:

backport --pr 5056

Questions ?

Please refer to the Backport tool documentation

plietar added a commit to plietar/CCF that referenced this pull request Mar 1, 2023
(cherry picked from commit 9461128)

# ------------------------ >8 ------------------------
# Do not modify or remove the line above.
# Everything below it will be ignored.
#
# Conflicts:
#	.daily_canary
#	CHANGELOG.md
wintersteiger pushed a commit to wintersteiger/CCF that referenced this pull request Mar 1, 2023
@shokouedamsr shokouedamsr added the backported This PR was successfully backported to LTS branch label Mar 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3.x-todo PRs which should be backported to 3.x auto-backport Automatically backport this PR to LTS branch backported This PR was successfully backported to LTS branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants