Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Permit Trace+Debug logging in all non-SGX builds #5375

Merged
merged 25 commits into from
Jul 7, 2023

Conversation

eddyashton
Copy link
Member

@eddyashton eddyashton commented Jun 16, 2023

We previously (#2404, and related follow-ups) made it so that logging verbosity was controlled at compile time. This is necessary for SGX, but not for other platforms. This PR loosens that behaviour, restoring a run-time logging.enclave_level configuration option that allows the enclave's verbosity to be controlled at launch. We still restrict the options which are available (and which are compiled into the binary), but only on SGX. On other platforms, we think it is safe to have a single build, which contains debug logging, and reliably disable that at launch (in an attested config).

This gives a few big wins for Virtual and SNP:

  • The release artifacts can be used for deeper debugging, as users can pass now pass --enclave-log-level to sandbox.sh to get verbose enclave logging (previously only available in the sgx_unsafe artifact).
  • Enabling and disabling CMake's VERBOSE_LOGGING on these platforms doesn't change the compile definitions, only the args passed to e2e tests. So you can quickly enable verbose logging to get more debug information, without a slow rebuild.

To avoid too many annoying changes/duplicated checks, I've retained the original definition names. This means there's potential confusion - there's a VERBOSE_LOGGING CMake option and a VERBOSE_LOGGING C++ preprocessor definition, but the latter is always-present unless we're on SGX. It may be worth renaming the latter to UNSAFE_LOGGING or something to avoid that confusion. VERBOSE_LOGGING is the CMake option, which also affects the runtime arg used by the e2e tests. If we're building for SGX and don't have VERBOSE_LOGGING, we add the compile definition CCF_DISABLE_VERBOSE_LOGGING, which is the inverse of the previously preproc def VERBOSE_LOGGING

ETA - This option is now passed as a CLI arg to cchost, where it should be covered by attestation, rather than in a potentially-unattested config file

@ghost
Copy link

ghost commented Jun 16, 2023

verbose_runtime_option@72660 aka 20230707.5 vs main ewma over 20 builds from 72383 to 72651

Click to see table

main

build_id build_number Commit latency factor ls_sgx_cft^ ls_sgx_cft_mem pi_ls_sgx_cft^ pi_ls_sgx_cft_mem pi_basic_sgx_cft^ pi_basic_sgx_cft_mem ls_virtual_cft^ ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem pi_ls_virtual_cft^ pi_basic_virtual_cft^ pi_ls_jwt_sgx_cft^ pi_ls_jwt_sgx_cft_mem ls_jwt_virtual_cft^ pi_ls_jwt_virtual_cft^ ls_js_virtual_cft^ ls_js_sgx_cft^ ls_js_sgx_cft_mem pi_basic_mt_virtual_cft^ pi_basic_mt_sgx_cft^ pi_basic_mt_sgx_cft_mem ls_full_js_virtual_cft^ ls_js_jwt_virtual_cft^ ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem hist_sgx_cft^ RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
72383 20230627.12 0.794654 19964.7 1.88908e+07 20174.9 1.25993e+07 22543.6 1.25993e+07 45741.5 6701.58 1.88908e+07 47328.1 54938.4 6508.1 6.30784e+06 12362.6 12833.8 4454.63 1744.34 1.05021e+07 60358.8 35562.3 2.51822e+07 3607.45 3230.74 1444.2 1.05021e+07 1432.1 1.05021e+07 48417.9 802075 1.1792e+06 8.17395e+06 3.07203e+07
72395 20230627.16 0.806536 19985.5 1.88908e+07 20126.5 1.25993e+07 22658.2 1.25993e+07 45773.8 6710.79 1.67936e+07 47459.1 54702.5 6562.6 6.30784e+06 12357.5 12867.5 4453.73 1745.46 1.05021e+07 55762.3 35673.8 2.51822e+07 3572.03 3380.15 1444.46 1.05021e+07 1425.05 1.05021e+07 48982.2 833741 1.17895e+06 8.15533e+06 3.08021e+07
72403 20230627.18 0.826649 19964 1.88908e+07 20263 1.25993e+07 22603 1.25993e+07 45807.8 6675.78 1.67936e+07 47641.5 56010.8 6522 6.30784e+06 12287.1 12818.2 4457.29 1745.29 1.05021e+07 57173.4 35745.4 2.51822e+07 3585.18 3377.04 1437.24 1.05021e+07 1432.06 1.05021e+07 48551.5 835700 1.17615e+06 8.17297e+06 3.07166e+07
72414 20230627.22 0.782261 19974.5 1.88908e+07 20296 1.05021e+07 22888.8 1.25993e+07 45570.4 6707.83 1.67936e+07 47412.7 55507.9 6508 6.30784e+06 12445.7 13048.7 4465.08 1762.63 1.05021e+07 58014.7 35567.1 2.30851e+07 3567.14 3392.01 1440.21 1.05021e+07 1426.65 1.05021e+07 48397.2 827058 1.18005e+06 8.17493e+06 3.06582e+07
72421 20230628.1 0.761085 19985.9 1.88908e+07 20171.1 1.25993e+07 22860.3 1.25993e+07 45793.6 6344.03 1.67936e+07 47897.3 55456.6 6568.5 6.30784e+06 12400.6 12990.9 4423.61 1751.62 1.05021e+07 63085.1 35513.1 2.30851e+07 3588.42 3303.7 1440.1 1.05021e+07 1429.72 1.05021e+07 43309.5 834665 1.17866e+06 8.15209e+06 3.07646e+07
72436 20230628.7 0.812082 19941.7 1.88908e+07 20191.5 1.25993e+07 22946.5 1.25993e+07 45756.4 6699.02 1.67936e+07 47828.3 55452.4 6519.4 6.30784e+06 12566.5 12921.1 4468.19 1762.47 1.05021e+07 60489.4 35354.6 2.51822e+07 3464.73 3378.16 1440.87 1.05021e+07 1431.31 1.05021e+07 45121.5 832343 1.18084e+06 8.17477e+06 3.10873e+07
72446 20230629.1 0.79726 19984.4 1.88908e+07 20130.2 1.25993e+07 22794.7 1.25993e+07 45706.1 6313.16 1.67936e+07 47595.5 55593.5 6476.1 6.30784e+06 12499.6 12672.7 4433.8 1743.35 1.05021e+07 65902.1 35235.7 2.51822e+07 3552.85 3364.21 1443.8 1.05021e+07 1431.7 1.05021e+07 47859.6 835992 1.17401e+06 8.13615e+06 3.2025e+07
72457 20230629.7 0.836172 19949.1 1.88908e+07 20124.7 1.05021e+07 22775.9 1.25993e+07 45903.5 6342.9 1.67936e+07 47464.2 55539.9 6510.9 6.30784e+06 12526.4 12919.9 4502.49 1746.41 1.05021e+07 56033.3 35999.6 2.51822e+07 3579.1 3257.95 1438.67 1.05021e+07 1426.47 1.05021e+07 50359.7 836001 1.18336e+06 8.17415e+06 3.0736e+07
72470 20230629.11 0.820991 19845.1 1.88908e+07 20161.5 1.25993e+07 22899.9 1.25993e+07 45669.2 6689.48 1.67936e+07 47537.5 55455.4 6470.9 6.30784e+06 12501.6 12979.3 4437.55 1745.17 1.05021e+07 64353.8 35582.9 2.51822e+07 3596.63 3385.29 1439.56 1.05021e+07 1426.72 1.05021e+07 48706.1 823851 1.18212e+06 8.13541e+06 3.15276e+07
72480 20230630.1 0.814747 19851.3 1.88908e+07 20203.8 1.25993e+07 22846.9 1.25993e+07 45723.9 6322.54 1.67936e+07 47528.8 55293.2 6510.5 6.30784e+06 12412.7 12703.4 4467.85 1737.33 1.05021e+07 55751.5 35599.4 2.51822e+07 3593.74 3383.35 1436.68 1.05021e+07 1428.85 1.05021e+07 42801.9 835136 1.17788e+06 8.12457e+06 3.08727e+07
72492 20230703.1 0.803381 19929 1.88908e+07 20125.3 1.25993e+07 22911.1 1.25993e+07 45839.6 6700.23 1.67936e+07 47420.8 52483.2 6462.7 6.30784e+06 12422.9 13003.4 4475.51 1748.42 1.05021e+07 65525.6 35541.5 2.51822e+07 3516.86 3256.88 1447.19 1.05021e+07 1433.01 1.05021e+07 48601.7 838528 1.18092e+06 8.13619e+06 3.0689e+07
72511 20230704.1 0.788467 20003 1.88908e+07 20249 1.25993e+07 23002.4 1.25993e+07 45730.3 6751.03 1.67936e+07 47716.5 55439.3 6565.7 6.30784e+06 12517.3 12827.1 4421.76 1766.68 1.05021e+07 66684.1 35778.7 2.51822e+07 3576.87 3215.16 1440.13 1.05021e+07 1434.55 1.05021e+07 48681.3 837219 1.1837e+06 8.15452e+06 3.09412e+07
72518 20230704.5 0.810358 19990.7 1.88908e+07 20224 1.25993e+07 22996 1.25993e+07 43590.3 6382.67 1.67936e+07 46999.3 54920.3 6515 6.30784e+06 12293.2 12863.1 4484.94 1745.43 1.05021e+07 54985.1 35577 2.51822e+07 3512.36 3296.63 1441.45 1.05021e+07 1433.36 1.05021e+07 43890.3 831120 1.17226e+06 8.15537e+06 3.0718e+07
72535 20230704.11 0.809873 19722.5 1.88908e+07 20094.7 1.25993e+07 22844.3 1.25993e+07 45757.3 6345.98 1.67936e+07 48362.4 55737.5 6424.8 6.30784e+06 12575.9 13184.8 4460.99 1735.62 1.05021e+07 55285.3 35866.1 2.51822e+07 3571.1 3230.93 1439.94 1.05021e+07 1427.56 1.05021e+07 51096.9 834507 1.17981e+06 8.15416e+06 3.07933e+07
72542 20230704.13 0.795832 19825.5 1.88908e+07 20265.3 1.05021e+07 23013.5 1.25993e+07 45881.4 6723.21 1.67936e+07 47876 55818.5 6469.8 6.30784e+06 12428.3 13009 4478.04 1744.77 1.05021e+07 75747.7 35761.7 2.30851e+07 3601.03 3416.26 1441.13 1.05021e+07 1431.19 1.05021e+07 48649.2 842097 1.17452e+06 8.17405e+06 3.08429e+07
72563 20230705.1 0.837403 19978.7 1.88908e+07 20194.9 1.25993e+07 22974.2 1.25993e+07 45762.3 6680.38 1.67936e+07 48395.9 55921.1 6525 6.30784e+06 12433 12883.3 4441 1742.76 1.05021e+07 63968.2 36005.7 2.51822e+07 3585.04 3387.97 1443.46 1.05021e+07 1433.91 1.05021e+07 48254.6 830201 1.18341e+06 8.17457e+06 3.08369e+07
72581 20230706.1 0.802693 20043.9 1.88908e+07 20253 1.05021e+07 23064.9 1.25993e+07 45757.9 6352.51 1.67936e+07 47804.3 55664.2 6520.5 6.30784e+06 12402.1 13046.5 4507.57 1747.78 1.05021e+07 57112.4 35948.3 2.51822e+07 3493.88 3301.02 1442.92 1.05021e+07 1429.37 1.05021e+07 48268.1 836151 1.18164e+06 8.15267e+06 3.08392e+07
72595 20230706.7 0.812809 19948.3 1.88908e+07 20174.7 1.25993e+07 22892.5 1.25993e+07 45751.5 6347.59 1.67936e+07 48020.4 55730.9 6513.6 6.30784e+06 12373.3 13088.3 4502.69 1745.91 1.05021e+07 54274.5 35566.7 2.51822e+07 3617.8 3329.05 1444.13 1.05021e+07 1431.13 1.05021e+07 43367.1 832506 1.1805e+06 8.15449e+06 3.08044e+07
72636 20230706.20 0.812909 19788.9 1.88908e+07 19954.7 1.25993e+07 22943 1.25993e+07 45879.2 6353.47 1.67936e+07 47828.4 55187 6433.5 6.30784e+06 12330.4 13116.5 4491.17 1746.38 1.05021e+07 55356.9 35630.6 2.51822e+07 3607.09 3294.02 1437.03 1.05021e+07 1414.03 1.05021e+07 44092.4 836826 1.18144e+06 8.15641e+06 3.18349e+07
72651 20230707.1 0.781317 19916.8 1.88908e+07 20104.6 1.25993e+07 22980.2 1.25993e+07 45641.9 6684.11 1.67936e+07 47893.2 55811.3 6569.2 6.30784e+06 12511.9 12907.1 4454.64 1763.19 1.05021e+07 63504.1 35620.8 2.51822e+07 3579.39 3411.35 1441.46 1.05021e+07 1433.71 1.05021e+07 48041.1 839165 1.18029e+06 8.15481e+06 3.10807e+07

verbose_runtime_option

build_id build_number Commit latency factor pi_basic_mt_sgx_cft^ pi_basic_mt_sgx_cft_mem ls_virtual_cft^ pi_ls_virtual_cft^ pi_basic_virtual_cft^ ls_sgx_cft^ ls_sgx_cft_mem ls_jwt_virtual_cft^ pi_ls_sgx_cft^ pi_ls_sgx_cft_mem pi_ls_jwt_virtual_cft^ pi_basic_sgx_cft^ pi_basic_sgx_cft_mem ls_js_virtual_cft^ ls_jwt_sgx_cft^ ls_jwt_sgx_cft_mem ls_full_js_virtual_cft^ pi_ls_jwt_sgx_cft^ pi_ls_jwt_sgx_cft_mem ls_js_jwt_virtual_cft^ pi_basic_mt_virtual_cft^ ls_js_sgx_cft^ ls_js_sgx_cft_mem hist_sgx_cft^ ls_full_js_sgx_cft^ ls_full_js_sgx_cft_mem ls_js_jwt_sgx_cft^ ls_js_jwt_sgx_cft_mem RB put (/s)^ CHAMP put (/s)^ RB get (/s)^ CHAMP get (/s)^
72622 20230706.14 0.814561 35714.2 2.51822e+07 45634.3 47371.1 55410.6 19913.2 1.88908e+07 12516.3 20147.5 1.25993e+07 12817.9 22927.8 1.25993e+07 4499.48 6693.51 1.67936e+07 3470.98 6514.8 6.30784e+06 3303.91 72946.9 1745.11 1.05021e+07 50916.9 1439.95 1.05021e+07 1423.92 1.05021e+07 832656 1.16982e+06 8.15397e+06 3.13875e+07
72646 20230706.22 0.808802 35631.8 2.51822e+07 45345.5 47705.6 54869.8 19898.9 1.88908e+07 12358.9 20232.4 1.25993e+07 12931.3 22716.8 1.25993e+07 4483.58 6685.01 1.67936e+07 3456.83 6510.8 6.30784e+06 3285.07 74320.1 1743.27 1.05021e+07 48469.9 1443.39 1.05021e+07 1432.66 1.05021e+07 835146 1.17508e+06 8.15228e+06 3.11015e+07
72648 20230706.23 0.80543 35378.2 2.51822e+07 45641.5 48093.6 54742 19899.1 1.88908e+07 12538.7 20061.9 1.25993e+07 12923.6 22725.1 1.25993e+07 4488.03 6317.62 1.67936e+07 3486.76 6468.1 6.30784e+06 3319.06 60055.9 1745.37 1.05021e+07 47677.6 1444.8 1.05021e+07 1423.23 1.05021e+07 834134 1.17592e+06 8.1517e+06 3.08485e+07
72660 20230707.5 0.852555 35470 2.51822e+07 45756.9 47725.8 54825.8 20109.2 1.88908e+07 12506 20246.2 1.25993e+07 12686.2 22946.1 1.25993e+07 4507.14 6744.52 1.67936e+07 3547.04 6566.6 6.30784e+06 3300.37 62785.5 1769.18 1.05021e+07 43512.8 1442.85 1.05021e+07 1374.13 1.05021e+07 833112 1.17588e+06 8.15481e+06 3.07328e+07

images

@eddyashton eddyashton marked this pull request as ready for review June 19, 2023 12:29
@eddyashton eddyashton requested a review from a team June 19, 2023 12:29
@eddyashton eddyashton added auto-backport Automatically backport this PR to LTS branch 4.x-todo PRs which should be backported to 4.x labels Jun 19, 2023
@achamayou
Copy link
Member

@eddyashton I am worried about bundling this in with items of configuration such as directories, IPs, domain names and ports, which may be node-specific. Even if the security policy correctly captures the configuration (which isn't a given, since existing implementations typically do not allow measuring mounts, except for the empty special case), it will differ by node, and effectively require that each node addition be preceded by governance to add the corresponding measurement.

It seems to me that the requirements for this bit of configuration are that:

  1. it is attested
  2. it does not vary per node, to avoid governance churn

A generic solution would be to create an attested configuration, which must come from somewhere that's measured (ie. the container right now, I don't believe there are other practical options). That's still relatively easy to get wrong.

A less generic, but low impact solution, is to make this a CLI argument (either directly, or add a CLI enclave log-level cap). CLI arguments are well captured by the security policy, it is difficult to get that wrong.

@eddyashton eddyashton marked this pull request as ready for review July 5, 2023 16:50
@eddyashton
Copy link
Member Author

@achamayou I've updated this PR, and the option is now passed on the CLI rather than in the JSON configuration file. Please re-review.

@achamayou
Copy link
Member

It may be worth renaming the latter to UNSAFE_LOGGING or something to avoid that confusion.

We should also take the opportunity to CCF_-prefix.

CMakeLists.txt Outdated Show resolved Hide resolved
@eddyashton
Copy link
Member Author

@achamayou I've renamed to separate the CMake option and the preprocessor definition. CMake's VERBOSE_LOGGING stays with the same name - this also affects how verbose the e2e tests are, so I figure its useful to retain the muscle memory here (with the win that it no longer requires a rebuild).

@achamayou
Copy link
Member

@eddyashton a changelog entry is probably worthwhile for this change.

@eddyashton eddyashton merged commit cd33041 into microsoft:main Jul 7, 2023
@ghost
Copy link

ghost commented Jul 7, 2023

💔 All backports failed

Status Branch Result
release/4.x Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

backport --pr 5375

Questions ?

Please refer to the Backport tool documentation and see the Github Action logs for details

@achamayou achamayou added auto-backport Automatically backport this PR to LTS branch 4.x-todo PRs which should be backported to 4.x and removed auto-backport Automatically backport this PR to LTS branch 4.x-todo PRs which should be backported to 4.x labels Jul 7, 2023
ghost pushed a commit that referenced this pull request Jul 7, 2023
@ghost ghost added the backported This PR was successfully backported to LTS branch label Jul 7, 2023
achamayou pushed a commit that referenced this pull request Jul 7, 2023
…builds (#5375) (#5424)

Co-authored-by: Eddy Ashton <edashton@microsoft.com>
@eddyashton eddyashton mentioned this pull request Jul 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
4.x-todo PRs which should be backported to 4.x auto-backport Automatically backport this PR to LTS branch backported This PR was successfully backported to LTS branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants