-
Notifications
You must be signed in to change notification settings - Fork 127
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Digest function now auto-detected from request #899
Digest function now auto-detected from request #899
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is dependent on the previous PR landing first.
Reviewable status: 0 of 1 LGTMs obtained, and pending CI: Analyze (javascript-typescript), Analyze (python), Bazel Dev / ubuntu-22.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Local / ubuntu-22.04, Publish image, Publish nativelink-worker-lre-cc, Remote / large-ubuntu-22.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04), integration-tests (20.04), integration-tests (22.04), macos-13, pre-commit-checks, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, vale, windows-2022 / stable (waiting on @adam-singer)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 11 of 11 files at r1, 12 of 12 files at r2, all commit messages.
Reviewable status: complete! 1 of 1 LGTMs obtained
7d93867
to
6c33e74
Compare
6c33e74
to
01a86dd
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is the final form.
Local testing shows it working, but still need more larger testing.
I can break this into 1 maybe 2 PRs if you prefer, but most of this is related.
Reviewable status: 1 of 1 LGTMs obtained, and pending CI: Vercel, pre-commit-checks
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 34 of 34 files at r3, all commit messages.
Reviewable status: 1 of 1 LGTMs obtained, and pending CI: Remote / large-ubuntu-22.04, and 1 discussions need to be resolved
nativelink-scheduler/tests/cache_lookup_scheduler_test.rs
line 119 at r3 (raw file):
let action_name = ActionInfoHashKey { instance_name: "instance".to_string(), digest_function: DigestHasherFunc::Sha256.proto_digest_func().try_into()?,
Why does this need to do a try_into
when the other passed ones in action_messages_test.rs
didnt?
01a86dd
to
465d536
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 1 of 1 LGTMs obtained, and pending CI: Analyze (javascript-typescript), Bazel Dev / ubuntu-22.04, Vercel, pre-commit-checks, ubuntu-20.04 / stable, ubuntu-22.04, vale, and 1 discussions need to be resolved
nativelink-scheduler/tests/cache_lookup_scheduler_test.rs
line 119 at r3 (raw file):
Previously, adam-singer (Adam Singer) wrote…
Why does this need to do a
try_into
when the other passed ones inaction_messages_test.rs
didnt?
Copy pasta failure. Fixed.
465d536
to
c6259b9
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 1 of 1 files at r4, all commit messages.
Reviewable status: 1 of 1 LGTMs obtained, and pending CI: Remote / large-ubuntu-22.04, and 1 discussions need to be resolved
nativelink-scheduler/tests/property_modifier_scheduler_test.rs
line 251 at r5 (raw file):
let action_name = ActionInfoHashKey { instance_name: "instance".to_string(), digest_function: DigestHasherFunc::Sha256.proto_digest_func().try_into()?,
nit: DigestHasherFunc::Sha256.proto_digest_func().try_into()?
-> DigestHasherFunc::Sha256
b8de3c3
to
85c190b
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 10 of 10 files at r6, all commit messages.
Reviewable status: 1 of 1 LGTMs obtained, and pending CI: Remote / large-ubuntu-22.04
85c190b
to
8a47ec3
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 1 of 1 LGTMs obtained, and pending CI: Analyze (javascript-typescript), Analyze (python), Bazel Dev / ubuntu-22.04, Cargo Dev / macos-13, Cargo Dev / ubuntu-22.04, Installation / macos-13, Installation / macos-14, Installation / ubuntu-22.04, Local / ubuntu-22.04, Publish image, Publish nativelink-worker-lre-cc, Remote / large-ubuntu-22.04, asan / ubuntu-22.04, docker-compose-compiles-nativelink (20.04), docker-compose-compiles-nativelink (22.04), integration-tests (20.04), integration-tests (22.04), macos-13, pre-commit-checks, ubuntu-20.04 / stable, ubuntu-22.04, ubuntu-22.04 / stable, vale, windows-2022 / stable
nativelink-scheduler/tests/property_modifier_scheduler_test.rs
line 251 at r5 (raw file):
Previously, adam-singer (Adam Singer) wrote…
nit:
DigestHasherFunc::Sha256.proto_digest_func().try_into()?
->DigestHasherFunc::Sha256
Done.
8a47ec3
to
a3ff0c3
Compare
The digest function the client requested is now properly implemented in the OriginContext. This allows us to inspect what digest-function the client requested from anywhere in the code. If you use `verify` store and use `hash_verification_function`, simply change it to `"verify_hash": true`. We will now auto-detect the hash function based on the request and if it is not set, we use the `default_digest_hash_function` in the `global` config.
a3ff0c3
to
f927801
Compare
…na#899) The digest function the client requested is now properly implemented in the OriginContext. This allows us to inspect what digest-function the client requested from anywhere in the code. If you use `verify` store and use `hash_verification_function`, simply change it to `"verify_hash": true`. We will now auto-detect the hash function based on the request and if it is not set, we use the `default_digest_hash_function` in the `global` config.
…na#899) The digest function the client requested is now properly implemented in the OriginContext. This allows us to inspect what digest-function the client requested from anywhere in the code. If you use `verify` store and use `hash_verification_function`, simply change it to `"verify_hash": true`. We will now auto-detect the hash function based on the request and if it is not set, we use the `default_digest_hash_function` in the `global` config.
author Nathan (Blaise) Bruer <github.blaise@allada.com> 1716060089 -0700 committer Marcus <marcuseagan@gmail.com> 1717216579 -0700 gpgsig -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapUMACgkQx/6q6dPS ZF2kPQ//ZYXhy26xNq5g2P6U7kf+dpcXok2rzntw9620kdNCLodCm7vceDvwa1mn JwVnXdGO7QcrLeSagHk6LewptncYq+hOc3ZA7wUtM3bSUTwPlwJue7zNVwpMSZHD 9pdu9CDgVWh6h7oR+JoRcyHjJ7NTkSU41aLEtwaHbDWatmtX2Cy4iX0BNk55nWaK BtzzuLLgiWrpOXNLL4MWlRdxLbN2YjbxW+2S3Q5eg7SnlERKU+COLoaHYUrk8v9J qzK35H05wDiiXHQpnWky895G3DDo/NB1znVpyOf/3sPw+OwUvACoZjORIbeh7+0W J5NqfX1ngWTtu0dVT1tehguAplVrBCWGJgJcVs03egVlOVSL/WthdkjW/ZkMfEk5 pUL9Gje8SyuyEMw3tUc8EV6owwLqqyvVsaO+baVZR5hAbxp/yDaT/fewic6OXa4X I58NE5a7w9b5Zsem5kW+yFD5GSa11sybrqffd+ofzPmmDvlVYmJ/CaQvTZ/lQpaA IIZLyPYcicC3v2I1+GA//QFPJsUlK0McW+MR7L6zlY395yvS6TTA3EoCBY5KrsYD inSfeY2IuMKjV81ihQWd56XiK289GKkvspUmjy84ytke0luqM7Xfpu3cii+gCwoq 8B0dUCJOqEKg1AJUlM4VOS7vAwc1tJHHrEm05/opIycQxkwNnhs= =6j1C -----END PGP SIGNATURE----- parent 0a33c83 author Nathan (Blaise) Bruer <github.blaise@allada.com> 1716060089 -0700 committer Marcus <marcuseagan@gmail.com> 1717216536 -0700 gpgsig -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapRgACgkQx/6q6dPS ZF1U1Q/9G9ZN2PkCUxBc+oTN1BjBbNB/52+xmi+Uao39vrGh2vPIOGYbjXPP36Vv NUQ5cckmlsoqC1VWWgJMts27Y6oSlP1JydRlgrXmhnNV/SjcAz6r2woP+3At58hk iZN0lqft7lZ1Viycq8bDDEDrT56BiGGbDqO0E3UMZncwp/4s5Cw8M+0DTapIz2PU UcxJk2goTlKwPIItlFBUF2dTPL+nxtI95rpIdaGRrP/pn7XaRuY72k5pGMVZHsS+ OVQBmXqYrSh3nUim+YWagvbDrYJU6lksI4ZDN6cnVgdEsF0xEWguCy3g6tR0WVUQ Z5Y9BkedyWt7v1tK+B5cBQgoP2m34pY8TzLFbQTLkv7mrxLIwyNqkDLVJOCPcamk dJ+yf5lp54vUkeRnVaTcp9IMEjF0Vjteg3WHTKNK8ZsBs9DnNzb+7HyjDa6bYcau e0Y5HVPuC5kcTMK95yTkAabdj1qtBr8QpJ7CErqTRBgbxpcgldd3DTNndHv5/Yjz QH/zctjfJcISk/Y+KDTBhZRsNF1VGAzpwsT3KnFUdcTg48xMFC6IBI/YkYywg4Ra ZIv6O9h0V7FivbXF5J0KqawpW4lQa6mLDzNwm7ZNODVNb5TEGYu8jsYMzkxaKVPR ZGEtFuBOPGgLVYcIbuAFeANBERzhw5KUMe7MBSP2DG7r+50pqQI= =zuq5 -----END PGP SIGNATURE----- parent 0a33c83 author Nathan (Blaise) Bruer <github.blaise@allada.com> 1716060089 -0700 committer Marcus <marcuseagan@gmail.com> 1717216503 -0700 gpgsig -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapPcACgkQx/6q6dPS ZF0xpQ/+ILuc9NSpJpS6BNw5IUyE4cQsquFZhKBTsX1l8/ZMkDZR6pKdO+gsUdKB DyA8OX8VspRAWNoxyiEfLRknwNtclr6XG1znX+WcyVjnaujcezau54dhoEAsImdn pgwfN/RpLh3ii1Z/MbpPlDzdDcYkOtDiqF2l4Cr2wQSs72o6qONUmCrycNdipyHa lED+VesZOZiZTEBOX56hMGQ/NVSPQ9hisIVjFq1J472v+opU5Ht0cu8KSCRA2lud 2DCVwpnyjIcG9Bf/XlCMzw9lODbOTqehaGsmES6CSLngjzeojhiHqABc5qYQT6Ip umWgSV2/AQsNcMG69WB+uPPT1aBwT//XnBfUoSazhGvFQLn9A7oLPLarhZGUP8M1 IND49L3OBpzbbdY3ipN9e22f4NcYg7irhy/dzLIHjtBd9CPlUTQTybmRz6KebJom 6Ha/9CXKPcaW23NqsMNlCoWqSL+cIAEvYgEWcZrceOvkeEHppI6pSLMLyWHw9epF Odd3AX8EP18axCNe+GJlCOZlRFq9cNXAYuboa0HpHhBbMP7FiVuQhapR/iWTA1Pv 2X+hjkWURzpwFz+R/ObTBMiI1YqrTHkD7fukb1ibELenCs8UeIznZ7ScV6YX72Ef rOoxoHMrDRbXsWpc3JG3TlEbYpdpiWN09sQQpktCwZtU/uh5JsQ= =DmRi -----END PGP SIGNATURE----- [Breaking] Digest function now auto-detected from request (TraceMachina#899) The digest function the client requested is now properly implemented in the OriginContext. This allows us to inspect what digest-function the client requested from anywhere in the code. If you use `verify` store and use `hash_verification_function`, simply change it to `"verify_hash": true`. We will now auto-detect the hash function based on the request and if it is not set, we use the `default_digest_hash_function` in the `global` config. Update images for docs (TraceMachina#930) Update Rust crate proc-macro2 to v1.0.84 (TraceMachina#916) Refactor EvictingMap so it does not use DigestInfo (TraceMachina#932) In prep to move to an abstract key instead of DigestInfo, this change cleans up EvictingMap and removes dependencies on DigestInfo. towards: TraceMachina#931 Update Rust crate mimalloc to v0.1.42 (TraceMachina#933) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Update Rust crate parking_lot to v0.12.3 (TraceMachina#936) Include UUID in ActionState (TraceMachina#927) Changes the field used for identifying which action an ActionState corresponds to include a uuid as well as the ActionInfoHashKey. All existing functionality is kept the same by making use of the nested ActionInfoHashKey contained within the Id. These changes will provide the basis for all usage of Id in followup changes to the scheduler. This breaks compatibility for forwarding an operation from one remote execution system to another that does not use our operation name format (ie: very unlikely, but possible). Co-authored-by: Zach Birenbaum <zach@tracemachina.com> Add Operation State Manager API (TraceMachina#937) Introduces API to be implemented by new scheduler state backends. WorkerStateManager is responsible for managing action updates sent to and from workers. ClientStateManager is responsible any state related to the clients client(s) requesting the action execution and can add or filter operations based on a given criteria. MatchingEnginerStateManager can filter, update, or remove actions. Co-authored-by: Zach Birenbaum <zach@tracemachina.com> Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Update dependency protobuf to v26.0.bcr.1 (TraceMachina#887) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Update Rust crate quote to v1.0.36 (TraceMachina#938) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Update Rust crate redis to v0.25.4 (TraceMachina#944) Update Rust crate syn to v2.0.66 (TraceMachina#946) Bump flake and Bazel modules (TraceMachina#947) Fixes TraceMachina#943 Bump trivially bumpable deps (TraceMachina#950) Notably, this changes the S3 schema for multipart upload starts/stops. Add kind-loadbalancer (TraceMachina#929) Provide an envoy loadbalancer container to access the local kind cluster. The loadbalancer adds the ability to reach the local kind cluster trough localhost, so that it is possible to work with a mac on the nativelink cluster. Docker on Mac uses a aarch64 linux vm unter the hood, so the builds in the cluster can now be tested on that architecture. Users can now query the loadbalancer instead of the gateway. The loadbalancer has the following endpoints: localhost:8080 -> hubble-ui localhost:8081 -> tekton-dashboard localhost:8082/eventlistener -> nativelink-eventlistener Fix pulumi ratelimiting build error (TraceMachina#953) On MacOS the pulumi testsuite hit the ratelimiting on github. Disable the pulumi tests with a patch on nixpkgs. Commit 0eed759 didn't fix this issue. Add drake toolchain configs (TraceMachina#942) Register metrics on PropertyModifierScheduler (TraceMachina#954) Fixes metrics being lost due to the underlying ActionScheduler not having it's metrics passed through the PropertyModifierScheduler. This should fix any scaling issues related to queued actions metrics not being available. Use single quotes for char (TraceMachina#955) Nightly clippy reports a style issue when using double quotes around a single char. Increase pre-commit timeout in CI (TraceMachina#956) The original setting of 5 minutes was a bit too flaky. Increase it to 10 minutes so that cache misses don't break the job. Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Update images for docs (TraceMachina#930) Refactor EvictingMap so it does not use DigestInfo (TraceMachina#932) In prep to move to an abstract key instead of DigestInfo, this change cleans up EvictingMap and removes dependencies on DigestInfo. towards: TraceMachina#931 Update Rust crate mimalloc to v0.1.42 (TraceMachina#933) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Update Rust crate parking_lot to v0.12.3 (TraceMachina#936) Include UUID in ActionState (TraceMachina#927) Changes the field used for identifying which action an ActionState corresponds to include a uuid as well as the ActionInfoHashKey. All existing functionality is kept the same by making use of the nested ActionInfoHashKey contained within the Id. These changes will provide the basis for all usage of Id in followup changes to the scheduler. This breaks compatibility for forwarding an operation from one remote execution system to another that does not use our operation name format (ie: very unlikely, but possible). Co-authored-by: Zach Birenbaum <zach@tracemachina.com> Add Operation State Manager API (TraceMachina#937) Introduces API to be implemented by new scheduler state backends. WorkerStateManager is responsible for managing action updates sent to and from workers. ClientStateManager is responsible any state related to the clients client(s) requesting the action execution and can add or filter operations based on a given criteria. MatchingEnginerStateManager can filter, update, or remove actions. Co-authored-by: Zach Birenbaum <zach@tracemachina.com> Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Update Rust crate redis to v0.25.4 (TraceMachina#944) Update Rust crate syn to v2.0.66 (TraceMachina#946) Bump trivially bumpable deps (TraceMachina#950) Notably, this changes the S3 schema for multipart upload starts/stops. Add kind-loadbalancer (TraceMachina#929) Provide an envoy loadbalancer container to access the local kind cluster. The loadbalancer adds the ability to reach the local kind cluster trough localhost, so that it is possible to work with a mac on the nativelink cluster. Docker on Mac uses a aarch64 linux vm unter the hood, so the builds in the cluster can now be tested on that architecture. Users can now query the loadbalancer instead of the gateway. The loadbalancer has the following endpoints: localhost:8080 -> hubble-ui localhost:8081 -> tekton-dashboard localhost:8082/eventlistener -> nativelink-eventlistener Fix pulumi ratelimiting build error (TraceMachina#953) On MacOS the pulumi testsuite hit the ratelimiting on github. Disable the pulumi tests with a patch on nixpkgs. Commit 0eed759 didn't fix this issue. Add drake toolchain configs (TraceMachina#942) Register metrics on PropertyModifierScheduler (TraceMachina#954) Fixes metrics being lost due to the underlying ActionScheduler not having it's metrics passed through the PropertyModifierScheduler. This should fix any scaling issues related to queued actions metrics not being available. Use single quotes for char (TraceMachina#955) Nightly clippy reports a style issue when using double quotes around a single char. Increase pre-commit timeout in CI (TraceMachina#956) The original setting of 5 minutes was a bit too flaky. Increase it to 10 minutes so that cache misses don't break the job.
author Nathan (Blaise) Bruer <github.blaise@allada.com> 1716060089 -0700 committer Marcus <marcuseagan@gmail.com> 1717216579 -0700 gpgsig -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapUMACgkQx/6q6dPS ZF2kPQ//ZYXhy26xNq5g2P6U7kf+dpcXok2rzntw9620kdNCLodCm7vceDvwa1mn JwVnXdGO7QcrLeSagHk6LewptncYq+hOc3ZA7wUtM3bSUTwPlwJue7zNVwpMSZHD 9pdu9CDgVWh6h7oR+JoRcyHjJ7NTkSU41aLEtwaHbDWatmtX2Cy4iX0BNk55nWaK BtzzuLLgiWrpOXNLL4MWlRdxLbN2YjbxW+2S3Q5eg7SnlERKU+COLoaHYUrk8v9J qzK35H05wDiiXHQpnWky895G3DDo/NB1znVpyOf/3sPw+OwUvACoZjORIbeh7+0W J5NqfX1ngWTtu0dVT1tehguAplVrBCWGJgJcVs03egVlOVSL/WthdkjW/ZkMfEk5 pUL9Gje8SyuyEMw3tUc8EV6owwLqqyvVsaO+baVZR5hAbxp/yDaT/fewic6OXa4X I58NE5a7w9b5Zsem5kW+yFD5GSa11sybrqffd+ofzPmmDvlVYmJ/CaQvTZ/lQpaA IIZLyPYcicC3v2I1+GA//QFPJsUlK0McW+MR7L6zlY395yvS6TTA3EoCBY5KrsYD inSfeY2IuMKjV81ihQWd56XiK289GKkvspUmjy84ytke0luqM7Xfpu3cii+gCwoq 8B0dUCJOqEKg1AJUlM4VOS7vAwc1tJHHrEm05/opIycQxkwNnhs= =6j1C -----END PGP SIGNATURE----- parent 0a33c83 author Nathan (Blaise) Bruer <github.blaise@allada.com> 1716060089 -0700 committer Marcus <marcuseagan@gmail.com> 1717216536 -0700 gpgsig -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapRgACgkQx/6q6dPS ZF1U1Q/9G9ZN2PkCUxBc+oTN1BjBbNB/52+xmi+Uao39vrGh2vPIOGYbjXPP36Vv NUQ5cckmlsoqC1VWWgJMts27Y6oSlP1JydRlgrXmhnNV/SjcAz6r2woP+3At58hk iZN0lqft7lZ1Viycq8bDDEDrT56BiGGbDqO0E3UMZncwp/4s5Cw8M+0DTapIz2PU UcxJk2goTlKwPIItlFBUF2dTPL+nxtI95rpIdaGRrP/pn7XaRuY72k5pGMVZHsS+ OVQBmXqYrSh3nUim+YWagvbDrYJU6lksI4ZDN6cnVgdEsF0xEWguCy3g6tR0WVUQ Z5Y9BkedyWt7v1tK+B5cBQgoP2m34pY8TzLFbQTLkv7mrxLIwyNqkDLVJOCPcamk dJ+yf5lp54vUkeRnVaTcp9IMEjF0Vjteg3WHTKNK8ZsBs9DnNzb+7HyjDa6bYcau e0Y5HVPuC5kcTMK95yTkAabdj1qtBr8QpJ7CErqTRBgbxpcgldd3DTNndHv5/Yjz QH/zctjfJcISk/Y+KDTBhZRsNF1VGAzpwsT3KnFUdcTg48xMFC6IBI/YkYywg4Ra ZIv6O9h0V7FivbXF5J0KqawpW4lQa6mLDzNwm7ZNODVNb5TEGYu8jsYMzkxaKVPR ZGEtFuBOPGgLVYcIbuAFeANBERzhw5KUMe7MBSP2DG7r+50pqQI= =zuq5 -----END PGP SIGNATURE----- parent 0a33c83 author Nathan (Blaise) Bruer <github.blaise@allada.com> 1716060089 -0700 committer Marcus <marcuseagan@gmail.com> 1717216503 -0700 gpgsig -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEEpvESN9ZnVKKULFG/x/6q6dPSZF0FAmZapPcACgkQx/6q6dPS ZF0xpQ/+ILuc9NSpJpS6BNw5IUyE4cQsquFZhKBTsX1l8/ZMkDZR6pKdO+gsUdKB DyA8OX8VspRAWNoxyiEfLRknwNtclr6XG1znX+WcyVjnaujcezau54dhoEAsImdn pgwfN/RpLh3ii1Z/MbpPlDzdDcYkOtDiqF2l4Cr2wQSs72o6qONUmCrycNdipyHa lED+VesZOZiZTEBOX56hMGQ/NVSPQ9hisIVjFq1J472v+opU5Ht0cu8KSCRA2lud 2DCVwpnyjIcG9Bf/XlCMzw9lODbOTqehaGsmES6CSLngjzeojhiHqABc5qYQT6Ip umWgSV2/AQsNcMG69WB+uPPT1aBwT//XnBfUoSazhGvFQLn9A7oLPLarhZGUP8M1 IND49L3OBpzbbdY3ipN9e22f4NcYg7irhy/dzLIHjtBd9CPlUTQTybmRz6KebJom 6Ha/9CXKPcaW23NqsMNlCoWqSL+cIAEvYgEWcZrceOvkeEHppI6pSLMLyWHw9epF Odd3AX8EP18axCNe+GJlCOZlRFq9cNXAYuboa0HpHhBbMP7FiVuQhapR/iWTA1Pv 2X+hjkWURzpwFz+R/ObTBMiI1YqrTHkD7fukb1ibELenCs8UeIznZ7ScV6YX72Ef rOoxoHMrDRbXsWpc3JG3TlEbYpdpiWN09sQQpktCwZtU/uh5JsQ= =DmRi -----END PGP SIGNATURE----- [Breaking] Digest function now auto-detected from request (TraceMachina#899) The digest function the client requested is now properly implemented in the OriginContext. This allows us to inspect what digest-function the client requested from anywhere in the code. If you use `verify` store and use `hash_verification_function`, simply change it to `"verify_hash": true`. We will now auto-detect the hash function based on the request and if it is not set, we use the `default_digest_hash_function` in the `global` config. Update images for docs (TraceMachina#930) Update Rust crate proc-macro2 to v1.0.84 (TraceMachina#916) Refactor EvictingMap so it does not use DigestInfo (TraceMachina#932) In prep to move to an abstract key instead of DigestInfo, this change cleans up EvictingMap and removes dependencies on DigestInfo. towards: TraceMachina#931 Update Rust crate mimalloc to v0.1.42 (TraceMachina#933) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Update Rust crate parking_lot to v0.12.3 (TraceMachina#936) Include UUID in ActionState (TraceMachina#927) Changes the field used for identifying which action an ActionState corresponds to include a uuid as well as the ActionInfoHashKey. All existing functionality is kept the same by making use of the nested ActionInfoHashKey contained within the Id. These changes will provide the basis for all usage of Id in followup changes to the scheduler. This breaks compatibility for forwarding an operation from one remote execution system to another that does not use our operation name format (ie: very unlikely, but possible). Co-authored-by: Zach Birenbaum <zach@tracemachina.com> Add Operation State Manager API (TraceMachina#937) Introduces API to be implemented by new scheduler state backends. WorkerStateManager is responsible for managing action updates sent to and from workers. ClientStateManager is responsible any state related to the clients client(s) requesting the action execution and can add or filter operations based on a given criteria. MatchingEnginerStateManager can filter, update, or remove actions. Co-authored-by: Zach Birenbaum <zach@tracemachina.com> Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Update dependency protobuf to v26.0.bcr.1 (TraceMachina#887) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Update Rust crate quote to v1.0.36 (TraceMachina#938) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Update Rust crate redis to v0.25.4 (TraceMachina#944) Update Rust crate syn to v2.0.66 (TraceMachina#946) Bump flake and Bazel modules (TraceMachina#947) Fixes TraceMachina#943 Bump trivially bumpable deps (TraceMachina#950) Notably, this changes the S3 schema for multipart upload starts/stops. Add kind-loadbalancer (TraceMachina#929) Provide an envoy loadbalancer container to access the local kind cluster. The loadbalancer adds the ability to reach the local kind cluster trough localhost, so that it is possible to work with a mac on the nativelink cluster. Docker on Mac uses a aarch64 linux vm unter the hood, so the builds in the cluster can now be tested on that architecture. Users can now query the loadbalancer instead of the gateway. The loadbalancer has the following endpoints: localhost:8080 -> hubble-ui localhost:8081 -> tekton-dashboard localhost:8082/eventlistener -> nativelink-eventlistener Fix pulumi ratelimiting build error (TraceMachina#953) On MacOS the pulumi testsuite hit the ratelimiting on github. Disable the pulumi tests with a patch on nixpkgs. Commit 0eed759 didn't fix this issue. Add drake toolchain configs (TraceMachina#942) Register metrics on PropertyModifierScheduler (TraceMachina#954) Fixes metrics being lost due to the underlying ActionScheduler not having it's metrics passed through the PropertyModifierScheduler. This should fix any scaling issues related to queued actions metrics not being available. Use single quotes for char (TraceMachina#955) Nightly clippy reports a style issue when using double quotes around a single char. Increase pre-commit timeout in CI (TraceMachina#956) The original setting of 5 minutes was a bit too flaky. Increase it to 10 minutes so that cache misses don't break the job. Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Update images for docs (TraceMachina#930) Refactor EvictingMap so it does not use DigestInfo (TraceMachina#932) In prep to move to an abstract key instead of DigestInfo, this change cleans up EvictingMap and removes dependencies on DigestInfo. towards: TraceMachina#931 Update Rust crate mimalloc to v0.1.42 (TraceMachina#933) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Update Rust crate parking_lot to v0.12.3 (TraceMachina#936) Include UUID in ActionState (TraceMachina#927) Changes the field used for identifying which action an ActionState corresponds to include a uuid as well as the ActionInfoHashKey. All existing functionality is kept the same by making use of the nested ActionInfoHashKey contained within the Id. These changes will provide the basis for all usage of Id in followup changes to the scheduler. This breaks compatibility for forwarding an operation from one remote execution system to another that does not use our operation name format (ie: very unlikely, but possible). Co-authored-by: Zach Birenbaum <zach@tracemachina.com> Add Operation State Manager API (TraceMachina#937) Introduces API to be implemented by new scheduler state backends. WorkerStateManager is responsible for managing action updates sent to and from workers. ClientStateManager is responsible any state related to the clients client(s) requesting the action execution and can add or filter operations based on a given criteria. MatchingEnginerStateManager can filter, update, or remove actions. Co-authored-by: Zach Birenbaum <zach@tracemachina.com> Unbreak docker-compose workflow (TraceMachina#940) Fixes TraceMachina#939 Update Rust crate redis to v0.25.4 (TraceMachina#944) Update Rust crate syn to v2.0.66 (TraceMachina#946) Bump trivially bumpable deps (TraceMachina#950) Notably, this changes the S3 schema for multipart upload starts/stops. Add kind-loadbalancer (TraceMachina#929) Provide an envoy loadbalancer container to access the local kind cluster. The loadbalancer adds the ability to reach the local kind cluster trough localhost, so that it is possible to work with a mac on the nativelink cluster. Docker on Mac uses a aarch64 linux vm unter the hood, so the builds in the cluster can now be tested on that architecture. Users can now query the loadbalancer instead of the gateway. The loadbalancer has the following endpoints: localhost:8080 -> hubble-ui localhost:8081 -> tekton-dashboard localhost:8082/eventlistener -> nativelink-eventlistener Fix pulumi ratelimiting build error (TraceMachina#953) On MacOS the pulumi testsuite hit the ratelimiting on github. Disable the pulumi tests with a patch on nixpkgs. Commit 0eed759 didn't fix this issue. Add drake toolchain configs (TraceMachina#942) Register metrics on PropertyModifierScheduler (TraceMachina#954) Fixes metrics being lost due to the underlying ActionScheduler not having it's metrics passed through the PropertyModifierScheduler. This should fix any scaling issues related to queued actions metrics not being available. Use single quotes for char (TraceMachina#955) Nightly clippy reports a style issue when using double quotes around a single char. Increase pre-commit timeout in CI (TraceMachina#956) The original setting of 5 minutes was a bit too flaky. Increase it to 10 minutes so that cache misses don't break the job.
The digest function the client requested is now properly implemented
in the OriginContext. This allows us to inspect what digest-function
the client requested from anywhere in the code.
This change is