Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[core] [3/N] Check cgroupv2 mount status #48833

Merged
merged 9 commits into from
Jan 7, 2025

Conversation

dentiny
Copy link
Contributor

@dentiny dentiny commented Nov 21, 2024

Before we prepare cgroup V2 environment at #48828 and setup cgroup for process at #48788, we should first check whether cgroupv2 is properly mounted.

Signed-off-by: dentiny <dentinyhao@gmail.com>
@jcotant1 jcotant1 added the core Issues that should be addressed in Ray Core label Nov 22, 2024

namespace ray {

bool IsCgroupV2MountedAsRw() {
Copy link

@lanbochen-anyscale lanbochen-anyscale Dec 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nits: I think you also need to check that the normal user in the container can modify the cgroup path, usually /sys/fs/cgroup is owned by root user even though it's writable, it is only writable by the root.

See https://github.com/anyscale/product/pull/32052/files#diff-005124dd6eac41dac5da8cac101b497921e0ee33ca99ec034fbb9fd82ddba508

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's a good point, to update cgroupv2, mounted as rw is not enough, current user should also have the permission to write. Add a test to check.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another thought, another way we could do is we don't do any precheck before actually setting cgroup, which fails directly due to lacking permission.

Comment on lines 23 to 27
// Precondition: cgroup V2 has already been mounted as rw.
//
// Setup command:
// sudo umount /sys/fs/cgroup/unified
// sudo mount -t cgroup2 cgroup2 /sys/fs/cgroup/unified -o rw

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

qq, I am not familiar but what requires to put these step in ray's cicd?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't think it should be placed under CI, which is a group of unit test, but for cgroup related feature they should be tested as integration tests.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's why you see me mark test as "manual".

@dentiny dentiny requested a review from rynewang December 20, 2024 09:26
@dentiny dentiny added the go add ONLY when ready to merge, run all tests label Dec 23, 2024
Signed-off-by: dentiny <dentinyhao@gmail.com>
@dentiny
Copy link
Contributor Author

dentiny commented Dec 23, 2024

@rynewang This PR is ready to review.

@rynewang
Copy link
Contributor

what's relationship of this PR with the other cgroup PRs? Can you give a roadmap of which PR does what?

@dentiny
Copy link
Contributor Author

dentiny commented Dec 24, 2024

what's relationship of this PR with the other cgroup PRs? Can you give a roadmap of which PR does what?

I think we just go through the implementation plan this morning?

For cgroup setup preparation, there're a few things:

For setting up cgroup for attempt:

@rynewang
Copy link
Contributor

rynewang commented Jan 6, 2025

  1. move to src/ray/common/cgroup/cgroup_setup.cc, or at least in src/ray/common/cgroup/
  2. call these 2 funcs in SetupCgroupsPreparation

Signed-off-by: dentiny <dentinyhao@gmail.com>
Signed-off-by: dentiny <dentinyhao@gmail.com>
Signed-off-by: dentiny <dentinyhao@gmail.com>
Signed-off-by: dentiny <dentinyhao@gmail.com>
@dentiny dentiny force-pushed the hjiang/cgroup-validate branch from a483c05 to c061931 Compare January 6, 2025 23:19
@rynewang rynewang merged commit e7e097f into ray-project:master Jan 7, 2025
5 checks passed
roshankathawate pushed a commit to roshankathawate/ray that referenced this pull request Jan 9, 2025
Before we prepare cgroup V2 environment at
ray-project#48828 and setup cgroup for
process at ray-project#48788, we should
first check whether cgroupv2 is properly mounted.

---------

Signed-off-by: dentiny <dentinyhao@gmail.com>
Signed-off-by: Roshan Kathawate <roshankathawate@gmail.com>
HYLcool pushed a commit to HYLcool/ray that referenced this pull request Jan 13, 2025
Before we prepare cgroup V2 environment at
ray-project#48828 and setup cgroup for
process at ray-project#48788, we should
first check whether cgroupv2 is properly mounted.

---------

Signed-off-by: dentiny <dentinyhao@gmail.com>
Signed-off-by: lielin.hyl <lielin.hyl@alibaba-inc.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Issues that should be addressed in Ray Core go add ONLY when ready to merge, run all tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants