Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] Update docker images tag to 20240105-165030-51bdaec6 #16344

Merged
merged 4 commits into from
Jan 7, 2024
Merged

[CI] Update docker images tag to 20240105-165030-51bdaec6 #16344

merged 4 commits into from
Jan 7, 2024

Conversation

vinx13
Copy link
Member

@vinx13 vinx13 commented Jan 3, 2024

This is to integrate changes in #16336

@junrushao
Copy link
Member

not sure about the CI failures...

@vinx13 vinx13 changed the title [CI] Update docker images tag to 20240103-070130-97f6e6507 [CI] Update docker images tag to 20240105-165030-51bdaec6 Jan 5, 2024
@vinx13
Copy link
Member Author

vinx13 commented Jan 6, 2024 •
edited
Loading

2024-01-06T00:34:29.983Z] docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
[2024-01-06T00:34:29.983Z] nvidia-container-cli: requirement error: unsatisfied condition: cuda>=11.8, please update your driver to a newer version, or use an earlier cuda container: unknown.

this seems related to ci gpu error. CI has cuda 11.7 installed while the docker image has 11.8, which can be run until the Nvidia updates on base image on November. We need to update the AMIs for GPU workers. @tqchen

@vinx13 vinx13 force-pushed the ci-images-20240103-070130-97f6e6507 branch from c2acc27 to 3e22f6e Compare January 6, 2024 22:51
@vinx13 vinx13 force-pushed the ci-images-20240103-070130-97f6e6507 branch from 3e22f6e to 9d23c57 Compare January 7, 2024 03:18
yongwww
yongwww approved these changes Jan 7, 2024
View reviewed changes
Copy link
Member

@yongwww yongwww left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

yongwww
yongwww reviewed Jan 7, 2024
View reviewed changes
ci/jenkins/generated/gpu_jenkinsfile.groovy
@@ -112,7 +112,7 @@ properties([
upstream_revision = null

// command to start a docker container
docker_run = 'docker/bash.sh --env CI --env PLATFORM --env TVM_SHARD_INDEX --env TVM_NUM_SHARDS --env RUN_DISPLAY_URL --env PLATFORM --env SKIP_SLOW_TESTS --env TEST_STEP_NAME'
docker_run = 'docker/bash.sh --env CI --env PLATFORM --env TVM_SHARD_INDEX --env TVM_NUM_SHARDS --env RUN_DISPLAY_URL --env PLATFORM --env SKIP_SLOW_TESTS --env TEST_STEP_NAME --env NVIDIA_DISABLE_REQUIRE=true'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this effort! doc about the NVIDIA_DISABLE_REQUIRE variable. We may consider disabling it when upgrading the image/cuda in the future.

@vinx13 vinx13 merged commit 9caa179 into apache:main Jan 7, 2024
7 checks passed
vinx13 added a commit to vinx13/tvm that referenced this pull request Jan 8, 2024
@vinx13
[CI] Update docker images tag to 20240105-165030-51bdaec6 (apache#16344)
131e8c8 
* [CI] Update docker images tag to 20240103-070130-97f6e6507

* Update images tag to 20240105-165030-51bdaec6

* Show cuda info in ci runner

* [CI] Set NVIDIA_DISABLE_REQUIRE=true
@vinx13 vinx13 mentioned this pull request Jan 8, 2024
Merged
@ysh329 ysh329 mentioned this pull request Apr 21, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yongwww yongwww yongwww approved these changes

@junrushao junrushao junrushao approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@vinx13 @junrushao @yongwww