Add GPU driver installation support in launcher #497

meetrajvala · 2024-09-25T11:02:50Z

This PR adds the following changes for supporting the GPU driver installation in launcher component:

Installation of open sourced GPU drivers using the cos-gpu-installer.
Changes in container_runnner to ensure that GPU device is accessible to the workload container.
Utility functions for listing the GPU device files and remounting the installation directory.
Image tests with required setup and validation scripts.
Unit tests

meetrajvala · 2024-09-25T13:33:17Z

/gcbrun

jkl73 · 2024-09-25T17:54:42Z

/gcbrun

meetrajvala · 2024-09-29T10:44:09Z

/gcbrun

launcher/image/testworkloads/gpu/cuda-vector-add/Dockerfile

launcher/container_runner.go

launcher/launcher/main.go

launcher/internal/gpu/driverinstaller.go

launcher/image/test/scripts/test_gpu_workload.sh

launcher/internal/gpu/driverinstaller.go

launcher/internal/experiments/experiments.go

yawangwang · 2024-10-09T18:04:38Z

Please squash your commits to have a cleaner history.

launcher/internal/gpu/driverinstaller.go

launcher/launcher/main.go

launcher/util_test.go

launcher/image/test/scripts/test_gpu_unsupported_gputype.sh

launcher/container_runner.go

launcher/util.go

jkl73 · 2024-10-10T06:06:20Z

launcher/internal/gpu/driverinstaller.go

+	code, _, _ := status.Result()
+	di.logger.Printf("Gpu driver installation task exited with status: %d\n", code)
+
+	err = remountAsExecutable(InstallationHostDir)


This feels weird,

can you try one thing? try to create the directory before launching the install container? And see if the result driver is executable without remounting

If we just create the directory with required permission before installation step, then it still fails. if we run the same remountAsExecutable function before the installation steps, then it works fine.

jkl73 · 2024-10-10T06:12:37Z

launcher/internal/gpu/driverinstaller.go

+}
+
+func getInstallerImageReference() (string, error) {
+	installerImageRefBytes, err := exec.Command("cos-extensions", "list", "--", "--gpu-installer").Output()


can we hardcode this ref? I run the command I saw "us.gcr.io/cos-cloud/cos-gpu-installer:v2.4.1"
I assume this won't change if in the same image?

As we use image family (e.g. cos-113-lts) field in cloudbuild, build will use the latest image from the family which may have the installer version updated and may break if we hardcode it. So it is recommended to get the version using cos-extensions for given base cos image as cos-gpu-installer is not guaranteed to be backward compatible.

This could be hardcoded by running this command at build time. That's because it will always be built on the same COS version. This should be fixed by GA; please add a TODO.

meetrajvala · 2024-10-10T06:49:54Z

/gcbrun

meetrajvala · 2024-10-10T07:45:16Z

/gcbrun

launcher/util.go

launcher/internal/gpu/driverinstaller.go

launcher/launcher/main.go

launcher/image/test/scripts/test_gpu_unsupported_gputype.sh

meetrajvala · 2024-10-10T20:47:22Z

/gcbrun

meetrajvala · 2024-10-10T21:04:11Z

/gcbrun

launcher/util.go

launcher/launcher/main.go

…upport

meetrajvala · 2024-10-11T00:09:58Z

/gcbrun

yawangwang

Please ensure to get approvals from other reviewers before merging this PR. Their expertise in CS image will also provide valuable insights.

yawangwang · 2024-10-11T00:12:54Z

launcher/cloudbuild.yaml

+    gcloud builds submit --config=test_gpu_driver_installation_cloudbuild.yaml --region us-west1 \
+      --substitutions _IMAGE_NAME=${OUTPUT_IMAGE_PREFIX}-hardened-${OUTPUT_IMAGE_SUFFIX},_IMAGE_PROJECT=${PROJECT_ID}
+    exit
+# TODO: Enable these tests for debug image once gpu qouta is setup for the build project.


nit: create a bug item to track TODOs.

meetrajvala · 2024-10-11T03:57:01Z

/gcbrun

alexmwu

Mostly minor comments.

alexmwu · 2024-10-07T21:11:46Z

launcher/cloudbuild.yaml

+- name: 'gcr.io/cloud-builders/gcloud'
+  id: GpuDriverInstallationDebugImageTests


How does this test work but GpuDriverInstallationHardenedImageTests not work?

launcher/container_runner.go

alexmwu · 2024-10-11T19:00:42Z

launcher/image/test/create_gpu_vm.sh

@@ -0,0 +1,97 @@
+#!/bin/bash
+local OPTIND


I don't understand why this needs to be a separate script rather than extending the existing util.

launcher/image/test/scripts/gpu/test_gpu_nogpu.sh

launcher/internal/gpu/driverinstaller.go

alexmwu · 2024-10-11T19:38:09Z

launcher/internal/gpu/driverinstaller.go

+}
+
+func getInstallerImageReference() (string, error) {
+	installerImageRefBytes, err := exec.Command("cos-extensions", "list", "--", "--gpu-installer").Output()


This could be hardcoded by running this command at build time. That's because it will always be built on the same COS version. This should be fixed by GA; please add a TODO.

launcher/internal/gpu/driverinstaller.go

alexmwu · 2024-10-11T19:44:09Z

launcher/internal/gpu/driverinstaller.go

+			oci.WithHostHostsFile,
+			oci.WithHostResolvconf,


since a lot of this is duplicated, it would be useful to have helper functions that help mount with hostnamespace, hosts file and resolve conf.

alexmwu · 2024-10-11T19:44:49Z

launcher/internal/gpu/driverinstaller.go

+			oci.WithHostDevices,
+			oci.WithMounts(mounts),
+			oci.WithHostNamespace(specs.NetworkNamespace),
+			oci.WithHostNamespace(specs.PIDNamespace),


Why do we need the same PID namespace?

meetrajvala marked this pull request as draft September 25, 2024 11:05

meetrajvala force-pushed the gpu-support branch 2 times, most recently from 103b6fd to 63548fb Compare September 27, 2024 17:36

meetrajvala marked this pull request as ready for review September 27, 2024 17:37

meetrajvala requested review from alexmwu and jkl73 September 27, 2024 17:37

meetrajvala force-pushed the gpu-support branch 5 times, most recently from e9588fb to 2654a31 Compare September 27, 2024 22:37

meetrajvala force-pushed the gpu-support branch 11 times, most recently from 494337b to f4c4fba Compare September 30, 2024 21:38

jkl73 reviewed Oct 1, 2024

View reviewed changes

meetrajvala force-pushed the gpu-support branch 4 times, most recently from 5774e43 to e155130 Compare October 8, 2024 22:31

yawangwang reviewed Oct 9, 2024

View reviewed changes

launcher/internal/gpu/driverinstaller.go Show resolved Hide resolved

launcher/launcher/main.go Outdated Show resolved Hide resolved

launcher/util_test.go Outdated Show resolved Hide resolved

launcher/image/test/scripts/test_gpu_unsupported_gputype.sh Outdated Show resolved Hide resolved

Add GPU driver installation

0ae2e12

meetrajvala force-pushed the gpu-support branch 2 times, most recently from 8919575 to a603a6d Compare October 9, 2024 23:24

meetrajvala requested a review from yawangwang October 9, 2024 23:26

meetrajvala force-pushed the gpu-support branch from a603a6d to 8a29735 Compare October 10, 2024 03:23

jkl73 reviewed Oct 10, 2024

View reviewed changes

meetrajvala force-pushed the gpu-support branch from 8a29735 to d658692 Compare October 10, 2024 06:42

meetrajvala force-pushed the gpu-support branch from d658692 to 2ed4e4a Compare October 10, 2024 07:38

meetrajvala requested a review from jkl73 October 10, 2024 10:29

yawangwang requested changes Oct 10, 2024

View reviewed changes

meetrajvala force-pushed the gpu-support branch from 2ed4e4a to 6e65eb6 Compare October 10, 2024 20:47

meetrajvala requested a review from yawangwang October 10, 2024 20:48

meetrajvala force-pushed the gpu-support branch from 6e65eb6 to bae1a53 Compare October 10, 2024 20:58

yawangwang reviewed Oct 10, 2024

View reviewed changes

launcher/util.go Outdated Show resolved Hide resolved

launcher/launcher/main.go Outdated Show resolved Hide resolved

launcher/launcher/main.go Outdated Show resolved Hide resolved

Merge branch 'main' of github.com:meetrajvala/go-tpm-tools into gpu-s…

91c74da

…upport

meetrajvala force-pushed the gpu-support branch from bae1a53 to 91c74da Compare October 11, 2024 00:06

meetrajvala requested a review from yawangwang October 11, 2024 00:10

yawangwang approved these changes Oct 11, 2024

View reviewed changes

meetrajvala changed the base branch from main to cs_gpu October 11, 2024 03:56

jkl73 approved these changes Oct 11, 2024

View reviewed changes

meetrajvala merged commit 796705b into google:cs_gpu Oct 11, 2024
12 checks passed

alexmwu reviewed Oct 11, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GPU driver installation support in launcher #497

Add GPU driver installation support in launcher #497

meetrajvala commented Sep 25, 2024

meetrajvala commented Sep 25, 2024

jkl73 commented Sep 25, 2024

meetrajvala commented Sep 29, 2024

yawangwang commented Oct 9, 2024

jkl73 Oct 10, 2024

meetrajvala Oct 11, 2024 •

edited

Loading

jkl73 Oct 10, 2024

meetrajvala Oct 10, 2024 •

edited

Loading

alexmwu Oct 11, 2024

meetrajvala commented Oct 10, 2024

meetrajvala commented Oct 10, 2024

meetrajvala commented Oct 10, 2024

meetrajvala commented Oct 10, 2024

meetrajvala commented Oct 11, 2024

yawangwang left a comment

yawangwang Oct 11, 2024

meetrajvala commented Oct 11, 2024

alexmwu left a comment

alexmwu Oct 7, 2024

alexmwu Oct 11, 2024

alexmwu Oct 11, 2024

alexmwu Oct 11, 2024

alexmwu Oct 11, 2024

		- name: 'gcr.io/cloud-builders/gcloud'
		id: GpuDriverInstallationDebugImageTests

Add GPU driver installation support in launcher #497

Add GPU driver installation support in launcher #497

Conversation

meetrajvala commented Sep 25, 2024

meetrajvala commented Sep 25, 2024

jkl73 commented Sep 25, 2024

meetrajvala commented Sep 29, 2024

yawangwang commented Oct 9, 2024

Choose a reason for hiding this comment

meetrajvala Oct 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

meetrajvala Oct 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

meetrajvala commented Oct 10, 2024

meetrajvala commented Oct 10, 2024

meetrajvala commented Oct 10, 2024

meetrajvala commented Oct 10, 2024

meetrajvala commented Oct 11, 2024

yawangwang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

meetrajvala commented Oct 11, 2024

alexmwu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

meetrajvala Oct 11, 2024 •

edited

Loading

meetrajvala Oct 10, 2024 •

edited

Loading