The SDK does not pick up the correct user project when running on Vertex Pipelines or CustomJobs

[Ark-kun]

Usually the Vertex SDK gets the project ID automatically (by calling google.auth.default()). This works when running on Kubeflow Pipelines.
However when running on Vertex Pipelines or CustomJobs, the detected project is not the user project and is not usable.
This leads to failure when trying to create any resource in the project:

google.api_core.exceptions.PermissionDenied: 403 Permission 'aiplatform.models.upload' denied on resource '//aiplatform.googleapis.com/projects/gbd40bc90c7804989-tp/locations/us-central1' (or it may not exist).

Fortunately there is a way to get project number from the Vertex environment. There is also a way to get project ID from the project number.

Deducing project number

project_number = os.environ.get("CLOUD_ML_PROJECT_ID")

Getting project ID:

    if not project:
        project_number = os.environ.get("CLOUD_ML_PROJECT_ID")
        if project_number:
            print(f"Inferred project number: {project_number}")
            project = project_number
            # To improve the naming we try to convert the project number into the user project ID.
            try:
                from googleapiclient import discovery

                cloud_resource_manager_service = discovery.build(
                    "cloudresourcemanager", "v3"
                )
                project_id = (
                    cloud_resource_manager_service.projects()
                    .get(name=f"projects/{project_number}")
                    .execute()["projectId"]
                )
                if project_id:
                    print(f"Inferred project ID: {project_id}")
                    project = project_id
            except Exception as e:
                print(e)

Perhaps we should improve the Vertex SDK initializer to use the project mentioned in CLOUD_ML_PROJECT_ID if it's available by default.

[Ark-kun]

I'm not sure this issue is fixed.

[Ark-kun]

/reopen