Access AWS Services from KServe

[surajkota]

Is your feature request related to a problem? Please describe.
Customers want to access AWS services for inference services like pull images for private ECR and download models from S3 bucket.

Describe the solution you'd like
Do the above using IRSA

[surajkota]

Thành Trần
16 hours ago
But I have another question: How can I add permission for my model server to pull image from my private registry ?
I am trying to test transformer: https://github.com/kserve/kserve/tree/release-0.7/docs/samples/v1beta1/transformer/torchserve_image_transformer
I build a docker image and push it into my private ECR registry. Then I create a secret (secret-registry) for my private ECR:

kubectl create secret docker-registry secret-registry \
--docker-server=80*******03.dkr.ecr.us-east-1.amazonaws.com \
--docker-username=AWS \
--docker-password=$(aws ecr get-login-password --region us-east-1) \
-n staging

Then, I create a torch_transformer.yaml file like:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: torchserve-transformer
spec:
  transformer:
    imagePullSecrets:
    - name: secret-registry
    containers:
    - image: 80*********03.dkr.ecr.us-east-1.amazonaws.com/cnndha:image-transformer-v2
      name: kfserving-container
      env:
        - name: STORAGE_URI
          value: gs://kfserving-examples/models/torchserve/image_classifier
  predictor:
    pytorch:
      storageUri: gs://kfserving-examples/models/torchserve/image_classifier

Then:
kubectl apply -n staing -f torch_transformer.yaml
Is that correct?

Thành Trần
16 hours ago
seem like still can not pull image from private registry
Can you give me instruction please? I do not find any document about this in kubeflow aws
image.png

image.png

Suraj Kota
16 minutes ago
I would recommend using IAM Role for ServiceAccount (IRSA) approach to pull images from private ECR.
They token you are trying put in the secret will expire and if the isvc scales on a different node, it will fail to pull the image again (edited)

Suraj Kota
8 minutes ago
Create a policy with following permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ecr:BatchCheckLayerAvailability",
                "ecr:BatchGetImage",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetAuthorizationToken"
            ],
            "Resource": "*"
        }
    ]
}

2. Create a IAM role and attach it to a service account using IRSA
   eksctl create iamserviceaccount --name image-pull-sa --namespace ${YOUR_PROFILE_NAMESPACE} --cluster ${CLUSTER_NAME} --region ${CLUSTER_REGION} --attach-policy-arn ${IAM_POLICY_ARN} --override-existing-serviceaccounts --approve
3. Specify this service account in the transformer spec

spec:
  transformer:
    serviceAccountName: image-pull-sa

docs.aws.amazon.comdocs.aws.amazon.com
IAM roles for service accounts - Amazon EKS
Learn how applications in your pods can access AWS services.

[AlexandreBrown]

@surajkota @jsitu777 What is the status on this ?

[surajkota]

@AlexandreBrown https://awslabs.github.io/kubeflow-manifests/docs/component-guides/kserve/access-aws-services-from-kserve/

Missed closing the issue