[Kube2iam] How to add annotations to the pods spawned by starboard jobs

[giovannirco]

We currently use kube2iam around here and moving to IRSA is proving to be a bit harder than I imagined, even testing with manual changes on our EKS cluster is a bit difficult and I am having some issues with IRSA mutating webhooks.

Is there a way I can edit the jobTemplate.spec that creates pods that run the vulnerability scan? If possible I could add the kube2iam annotation that will enable the pods spawned by starboard jobs to have access to ECR images and trivy will be able to run properly.

[giovannirco]

I have edited the vulnerabilityreport/scanner.go and added the annotation there then compiled the binary.

Running it got the annotation there on the pod

but I still get the same 401 error getting from ecr url and a docker error :(

> bin/starboard -v4 scan vulnerabilityreports deploy/bitcoin-indexer --namespace bitcoin --delete-scan-job=false
I0120 21:00:42.308575   37811 scanner.go:66] Getting Pod template for workload: {Deployment bitcoin-indexer bitcoin}
I0120 21:00:42.470477   37811 scanner.go:72] Scanning with options: {ScanJobTimeout:0s DeleteScanJob:false}
I0120 21:00:42.626617   37811 runner.go:79] Running task and waiting forever
I0120 21:00:42.626704   37811 runnable_job.go:76] Creating job "starboard/dd025e31-e274-4136-8c1f-56f69ce8743b"
I0120 21:00:43.258625   37811 runnable_job.go:132] Event: Created pod: dd025e31-e274-4136-8c1f-56f69ce8743b-47l56 (SuccessfulCreate)
I0120 21:00:47.903264   37811 runnable_job.go:114] Stopping runnable job on task failure with status: Failed
E0120 21:00:48.210516   37811 runnable_job.go:164] Container bitcoin-indexer terminated with Error: 2021-01-21T00:00:47.214Z	FATAL	unable to initialize a scanner: unable to initialize a docker scanner: 2 errors occurred:
	* unable to inspect the image (xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/magnifier/magnifier/xxxxxxxx:latest): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
	* GET https://xxxxxxxx.dkr.ecr.us-east-1.amazonaws.com/v2/magnifier/magnifier/xxxxxxxx/manifests/latest: unsupported status code 401; body: Not Authorized



I0120 21:00:48.210544   37811 runner.go:83] Stopping runner on task completion with error: warning event received: Job has reached the specified backoff limit (BackoffLimitExceeded)
error: running scan job: warning event received: Job has reached the specified backoff limit (BackoffLimitExceeded)

When running a scan on a public image it works just fine:

> bin/starboard -v4 scan vulnerabilityreports deploy/bitcoin-ingress-nginx-controller --namespace bitcoin --delete-scan-job=false
I0120 21:00:11.358294   37784 scanner.go:66] Getting Pod template for workload: {Deployment bitcoin-ingress-nginx-controller bitcoin}
I0120 21:00:11.521589   37784 scanner.go:72] Scanning with options: {ScanJobTimeout:0s DeleteScanJob:false}
I0120 21:00:11.679614   37784 runner.go:79] Running task and waiting forever
I0120 21:00:11.679683   37784 runnable_job.go:76] Creating job "starboard/a00b6673-7be0-4adb-a261-ace894602f41"
I0120 21:00:12.318888   37784 runnable_job.go:132] Event: Created pod: a00b6673-7be0-4adb-a261-ace894602f41-24d2n (SuccessfulCreate)
I0120 21:00:25.431082   37784 runnable_job.go:111] Stopping runnable job on task completion with status: Complete
I0120 21:00:25.431218   37784 runner.go:83] Stopping runner on task completion with error: <nil>
I0120 21:00:25.431235   37784 scanner.go:101] Scan job completed: starboard/a00b6673-7be0-4adb-a261-ace894602f41
I0120 21:00:25.431270   37784 scanner.go:176] Getting logs for controller container in job: starboard/a00b6673-7be0-4adb-a261-ace894602f41
I0120 21:00:26.238813   37784 scanner.go:91] Skipping scan job deletion: starboard/a00b6673-7be0-4adb-a261-ace894602f41
I0120 21:00:26.397809   37784 io.go:76] Creating VulnerabilityReport "bitcoin/deployment-bitcoin-ingress-nginx-controller-controller"

Checking the roles on AWS console I see that the role never got used :(
I will keep researching but I would love to see this working with kube2iam

[giovannirco]

@mmerrill3 did when you were trying to setup this with Kiam what was your blocker? Do you mind taking a look at this and help me out? After getting the annotation on the job pod kube2iam should deal with this just fine and I believe from the application point of view there shouldn't be any difference between IRSA and Kube2iam. What kube2iam pod is doing here is just redirect traffic to AWS resources.

[mmerrill3]

Hi @giovannirco ,
When trivy is started by starboard, its started by the job controller running inside starboard. Starboard will pass the service account token from itself into the trivy job. The annotations don't get passed into the trivy job. That's why the annotation based solutions won't work here. You have to use the service account.

What I didn't try is what will happen if I set nothing. If I don't use IRSA, and I don't set annotations for kiam. Will trivy just use the ec2 instance's IAM role? I think so. That should be the last provider checked by the AWS SDK when going to ECR. I didn't try that b/c I configured kiam to never give back the EC2 instance's IAM role. Rather, kiam would always give back a zero-permissions role if EC2 was checked by a pod. I think that if you don't have kiam, or kube2iam, you could use the EC2 instance's IAM role. If you do have kiam, or kube2iam, and you are giving back a default role that is not the EC2's IAM role, add the ECR permissions to the default role that is being given back. I think that would work.

All of that experience ended in this for me, maybe it will help you: https://github.com/mmerrill3/happy-hour/blob/master/irsa.md

[danielpacak]

To answer the original question: it's not possible to annotate job spawned by Starboard (v0.10). I've created #539 to add such capability.