Using Trivy plugin with insecure registry and / or custom certificate authority

[awsompankaj]

Hi, I am running my kubernetes cluster 1.19.3 with containerd runtime, when i run the vulnerabilityreports for any deployment my job fails with below errors, seems it requires only docker daemon to run which is not present in any of the host in cluster.
my images are stored in private harbor repo.

starboard scan vulnerabilityreports deployment.apps/wordpress -v5
I0405 12:12:33.522748 3085368 scanner.go:66] Getting Pod template for workload: {Deployment wordpress wordpress}
I0405 12:12:33.530500 3085368 scanner.go:72] Scanning with options: {ScanJobTimeout:0s DeleteScanJob:true}
I0405 12:12:33.533791 3085368 runner.go:79] Running task and waiting forever
I0405 12:12:33.533967 3085368 runnable_job.go:76] Creating job "starboard/00106fc0-9f57-4dbe-ac0f-6ad75bd435a6"
I0405 12:12:33.611331 3085368 runnable_job.go:132] Event: Created pod: 00106fc0-9f57-4dbe-ac0f-6ad75bd435a6-gpc48 (SuccessfulCreate)
I0405 12:13:14.479651 3085368 runnable_job.go:114] Stopping runnable job on task failure with status: Failed
E0405 12:13:14.495565 3085368 runnable_job.go:164] Container wordpress terminated with Error: 2021-04-05T12:13:14.333Z  FATAL   unable to initialize a scanner: unable to initialize a docker scanner: 3 errors occurred:
       * unable to inspect the image (harbor-xxxxxxxx/wordpress:4.8-apache): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
       * unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory
       * Get "https://harbor-xxxxxxxx/v2/": Forbidden


I0405 12:13:14.496341 3085368 runner.go:83] Stopping runner on task completion with error: warning event received: Job has reached the specified backoff limit (BackoffLimitExceeded)
error: running scan job: warning event received: Job has reached the specified backoff limit (BackoffLimitExceeded)

[danielpacak]

👋 @awsompankaj Thank you for trying out Starboard and reporting this issue. The last relevant error message in your logs is:

Get "https://harbor-xxxxxxxx/v2/": Forbidden

which might indicate registry authentication / authorisation issues. Neglect Docker socket / daemon error reported by Trivy as it's not relevant in this case. (Trivy does not require Docker engine to scan container images at all.)

I assume that your Deployment refers to a container image stored in a private Harbor registry and you provided image pull secret at a service account or pod level. Is that correct? Those are the only supported options to scan private container images with Starboard / Trivy.

If you're authorising K8s nodes to pull images this option is not supported yet, see #120

[awsompankaj]

@danielpacak Thanks for your response.
We are using private harbor repo but the image which we are using in deployment is public and not using any pull secrets.

is the issue related to certificated which is used in harbor URL which is selfsigned? any suggestion?

[danielpacak]

This might be the root cause. Can you try scanning this image with trivy directly and see if you can reproduce the error?

$ trivy image <image ref>

[awsompankaj]

@danielpacak using trivy binary i am able to scan the image from harbor repo.

is there any way to import the CA cert for harbor in trivy job or add environment variable TRIVY_INSECURE=true when we run the starboard scan?

 trivy -d image harbor-ixxxxx/nginx:1.17.6
2021-04-06T14:50:22.795Z        DEBUG   Severities: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
2021-04-06T14:50:22.796Z        DEBUG   cache dir:  /home/xxx/.cache/trivy
2021-04-06T14:50:22.796Z        DEBUG   DB update was skipped because DB is the latest
2021-04-06T14:50:22.796Z        DEBUG   DB Schema: 1, Type: 1, UpdatedAt: 2021-04-06 12:06:54.473025742 +0000 UTC, NextUpdate: 2021-04-07 00:06:54.473025442 +0000 UTC, DownloadedAt: 2021-04-06 14:45:04.687866644 +0000 UTC
2021-04-06T14:50:22.799Z        DEBUG   Vulnerability type:  [os library]
2021-04-06T14:50:22.801Z        DEBUG   Artifact ID: sha256:f7bb5701a33c0e572ed06ca554edca1bee96cbbc1f76f3b01c985de7e19d0657
2021-04-06T14:50:22.802Z        DEBUG   Blob IDs: [sha256:556c5fb0d91b726083a8ce42e2faaed99f11bc68d3f70e2c7bbce87e7e0b3e10 sha256:49434cc20e95909131e0246cd70999bc362a1b6f511a31b20c4c9c9ad40c736b sha256:75248c0d543896c4ba05f1624b740d73a1642d9197f9608033196061e4f1d6bb]
2021-04-06T14:50:22.804Z        INFO    Detecting Debian vulnerabilities...
2021-04-06T14:50:22.805Z        DEBUG   debian: os version: 10
2021-04-06T14:50:22.805Z        DEBUG   debian: the number of packages: 118
2021-04-06T14:50:22.817Z        INFO    Trivy skips scanning programming language libraries because no supported file was detected

harbor-xxxxxx/nginx:1.17.6 (debian 10.2)
========================================================================
Total: 169 (UNKNOWN: 0, LOW: 91, MEDIUM: 29, HIGH: 43, CRITICAL: 6)

[awsompankaj]

Hi @danielpacak Is there any way I can edit the job template or use TRIVY_INSECURE=true any where in the job env or this parameter is not supported yet with starboard.

[danielpacak]

Currently we do not support custom registry CA nor TRIVY_INSECURE flag

[awsompankaj]

Thank you @danielpacak for your response, can this be a feature in coming releases.

[danielpacak]

We're actually preparing release v0.10.0 so these request would be out of scope, but we can probably ship TRIVY_INSECURE config flag in v0.10.0. Custom CA requires more work so I'd postpone it for v0.11.0

[danielpacak]

@awsompankaj I've created #545 and #546 to follow up on this discussion. Notice also that a quick workaround for custom CA might be to add it to a custom Trivy scanner image and use as as trivy.imageRef config param:

FROM docker.io/aquasec/trivy:0.16.0

RUN add_your_ca_here.sh