Trivy update fails

[gwitsch]

Question

Hello,

recently trivy is having issues updating the database before scanning container images.

To isolate the problem, I have introduced a separate update task which runs the update-only commands.
The problem only occurs at our nightly build pipeline run betwenn 8pm - 11pm.

Here is the update commands output:

21:00:20 Trivy: updating vulnerability DB ...
[Pipeline] sh
21:00:21 + trivy image --debug --download-db-only --cache-dir /home/jenkins/trivy_cache --timeout 15m --no-progress
21:00:21 2024-09-16T19:00:21Z DEBUG Cache dir dir="/home/jenkins/trivy_cache"
21:00:21 2024-09-16T19:00:21Z DEBUG Parsed severities severities=[UNKNOWN LOW MEDIUM HIGH CRITICAL]
21:00:21 2024-09-16T19:00:21Z DEBUG Ignore statuses statuses=[]
21:00:21 2024-09-16T19:00:21Z INFO [db] Need to update DB
21:00:21 2024-09-16T19:00:21Z INFO [db] Downloading DB... repository="ghcr.io/aquasecurity/trivy-db:2"
21:00:22 2024-09-16T19:00:22Z FATAL Fatal error
21:00:22 - init error:
21:00:22 github.com/aquasecurity/trivy/pkg/commands/artifact.Run
21:00:22 /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:358
21:00:22 - DB error:
21:00:22 github.com/aquasecurity/trivy/pkg/commands/artifact.NewRunner
21:00:22 /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:119
21:00:22 - failed to download vulnerability DB:
21:00:22 github.com/aquasecurity/trivy/pkg/commands/operation.DownloadDB
21:00:22 /home/runner/work/trivy/trivy/pkg/commands/operation/operation.go:41
21:00:22 - database download error:
21:00:22 github.com/aquasecurity/trivy/pkg/db.(*Client).Download
21:00:22 /home/runner/work/trivy/trivy/pkg/db/db.go:162
21:00:22 - oci download error:
21:00:22 github.com/aquasecurity/trivy/pkg/oci.(*Artifact).Download
21:00:22 /home/runner/work/trivy/trivy/pkg/oci/artifact.go:144
21:00:22 - failed to fetch the layer:
21:00:22 github.com/aquasecurity/trivy/pkg/oci.(*Artifact).download
21:00:22 /home/runner/work/trivy/trivy/pkg/oci/artifact.go:158
21:00:22 - GET https://ghcr.io/v2/aquasecurity/trivy-db/blobs/sha256:15a6fd375dca7d4d11fde458d5ba5f3fa60832e93da70143ad9e9e188d8d8c07: TOOMANYREQUESTS: retry-after: 576.466µs, allowed: 44000/minute

From my point of view, it looks like it's a problem related to the Github container registry. I guess there is some kind of DDoS protection in place.

Has anyone already faced the same issue? Or already found a solution?

Best regards,
Gotthard

Target

Container Image

Scanner

None

Output Format

None

Mode

Standalone

Operating System

Ubuntu

Version

0.54.1

[nikpivkin]

Hi @gwitsch !

We have encountered the same thing at CI and are still looking into the problem.

[synack-security]

I have seen this as well

[simar7]

Please see https://aquasecurity.github.io/trivy/v0.55/docs/references/troubleshooting/#github-rate-limiting

[acastong]

For those that my have a on site registry that can act as a cache for remote content, the following environment variables can prevent the problem by making Trivy pull from the cache on not hit the ghcr pull limit:

TRIVY_DB_REPOSITORY=${registryCacheHost}/${ghcr-cache-path}/aquasecurity/trivy-db \
TRIVY_JAVA_DB_REPOSITORY=${registryCacheHost}/${ghcr-cache-path}/aquasecurity/trivy-java-db

where registryCacheHost is you caching registry fqdn and ghcr-cache-path is the project/repo in your registry that is configured to cache ghcr.io

[gwitsch]

Adding a Github token seems to be working. At least the last couple of days, the update process worked as expected.

[synack-security]

I added a token and 1 out of 2 scans still failed. Haven’t seen any failures since Thursday though without using token.

[gwitsch]

Sadly, Github tokens don't fix the issue.
Our nightly builds failed with the same exception:

23:01:35    - 1 error occurred:
23:01:35  	* GET https://ghcr.io/v2/aquasecurity/trivy-java-db/manifests/1: TOOMANYREQUESTS: retry-after: 140.905µs, allowed: 44000/minute

[knqyf263]

Same here.
#7580 (comment)

[knqyf263]

We published trivy-db in Amazon ECR Public Gallery as well as GHCR.
aquasecurity/trivy-db#440

The following should work now. We'll update the documentation tomorrow.

trivy image --db-repository public.ecr.aws/aquasecurity/trivy-db:2 YOUR_IMAGE

[thomascaillier]

@ryan-cpi --db-repository and --java-db-repository now accept a list (see 3562529)

[thomascaillier]

@knqyf263 is it planned to set these new repositories as default ones ? (in addition of the GHCR one, or not)

[knqyf263]

We're careful about that.
#7679 (comment)

[Connor-Wood]

@thomascaillier

We tried the list: trivy image --db-repository ghcr.io/aquasecurity/trivy-db:2 public.ecr.aws/aquasecurity/trivy-db:2
and are seeing an error on only the first one. In what cases are the following repos tried?

2024-10-14T14:45:11Z INFO [db] Need to update DB
2024-10-14T14:45:11Z INFO [db] Downloading DB... repository="ghcr.io/aquasecurity/trivy-db:2"
2024-10-14T14:45:11Z FATAL Fatal error init error: DB error: failed to download vulnerability DB: database download error: oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-db/blobs/sha256:8dcb18f131bc5d8d17159f7682ebd49b2ea9c784edbb03aa87fe92927ff1d851: TOOMANYREQUESTS: retry-after: 36.478µs, allowed: 44000/minute

[thomascaillier]

@Connor-Wood i'm sorry i don't know, i just saw it in the docs

[ryan-cpi]

I've not been able to get a successful Trivy run in GitLab for the last few hours due to this issue. We've seen it on and off in the past but all of our installs fail 100% now. This is causing our pipelines to fail so we can no longer merge code.

2024-10-01T12:18:51Z	FATAL	Fatal error	init error: DB error: failed to download vulnerability DB: database download error: OCI repository error: 1 error occurred:
	* GET https://ghcr.io/v2/aquasecurity/trivy-db/manifests/2: TOOMANYREQUESTS: retry-after: 545.68µs, allowed: 44000/minute
Python Version 3.12.6 (main, Sep 12 2024, 21:15:48) [GCC 13.2.1 20240309]

[knqyf263]

@prezha As @ryan-cpi pointed out, the situation may not improve within a few hours. We're very careful about retrying because retrying may increase the number of requests and make the situation worse. We'll probably introduce retry in the future, but we want to add more mitigations first.

[knqyf263]

This has been happening for a few hours and none of the manual retrys seem to help. For the long term fix of using a cache is there any recommended way that's documented for doing this with GitLab runners?

I'm not familiar with GitLab, so we need help from someone who knows it well. However, according to the document, the cache seems to be shared between pipelines. Am I missing something?

Subsequent pipelines can use the cache.

https://docs.gitlab.com/ee/ci/caching/#cache

[prezha]

@prezha As @ryan-cpi pointed out, the situation may not improve within a few hours. We're very careful about retrying because retrying may increase the number of requests and make the situation worse. We'll probably introduce retry in the future, but we want to add more mitigations first.

that makes sense, thank you for providing the context

[ryan-cpi]

For the short term, try #7538 (comment). For the long term, please consider storing the database in the cache.

The long term solution doesn't seem to be working for us as we had this in place long before these issues have been popping up. For the short term solution being using ECR how frequently is this image updated as I don't want to be pointing to an image that will only be updated short term?

Is there a long-term solution we can use or are we good to permanently use the ECR image from now on?

[bartvollebregt]

I'm not familiar with GitLab, so we need help from someone who knows it well. However, according to the document, the cache seems to be shared between pipelines. Am I missing something?

@knqyf263 If you need any help with Gitlab configuration, let me know.

This is what our Gitlab job looks like:

container_scanning:
  stage: verify
  image:
    name: ${CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX}/aquasec/trivy:latest
    entrypoint: [""]
  only:
    - schedules
  variables:
    GIT_STRATEGY: none
    TRIVY_USERNAME: "$CI_REGISTRY_USER"
    TRIVY_PASSWORD: "$CI_REGISTRY_PASSWORD"
    TRIVY_AUTH_URL: "$CI_REGISTRY"
    TRIVY_NO_PROGRESS: "true"
    TRIVY_CACHE_DIR: ".trivycache/"
    TRIVY_DB_REPOSITORY: ghcr.io/aquasecurity/trivy-db,public.ecr.aws/aquasecurity/trivy-db,public.ecr.aws/aquasecurity/trivy-db:2
    FULL_IMAGE_NAME: $CI_REGISTRY_IMAGE:latest
  before_script:
    - apk --no-cache add curl
  script:
    - trivy --version
    - time trivy clean --scan-cache
    - time trivy image --download-db-only
    - trivy image --scanners vuln --vuln-type os --exit-code=1 --severity='HIGH,CRITICAL' -q "$FULL_IMAGE_NAME"
  cache:
    key: trivy-cache
    paths:
      - .trivycache/

We're still facing issues with the TOOMANYREQUESTS error though.

As you can see we're using a cache:key: value here. This defines when the cache is re-used in another job. In this case I'm using a hardcoded string value, so each pipeline job in the same repository will use the same cache. However, we have a lot of different projects and repository and it's not possible to share cache across repositories in Gitlab (due to security reasons and cache poisoning).

[bryanhermass]

I found an option to create a local cache by setting the environment variable TRIVY_CACHE_DIR= and using the command trivy --download-db-only. It downloads the database to the specified path, and when scanning images, it won't connect to https://ghcr.io/v2/aquasecurity/trivy-db/manifests/2.

I think this could solve the issue.

Doc : https://aquasecurity.github.io/trivy/v0.56/docs/advanced/air-gap/#populating-the-trivy-cache