Unsupported MediaType when scanning image from private registry

[sukhinin]

Description

I try to scan an image from our private registry and get "Unsupported MediaType" error.

When I inspect HTTP requests, I see that Accept header is set to application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json.

Because of this our registry replies with Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws, which Trivy does not support. I believe it's a bug and unsupported content types should not be listed in Accept header.

If I rewrite HTTP request on the fly (via mitm proxy) and set Accept header to application/vnd.docker.distribution.manifest.v2+json, the correct manifest is returned and scan proceeds normally.

Output of run with -debug:

 % ./trivy -d image our.private.registry/image:tag
2020-08-05T14:49:48.014+0300	DEBUG	Severities: UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL
2020-08-05T14:49:48.032+0300	DEBUG	cache dir:  /Users/sukhinin/Library/Caches/trivy
2020-08-05T14:49:48.032+0300	DEBUG	DB update was skipped because DB is the latest
2020-08-05T14:49:48.032+0300	DEBUG	DB Schema: 1, Type: 1, UpdatedAt: 2020-08-05 00:26:27.623076822 +0000 UTC, NextUpdate: 2020-08-05 12:26:27.623076422 +0000 UTC
2020-08-05T14:49:48.511+0300	FATAL	unable to initialize a scanner:
    github.com/aquasecurity/trivy/internal/artifact.run
        /home/circleci/project/internal/artifact/run.go:72
  - unable to initialize a docker scanner:
    github.com/aquasecurity/trivy/internal/artifact.dockerScanner
        /home/circleci/project/internal/artifact/image.go:28
  - 2 errors occurred:
	* unable to inspect the image (our.private.registry/image:tag): Error: No such image: our.private.registry/image:tag
	* unsupported MediaType: "application/vnd.docker.distribution.manifest.v1+prettyjws", see https://github.com/google/go-containerregistry/issues/377

Output of trivy -v:

% trivy -v
Version: 0.10.1
Vulnerability DB:
  Type: Light
  Version: 1
  UpdatedAt: 2020-08-05 00:26:27.623076822 +0000 UTC
  NextUpdate: 2020-08-05 12:26:27.623076422 +0000 UTC

[knqyf263]

Thank you for looking into the problem. I raised an issue in google/go-containerregistry.
google/go-containerregistry#754

[jonjohnsonjr]

Do you happen to know what criteria the registry uses when selecting a media type? I wonder if we can use content negotiation or just ordering of the accept header to indicate which media type we prefer. I would think that, given the choice between two options, registries should always reply with the most recent supported format... but I understand that changing the registry is probably harder than changing a client.

We added schema 1 to the accept header originally because it was useful for one of our clients to distinguish between the image not existing vs the image being schema 1. Your argument that we shouldn't include it at all does make sense, though.

[sukhinin]

Sorry for the delay. We're using a quite old version of docker-distribution as our registry. I've ran a couple of tests to see how registry responds to different Accept header values. It seems v1 manifests are always preferred.

1. No header → Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws

% curl -I https://our.private.registry/v2/<image>/manifests/<tag>
HTTP/1.1 200 OK
Server: Apache
Date: Thu, 20 Aug 2020 14:07:04 GMT
Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
Content-Length: 10769
Connection: keep-alive
Docker-Content-Digest: sha256:35bc7d7c09a531da85b71b5bfb0871d9d09c0eadbd41398f20926a27b96698fd
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:35bc7d7c09a531da85b71b5bfb0871d9d09c0eadbd41398f20926a27b96698fd"
Accept-Ranges: bytes

2. Accept: application/vnd.docker.distribution.manifest.v1+json → Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws

% curl -I -H "Accept: application/vnd.docker.distribution.manifest.v1+json" https://our.private.registry/v2/<image>/manifests/<tag>
HTTP/1.1 200 OK
Server: Apache
Date: Thu, 20 Aug 2020 14:07:29 GMT
Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
Content-Length: 10769
Connection: keep-alive
Docker-Content-Digest: sha256:35bc7d7c09a531da85b71b5bfb0871d9d09c0eadbd41398f20926a27b96698fd
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:35bc7d7c09a531da85b71b5bfb0871d9d09c0eadbd41398f20926a27b96698fd"
Accept-Ranges: bytes

3. Accept: application/vnd.docker.distribution.manifest.v2+json → Content-Type: application/vnd.docker.distribution.manifest.v2+json

% curl -I -H "Accept: application/vnd.docker.distribution.manifest.v2+json" https://our.private.registry/v2/<image>/manifests/<tag>
HTTP/1.1 200 OK
Server: Apache
Date: Thu, 20 Aug 2020 14:07:52 GMT
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Content-Length: 1995
Connection: keep-alive
Docker-Content-Digest: sha256:6b62b4bc1fcf5c35008b2bd9cf9e5450456d038af2f92ef72bcbd27a584582e5
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:6b62b4bc1fcf5c35008b2bd9cf9e5450456d038af2f92ef72bcbd27a584582e5"
Accept-Ranges: bytes

4. Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json → Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws

% curl -I -H "Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v2+json" https://our.private.registry/v2/<image>/manifests/<tag>
HTTP/1.1 200 OK
Server: Apache
Date: Thu, 20 Aug 2020 14:08:06 GMT
Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
Content-Length: 10769
Connection: keep-alive
Docker-Content-Digest: sha256:35bc7d7c09a531da85b71b5bfb0871d9d09c0eadbd41398f20926a27b96698fd
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:35bc7d7c09a531da85b71b5bfb0871d9d09c0eadbd41398f20926a27b96698fd"
Accept-Ranges: bytes

5. Accept: application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json → Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws

% curl -I -H "Accept: application/vnd.docker.distribution.manifest.v2+json,application/vnd.docker.distribution.manifest.v1+json" https://our.private.registry/v2/<image>/manifests/<tag>
HTTP/1.1 200 OK
Server: Apache
Date: Thu, 20 Aug 2020 14:08:25 GMT
Content-Type: application/vnd.docker.distribution.manifest.v1+prettyjws
Content-Length: 10769
Connection: keep-alive
Docker-Content-Digest: sha256:35bc7d7c09a531da85b71b5bfb0871d9d09c0eadbd41398f20926a27b96698fd
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:35bc7d7c09a531da85b71b5bfb0871d9d09c0eadbd41398f20926a27b96698fd"
Accept-Ranges: bytes

Proxying requests to registry via nginx (running on the same host as trivy and listening on a loopback interface) with the following config allows to work around the problem.

events {
  worker_connections   1024;
}

http {
  server {
    listen  80;

    location / {
      proxy_pass https://our.private.registry;
      proxy_set_header Host our.private.registry;
    }

    location ~ ^/v2/(.+)/manifests/(.+)$ {
      proxy_pass https://our.private.registry;
      proxy_set_header Host our.private.registry;
      proxy_set_header Accept application/vnd.docker.distribution.manifest.v2+json;
    }
  }
}

[github-actions]

This issue is stale because it has been labeled with inactivity.

[knqyf263]

Please watch google/go-containerregistry#754.