isManifestUnknownError fails against Harbor registries, breaking sigstore signature upload

[Plat0nicS0lid]

Skopeo is planned to be used to copy already signed images between two or more private registries.
In my situation, the images are already signed in a private source registry. This signature is also shown as existing within the source registry.
If this signed image is copied directly between the two registries - there is a graphical front end in the registry product for this - the image includes the signature appearing on the target registry.
However, this solution cannot be used due to other disadvantages. Thats the reason trying skopeo for a potential solution.
A $skopeo copy docker://192.168.x.y/images/test:0.1 dir:test_0.1 copies the image including the signature into the test_0.1 directory - into signature-1.
However, a copy directly between two registries does not work:
$skopeo copy docker://192.168.x.y/images/test:0.1 docker://192.168.y.z/images/test:0.1
Getting image source signatures
Checking if image destination supports signatures
Copying blob 96526aa774ef skipped: already exists
Copying blob 5b088f1e648c skipped: already exists
Copying config 33b8df73a9 done
Writing manifest to image destination
Storing signatures
FATA[0000] writing signatures: reading manifest sha256-blablabla.sig in 192.168.y.z/images/test: unknown: artifact images/test:sha256-blablabla.sig not found

Also, there is no positiv result trying to copy the content from inside the local directory. The same message is the result.

My question is: might it be that Skopeo cannot be used to copy images between two or more registries that are already signed in the source registry? There are no plans to re-sign the image with the private key in between again. This signature should be adopted 1:1 into the target registry - just as the registry product itself can do, although there are other reasons against using this solution.

May I ask for assistance or is this some kind of issue or enhancement?

[mtrmac]

Thanks for your report.

As you see, the code is trying to copy signatures. This is supposed to work.

AFAICS the issue here is interoperability with the specific destination registry implementation; a signed push with a sigstore signature to that registry would fail just the same.

---

First, please make sure you are using the recently-released Skopeo 1.14.0; the behavior in this area has changed, quite possibly fixing this.

If that’s not the case, we’ll need to fix that in c/image; for starters, please provide a full log from skopeo --debug copy …, and there might be more debugging information necessary.

[Plat0nicS0lid]

Hello,
tried the skopeo version within arch linux docker image.
Please see the skopeo -v output:
[root@6bbacdc9b6e4 ~]# skopeo -v
skopeo version 1.14.0 commit: 6abf96bb82666fbb3d4ad9faf1812e5ae2d31a74
[root@6bbacdc9b6e4 ~]#

To give you some more information about the environment. Both source and destination are harbor registries - please see goharbor.io if you need more information about the product.
Harbor is able to handle cosign signed images per default. As stated in my first entry, using the internal image replication leads to the result that signed images within the source registry will be copied together with the signature to the target registry. But for our use case, using the graphical tool is not an option.

The "test:0.1" image is a very simple image only containing an alpine image with a simple echo hello line as cmd. That image has been signed with cosign and a local self created key. The image has been uploaded to registry 1 into the project "images".
That is the starting situation.

Please see the following sysouts. First one copying the image into a local dir into the running container that provides skopeo version 1.14.0:

---snip---

[root@6bbacdc9b6e4 ~]# skopeo copy docker://192.168.a.b/images/test:0.1 dir:test_0-1
Getting image source signatures
Checking if image destination supports signatures
Copying blob 5b088f1e648c done |
Copying blob 96526aa774ef done |
Copying config 33b8df73a9 done |
Writing manifest to image destination
Storing signatures
[root@6bbacdc9b6e4 ~]# cd test_0-1/
[root@6bbacdc9b6e4 test_0-1]# ls -la
total 49052
drwxr-xr-x 2 root root 4096 Nov 29 21:29 .
drwxr-x--- 1 root root 4096 Nov 29 21:29 ..
-rw-r--r-- 1 root root 1127 Nov 29 21:29 33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
-rw-r--r-- 1 root root 46789009 Nov 29 21:29 5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef
-rw-r--r-- 1 root root 3401967 Nov 29 21:29 96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa
-rw-r--r-- 1 root root 740 Nov 29 21:29 manifest.json
-rw-r--r-- 1 root root 1629 Nov 29 21:29 signature-1
-rw-r--r-- 1 root root 33 Nov 29 21:29 version
[root@6bbacdc9b6e4 test_0-1]#

[root@6bbacdc9b6e4 ~]# skopeo --debug copy dir:test_0-1 docker://192.168.y.z/images/test:0.1
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Loading registries configuration "/root/.config/containers/registries.conf"
DEBU[0000] Found credentials for 192.168.y.z/images/test in credential helper containers-auth.json in file /run/containers/0/auth.json
DEBU[0000] Lookaside configuration: using "docker" namespace 192.168.y.z
DEBU[0000] No signature storage configuration found for 192.168.y.z/images/test:0.1, using built-in default file:///var/lib/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/192.168.y.z
DEBU[0000] Sigstore attachments: using "docker" namespace 192.168.y.z
DEBU[0000] Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite
DEBU[0000] IsRunningImageAllowed for image dir:/root/test_0-1
DEBU[0000] Using default policy section
DEBU[0000] Requirement 0: allowed
DEBU[0000] Overall: allowed
Getting image source signatures
Checking if image destination supports signatures
DEBU[0000] GET https://192.168.y.z/v2/
DEBU[0000] Ping https://192.168.y.z/v2/ err Get "https://192.168.y.z/v2/": dial tcp 192.168.y.z:443: connect: no route to host (&url.Error{Op:"Get", URL:"https://192.168.y.z/v2/", Err:(*net.OpError)(0xc0000403c0)})
DEBU[0000] GET http://192.168.y.z/v2/
DEBU[0000] Ping http://192.168.y.z/v2/ status 401
DEBU[0000] We can't modify the manifest, hoping for the best...
DEBU[0000] Checking if we can reuse blob sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa: general substitution = false, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0000] Checking /v2/images/test/blobs/sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa
DEBU[0000] GET http://nuc04/service/token?account=admin&scope=repository%3Aimages%2Ftest%3Apull%2Cpush&service=harbor-registry
DEBU[0000] HEAD http://192.168.y.z/v2/images/test/blobs/sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa
DEBU[0000] ... already exists
DEBU[0000] Skipping blob sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa (already present):
DEBU[0000] Checking if we can reuse blob sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef: general substitution = false, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0000] Checking /v2/images/test/blobs/sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef
DEBU[0000] HEAD http://192.168.y.z/v2/images/test/blobs/sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef
Copying blob 96526aa774ef skipped: already exists
DEBU[0000] ... already exists
Copying blob 96526aa774ef skipped: already exists
Copying blob 5b088f1e648c skipped: already exists
DEBU[0000] No compression detected
DEBU[0000] Compression change for blob sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2 ("application/vnd.docker.container.image.v1+json") not supported
DEBU[0000] Using original blob without modification
DEBU[0000] Checking /v2/images/test/blobs/sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
DEBU[0000] HEAD http://192.168.y.z/v2/images/test/blobs/sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
DEBU[0000] ... already exists
Copying config 33b8df73a9 done |
Writing manifest to image destination
DEBU[0000] PUT http://192.168.y.z/v2/images/test/manifests/0.1
Storing signatures
DEBU[0000] Looking for sigstore attachments in 192.168.y.z/images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0000] GET http://192.168.y.z/v2/images/test/manifests/sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0000] Content-Type from manifest GET is "application/json; charset=utf-8"
DEBU[0000] Fetching sigstore attachment manifest failed: reading manifest sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig in 192.168.y.z/images/test: unknown: artifact images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig not found
FATA[0000] writing signatures: reading manifest sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig in 192.168.y.z/images/test: unknown: artifact images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig not found
[root@6bbacdc9b6e4 ~]#

---snap---

If I understand the last line correctly, instead of "signature-1" a file named sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig will be expected - is this correct?

Second one showing sysout copying the image directly between harbor 1 and harbor 2:

---snip---

[root@6bbacdc9b6e4 ~]# skopeo --debug copy docker://192.168.a.b/images/test:0.1 docker://192.168.y.z/images/test:0.1
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Loading registries configuration "/root/.config/containers/registries.conf"
DEBU[0000] Found credentials for 192.168.y.z/images/test in credential helper containers-auth.json in file /run/containers/0/auth.json
DEBU[0000] Lookaside configuration: using "docker" namespace 192.168.y.z
DEBU[0000] No signature storage configuration found for 192.168.y.z/images/test:0.1, using built-in default file:///var/lib/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/192.168.y.z
DEBU[0000] Sigstore attachments: using "docker" namespace 192.168.y.z
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Trying to access "192.168.a.b/images/test:0.1"
DEBU[0000] Found credentials for 192.168.a.b/images/test in credential helper containers-auth.json in file /run/containers/0/auth.json
DEBU[0000] Lookaside configuration: using "docker" namespace 192.168.a.b
DEBU[0000] No signature storage configuration found for 192.168.a.b/images/test:0.1, using built-in default file:///var/lib/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/192.168.a.b
DEBU[0000] Sigstore attachments: using "docker" namespace 192.168.a.b
DEBU[0000] GET https://192.168.a.b/v2/
DEBU[0001] Ping https://192.168.a.b/v2/ err Get "https://192.168.a.b/v2/": dial tcp 192.168.a.b:443: connect: no route to host (&url.Error{Op:"Get", URL:"https://192.168.a.b/v2/", Err:(*net.OpError)(0xc000578780)})
DEBU[0001] GET http://192.168.a.b/v2/
DEBU[0001] Ping http://192.168.a.b/v2/ status 401
DEBU[0001] GET http://nuc02/service/token?account=admin&scope=repository%3Aimages%2Ftest%3Apull&service=harbor-registry
DEBU[0001] GET http://192.168.a.b/v2/images/test/manifests/0.1
DEBU[0001] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json"
DEBU[0001] Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite
DEBU[0001] IsRunningImageAllowed for image docker:192.168.a.b/images/test:0.1
DEBU[0001] Using default policy section
DEBU[0001] Requirement 0: allowed
DEBU[0001] Overall: allowed
Getting image source signatures
DEBU[0001] Reading /var/lib/containers/sigstore/images/test@sha256=3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db/signature-1
DEBU[0001] Looking for sigstore attachments in 192.168.a.b/images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0001] GET http://192.168.a.b/v2/images/test/manifests/sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0001] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json"
DEBU[0001] Found a sigstore attachment manifest with 1 layers
DEBU[0001] Fetching sigstore attachment 1/1: sha256:73d5db8bd77407d1cc1863c88fb584f3b658807e934c41d9b9f1edd8166abae0
DEBU[0001] Downloading /v2/images/test/blobs/sha256:73d5db8bd77407d1cc1863c88fb584f3b658807e934c41d9b9f1edd8166abae0
DEBU[0001] GET http://192.168.a.b/v2/images/test/blobs/sha256:73d5db8bd77407d1cc1863c88fb584f3b658807e934c41d9b9f1edd8166abae0
Checking if image destination supports signatures
DEBU[0001] GET https://192.168.y.z/v2/
DEBU[0002] Ping https://192.168.y.z/v2/ err Get "https://192.168.y.z/v2/": dial tcp 192.168.y.z:443: connect: no route to host (&url.Error{Op:"Get", URL:"https://192.168.y.z/v2/", Err:(*net.OpError)(0xc000578050)})
DEBU[0002] GET http://192.168.y.z/v2/
DEBU[0002] Ping http://192.168.y.z/v2/ status 401
DEBU[0002] We can't modify the manifest, hoping for the best...
DEBU[0002] Checking if we can reuse blob sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef: general substitution = false, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0002] Checking /v2/images/test/blobs/sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef
DEBU[0002] GET http://nuc04/service/token?account=admin&scope=repository%3Aimages%2Ftest%3Apull%2Cpush&service=harbor-registry
DEBU[0002] Checking if we can reuse blob sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa: general substitution = false, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0002] Checking /v2/images/test/blobs/sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa
DEBU[0002] GET http://nuc04/service/token?account=admin&scope=repository%3Aimages%2Ftest%3Apull%2Cpush&service=harbor-registry
DEBU[0002] HEAD http://192.168.y.z/v2/images/test/blobs/sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa
DEBU[0002] HEAD http://192.168.y.z/v2/images/test/blobs/sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef
DEBU[0002] ... already exists
DEBU[0002] ... already exists
DEBU[0002] Skipping blob sha256:5b088f1e648c9ea3e0260bfa316afe24d9dc938a43c9a69bdad8c06f9ed586ef (already present):
Copying blob 5b088f1e648c skipped: already exists
Copying blob 5b088f1e648c skipped: already exists
Copying blob 96526aa774ef skipped: already exists
DEBU[0002] Downloading /v2/images/test/blobs/sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
DEBU[0002] GET http://192.168.a.b/v2/images/test/blobs/sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
DEBU[0002] No compression detected
DEBU[0002] Compression change for blob sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2 ("application/vnd.docker.container.image.v1+json") not supported
DEBU[0002] Using original blob without modification
DEBU[0002] Checking /v2/images/test/blobs/sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
DEBU[0002] HEAD http://192.168.y.z/v2/images/test/blobs/sha256:33b8df73a94292833e0ca99d089e29c3be6a52df5626ab51bcda375df4b339b2
DEBU[0002] ... already exists
Copying config 33b8df73a9 done |
Writing manifest to image destination
DEBU[0002] PUT http://192.168.y.z/v2/images/test/manifests/0.1
Storing signatures
DEBU[0002] Looking for sigstore attachments in 192.168.y.z/images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0002] GET http://192.168.y.z/v2/images/test/manifests/sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0002] Content-Type from manifest GET is "application/json; charset=utf-8"
DEBU[0002] Fetching sigstore attachment manifest failed: reading manifest sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig in 192.168.y.z/images/test: unknown: artifact images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig not found
FATA[0002] writing signatures: reading manifest sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig in 192.168.y.z/images/test: unknown: artifact images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig not found
[root@6bbacdc9b6e4 ~]#

---snap---

Second attempt showing the same result.

Added the following lines to /etc/containers/registries.d/default.yaml:
docker:
192.168.a.b:
use-sigstore-attachments: true
192.168.y.z:
use-sigstore-attachments: true

Hope that helps. Please let me know if any further information is needed.

[mtrmac]

Thanks!

Storing signatures
DEBU[0000] Looking for sigstore attachments in 192.168.y.z/images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig
DEBU[0000] GET http://192.168.y.z/v2/images/test/manifests/sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig 
DEBU[0000] Content-Type from manifest GET is "application/json; charset=utf-8"
DEBU[0000] Fetching sigstore attachment manifest failed: reading manifest sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig in 192.168.y.z/images/test: unknown: artifact images/test:sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig not found 

OK, the registry (ref. https://github.com/goharbor/harbor/blob/7cef4217b03db1cb61d9fabded22540fafbff4b2/src/controller/artifact/controller.go#L298) apparently uses a non-standard "code" value NOT_FOUND: https://github.com/goharbor/harbor/blob/7cef4217b03db1cb61d9fabded22540fafbff4b2/src/lib/errors/const.go#L19 (and that turns into unknown) but that value should turn into a 404, looking at https://github.com/goharbor/harbor/blob/7cef4217b03db1cb61d9fabded22540fafbff4b2/src/lib/http/error.go#L39.

If I understand the last line correctly, instead of "signature-1" a file named sha256-3a1332abd837b013a0378d662f403387f64a5bdc21c174031705d9de07d1a0db.sig will be expected - is this correct?

No, that name refers to a tag on the destination registry (and a component of the URLs quoted above). That works as expected.

---

Either way, we’ll want to capture the full HTTP response of the above-quoted HTTP request (with the relevant authentication headers). Is that easy to do for you in the current environment? If not, we’ll need a source code patch to capture that in Skopeo.

[mtrmac]

… looking more closely (and compare goharbor/harbor#19179 , not the same request but a sample of the error message):

• The registry returns a 404, as requested
• The registry chooses to return JSON, using the standard format per https://github.com/opencontainers/distribution-spec/blob/main/spec.md#error-codes . But it violates the spec in not using the right "code" value.

our isManifestUknown can, currently, only check for status 404 if the registry does not use the standard JSON format; if the standard JSON format is used, the HTTP status value is not currently available in isManifestUnknown.

[Plat0nicS0lid]

Hello again,
many thanks for your help and looking for the cause. Apologies for the delay and not answered earlier. May I ask for some guidance how to deliver the full http response? If this is something a sopeo option handles, could you please tell me how to manage it? Or is this something that can be recorded with a running tcpdump, or similar? If yes, which options have I to use, to record the necessary information for you?

If I understand you correctly, the situation is that a not expected response will be returned to prevent the signature to be delivered to harbor. Is this something that can be shipped around inside skopeo or must this be changed on harbor side? As the image copy works between two harbor registries directly, also if the source image has a signature, the functionality is available in general.
Just to see the copy result between two registries working if copying with skopeo, could you help me out with a registry product which would work? Basically, there will be no option to switch to another product than harbor, but it is my personal interest to see that process working in general.

[STARRY-S]

Hi @mtrmac, I use the Harbor server (v2.10.2) and debug the skopeo to capture the HTTP response body content when the non-standard "code" value NOT_FOUND appared:

Hope this information helps and feel free to ask me if you need me to capture additional debug information, thanks.

---

By refer to goharbor/harbor#19179, if I understand correctly, it seems like this will be fixed in harbor v2.11.0.

[mtrmac]

@STARRY-S Thanks; that’s the request body, but not full request headers. Could you perhaps apply https://github.com/containers/image/pull/201/files (with s/Trace/Error/g) to Skopeo, and capture the response from the output of that?

(I don’t expect the response to contain credentials or secrets, but please double-check.)

[STARRY-S]

@mtrmac Hi, here's the log with captured response output:

DEBU[0000] Reading user-specified passphrase for signing from password.txt
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf"
DEBU[0000] Found credentials for harbor.example.local:30080/library/alpine in credential helper containers-auth.json in file /Users/demo/.config/containers/auth.json
DEBU[0000]  Lookaside configuration: using "default-docker" configuration
DEBU[0000]  No signature storage configuration found for harbor.example.local:30080/library/alpine:3.1, using built-in default file:///Users/demo/.local/share/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/harbor.example.local:30080
DEBU[0000]  Sigstore attachments: using "default-docker" configuration
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Trying to access "harbor.example.local:30080/test/alpine:3.1"
DEBU[0000] Found credentials for harbor.example.local:30080/test/alpine in credential helper containers-auth.json in file /Users/demo/.config/containers/auth.json
DEBU[0000]  Lookaside configuration: using "default-docker" configuration
DEBU[0000]  No signature storage configuration found for harbor.example.local:30080/test/alpine:3.1, using built-in default file:///Users/demo/.local/share/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/harbor.example.local:30080
DEBU[0000]  Sigstore attachments: using "default-docker" configuration
DEBU[0000] GET https://harbor.example.local:30080/v2/
ERRO[0000] ===REQ===
GET /v2/ HTTP/1.1
Host: harbor.example.local:30080
User-Agent: skopeo/1.16.0-dev
Docker-Distribution-Api-Version: registry/2.0
Accept-Encoding: gzip


===REQ===
ERRO[0005] ===RES error: Get "https://harbor.example.local:30080/v2/": http: server gave HTTP response to HTTPS client
DEBU[0005] Ping https://harbor.example.local:30080/v2/ err Get "https://harbor.example.local:30080/v2/": http: server gave HTTP response to HTTPS client (&url.Error{Op:"Get", URL:"https://harbor.example.local:30080/v2/", Err:(*errors.errorString)(0x10223e550)})
DEBU[0005] GET http://harbor.example.local:30080/v2/
ERRO[0005] ===REQ===
GET /v2/ HTTP/1.1
Host: harbor.example.local:30080
User-Agent: skopeo/1.16.0-dev
Docker-Distribution-Api-Version: registry/2.0
Accept-Encoding: gzip


===REQ===
ERRO[0010] ===RES===
HTTP/1.1 401 Unauthorized
Content-Length: 76
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Wed, 08 May 2024 08:14:54 GMT
Docker-Distribution-Api-Version: registry/2.0
Server: nginx
Set-Cookie: sid=7a4ac349c283447409991ecf3f3a9fc8; Path=/; HttpOnly
Www-Authenticate: Bearer realm="http://harbor.example.local:30080/service/token",service="harbor-registry"
X-Request-Id: f1750081-2187-4bb4-92f9-8b53158edba2


===RES===
DEBU[0010] Ping http://harbor.example.local:30080/v2/ status 401
DEBU[0010] GET http://harbor.example.local:30080/service/token?account=admin&scope=repository%3Atest%2Falpine%3Apull&service=harbor-registry
ERRO[0010] ===REQ===
GET /service/token?account=admin&scope=repository%3Atest%2Falpine%3Apull&service=harbor-registry HTTP/1.1
Host: harbor.example.local:30080
User-Agent: skopeo/1.16.0-dev
Authorization: Basic YWRta ......
Accept-Encoding: gzip


===REQ===
ERRO[0015] ===RES===
HTTP/1.1 200 OK
Connection: keep-alive
Content-Security-Policy: frame-ancestors 'none'
Content-Type: application/json; charset=utf-8
Date: Wed, 08 May 2024 08:14:59 GMT
Server: nginx
Set-Cookie: sid=7a50d7206192e46800a72b59a2ba57ae; Path=/; HttpOnly
X-Frame-Options: DENY
X-Request-Id: aa719dc7-a408-4913-9591-a96eefbab122


===RES===
DEBU[0015] GET http://harbor.example.local:30080/v2/test/alpine/manifests/3.1
ERRO[0015] ===REQ===
GET /v2/test/alpine/manifests/3.1 HTTP/1.1
Host: harbor.example.local:30080
User-Agent: skopeo/1.16.0-dev
Accept: application/vnd.oci.image.manifest.v1+json
Accept: application/vnd.docker.distribution.manifest.v2+json
Accept: application/vnd.docker.distribution.manifest.v1+prettyjws
Accept: application/vnd.docker.distribution.manifest.v1+json
Accept: application/vnd.docker.distribution.manifest.list.v2+json
Accept: application/vnd.oci.image.index.v1+json
Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjJCRE46WFBBSjpLT0VBOkxJNEg6WUw2QTpTM0pIOkRMUU46MjVGTDpZSDNUOlBQN1I6NFhHTzpXTjNVIiwidHlwIjoiSldUIn0.eyJpc3MiOiJoYXJib3ItdG9rZW4taXNzdWVyIiwic3ViIjoiYWRtaW4iLCJhdWQiOiJoYXJib3ItcmVnaXN0cnkiLCJleHAiOjE3MTUxNTc4OTksIm5iZiI6MTcxNTE1NjA5OSwiaWF0IjoxNzE1MTU2MDk5LCJqdGkiOiJ6eFRxM3pCQ21BaXpvbk5OIiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJ0ZXN0L2FscGluZSIsImFjdGlvbnMiOlsiZGVsZXRlIiwicHVsbCIsInB1c2giXX1dfQ.ZfSStFow3rSvADnAZN2H2odbXylWRE3V3bTcfj2ZNiScI_VS9IOUmymyf5jV8WK4Ip2uuyb4wYRwz1RLyUbeAUfORArs24pjR9UQQmv-TuHVPbb6bJ-wBCZjP7XNe0ygQhoARO9HQVeZhsE0UCZXBmlscuEQnrDG-pdVtSysV3Rp22E8GDNiAuuJGZFF9a60QrGgX_osIztIJVpLzo7e7Lviw94Vh0w-L1lbupsioRCy3tRcVGimIQd7OcIN3MV8lG8fBgg6AvWvDjO24rDp8Gf19srDDv2z8FTrYQLfigSlA9eMInsSP_n661VVTM-Bkg92kB5ghZWRlQTs7Rj3vQ
Docker-Distribution-Api-Version: registry/2.0
Accept-Encoding: gzip


===REQ===
ERRO[0015] ===RES===
HTTP/1.1 200 OK
Content-Length: 402
Connection: keep-alive
Content-Security-Policy: frame-ancestors 'none'
Content-Type: application/vnd.docker.distribution.manifest.list.v2+json
Date: Wed, 08 May 2024 08:14:59 GMT
Docker-Content-Digest: sha256:443205b0cfcc78444321d56a2fe273f06e27b2c72b5058f8d7e975997d45b015
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:443205b0cfcc78444321d56a2fe273f06e27b2c72b5058f8d7e975997d45b015"
Server: nginx
Set-Cookie: sid=8c840d44eee862b615b0f2a3a0bfd565; Path=/; HttpOnly
X-Frame-Options: DENY
X-Request-Id: 36d3ad30-7a6d-4987-84a6-1c32c3d6a03a


===RES===
DEBU[0015] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.list.v2+json"
DEBU[0015] Using SQLite blob info cache at /Users/demo/.local/share/containers/cache/blob-info-cache-v1.sqlite
DEBU[0015] Source is a manifest list; copying all instances
Getting image list signatures
DEBU[0015] Reading /Users/demo/.local/share/containers/sigstore/test/alpine@sha256=443205b0cfcc78444321d56a2fe273f06e27b2c72b5058f8d7e975997d45b015/signature-1
DEBU[0015] Looking for sigstore attachments in harbor.example.local:30080/test/alpine:sha256-443205b0cfcc78444321d56a2fe273f06e27b2c72b5058f8d7e975997d45b015.sig
DEBU[0015] GET http://harbor.example.local:30080/v2/test/alpine/manifests/sha256-443205b0cfcc78444321d56a2fe273f06e27b2c72b5058f8d7e975997d45b015.sig
ERRO[0015] ===REQ===
GET /v2/test/alpine/manifests/sha256-443205b0cfcc78444321d56a2fe273f06e27b2c72b5058f8d7e975997d45b015.sig HTTP/1.1
Host: harbor.example.local:30080
User-Agent: skopeo/1.16.0-dev
Accept: application/vnd.oci.image.manifest.v1+json
Accept: application/vnd.docker.distribution.manifest.v2+json
Accept: application/vnd.docker.distribution.manifest.v1+prettyjws
Accept: application/vnd.docker.distribution.manifest.v1+json
Accept: application/vnd.docker.distribution.manifest.list.v2+json
Accept: application/vnd.oci.image.index.v1+json
Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IjJCRE46WFBBSjpLT0VBOkxJNEg6WUw2QTpTM0pIOkRMUU46MjVGTDpZSDNUOlBQN1I6NFhHTzpXTjNVIiwidHlwIjoiSldUIn0.eyJpc3MiOiJoYXJib3ItdG9rZW4taXNzdWVyIiwic3ViIjoiYWRtaW4iLCJhdWQiOiJoYXJib3ItcmVnaXN0cnkiLCJleHAiOjE3MTUxNTc4OTksIm5iZiI6MTcxNTE1NjA5OSwiaWF0IjoxNzE1MTU2MDk5LCJqdGkiOiJ6eFRxM3pCQ21BaXpvbk5OIiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJ0ZXN0L2FscGluZSIsImFjdGlvbnMiOlsiZGVsZXRlIiwicHVsbCIsInB1c2giXX1dfQ.ZfSStFow3rSvADnAZN2H2odbXylWRE3V3bTcfj2ZNiScI_VS9IOUmymyf5jV8WK4Ip2uuyb4wYRwz1RLyUbeAUfORArs24pjR9UQQmv-TuHVPbb6bJ-wBCZjP7XNe0ygQhoARO9HQVeZhsE0UCZXBmlscuEQnrDG-pdVtSysV3Rp22E8GDNiAuuJGZFF9a60QrGgX_osIztIJVpLzo7e7Lviw94Vh0w-L1lbupsioRCy3tRcVGimIQd7OcIN3MV8lG8fBgg6AvWvDjO24rDp8Gf19srDDv2z8FTrYQLfigSlA9eMInsSP_n661VVTM-Bkg92kB5ghZWRlQTs7Rj3vQ
Docker-Distribution-Api-Version: registry/2.0
Accept-Encoding: gzip


===REQ===
ERRO[0015] ===RES===
HTTP/1.1 404 Not Found
Content-Length: 153
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Wed, 08 May 2024 08:14:59 GMT
Server: nginx
Set-Cookie: sid=f617c257877837614ada2561513d6827; Path=/; HttpOnly
X-Request-Id: 1b151fb1-c943-4190-a9ce-5156ed5e3200


===RES===
DEBU[0015] Content-Type from manifest GET is "application/json; charset=utf-8"
DEBU[0015] Fetching sigstore attachment manifest failed: reading manifest sha256-443205b0cfcc78444321d56a2fe273f06e27b2c72b5058f8d7e975997d45b015.sig in harbor.example.local:30080/test/alpine: unknown: artifact test/alpine:sha256-443205b0cfcc78444321d56a2fe273f06e27b2c72b5058f8d7e975997d45b015.sig not found
FATA[0015] reading signatures: reading manifest sha256-443205b0cfcc78444321d56a2fe273f06e27b2c72b5058f8d7e975997d45b015.sig in harbor.example.local:30080/test/alpine: unknown: artifact test/alpine:sha256-443205b0cfcc78444321d56a2fe273f06e27b2c72b5058f8d7e975997d45b015.sig not found
Exiting.

HTTP response header debug:

HTTP response body:

"{\"errors\":[{\"code\":\"NOT_FOUND\",\"message\":\"artifact test/alpine:sha256-443205b0cfcc78444321d56a2fe273f06e27b2c72b5058f8d7e975997d45b015.sig not found\"}]}\n"

[mtrmac]

@STARRY-S Thanks for the data!

#2413 should help fix this. Would you be willing to test and confirm, please?

[STARRY-S]

@STARRY-S Thanks for the data!

#2413 should help fix this. Would you be willing to test and confirm, please?

Using PR #2413, replacing the github.com/containers/image/v5 depency with github.com/mtrmac/image/v5@v5.0.0-20240514091713-60b732e62411, test code.

Test Result

Command:

skopeo copy \
    docker://docker.io/library/alpine:3 \
    docker://harbor.example.local/library/alpine:3 \
    --src-tls-verify=false --dest-tls-verify=false \
    --sign-by-sigstore-private-key=sigstore.private \
    --sign-passphrase-file=password.txt \
    --multi-arch=all \
    --preserve-digests

Log outputs:

Getting image list signatures
Copying 7 images generated from 7 images in list
Copying image sha256:6457d53fb065d6f250e1504b9bc42d5b6c65941d57532c072d929dd0628977d0 (1/7)
Getting image source signatures
Copying blob 4abcf2066143 skipped: already exists
Copying config 05455a0888 done   |
Writing manifest to image destination
Creating signature: Signing image using a sigstore signature
Storing signatures
Copying image sha256:b229a85166aadbde58e73e03c5e2b9737fb4642ffb2d98ba453adc90d144c1d8 (2/7)
Getting image source signatures
Copying blob 0dc2e6c0f9de skipped: already exists
Copying config 935b61847f done   |
Writing manifest to image destination
Creating signature: Signing image using a sigstore signature
Storing signatures
Copying image sha256:ec299a7ba3c670e38642b0b62a0c779d84b249a3c889757e2b6f841433b4c6fe (3/7)
Getting image source signatures
Copying blob fda0ff469afd skipped: already exists
Copying config 30c69795e4 done   |
Writing manifest to image destination
Creating signature: Signing image using a sigstore signature
Storing signatures
Copying image sha256:a0264d60f80df12bc1e6dd98bae6c43debe6667c0ba482711f0d806493467a46 (4/7)
Getting image source signatures
Copying blob bca4290a9639 skipped: already exists
Copying config ace17d5d88 done   |
Writing manifest to image destination
Creating signature: Signing image using a sigstore signature
Storing signatures
Copying image sha256:15c46ced65c6abed6a27472a7904b04273e9a8091a5627badd6ff016ab073171 (5/7)
Getting image source signatures
Copying blob 4a0759b5afbf skipped: already exists
Copying config 2d433224a9 done   |
Writing manifest to image destination
Creating signature: Signing image using a sigstore signature
Storing signatures
Copying image sha256:b12b826de1ec8c4237aa09a0287e7be8bd317586f32bf6cd9395ec5dba52a3a2 (6/7)
Getting image source signatures
Copying blob f4968021da4f skipped: already exists
Copying config 5b984dd032 done   |
Writing manifest to image destination
Creating signature: Signing image using a sigstore signature
Storing signatures
Copying image sha256:5d0da60400afb021f2d8dbfec8b7d26457e77eb8825cba90eba84319133f0efe (7/7)
Getting image source signatures
Copying blob eb8fba61d864 skipped: already exists
Copying config 8fc740d8c4 done   |
Writing manifest to image destination
Creating signature: Signing image using a sigstore signature
Storing signatures
Writing manifest list to image destination
Creating signature: Signing image using a sigstore signature
Storing list signatures

Result

The error message unknown: artifact images/test:sha256-xxx.sig not found no longer exists, passed.

[mtrmac]

Thanks for the testing! #2413 was merged.

[Plat0nicS0lid]

Hello @mtrmac, Hello @STARRY-S ,

unfortunatelly, I cannot agree that the problem is solved.
I also tested the image replication from one harbor instance to a second one. Basically the same situation as from the beginning.

Let me describe the setup:
VM 0: Docker host - Scopeo Image v 1.15.0 - pulled via quay.io/skopeo/stable:v1.15.0, current latest one (28 days ago)
VM 1: Docker host - Docker Image(s) Harbor version 2.10.2
VM 2: Docker host - Docker Image(s) Harbor version 2.10.2

Test case:

• pulled alpine image to vm0, tagged it with vm1 docker suffix and pushed it to vm1 harbor - into project images.
• signed image in vm1 harbor with cosign
• image signature is visible in vm1 harbor
• now, using skopeo:v1.15.0 image to replicate image from vm1 harbor to vm2 harbor - expecting image will be replicated together with the existing signature - it is not planned to (re-)sign the image again after replication. This is a different approach as tested by @STARRY-S.

Question: is the skopeo image v.1.15.0 to old and does not contain the changes described above, or is the working outcome the result of the different steps, replicating an image from docker hub to an own harbor instance and just sign that one after replication. This is another approach as mine.
If I need a newer skopeo image, please tell me where to pull it from. As the v1.15.0 is older than 28 days, I assume that this latest one does not contain the changes.

skopeo copy --debug docker://192.168.210.91/images/alpine:3.18.4 docker://192.168.210.93/images/alpine:3.18.4
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Loading registries configuration "/root/.config/containers/registries.conf"
DEBU[0000] Found credentials for 192.168.210.93/images/alpine in credential helper containers-auth.json in file /tmp/auth.json
DEBU[0000]  Lookaside configuration: using "docker" namespace 192.168.210.93
DEBU[0000]  No signature storage configuration found for 192.168.210.93/images/alpine:3.18.4, using built-in default file:///var/lib/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/192.168.210.93
DEBU[0000]  Sigstore attachments: using "docker" namespace 192.168.210.93
DEBU[0000] Using registries.d directory /etc/containers/registries.d
DEBU[0000] Trying to access "192.168.210.91/images/alpine:3.18.4"
DEBU[0000] Found credentials for 192.168.210.91/images/alpine in credential helper containers-auth.json in file /tmp/auth.json
DEBU[0000]  Lookaside configuration: using "docker" namespace 192.168.210.91
DEBU[0000]  No signature storage configuration found for 192.168.210.91/images/alpine:3.18.4, using built-in default file:///var/lib/containers/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/192.168.210.91
DEBU[0000]  Sigstore attachments: using "docker" namespace 192.168.210.91
DEBU[0000] GET https://192.168.210.91/v2/
DEBU[0000] Ping https://192.168.210.91/v2/ err Get "https://192.168.210.91/v2/": dial tcp 192.168.210.91:443: connect: no route to host (&url.Error{Op:"Get", URL:"https://192.168.210.91/v2/", Err:(*net.OpError)(0xc000516190)})
DEBU[0000] GET http://192.168.210.91/v2/
DEBU[0000] Ping http://192.168.210.91/v2/ status 401
DEBU[0000] GET http://nuc02/service/token?account=sascha&scope=repository%3Aimages%2Falpine%3Apull&service=harbor-registry
DEBU[0000] GET http://192.168.210.91/v2/images/alpine/manifests/3.18.4
DEBU[0000] Content-Type from manifest GET is "application/vnd.docker.distribution.manifest.v2+json"
DEBU[0000] Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite
DEBU[0000] IsRunningImageAllowed for image docker:192.168.210.91/images/alpine:3.18.4
DEBU[0000]  Using default policy section
DEBU[0000]  Requirement 0: allowed
DEBU[0000] Overall: allowed
Getting image source signatures
DEBU[0000] Reading /var/lib/containers/sigstore/images/alpine@sha256=48d9183eb12a05c99bcc0bf44a003607b8e941e1d4f41f9ad12bdcc4b5672f86/signature-1
DEBU[0000] Looking for sigstore attachments in 192.168.210.91/images/alpine:sha256-48d9183eb12a05c99bcc0bf44a003607b8e941e1d4f41f9ad12bdcc4b5672f86.sig
DEBU[0000] GET http://192.168.210.91/v2/images/alpine/manifests/sha256-48d9183eb12a05c99bcc0bf44a003607b8e941e1d4f41f9ad12bdcc4b5672f86.sig
DEBU[0000] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json"
DEBU[0000] Found a sigstore attachment manifest with 1 layers
DEBU[0000] Fetching sigstore attachment 1/1: sha256:73915660087de524a0f777380d9d6f6b7b7df31c9c63f77d5375bdb5d3c1ad54
DEBU[0000] Downloading /v2/images/alpine/blobs/sha256:73915660087de524a0f777380d9d6f6b7b7df31c9c63f77d5375bdb5d3c1ad54
DEBU[0000] GET http://192.168.210.91/v2/images/alpine/blobs/sha256:73915660087de524a0f777380d9d6f6b7b7df31c9c63f77d5375bdb5d3c1ad54
Checking if image destination supports signatures
DEBU[0000] GET https://192.168.210.93/v2/
DEBU[0000] Ping https://192.168.210.93/v2/ err Get "https://192.168.210.93/v2/": dial tcp 192.168.210.93:443: connect: no route to host (&url.Error{Op:"Get", URL:"https://192.168.210.93/v2/", Err:(*net.OpError)(0xc0005163c0)})
DEBU[0000] GET http://192.168.210.93/v2/
DEBU[0000] Ping http://192.168.210.93/v2/ status 401
DEBU[0000] We can't modify the manifest, hoping for the best...
DEBU[0000] Checking if we can reuse blob sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa: general substitution = false, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0000] Checking /v2/images/alpine/blobs/sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa
DEBU[0000] GET http://nuc07/service/token?account=sascha&scope=repository%3Aimages%2Falpine%3Apull%2Cpush&service=harbor-registry
DEBU[0000] HEAD http://192.168.210.93/v2/images/alpine/blobs/sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa
DEBU[0000] ... already exists
DEBU[0000] Skipping blob sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa (already present):
Copying blob 96526aa774ef skipped: already exists
DEBU[0000] Downloading /v2/images/alpine/blobs/sha256:8ca4688f4f356596b5ae539337c9941abc78eda10021d35cbc52659c74d9b443
DEBU[0000] GET http://192.168.210.91/v2/images/alpine/blobs/sha256:8ca4688f4f356596b5ae539337c9941abc78eda10021d35cbc52659c74d9b443
DEBU[0000] No compression detected
DEBU[0000] Compression change for blob sha256:8ca4688f4f356596b5ae539337c9941abc78eda10021d35cbc52659c74d9b443 ("application/vnd.docker.container.image.v1+json") not supported
DEBU[0000] Using original blob without modification
DEBU[0000] Checking /v2/images/alpine/blobs/sha256:8ca4688f4f356596b5ae539337c9941abc78eda10021d35cbc52659c74d9b443
DEBU[0000] HEAD http://192.168.210.93/v2/images/alpine/blobs/sha256:8ca4688f4f356596b5ae539337c9941abc78eda10021d35cbc52659c74d9b443
DEBU[0000] ... already exists
Copying config 8ca4688f4f done   |
Writing manifest to image destination
DEBU[0000] PUT http://192.168.210.93/v2/images/alpine/manifests/3.18.4
Storing signatures
DEBU[0000] Looking for sigstore attachments in 192.168.210.93/images/alpine:sha256-48d9183eb12a05c99bcc0bf44a003607b8e941e1d4f41f9ad12bdcc4b5672f86.sig
DEBU[0000] GET http://192.168.210.93/v2/images/alpine/manifests/sha256-48d9183eb12a05c99bcc0bf44a003607b8e941e1d4f41f9ad12bdcc4b5672f86.sig
DEBU[0000] Content-Type from manifest GET is "application/json; charset=utf-8"
DEBU[0000] Fetching sigstore attachment manifest failed: reading manifest sha256-48d9183eb12a05c99bcc0bf44a003607b8e941e1d4f41f9ad12bdcc4b5672f86.sig in 192.168.210.93/images/alpine: unknown: artifact images/alpine:sha256-48d9183eb12a05c99bcc0bf44a003607b8e941e1d4f41f9ad12bdcc4b5672f86.sig not found
FATA[0000] writing signatures: reading manifest sha256-48d9183eb12a05c99bcc0bf44a003607b8e941e1d4f41f9ad12bdcc4b5672f86.sig in 192.168.210.93/images/alpine: unknown: artifact images/alpine:sha256-48d9183eb12a05c99bcc0bf44a003607b8e941e1d4f41f9ad12bdcc4b5672f86.sig not found
[root@c86d2ed07d47 /]#

skopeo --version
skopeo version 1.15.0

If I copy within the container from one directory to another, the signature is available:

skopeo copy --debug dir:alpine_3-18-4 dir:alpine_3.18.4
DEBU[0000] overwriting existing container image directory "/root/alpine_3.18.4"
DEBU[0000] Using SQLite blob info cache at /var/lib/containers/cache/blob-info-cache-v1.sqlite
DEBU[0000] IsRunningImageAllowed for image dir:/root/alpine_3-18-4
DEBU[0000]  Using default policy section
DEBU[0000]  Requirement 0: allowed
DEBU[0000] Overall: allowed
Getting image source signatures
Checking if image destination supports signatures
DEBU[0000] We can't modify the manifest, hoping for the best...
DEBU[0000] Checking if we can reuse blob sha256:96526aa774ef0126ad0fe9e9a95764c5fc37f409ab9e97021e7b4775d82bf6fa: general substitution = false, compression for MIME type "application/vnd.docker.image.rootfs.diff.tar.gzip" = true
DEBU[0000] Detected compression format gzip
DEBU[0000] Using original blob without modification
Copying blob 96526aa774ef done   |
DEBU[0000] No compression detected
DEBU[0000] Compression change for blob sha256:8ca4688f4f356596b5ae539337c9941abc78eda10021d35cbc52659c74d9b443 ("application/vnd.docker.container.image.v1+json") not supported
DEBU[0000] Using original blob without modification
Copying config 8ca4688f4f done   |
Writing manifest to image destination
Storing signatures

ll
total 16
drwxr-xr-x. 2 root root 4096 May 30 23:57 alpine_3-18-4
drwxr-xr-x. 2 root root 4096 May 31 12:22 alpine_3.18.4
[root@c86d2ed07d47 ~]# ls alpin* | grep sig
signature-1
signature-1
[root@c86d2ed07d47 ~]# diff alpine_3.18.4/signature-1 alpine_3-18-4/signature-1
[root@c86d2ed07d47 ~]#

May I ask for another test or check?

Best regards

[mtrmac]

is the skopeo image v.1.15.0 to old and does not contain the changes described above

Yes; this was included in c/image v5.31.0, and that isn’t in a Skopeo release yet.

Per https://github.com/containers/image_build/blob/main/README.md , those unreleased versions should have (unstable! testing-only!) builds at https://quay.io/repository/skopeo/upstream, but https://quay.io/repository/skopeo/upstream?tab=history shows the tag being deleted. @cevich is that just me forgetting about an announced change, or is that unexpected?

[cevich]

Oh! That's wrong, there should always be a "latest" tag there. Checking podman and buildah, it's the same deal. Something is wrong. I'll look into it.

[Plat0nicS0lid]

Thanks @mtrmac and @cevich,
found and pulled quay.io/skopeo/upstream:latest which shows version:

skopeo --version
skopeo version 1.16.0-dev

Test result:
Used command:

skopeo copy docker://192.168.210.91/images/alpine:3.18.4 docker://192.168.210.93/images/alpine:3.18.4

.
The image will be replicated with the already existing signature to vm02 harbor instance. Inside the project, the image now shows an existing signature.

Many thanks for your help.