Test cleartext signature support in gpgme and sequoia backends #307

jlebon · 2025-09-03T11:53:44Z

Currently, simple signatures are expected to be in the binary GPG format, and that's what e.g. podman push --sign-by produces as well. But the code for all backends work today with cleartext signatures.

Add a new test to cover this case. But only in the GPGME and Sequoia backends since the OpenPGP backend does not support it and cannot easily be supported (see also 1).

The reason why I'm interested in this is that I'd like to make use of it for signing Fedora CoreOS container images. The end-goal is to move to Sigstore signing, but until that's ready, we'd like to use GPG signing.

We use Robosignatory, the Fedora signing service, which only supports detached signatures, and while it's theoretically possible to convert the detached signatures we get back into inline binary signatures, it's much less cumbersome and error-prone to convert it to cleartext signatures.

It's worth noting that while Fedora's signing server (Sigul) does support container image signing, Robosignatory does not surface it yet (see https://pagure.io/robosignatory/issue/22).

Fixing that wouldn't be too hard I think, but all this is ideally short-term anyway until we can move to Sigstore signing + Konflux. There's work in progress on that (see e.g. https://discussion.fedoraproject.org/t/148999).

The primary goal here is just ensuring that this keeps working until we move off of it.

jlebon · 2025-09-03T11:54:16Z

I only tested the gpgme backend, but AIUI I think CI will test the other backends. (Right?)

mtrmac

I only tested the gpgme backend, but AIUI I think CI will test the other backends. (Right?)

Yes, and it is failing in the openpgp implementation, as suspected.

This then is implying that this is a supported use case.

I’m somewhat tempted to make it a tested/supported but not very documented use case . Well, that’s what this PR is doing anyway, by adding a test and not extending the signature format documentation. :)

The added test LGTM. If the OpenPGP mechanism will need extending, we might need to add extra unit tests if the existing tests don’t exercise all lines.

jlebon · 2025-09-03T16:26:38Z

I only tested the gpgme backend, but AIUI I think CI will test the other backends. (Right?)

Yes, and it is failing in the openpgp implementation, as suspected.

This then is implying that this is a supported use case.

I’m somewhat tempted to make it a tested/supported but not very documented use case . Well, that’s what this PR is doing anyway, by adding a test and not extending the signature format documentation. :)

Yup, that's fine with me! :) Happy to add a marker anywhere else as needed to signal this.

The added test LGTM. If the OpenPGP mechanism will need extending, we might need to add extra unit tests if the existing tests don’t exercise all lines.

Hmm, right. I do see https://pkg.go.dev/golang.org/x/crypto@v0.41.0/openpgp/clearsign and could try to get that working. Though given golang/go#44226, I wonder if it'd also be reasonable to just... not support it there?

mtrmac · 2025-09-03T16:37:16Z

I wonder if it'd also be reasonable to just... not support it there?

That would make the clear-signed variant a clearly-unsupported format. That might be fine? To an extent, what matters is the consumers that are relevant to your workflow.

The openpgp variant was originally built for OpenShift, which wanted a pure-Go implementation. That might not be directly relevant.

OTOH, for some uses, third-party SBOM / policy checkers and image audit workflows can use ~arbitrary implementations, perhaps not even c/image, and in that case it would be hard to argue that a format not universally accepted even by c/image should be supported by all such software.

jlebon · 2025-09-04T03:00:17Z

OK added support for it in the openpgp backend as well!

jlebon · 2025-09-05T20:00:39Z

Anything else on this one?

mtrmac

I’m afraid there’s a rather major snag with this plan: the CheckDetachedSignature code path is not enforcing everything the existing code does.

In particular, the SigLifetimeSecs check is, AFAICS, impossible to do with CheckDetachedSignature because the function does not return the signature packet.

(Ultimately, for confidence, a lot of TestGPGSigningMechanismVerify should be duplicated with clear-signed signatures, to ensure parity. Purely by code review, it seems mostly fine — but the signature expiration is a clear divergence.)

I don’t know what’s an easy path forward here, given the deprecated/frozen status of x/crypto/openpgp. Moving away from that backend, maybe to one of its forks, seems possible (e.g., from a random look which is by no means a recommendation, ProtonMail/go-crypto seems to have code that both returns the data, and enforces validity automatically), but that’s a somewhat different project / PR.

mtrmac · 2025-09-19T18:06:27Z

image/signature/mechanism_openpgp.go

 // or the mechanism should implement signingMechanismWithVerificationIdentityLookup.
 func (m *openpgpSigningMechanism) Verify(unverifiedSignature []byte) (contents []byte, keyIdentity string, err error) {
+	// First, try to decode it as a clearsigned message.
+	block, _ := clearsign.Decode(unverifiedSignature)


(Purely aesthetically, I’d prefer the expected code path, i.e. a non-clearsigned signature, to be tried first. Ultimately that makes no difference to security.)

mtrmac · 2025-09-19T18:06:45Z

image/signature/mechanism_openpgp.go

+		// It is a clearsigned message.
+		signer, err := openpgp.CheckDetachedSignature(m.keyring, bytes.NewReader(block.Bytes), block.ArmoredSignature.Body)
+		if err != nil {
+			return nil, "", err


This failure path does not have test coverage.

jlebon · 2025-09-19T20:08:50Z

Ahh, tough.

So let me narrow this down a bit more. My main goal here is to make sure signature verification of update payloads in FCOS clients don't break in the future before we switch over to Sigstore. So I'm most particularly interested in the paths relevant for Fedora, which I think today is the gpgme backend (but possibly in the future Sequoia?), and those are covered.

I.e. going back to this conversation:

I wonder if it'd also be reasonable to just... not support it there?

That would make the clear-signed variant a clearly-unsupported format. That might be fine? To an extent, what matters is the consumers that are relevant to your workflow.

...

OTOH, for some uses, third-party SBOM / policy checkers and image audit workflows can use ~arbitrary implementations, perhaps not even c/image, and in that case it would be hard to argue that a format not universally accepted even by c/image should be supported by all such software.

Yeah, I'm personally OK with us not claiming full support for it and with the third-party caveat.

What I mostly want to avoid is a change that lands here that breaks the FCOS workflow and only finding out when we're about to do a release and fail update testing.

mtrmac · 2025-10-06T12:42:54Z

Per the above, I think this should work — assuming there are no relevant consumers of these signatures choosing to use OpenPGP:

Don’t promise anything formally; this is a “happens to work currently” situation. (In particular, if we ended up adding another mechanism, or replacing the default in 5 years, there’s no commitment to accepting the cleartext signatures in the new one. I think that’s very unlikely, but 2 years ago I wouldn’t have predicted the move to Sequoia.)
Drop the new mechanism_openpgp implementation.
Take the newly added test, and move it from the generic mechanism_test.go to mechanism_gpgme_test.go and mechamism_sequoia_test.go.

That would ensure that we’d see blocking test failure if this broke (which would be an API break of GPGME / Sequoia, and I think very unlikely), and in that unlikely case we could discuss what to do.

Currently, simple signatures are expected to be in the binary GPG format, and that's what e.g. `podman push --sign-by` produces as well. But the code for all backends work today with cleartext signatures. Add a new test to cover this case. But only in the GPGME and Sequoia backends since the OpenPGP backend does not support it and cannot easily be supported (see also [1]). The reason why I'm interested in this is that I'd like to make use of it for signing Fedora CoreOS container images. The end-goal is to move to Sigstore signing, but until that's ready, we'd like to use GPG signing. We use Robosignatory, the Fedora signing service, which only supports detached signatures, and while it's theoretically possible to convert the detached signatures we get back into inline binary signatures, it's much less cumbersome and error-prone to convert it to cleartext signatures. It's worth noting that while Fedora's signing server (Sigul) does support container image signing, Robosignatory does not surface it yet (see https://pagure.io/robosignatory/issue/22). Fixing that wouldn't be too hard I think, but all this is ideally short-term anyway until we can move to Sigstore signing + Konflux. There's work in progress on that (see e.g. https://discussion.fedoraproject.org/t/148999). The primary goal here is just ensuring that this keeps working until we move off of it. Signed-off-by: Jonathan Lebon <jonathan@jlebon.com> [1]: containers#307 (review)

jlebon · 2025-10-06T15:51:09Z

Updated to drop the "supported" wording, the openpgp implementation, and moved the tests to the gpgme and sequoia backends!

jlebon · 2025-10-07T19:44:40Z

Hmm, the Sequoia CI issue I think is unrelated. Might just be infra flake. I did test the sequoia backend locally and tests passed.

mtrmac · 2025-10-07T20:42:03Z

Just a quick note now,

Expect "time="2025-10-06T10:59:58-05:00" level=fatal msg="Source image rejected: E60EC577E915C82C82D469E9676112589F943F38 was not found"
            	            	" to match "(?s).*Source image rejected: (Invalid GPG signature|Missing key:).*"

is a drift in the expected test message caused by differing version of Podman-sequoia, and IIRC we have updated for that.

Can you try rebasing, first?

Currently, simple signatures are expected to be in the binary GPG format, and that's what e.g. `podman push --sign-by` produces as well. But the code for all backends work today with cleartext signatures. Add a new test to cover this case. But only in the GPGME and Sequoia backends since the OpenPGP backend does not support it and cannot easily be supported (see also [1]). The reason why I'm interested in this is that I'd like to make use of it for signing Fedora CoreOS container images. The end-goal is to move to Sigstore signing, but until that's ready, we'd like to use GPG signing. We use Robosignatory, the Fedora signing service, which only supports detached signatures, and while it's theoretically possible to convert the detached signatures we get back into inline binary signatures, it's much less cumbersome and error-prone to convert it to cleartext signatures. It's worth noting that while Fedora's signing server (Sigul) does support container image signing, Robosignatory does not surface it yet (see https://pagure.io/robosignatory/issue/22). Fixing that wouldn't be too hard I think, but all this is ideally short-term anyway until we can move to Sigstore signing + Konflux. There's work in progress on that (see e.g. https://discussion.fedoraproject.org/t/148999). The primary goal here is just ensuring that this keeps working until we move off of it. Signed-off-by: Jonathan Lebon <jonathan@jlebon.com> [1]: containers#307 (review)

github-actions bot added the image Related to "image" package label Sep 3, 2025

jlebon force-pushed the pr/simple-sign-cleartext branch from d2f8d3b to a4a1c23 Compare September 3, 2025 12:16

mtrmac reviewed Sep 3, 2025

View reviewed changes

jlebon mentioned this pull request Sep 3, 2025

Build FCOS based on Fedora 43 using podman build coreos/fedora-coreos-tracker#1969

Closed

15 tasks

jlebon force-pushed the pr/simple-sign-cleartext branch from a4a1c23 to 310afd4 Compare September 4, 2025 02:56

jlebon changed the title ~~Add test for cleartext signatures~~ Declare cleartext signature support and add test Sep 4, 2025

mtrmac requested changes Sep 19, 2025

View reviewed changes

jlebon force-pushed the pr/simple-sign-cleartext branch from 310afd4 to 44e128f Compare October 6, 2025 15:48

jlebon changed the title ~~Declare cleartext signature support and add test~~ Test cleartext signature support in gpgme and sequoia backends Oct 6, 2025

jlebon force-pushed the pr/simple-sign-cleartext branch from 44e128f to 40ee327 Compare October 7, 2025 20:50

Test cleartext signature support in gpgme and sequoia backends #307

Are you sure you want to change the base?

Test cleartext signature support in gpgme and sequoia backends #307

Uh oh!

Conversation

jlebon commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jlebon commented Sep 3, 2025

Uh oh!

mtrmac left a comment

Choose a reason for hiding this comment

Uh oh!

jlebon commented Sep 3, 2025

Uh oh!

mtrmac commented Sep 3, 2025

Uh oh!

jlebon commented Sep 4, 2025

Uh oh!

jlebon commented Sep 5, 2025

Uh oh!

mtrmac left a comment

Choose a reason for hiding this comment

Uh oh!

mtrmac Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

mtrmac Sep 19, 2025

Choose a reason for hiding this comment

Uh oh!

jlebon commented Sep 19, 2025

Uh oh!

mtrmac commented Oct 6, 2025

Uh oh!

jlebon commented Oct 6, 2025

Uh oh!

jlebon commented Oct 7, 2025

Uh oh!

mtrmac commented Oct 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jlebon commented Sep 3, 2025 •

edited

Loading