[Bug]: Receiving error verifying x5cInsecure certificate chain: x509: certificate specifies an incompatible key usage only after expiry

[andsens]

I made a Dockerfile for an easy reproduce in the next comment.

Steps to Reproduce

So I'm trying to create an authority for issuing and renewing Kubernetes client certificates in k3s. To that end I am hostpath mounting the k3s client-ca into step-ca and specifying it both as root and intermediate.
The template is configured so that the subject cannot be changed:

{
	"subject": {{ toJson .AuthorizationCrt.Subject }},
	"keyUsage": ["keyEncipherment", "digitalSignature"],
	"extKeyUsage": ["clientAuth"]
}

To bootstrap it all, the first cert is issued by running step certificate create using an inline template.

To renew certificates I added an X5C provisioner with the same client-ca. The relevant parts of the config look like this:

{
  "root": "/home/step/certs/kube_apiserver_client_ca.crt",
  "crt": "/home/step/certs/kube_apiserver_client_ca.crt",
  "key": "/home/step/certs/kube_apiserver_client_ca_key",
  "authority": {
    "provisioners": [
      {
        "type": "X5C",
        "name": "kube-apiserver-client-ca",
        "claims": {
          "allowRenewalAfterExpiry": true
        },
        "options": {
          "x509": {
            "templateFile": "/var/lib/home-cluster/config/smallstep/templates/kube-apiserver-client-cert.tpl"
          }
        },
        "roots": "<BASE64"
      }
    ],
    "template": {},
    "backdate": "1m0s"
  }
}

The bootstrapped certificate works and I am even able to renew it with the CA when the client certificate is still valid. However, once the certificate has expired, step ca renew --force system:admin.crt system:admin_key fails on the server with:

time="2024-05-13T21:34:31Z" level=warning duration=1.524841ms duration-ns=1524841 error="error verifying x5cInsecure certificate chain: x509: certificate specifies an incompatible key usage" fields.time="2024-05-13T21:34:31Z" method=POST name=ca path=/renew protocol=HTTP/1.1 referer= remote-address="fe80:myIPv6" request-id=8c1c4680-a57d-4fe1-bab2-57169fb68437 size=56 status=401 user-agent="Smallstep CLI/0.26.0 (linux/amd64)" user-id=

client output:

error validating renew token
error renewing certificate
github.com/smallstep/cli/command/ca.(*renewer).Renew
        github.com/smallstep/cli/command/ca/renew.go:472
github.com/smallstep/cli/command/ca.renewCertificateAction
        github.com/smallstep/cli/command/ca/renew.go:329
github.com/smallstep/cli/command/ca.renewCertificateCommand.ActionFunc.func1
        go.step.sm/cli-utils@v0.9.0/command/command.go:37
github.com/urfave/cli.HandleAction
        github.com/urfave/cli@v1.22.14/app.go:522
github.com/urfave/cli.Command.Run
        github.com/urfave/cli@v1.22.14/command.go:175
github.com/urfave/cli.(*App).RunAsSubcommand
        github.com/urfave/cli@v1.22.14/app.go:405
github.com/urfave/cli.Command.startApp
        github.com/urfave/cli@v1.22.14/command.go:380
github.com/urfave/cli.Command.Run
        github.com/urfave/cli@v1.22.14/command.go:103
github.com/urfave/cli.(*App).Run
        github.com/urfave/cli@v1.22.14/app.go:277
main.main
        ./main.go:124
runtime.main
        runtime/proc.go:271
runtime.goexit
        runtime/asm_amd64.s:1695

error validating renew token

A client cert looks like this:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 251184018212606555757190585672924233198 (0xbcf8483fbf7ac01e5ee7812c43817dee)
    Signature Algorithm: ECDSA-SHA256
        Issuer: CN=k3s-client-ca@1714937196
        Validity
            Not Before: May 13 21:17:36 2024 UTC
            Not After : May 13 21:19:36 2024 UTC
        Subject: CN=system:admin,O=system:masters
        Subject Public Key Info:
            Public Key Algorithm: ECDSA
                Public-Key: (256 bit)
                X:
                    32:e9:00:1c:8a:15:a1:91:94:e8:1c:02:7d:4d:b4:
                    cb:fb:8b:ad:9a:59:39:72:01:61:40:2e:68:52:fd:
                    3b:65
                Y:
                    23:ae:e8:98:54:03:63:b5:f8:7e:46:b6:b7:91:ad:
                    81:66:75:3f:1a:20:d7:de:f2:5a:3d:a1:0b:23:da:
                    e2:d4
                Curve: P-256
        X509v3 extensions:
            X509v3 Authority Key Identifier:
                keyid:17:E0:10:F6:11:7E:B2:16:9F:DB:21:A1:36:D7:10:06:57:F1:8B:F4
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                Client Authentication
            X509v3 Subject Key Identifier:
                D9:54:C3:DC:52:9F:C7:18:BF:F2:88:FE:DE:DB:C8:70:7A:F5:59:9B
    Signature Algorithm: ECDSA-SHA256
         30:44:02:20:45:56:43:1f:29:57:b0:6c:2b:9a:89:08:e0:05:
         c2:48:6a:ca:20:9a:59:15:74:1e:bf:c6:b9:84:5b:30:5e:5e:
         02:20:63:a1:87:85:b4:cd:2c:d7:b0:f2:c2:96:3b:4f:c4:25:
         ad:49:fc:75:50:f8:62:5b:31:26:51:08:60:fc:1c:af

The kubernetes client ca looks like this:

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 0 ()
    Signature Algorithm: ECDSA-SHA256
        Issuer: CN=k3s-client-ca@1714937196
        Validity
            Not Before: May 5 19:26:36 2024 UTC
            Not After : May 3 19:26:36 2034 UTC
        Subject: CN=k3s-client-ca@1714937196
        Subject Public Key Info:
            Public Key Algorithm: ECDSA
                Public-Key: (256 bit)
                X:
                    b5:60:a0:c7:97:47:5f:33:86:74:ac:24:b4:2f:3e:
                    22:f6:5f:fe:ab:8f:29:f9:e4:c1:4d:fd:9c:65:d2:
                    a3:ed
                Y:
                    46:bf:b0:42:6b:70:10:5c:6e:4b:e5:de:64:b0:0f:
                    81:4e:1f:8e:da:d3:78:80:1f:24:53:78:e7:c6:f4:
                    0e:37
                Curve: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier:
                17:E0:10:F6:11:7E:B2:16:9F:DB:21:A1:36:D7:10:06:57:F1:8B:F4
    Signature Algorithm: ECDSA-SHA256
         30:44:02:20:0f:8f:ad:e5:d2:f7:32:b1:3b:94:64:49:0c:1f:
         0e:95:d2:56:e9:47:47:ad:e5:bc:30:48:c9:c6:96:3c:c9:ec:
         02:20:77:b9:ee:9d:df:22:23:71:fa:f7:c1:58:4f:bc:a1:f7:
         db:65:7e:c2:22:ea:00:09:7c:3b:73:db:ba:97:c5:d3

(Note that there is no PathLength restriction, so setting it as root should be OK I believe)

Your Environment

• OS - Windows client, k3s on debian bookworm running the official docker image
• step-ca Version - 0.26.1

Expected Behavior

Since allowRenewalAfterExpiry is true, the certificate should be renewed.

Actual Behavior

The certificate renewal request fails.

Additional Context

So far I have traced the error from renew.go, through authorize.go to crytpto/jose/parse.go, to zcrypto/x509/verify.go (the only place I found that actually returns IncompatibleUsage) to checkChainForKeyUsage() in that same file.

Now, at first glance this seems to be similar to smallstep/cli#467, but no certificates involved in the defaults templates from step ca init specify the ServerAuth ext key usage either, so obviously I'm reading some code wrong. Additionally, the date check happens after the key usage check, which confuses me even more as to how that error is the one I'm seeing. All in all, I really tried to see what I'm doing wrong or how I can work around a potential limitation, but I'm just not grok'ing the code well enough to follow the thread.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

p,s,: What I'm trying to do is an ideal use-case for #186 btw. Right now I'm running a separate step-ca to issue SSH/HTTP certs etc.

[andsens]

Here's a minimal reproduce in a self-contained Dockerfile (though now I'm not getting the incomaptible key usage in the log. The renewal still fails however, and changing the --not-after=1s on line 14 to e.g. 1m makes everything work:

FROM cr.step.sm/smallstep/step-ca-bootstrap@sha256:5270356cf91596afe18478eab60c6c0866b2cc62618f282d42827e58f84d6eae as bootstrap

USER step
RUN mkdir certs && step certificate create --no-password --insecure --not-after=1h --template=<(echo '{\
  "subject": { "commonName": {{ toJson .Subject.CommonName }} }, \
  "keyUsage": ["keyEncipherment", "digitalSignature", "certSign"], \
  "basicConstraints": {"isCA": true, "maxPathLen": 15}}') k3s-client-ca@1714937196 certs/root_ca.crt certs/root_ca_key
RUN mkdir config && echo '{"address": ":9000","commonName": "k3s-client-ca@1714937196","dnsNames": ["127.0.0.1"],"db": {"type": "badgerv2","dataSource": "/home/step/db"},\
  "root": "/home/step/certs/root_ca.crt", "crt": "/home/step/certs/root_ca.crt", "key": "/home/step/certs/root_ca_key"}'>config/ca.json
RUN echo  '{"subject": {{ toJson .AuthorizationCrt.Subject }},\
  "keyUsage": ["keyEncipherment", "digitalSignature"], "extKeyUsage": ["clientAuth"]}' >x5c.tpl
RUN step ca provisioner add repro --allow-renewal-after-expiry --type X5C --x5c-root certs/root_ca.crt --x509-template x5c.tpl

RUN step certificate create --no-password --insecure --not-before=-24h --not-after=1s --template=<(echo '{ \
  "subject": {"commonName": {{ toJson .Subject.CommonName }},"extraNames": [{"type":"2.5.4.10", "value": "system:masters"}]}, \
  "keyUsage": ["keyEncipherment", "digitalSignature"], \
  "extKeyUsage": ["clientAuth"]}') \
  system:admin --ca=certs/root_ca.crt --ca-key=certs/root_ca_key peer.crt peer_key

FROM cr.step.sm/smallstep/step-ca:0.26.1
COPY --from=bootstrap /home/step /home/step
ENTRYPOINT ["/usr/bin/env", "bash", "-c"]
CMD ["\
  cat config/ca.json && \
  step certificate inspect certs/root_ca.crt ; \
  step certificate inspect peer.crt; \
  STEPDEBUG=1 step-ca &>log & pid=$! && sleep 2 && \
  STEPDEBUG=1 step ca renew --ca-url 127.0.0.1:9000 --force peer.crt peer_key; \
  kill $pid && wait && echo && cat log \
"]

[andsens]

Oooh, so because the certificate is expired authentication uses X5CInsecure with JWT instead of mTLS, which is what triggers this codepath?

[hslatman]

@andsens exactly 🙂

[maraino]

Hi @andsens, I've just pushed a commit to the master branch that fixes this issue.

Due to this bug, there's a slight change in behavior. Without ClientAuth, a certificate cannot renew itself using mTLS, but this bug allowed the certificate to be renewed using the X5CInsercure flow, used for expired certificates or --mtls=false. This was not our intention, and right now, mTLS and non-mTLS requirements are the same.

[andsens]

But that's fine, the client cert has client auth:

        X509v3 extensions:
            X509v3 Authority Key Identifier:
                keyid:17:E0:10:F6:11:7E:B2:16:9F:DB:21:A1:36:D7:10:06:57:F1:8B:F4
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                Client Authentication
            X509v3 Subject Key Identifier:
                D9:54:C3:DC:52:9F:C7:18:BF:F2:88:FE:DE:DB:C8:70:7A:F5:59:9B

:-D