Issuing certificates fails with "provisioner not found or invalid audience"

[notself]

I'm trying to get step-issuer to use a self hosted CA to issue certificates for my Kubernetes cluster. However, when the issuer tries to obtain a certificate, it always fails with

The request lacked necessary authorization to be completed. Please see the certificate authority logs for more info.

On the CA side, there is a log entry with

authority.Authorize: authority.authorizeSign: provisioner not found or invalid audience (https://ca.example.com/1.0/sign)

As far as I can see, the provisioner is correct (see below). The audience also seems to be correct. The only thing I can think of is the /1.0/sign path that may be causing issues, but I don't know what I'm doing wrong.

CA version: 0.25.0
Step Issuer version: 0.8.0 (also tried to build an image from master which depends on github.com/smallstep/certificates v0.25.0 instead of v0.24.2)

Full details below:

CA Details

The CA is running on a host with IP 192.168.1.254, listening on port 443. ca.example.com and hostname both resolve to 192.168.1.254

$ cat /usr/local/etc/step/config/ca.json
{
    (...)
        "address": "192.168.1.254:443",
        "dnsNames": [
                "ca.example.com",
                "hostname",
                "192.168.1.254"
        ],
    (...)

I have a JWK provisioner named k8s and I am able to issue certificates with this provisioner from another host.

$ cat ~/.step/config/defaults.json
{
  "ca-url": "https://ca.example.com",
  "fingerprint": "...",
  "root": "...",
  "redirect-url": ""
}

$ step ca provisioner list
[
   {
      "type": "JWK",
      "name": "k8s",
      "key": {
         "use": "sig",
         "kty": "EC",
         "kid": "JUD6XUGQGanbBQgpckAaLIOwVgYxILTwc-80fzhUIyI",
(...)

$ step ca certificate example@k8s crt key --provisioner k8s
✔ Provisioner: k8s (JWK) [kid: JUD6XUGQGanbBQgpckAaLIOwVgYxILTwc-80fzhUIyI]
Please enter the password to decrypt the provisioner key:
✔ CA: https://ca.example.com
✔ Certificate: crt
✔ Private Key: key

step-issuer details

step-issuer is running on my kubernetes cluster and configured to use my internal CA.

apiVersion: certmanager.step.sm/v1beta1
kind: StepClusterIssuer
metadata:
  name: "step-issuer"
spec:
  url: https://ca.example.com
  caBundle: ...
  provisioner:
    name: cert-issuer-internal-cluster
    kid: JUD6XUGQGanbBQgpckAaLIOwVgYxILTwc-80fzhUIyI
    passwordRef:
      name: step-issuer-secrets
      namespace: default
      key: key_password

The issuer is running with the correct configuration and is ready to issue certificates. From what I can see, this means that both the kid and key password are set correctly, otherwise the issuer does not become ready.

$ kubectl logs step-issuer-599b6dfdd7-ft7vn manager
{"level":"info","ts":"2023-11-03T12:17:36Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2023-11-03T12:17:36Z","logger":"controller-runtime.metrics","msg":"Starting metrics server"}
I1103 12:17:36.427304       1 leaderelection.go:250] attempting to acquire leader lease default/step-issuer-operator-lock...
{"level":"info","ts":"2023-11-03T12:17:36Z","logger":"controller-runtime.metrics","msg":"Serving metrics server","bindAddress":"127.0.0.1:8080","secure":false}
I1103 12:17:51.825609       1 leaderelection.go:260] successfully acquired lease default/step-issuer-operator-lock
{"level":"info","ts":"2023-11-03T12:17:51Z","msg":"Starting EventSource","controller":"stepissuer","controllerGroup":"certmanager.step.sm","controllerKind":"StepIssuer","source":"kind source: *v1beta1.StepIssuer"}
{"level":"info","ts":"2023-11-03T12:17:51Z","msg":"Starting Controller","controller":"stepissuer","controllerGroup":"certmanager.step.sm","controllerKind":"StepIssuer"}
{"level":"info","ts":"2023-11-03T12:17:51Z","msg":"Starting EventSource","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","source":"kind source: *v1.CertificateRequest"}
{"level":"info","ts":"2023-11-03T12:17:51Z","msg":"Starting Controller","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest"}
{"level":"info","ts":"2023-11-03T12:17:51Z","msg":"Starting EventSource","controller":"stepclusterissuer","controllerGroup":"certmanager.step.sm","controllerKind":"StepClusterIssuer","source":"kind source: *v1beta1.StepClusterIssuer"}
{"level":"info","ts":"2023-11-03T12:17:51Z","msg":"Starting Controller","controller":"stepclusterissuer","controllerGroup":"certmanager.step.sm","controllerKind":"StepClusterIssuer"}
{"level":"info","ts":"2023-11-03T12:17:53Z","msg":"Starting workers","controller":"stepclusterissuer","controllerGroup":"certmanager.step.sm","controllerKind":"StepClusterIssuer","worker count":1}
{"level":"info","ts":"2023-11-03T12:17:53Z","msg":"Starting workers","controller":"certificaterequest","controllerGroup":"cert-manager.io","controllerKind":"CertificateRequest","worker count":1}
{"level":"info","ts":"2023-11-03T12:17:53Z","msg":"Starting workers","controller":"stepissuer","controllerGroup":"certmanager.step.sm","controllerKind":"StepIssuer","worker count":1}
{"level":"info","ts":"2023-11-03T12:18:01Z","logger":"controllers.StepClusterIssuer","msg":"found status change for StepIssuer condition; setting lastTransitionTime","stepclusterissuer":{"name":"step-issuer"},"condition":"Ready","old_status":"False","new_status":"True","time":"2023-11-03T12:18:01Z"}

$ kubectl get stepclusterissuers.certmanager.step.sm step-issuer -o yaml
(...)
  provisioner:
    kid: JUD6XUGQGanbBQgpckAaLIOwVgYxILTwc-80fzhUIyI
    name: cert-issuer-internal-cluster
    passwordRef:
      key: key_password
      name: step-issuer-secrets
      namespace: default
  url: https://ca.example.com
status:
  conditions:
  - lastTransitionTime: "2023-11-03T11:34:36Z"
    message: StepClusterIssuer verified and ready to sign certificates
    reason: Verified
    status: "True"
    type: Ready

Creating a certificate

I then create a certificate in kubernetes

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: example-client
  namespace: default
spec:
  secretName: example-client-tls
  issuerRef:
    group: certmanager.step.sm
    kind: StepClusterIssuer
    name: step-issuer
  usages:
    - digital signature
    - key encipherment
    - client auth
  commonName: example@k8s

A certificaterequest is created but it fails to obtain the certificate

$ kubectl get certificaterequest example-client-fvj6s -o yaml
status:
  conditions:
  - lastTransitionTime: "2023-11-03T12:25:25Z"
    message: Certificate request has been approved by cert-manager.io
    reason: cert-manager.io
    status: "True"
    type: Approved
  - lastTransitionTime: "2023-11-03T12:25:26Z"
    message: 'Failed to sign certificate request: The request lacked necessary authorization
      to be completed. Please see the certificate authority logs for more info.'
    reason: Failed
    status: "False"
    type: Ready

The CA logs an error

(...) level=warning (...) error="authority.Authorize: authority.authorizeSign: provisioner not found or invalid audience (https://ca.example.com/1.0/sign)" ott=(...) path=/sign (...) status=401

Decoding the token in ott shows the correct kid

Header

{
  "alg": "ES256",
  "kid": "JUD6XUGQGanbBQgpckAaLIOwVgYxILTwc-80fzhUIyI",
  "typ": "JWT"
}

Payload

{
  "aud": "https://ca.example.com/1.0/sign",
  "exp": 1699014627,
  "iat": 1699014327,
  "iss": "cert-issuer-internal-cluster",
  "jti": "d9fad6e1b640cf416e7f4a43b23ed1aacc7c991ad319f9aad510a2b60dc28368",
  "nbf": 1699014327,
  "sans": [
    "example@k8s"
  ],
  "sha": "b321a15add56f6d04d9fa4b7308da1174204bf11d4620029ea963e6cee563b06",
  "sub": "example@k8s"
}

[maraino]

Hi, @notself, the problem is the provisioner name, it should be k8s and right now is cert-issuer-internal-cluster:

apiVersion: certmanager.step.sm/v1beta1
kind: StepClusterIssuer
metadata:
  name: "step-issuer"
spec:
  url: https://ca.example.com
  caBundle: ...
  provisioner:
    name: cert-issuer-internal-cluster
    kid: JUD6XUGQGanbBQgpckAaLIOwVgYxILTwc-80fzhUIyI
    passwordRef:
      name: step-issuer-secrets
      namespace: default
      key: key_password

[notself]

hi @maraino, thank you so much 🙇 I'm not sure how I missed that 🤦