[backend] cache-deployer generate CSR with wrong usage

[jomenxiao]

Environment

• How did you deploy Kubeflow Pipelines (KFP)?
  kind
• KFP version:
  brunch master

Steps to reproduce

follow README kustomize
https://github.com/kubeflow/pipelines/blob/master/manifests/kustomize/README.md

• error message
  "message": "invalid usage for client certificate: server auth",

describe csr

➜  .kind kubectl get csr cache-server.kubeflow
NAME                    AGE     SIGNERNAME                            REQUESTOR                                                             CONDITION
cache-server.kubeflow   6m38s   kubernetes.io/kube-apiserver-client   system:serviceaccount:kubeflow:kubeflow-pipelines-cache-deployer-sa   Approved,Failed
➜  .kind kubectl get csr cache-server.kubeflow  -o json
{
    "apiVersion": "certificates.k8s.io/v1",
    "kind": "CertificateSigningRequest",
    "metadata": {
        "creationTimestamp": "2021-12-21T06:48:46Z",
        "name": "cache-server.kubeflow",
        "resourceVersion": "1485",
        "uid": "bece32dd-b0f2-4d31-9e1c-2aafa656945e"
    },
    "spec": {
        "extra": {
            "authentication.kubernetes.io/pod-name": [
                "cache-deployer-deployment-578ffc9d46-5bjml"
            ],
            "authentication.kubernetes.io/pod-uid": [
                "4866ee10-fb48-4034-9c36-bf519e0b81f1"
            ]
        },
        "groups": [
            "system:serviceaccounts",
            "system:serviceaccounts:kubeflow",
            "system:authenticated"
        ],
        "request": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQzlEQ0NBZHdDQVFBd0pERWlNQ0FHQTFVRUF3d1pZMkZqYUdVdGMyVnlkbVZ5TG10MVltVm1iRzkzTG5OMgpZekNDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFLajcrZWtTRDA0Tm54YTczeXNLCnF6eVV4T1lnczEvUzJoZU9BMG1TVlVucXJsNm5ZVDRZWjBDWnViaFRvWUp0UEN2b0dSVXdoQ2ZsNE9nMFIzc1YKQ2VBdGpHT0RPR1BFdituUjJoTEgrSkFYZFIyM01sdzduOWhiQ0VhVWVONFNxWVdUK1BWbVhCV0RkdC90WVJPWQpuZnVzOFRqejJNbWgzN1ZUaUhqUGdDVlVoQTBIZHFkTms0VUMzb3UyL21PMHZYNXRLVTF5bWE2cEYyTUhMM1BtCnJzdDhzMmZGbnhkd0xlWFJlTzlPRm5xMlZzTHN2NGR5azE0SG5SWW82dER5eGwvWkpDYWNnTmNaYmVLeWNCVjcKYkhPOGVUbktoN20ydmppR0p2QnRnVG1NTzlpOWNLbzdzVS9YakROdyt0MVFMaFBxcDNHUkZITXVXUzEyMFAwNAovMFVDQXdFQUFhQ0JpakNCaHdZSktvWklodmNOQVFrT01Yb3dlREFKQmdOVkhSTUVBakFBTUFzR0ExVWREd1FFCkF3SUY0REFUQmdOVkhTVUVEREFLQmdnckJnRUZCUWNEQVRCSkJnTlZIUkVFUWpCQWdneGpZV05vWlMxelpYSjIKWlhLQ0ZXTmhZMmhsTFhObGNuWmxjaTVyZFdKbFpteHZkNElaWTJGamFHVXRjMlZ5ZG1WeUxtdDFZbVZtYkc5MwpMbk4yWXpBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQVRXL1dvdHpQRGtnNE1YbndyNUhxU2wxZ0FWOHBJM1dnCkpZRGdNMXRvdU1pTkZZaWZjVkRkRDZVNGhlblZBR0pTWSt2T1pXQkRoYTIvbVpNemROUXkybEZ1MDlOOHJ5eVYKcFpPTnBIWnRVZXloNTU1ZDdwWjVzNEtiRGFhd0RXV25tejBIWEk0SUJlY3FYSzRNVENsSVgrNXlLL2ZSZFd6RQo1RFNvcE9pWHQvMGxuMTJYNzZNcEV5WG5obDRQenpieG5wdFRJUjFPRU9Gb1pFUzdETkltbTcrQzRMNDVpSDhkCnBFSnhqQ0grcFVueVBWMkZLYUJnTHAzR2JHUDZlaTJ2eFdiRFVGWjhJdHNyNlpKdXNlU3Fib3pGV2lQOHBGYUMKdDhrUXNsRWlKSjFEellKdk5reXE3Wm5iMURRK09ESGZiV1IxdEtOdnJJaW1aSGJ4YVBxV3NBPT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUgUkVRVUVTVC0tLS0tCg==",
        "signerName": "kubernetes.io/kube-apiserver-client",
        "uid": "3796d0f3-65b1-443a-9b13-97e9a61e381b",
        "usages": [
            "digital signature",
            "key encipherment",
            "server auth"
        ],
        "username": "system:serviceaccount:kubeflow:kubeflow-pipelines-cache-deployer-sa"
    },
    "status": {
        "conditions": [
            {
                "lastTransitionTime": "2021-12-21T06:48:46Z",
                "lastUpdateTime": "2021-12-21T06:48:46Z",
                "message": "This CSR was approved by kubectl certificate approve.",
                "reason": "KubectlApprove",
                "status": "True",
                "type": "Approved"
            },
            {
                "lastTransitionTime": "2021-12-21T06:48:46Z",
                "lastUpdateTime": "2021-12-21T06:48:46Z",
                "message": "invalid usage for client certificate: server auth",
                "reason": "SignerValidationFailure",
                "status": "True",
                "type": "Failed"
            }
        ]
    }
}

[zijianjoy]

cc @chensun

[kimwnasptd]

I also bumped into the exact same issue while testing the KF 1.5 RC0 manifests kubeflow/manifests#2099

I think this has definitely something to do with KinD, but I couldn't get to the bottom of it. For me it was:

• KinD cluster with K8s 1.20.7
• 1.8.0-rc.1 commit

BUT, when testing this with:

• EKS with K8s 1.19
• KFP 1.7.0

Then the CertificateSigningRequest would get into Approved state, but the cache-deployer would still complain that a certificate would not appear.

ERROR: After approving csr cache-server.kubeflow, the signed certificate did not appear on the resource. Giving up after 10 attempts.