Default certificate does not exist #216

Roxyrob · 2024-01-08T16:23:29Z

ref: Default certificate does not exist #601

On haproxy ingress controller v1.10.10 I get this error in logs:

ERROR global.go:272 default certificate: annotation 'ssl-certificate': secret
'namespace/namespace-default-cert' does not exist
namespace is custom namespace (not default).

I cannot use this ingress controller in https.
If I manually create requested certificate that controller will remove it.

Sometimes ingress controller seems to deploy and auto create default certificates, with the same ingress configuration.

I know I can create custom one, but I need to know how haproxy ingress controller check and choose to create or not its default certificate to support tls without custom one ?

Roxyrob · 2024-01-08T16:26:43Z

As written in the referenced haproxy ingress controller issue, Until this issue is identified and resolved I found a WORKAROUND that seems stable enough.

Following this link: Haproxy document SSL / TLS

Manually create a certificate

openssl req -x509 -newkey rsa:2048 -keyout example.key -out example.crt -days 365 -nodes -subj "/C=US/ST=Ohio/L=Columbus/O=MyCompany/CN=example.com"

Manually create a secret

kubectl create secret tls -n {namespace} haproxytempcert --cert="example.crt" --key="example.key"

Temporary Change haproxy ingress ConfigMap

     kubectl edit configmap -n {namespace} {configmap_name}

       apiVersion: v1
       kind: ConfigMap
       metadata:
         name: ...
         namespace: {namespace}
       data:
	 ...
         ssl-certificate: "{namespace}/haproxytempcert"    <===  (Add)

Doing this "sometimes" seems "to wakeup" ingress controller that create default cert using haproxytempcert.
Default certificate now exists and contains my Self Signed certificate.

If ingress controller DOES NOT wake-up, leave the haproxytempcert secret and ConfigMap annotation (ssl-certificate) to be used by haproxy ingress controller.

If Yes now you can delete temporary secrets and ConfigMap annotation:

Delete "ssl-certificate" from ConfigMap

     kubectl edit configmap -n {namespace} {configmap_name}

       apiVersion: v1
       kind: ConfigMap
       metadata:
         name: ...
         namespace: {namespace}
       data:
         ...

Delete secret

kubectl delete secret -n {namespace} haproxytempcert

Default certificate survive and now contains default haproxy Self-Signed certificate.
What one expect from start.

Better to know why and where this happen.

dkorunic · 2024-01-09T08:44:54Z

@Roxyrob Can you try with the following custom settings, where controller.defaultTLSSecret.secretNamespace refers to your custom namespace?

controller:
  defaultTLSSecret:
    enabled: true
    secretNamespace: namespace

Roxyrob · 2024-01-11T18:20:22Z

@dkorunic, was the first try.

Indeed deployment manifest has "--default-ssl-certificate" correctly set but no default certificate was created anyway:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    ...
  labels:
    app.kubernetes.io/instance: ingress-haproxy-internal
    app.kubernetes.io/name: kubernetes-ingress
    app.kubernetes.io/version: 1.10.10
    helm.sh/chart: kubernetes-ingress-1.35.5
  name: ingress-haproxy-internal
  namespace: ingress-haproxy-internal
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: ingress-haproxy-internal
      app.kubernetes.io/name: kubernetes-ingress
    ...
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: ingress-haproxy-internal
        app.kubernetes.io/name: kubernetes-ingress
    spec:
      containers:
        - args:
            - >-
              --default-ssl-certificate=ingress-haproxy-internal/ingress-haproxy-internal-default-cert
            - '--configmap=ingress-haproxy-internal/ingress-haproxy-internal'
            ...

When deployment do not create default certificate the only way is the WORKAROUND above.

dkorunic · 2024-01-16T16:09:34Z

I haven't been able to reproduce your issue, it works for me without any issues. Configuration override used:

controller:
  defaultTLSSecret:
    enabled: true
    secretNamespace: testspace

Helm command to install:

$ helm install testissue216 --create-namespace --namespace testspace -f myvalues.yaml ./kubernetes-ingress

Verify secrets:

$ kubectl get secrets --namespace testspace                                       NAME                                           TYPE                 DATA   AGE
sh.helm.release.v1.testissue216.v1             helm.sh/release.v1   1      30s
testissue216-kubernetes-ingress-default-cert   kubernetes.io/tls    2      29s

$ kubectl describe secret testissue216-kubernetes-ingress-default-cert --namespace testspace
Name:         testissue216-kubernetes-ingress-default-cert
Namespace:    testspace
Labels:       app.kubernetes.io/instance=testissue216
             app.kubernetes.io/managed-by=Helm
             app.kubernetes.io/name=kubernetes-ingress
             app.kubernetes.io/version=1.10.10
             helm.sh/chart=kubernetes-ingress-1.36.0
Annotations:  helm.sh/hook: pre-install
             helm.sh/hook-delete-policy: before-hook-creation

Type:  kubernetes.io/tls

Data
====
tls.crt:  1159 bytes
tls.key:  1679 bytes

Check IC logs:

$ kubectl logs testissue216-kubernetes-ingress-d86cf687d-jkwjn --namespace testspace| grep certificate
2024/01/16 16:04:54 Default ssl certificate: testspace/testissue216-kubernetes-ingress-default-cert

Everything looks correct. To me it looks like you haven't matched secretNamespace and namespace while installing your Ingress Controller instance. Sadly due to how Helm renders values from values.yaml we cannot match that automatically, it has to be specified by hand.

dkorunic · 2024-01-17T12:39:40Z

In fact, I was wrong in thinking that we need to match controller.defaultTLSSecret.secretNamespace, it contains '{{ include "kubernetes-ingress.namespace" . }}' which will correctly render into string with {{ tpl .Values.controller.defaultTLSSecret.secretNamespace . }}.

As shown below it works as expected, no visible issues whatsoever. Required ingress controller service arguments are enabled, default SSL certificate is being created and it gets exported to the filesystem as needed:


$ helm install testissue216 --create-namespace --namespace testspace2 -f myvalues.yaml ./kubernetes-ingress

$ kubectl get secrets --namespace testspace2
NAME                                           TYPE                 DATA   AGE
sh.helm.release.v1.testissue216.v1             helm.sh/release.v1   1      2m54s
testissue216-kubernetes-ingress-default-cert   kubernetes.io/tls    2      2m54s

$ kubectl describe secret testissue216-kubernetes-ingress-default-cert --namespace testspace2
Name:         testissue216-kubernetes-ingress-default-cert
Namespace:    testspace2
Labels:       app.kubernetes.io/instance=testissue216
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=kubernetes-ingress
              app.kubernetes.io/version=1.10.10
              helm.sh/chart=kubernetes-ingress-1.36.0
Annotations:  helm.sh/hook: pre-install
              helm.sh/hook-delete-policy: before-hook-creation

Type:  kubernetes.io/tls

Data
====
tls.crt:  1164 bytes
tls.key:  1679 bytes

$ kubectl logs testissue216-kubernetes-ingress-b556d758d-pfd4g --namespace testspace2| grep ssl
2024/01/17 12:33:09 Default ssl certificate: testspace2/testissue216-kubernetes-ingress-default-cert

$ kubectl exec -it testissue216-kubernetes-ingress-b556d758d-pfd4g --namespace testspace2 -- ls -al /etc/haproxy/certs/frontend/0_testspace2_testissue216-kubernetes-ingress-default-cert.pem
-rw-r--r--    1 haproxy  haproxy       2843 Jan 17 12:33 /etc/haproxy/certs/frontend/0_testspace2_testissue216-kubernetes-ingress-default-cert.pem

dkorunic · 2024-01-17T12:41:44Z

Closing the ticket as no-op, nothing to fix here.

Roxyrob · 2024-01-17T13:31:34Z

Ok. I'll go deep on this, probably misconfiguration or issue on ArgoCD part creating secret resource .yaml from parameters.

dkorunic · 2024-01-17T13:47:04Z

@Roxyrob We have been doing quite a few ArgoCD-related workarounds lately (#214, #211 and possibly more), so it is quite possible.

rcazzatoApk · 2024-02-28T07:35:21Z

This issue is related to this ArgoCD behavior/bug interpreting Helm Hooks:

Argo hook not running on auto sync only on manual sync #9830

As explained here:

"automated sync doesn't trigger synchooks when it's a self-heal action"

As soon as ArgoCD will correct to look at "initial deployment" as "self-heal action", Job creating CRDs and Self-Signed certificate (Jobs annotated with helm hooks) will be run correctly in PreSync and GitOps with ArgoCD with syncPolicy.automated will work correctly.

This is a common issue for every deployment using Helm Hooks for environment preparation and actually the WORKAROUND is to start a Manual Sync after deplopyment (this can be annoying as haproxy in internal version to reach ArgoCD WebUI will not be immediately ready after deployment - it is needed to temporary use kubectl port-forward or argocd app sync cli).

dkorunic · 2024-02-28T07:48:48Z

Let us keep this issue open for now and I'll keep tracking argoproj/argo-cd#9830.

dkorunic self-assigned this Jan 9, 2024

dkorunic added the question Further information is requested label Jan 9, 2024

dkorunic added the invalid This doesn't seem right label Jan 17, 2024

dkorunic closed this as completed Jan 17, 2024

dkorunic reopened this Feb 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Default certificate does not exist #216

Default certificate does not exist #216

Roxyrob commented Jan 8, 2024

Roxyrob commented Jan 8, 2024 •

edited

Loading

dkorunic commented Jan 9, 2024

Roxyrob commented Jan 11, 2024 •

edited

Loading

dkorunic commented Jan 16, 2024

dkorunic commented Jan 17, 2024 •

edited

Loading

dkorunic commented Jan 17, 2024

Roxyrob commented Jan 17, 2024

dkorunic commented Jan 17, 2024

rcazzatoApk commented Feb 28, 2024 •

edited

Loading

dkorunic commented Feb 28, 2024

Default certificate does not exist #216

Default certificate does not exist #216

Comments

Roxyrob commented Jan 8, 2024

Roxyrob commented Jan 8, 2024 • edited Loading

dkorunic commented Jan 9, 2024

Roxyrob commented Jan 11, 2024 • edited Loading

dkorunic commented Jan 16, 2024

dkorunic commented Jan 17, 2024 • edited Loading

dkorunic commented Jan 17, 2024

Roxyrob commented Jan 17, 2024

dkorunic commented Jan 17, 2024

rcazzatoApk commented Feb 28, 2024 • edited Loading

dkorunic commented Feb 28, 2024

Roxyrob commented Jan 8, 2024 •

edited

Loading

Roxyrob commented Jan 11, 2024 •

edited

Loading

dkorunic commented Jan 17, 2024 •

edited

Loading

rcazzatoApk commented Feb 28, 2024 •

edited

Loading