fix: CrashLoopBackOff once tlsProfile changed #640

opokornyy · 2023-08-01T08:53:28Z

What this PR does / why we need it:

This PR adds the usage of callbacks to Prometheus and the webhook server, enabling the dynamic reloading of the TLS configuration. This eliminates the necessity of restarting the SSP pod, which had previously led to the pod entering the CrashLoopBackOff state.

Fixes #

https://bugzilla.redhat.com/show_bug.cgi?id=2151248

Release note:

None

openshift-ci · 2023-08-01T08:53:31Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

opokornyy · 2023-08-01T08:54:54Z

/cc @0xFelix @akrejcir

codingben · 2023-08-01T09:17:22Z

Hi, if this PR is still under work, can you please convert it to Draft? so CI won't run.

0xFelix

In general this goes into the right direction. The GetConfigForClient functions still need to be deduplicated, improved and should not be lambda functions.

0xFelix · 2023-08-01T09:11:40Z

controllers/ssp_controller.go

@@ -239,9 +239,6 @@ func (r *sspReconciler) isRestartNeeded(sspObj *ssp.SSP) bool {
 	if reflect.DeepEqual(r.lastSspSpec, ssp.SSPSpec{}) {


This func is never returning true, so it can likely be dropped.

I've removed my comment here, because it was irrelevant. Sorry for the noise.

0xFelix · 2023-08-01T09:13:36Z

main.go

 		os.Exit(1)
 	}

+	var webhookServer *webhook.Server
+	if tlsOptions.IsEmpty() {


Not sure we can do this. I don't think this allows to change the config later without restart?

0xFelix · 2023-08-01T09:13:59Z

tests/crypto_policy_test.go

@@ -65,7 +64,7 @@ var _ = Describe("Crypto Policy", func() {
 				},
 			},
 		}
-
+		// is not supported anymore


What does it mean?

0xFelix · 2023-08-01T09:14:34Z

tests/crypto_policy_test.go

+			Namespace: strategy.GetSSPDeploymentNameSpace(),
+		}, deployment)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(deployment.Status.ReadyReplicas).To(BeNumerically(">=", 1))


Ready replicas is dangerous, might better use UpdatedReplicas. @jcanocan

You can check both just to be sure.

0xFelix · 2023-08-01T09:16:39Z

main.go

+				tlsOptions, err := common.GetSspTlsOptions(ctx)
+				if err != nil {
+					setupLog.Error(err, "Error while getting tls profile")
+					os.Exit(1)


Do not exit here, better return an error.

0xFelix · 2023-08-01T09:17:24Z

main.go

+				}
+				cfg.CipherSuites = tlsOptions.CipherIDs(&setupLog)
+				cfg.MinVersion, _ = minTLSVersionId(tlsOptions.MinTLSVersion)
+				certificate, err := loadCertificates(path.Join(sdkTLSDir, sdkTLSCrt), path.Join(sdkTLSDir, sdkTLSKey))


Not sure how expensive it is to load certs on every request. You might cache some of this.

Agreed.
There is already a certificate watcher in the controller-runtime library. Maybe we can use that, I will check...

Here it is: https://github.com/kubevirt/ssp-operator/blob/9d3869accf1c183cdf66b3fdf572a3d8e9ec5a2d/vendor/sigs.k8s.io/controller-runtime/pkg/certwatcher/certwatcher.go#L45-L44

You can simply create a new CertWatcher and then add it to the manager to start it. Or call Start() manually in a separate goroutine.

codingben · 2023-08-01T09:35:44Z

main.go

-	// webhook server configuration is used.
-	if sspTLSOptions.IsEmpty() {
-		return nil
+// vm-console-proxy function


Can you please add a link to that function? e.g. so I can compare the changes here and there, if there any changes.

codingben · 2023-08-01T09:36:12Z

main.go


-	funcs := []func(*tls.Config){tlsCfgFunc}
-	return &webhook.Server{Port: webhookPort, TLSMinVersion: sspTLSOptions.MinTLSVersion, TLSOpts: funcs}
+// crypto_policy.go function


Can you please add a link to that function, in that file?

codingben · 2023-08-01T09:40:37Z

main.go

+		Addr:    metricsAddr,
+		Handler: mux,
+		TLSConfig: &tls.Config{
+			GetConfigForClient: func(_ *tls.ClientHelloInfo) (*tls.Config, error) {


Can you please extract it to a new function, to keep runPrometheusServer() short?

codingben · 2023-08-01T09:42:16Z

main.go

@@ -138,26 +180,63 @@ func main() {

 	ctx := ctrl.SetupSignalHandler()

-	tlsOptions, err := common.GetSspTlsOptions(ctx)
+	tlsCfgFunc := func(cfg *tls.Config) {


Is it possible to keep that function outside of main()? I think it's a good practice to keep main() short function, it's easier to read the code that way.

Currently main() is very long and should be very short, in general.

codingben · 2023-08-01T09:54:41Z

Hi, if this PR is still under work, can you please convert it to Draft? so CI won't run.

Also I'd suggest to have shorter commit title, and update the PR description to be the same as commit description (so you won't forget to sync them later).

Now you have:

fix: SSP operator moving to CrashLoopBackOff after tlsProfile change

This commit adds the usage of callbacks to Prometheus and
Webhook server to fetch TLS config on every request.

Signed-off-by: Ondrej Pokorny <opokorny@redhat.com>

I'd suggest to give it a name according to "This commit will fix: CrashLoopBackOff once tlsProfile changed" sentence. For example:

fix: CrashLoopBackOff once tlsProfile changed

Add usage of callbacks to Prometheus and webhook
server to fetch TLS config on each client request.

Signed-off-by: Ondrej Pokorny <opokorny@redhat.com>

jcanocan · 2023-08-01T15:34:02Z

main.go

+				cfg.MinVersion, _ = minTLSVersionId(tlsOptions.MinTLSVersion)
+				certificate, err := loadCertificates(path.Join(sdkTLSDir, sdkTLSCrt), path.Join(sdkTLSDir, sdkTLSKey))
+				if err != nil {
+					setupLog.Error(err, "Prometheus server error")


No need to log the error here. Just return the error and the calling function will log it for you

jcanocan · 2023-08-01T16:01:18Z

tests/crypto_policy_test.go

+			Namespace: strategy.GetSSPDeploymentNameSpace(),
+		}, deployment)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(deployment.Status.ReadyReplicas).To(BeNumerically(">=", 1))


You can check both just to be sure.

0xFelix · 2023-08-04T07:32:07Z

main.go

@@ -66,53 +68,115 @@ const (
 	webhookPort = 9443
 )

-func runPrometheusServer(metricsAddr string, tlsOptions common.SSPTLSOptions) error {
+// crypto_policy.go function
+func minTLSVersionId(s string) (uint16, error) {


Why not use the func in internal/common/crypto_policy.go?

0xFelix · 2023-08-04T07:34:19Z

main.go

+	return &tls.Config{
+		GetConfigForClient: func(_ *tls.ClientHelloInfo) (*tls.Config, error) {
+			cfg := &tls.Config{}
+			tlsOptions, err := common.GetSspTlsOptions(ctx)


common.GetSspTlsOptions is making API calls for every request if it stays like this. IMO we should have a Watcher/SharedInformer for the SSP CR and use it to retrieve the SSP if it was changed. Otherwise the SSP CR should be cached.

0xFelix · 2023-08-04T07:34:56Z

main.go

+				return cfg, nil
+			}
+			cfg.CipherSuites = tlsOptions.CipherIDs(&setupLog)
+			setupLog.Info("Configured ciphers", "ciphers", cfg.CipherSuites)


IMO we don't need to log this on every request.

0xFelix · 2023-08-04T07:35:32Z

main.go

+		// please be aware that the APIServer is using http keepalive so this is going to
+		// be executed only after a while for fresh connections and not on existing ones
+		cfg.GetConfigForClient = func(_ *tls.ClientHelloInfo) (*tls.Config, error) {
+			tlsOptions, err := common.GetSspTlsOptions(ctx)


Same issue as above, SSP CR needs to be cached.

0xFelix · 2023-08-04T07:35:42Z

main.go

+			}
+
+			cfg.CipherSuites = tlsOptions.CipherIDs(&setupLog)
+			setupLog.Info("Configured ciphers", "ciphers", cfg.CipherSuites)


Same as above.

jcanocan · 2023-08-08T09:38:42Z

main.go

+		GetConfigForClient: func(_ *tls.ClientHelloInfo) (*tls.Config, error) {
+			cfg := &tls.Config{}
+			var err error
+			cfg.GetCertificate = certWatcher.GetCertificate
+
+			if controllers.TLSProfile == nil {
+				cfg.MinVersion = crypto.DefaultTLSVersion()
+				cfg.CipherSuites = nil
+				return cfg, nil
+			}
+
+			if controllers.TLSProfile.Type == ocpconfigv1.TLSProfileCustomType {
+				cfg.CipherSuites = common.CipherIDs(controllers.TLSProfile.Custom.Ciphers)
+				cfg.MinVersion, err = crypto.TLSVersion(string(controllers.TLSProfile.Custom.MinTLSVersion))
+				if err != nil {
+					return nil, err
+				}
+				return cfg, nil
+			}
+
+			cipherNames, minTypedTLSVersion := ocpconfigv1.TLSProfiles[controllers.TLSProfile.Type].Ciphers, ocpconfigv1.TLSProfiles[controllers.TLSProfile.Type].MinTLSVersion
+			cfg.CipherSuites = common.CipherIDs(cipherNames)
+			cfg.MinVersion, err = crypto.TLSVersion(string(minTypedTLSVersion))
+			if err != nil {
+				return nil, err
+			}
+			return cfg, nil
+		},


Can we extract this to an external function? It is duplicated on getTLSConfigFunc().

0xFelix · 2023-08-08T09:58:50Z

controllers/ssp_controller.go

@@ -67,6 +66,8 @@ var kvsspCRDs = map[string]string{
 	"kubevirtcommontemplatesbundles.ssp.kubevirt.io": "KubevirtCommonTemplatesBundle",
 }

+var TLSProfile *osconfv1.TLSSecurityProfile


Please avoid globals.

0xFelix · 2023-08-08T10:00:32Z

internal/template-validator/tlsinfo/tlsinfo.go

@@ -192,8 +193,8 @@ func (ti *TLSInfo) CreateTlsConfig() *tls.Config {
 	}

 	if !ti.sspTLSOptions.IsEmpty() {
-		tlsConfig.CipherSuites = ti.sspTLSOptions.CipherIDs(nil)
-		minVersion, err := ti.sspTLSOptions.MinTLSVersionId()
+		tlsConfig.CipherSuites = common.CipherIDs(ti.sspTLSOptions.OpenSSLCipherNames)


Is there a lib for this too?

There is this function, but the problem is that it causes panic for some cipher names and we just want to log that the cipher is not supported

ssp-operator/vendor/github.com/openshift/library-go/pkg/crypto/crypto.go

Lines 233 to 245 in fc1d67c

func CipherSuitesOrDie(cipherNames []string) []uint16 {

if len(cipherNames) == 0 {

return DefaultCiphers()

}

cipherValues := []uint16{}

for _, cipherName := range cipherNames {

cipher, err := CipherSuite(cipherName)

if err != nil {

panic(err)

}

cipherValues = append(cipherValues, cipher)

}

return cipherValues

0xFelix · 2023-08-08T10:01:35Z

main.go

+			}
+
+			if controllers.TLSProfile.Type == ocpconfigv1.TLSProfileCustomType {
+				cfg.CipherSuites = common.CipherIDs(controllers.TLSProfile.Custom.Ciphers)


Can you dedup the calls to common.CipherIDs and crypto.TLSVersion? This block and the block below look very similar.

0xFelix · 2023-08-08T10:01:46Z

main.go

+		GetConfigForClient: func(_ *tls.ClientHelloInfo) (*tls.Config, error) {
+			cfg := &tls.Config{}
+			var err error
+			cfg.GetCertificate = certWatcher.GetCertificate
+
+			if controllers.TLSProfile == nil {
+				cfg.MinVersion = crypto.DefaultTLSVersion()
+				cfg.CipherSuites = nil
+				return cfg, nil
+			}
+
+			if controllers.TLSProfile.Type == ocpconfigv1.TLSProfileCustomType {
+				cfg.CipherSuites = common.CipherIDs(controllers.TLSProfile.Custom.Ciphers)
+				cfg.MinVersion, err = crypto.TLSVersion(string(controllers.TLSProfile.Custom.MinTLSVersion))
+				if err != nil {
+					return nil, err
+				}
+				return cfg, nil
+			}
+
+			cipherNames, minTypedTLSVersion := ocpconfigv1.TLSProfiles[controllers.TLSProfile.Type].Ciphers, ocpconfigv1.TLSProfiles[controllers.TLSProfile.Type].MinTLSVersion
+			cfg.CipherSuites = common.CipherIDs(cipherNames)
+			cfg.MinVersion, err = crypto.TLSVersion(string(minTypedTLSVersion))
+			if err != nil {
+				return nil, err
+			}
+			return cfg, nil
+		},


lyarwood · 2023-08-16T11:39:19Z

/cc

akrejcir

A partial review. I will look at it more later.

akrejcir · 2023-08-16T14:13:54Z

internal/template-validator/tlsinfo/tlsinfo.go

@@ -192,8 +193,8 @@ func (ti *TLSInfo) CreateTlsConfig() *tls.Config {
 	}

 	if !ti.sspTLSOptions.IsEmpty() {
-		tlsConfig.CipherSuites = ti.sspTLSOptions.CipherIDs(nil)
-		minVersion, err := ti.sspTLSOptions.MinTLSVersionId()
+		tlsConfig.CipherSuites = common.CipherIDs(ti.sspTLSOptions.OpenSSLCipherNames, &logger.Log)


Please keep the nil log here, because the change is not part of this PR. If you want, you can open a new PR to add the log.

akrejcir · 2023-08-16T14:15:48Z

internal/template-validator/tlsinfo/tlsinfo.go

-		tlsConfig.CipherSuites = ti.sspTLSOptions.CipherIDs(nil)
-		minVersion, err := ti.sspTLSOptions.MinTLSVersionId()
+		tlsConfig.CipherSuites = common.CipherIDs(ti.sspTLSOptions.OpenSSLCipherNames, &logger.Log)
+		minVersion, err := crypto.TLSVersion(ti.sspTLSOptions.MinTLSVersion)


This is not correct, because crypto.TLSVersion() function expects one of these names:

ssp-operator/vendor/github.com/openshift/library-go/pkg/crypto/crypto.go

Lines 37 to 42 in 9e99e78

var versions = map[string]uint16{

"VersionTLS10": tls.VersionTLS10,

"VersionTLS11": tls.VersionTLS11,

"VersionTLS12": tls.VersionTLS12,

"VersionTLS13": tls.VersionTLS13,

}

So a string like: VersionTLS12, but ti.sspTLSOptions.MinTLSVersion is a string returned by this function:

ssp-operator/internal/common/crypto_policy.go

Lines 119 to 134 in 11c2e89

func tlsVersionToHumanReadable(version ocpv1.TLSProtocolVersion) (string, error) {

switch version {

case "":

return "", nil

case ocpv1.VersionTLS10:

return "1.0", nil

case ocpv1.VersionTLS11:

return "1.1", nil

case ocpv1.VersionTLS12:

return "1.2", nil

case ocpv1.VersionTLS13:

return "1.3", nil

default:

return "", fmt.Errorf("invalid ocpv1.VersionTLS %v", version)

}

}

For example: 1.2.

akrejcir · 2023-08-16T14:16:16Z

main.go

 	"github.com/prometheus/client_golang/prometheus/promhttp"
+	ssp "kubevirt.io/ssp-operator/api/v1beta2"


Please move local imports to the block below.

akrejcir · 2023-08-16T14:19:06Z

main.go

+// This callback executes on each client call returning a new config to be used
+// please be aware that the APIServer is using http keepalive so this is going to
+// be executed only after a while for fresh connections and not on existing ones
+func getConfigForClientCallback(cfg *tls.Config, apiClient client.Client, ctx context.Context) (*tls.Config, error) {


Three comments for this function:

After thinking about this more, I think it is better to have a parameter type cache.Cache instead of client.Client. It has the same reading methods as Client, but cannot be used for writing.

Usually the context.Context parameter is the first.

Please don't include the word Callback in a function name. In my opinion, it's better to name a function, by what it does, then by how it is used.

akrejcir · 2023-08-16T14:19:47Z

main.go

+// please be aware that the APIServer is using http keepalive so this is going to
+// be executed only after a while for fresh connections and not on existing ones
+func getConfigForClientCallback(cfg *tls.Config, apiClient client.Client, ctx context.Context) (*tls.Config, error) {
+	var err error


This line can be removed, because err is redefined below.

akrejcir · 2023-08-16T14:23:44Z

main.go

+		cfg.CipherSuites = common.CipherIDs(tlsProfile.Custom.Ciphers, &setupLog)
+		cfg.MinVersion, err = crypto.TLSVersion(string(tlsProfile.Custom.MinTLSVersion))
+		if err != nil {
+			return nil, err


When this error is returned, the cfg parameter has already been modified. Can you change the code, so that on error, the cfg parameter is kept unchanged?

akrejcir · 2023-08-16T14:24:22Z

main.go

+	cfg.CipherSuites = common.CipherIDs(cipherNames, &setupLog)
+	cfg.MinVersion, err = crypto.TLSVersion(string(minTypedTLSVersion))
+	if err != nil {
+		return nil, err


Similarly here, please keep the cfg unchanged on error.

akrejcir · 2023-08-16T14:25:48Z

main.go

+	}
+}
+
+func getPrometheusServer(metricsAddr string, ctx context.Context) (*prometheusServer, error) {


Can you rename this function to newPrometheusServer()? Because it is a constructor.

akrejcir · 2023-08-16T14:26:47Z

main.go

+	setupLog.Info("Starting Prometheus metrics endpoint server with TLS")
+	metrics.Registry.MustRegister(common_templates.CommonTemplatesRestored)
+	metrics.Registry.MustRegister(common.SSPOperatorReconcileSucceeded)
+	handler := promhttp.HandlerFor(metrics.Registry, promhttp.HandlerOpts{})
+	mux := http.NewServeMux()
+	mux.Handle("/metrics", handler)


Can you move this logic to prometheusServer.Start() method? It makes sense to call it when the server is started, not when it's created.

akrejcir · 2023-08-16T14:27:25Z

main.go

+	certWatcher, err := certwatcher.New(metricsServer.certPath, metricsServer.keyPath)
+	if err != nil {
+		return nil, err
 	}

-	tlsCfgFunc := func(cfg *tls.Config) {
-		cfg.CipherSuites = sspTLSOptions.CipherIDs(&setupLog)
-		setupLog.Info("Configured ciphers", "ciphers", cfg.CipherSuites)
+	go func() {
+		if err := certWatcher.Start(ctx); err != nil {
+			setupLog.Error(err, "certificate watcher error")
+		}
+	}()


Similarly, can you move the certWatcher creation and start to the prometheusServer.Start() method?

0xFelix

Added some comments

0xFelix · 2023-08-21T08:24:42Z

main.go

+	var tlsProfile *ocpconfigv1.TLSSecurityProfile
+
+	if len(sspList.Items) != 0 {
+		tlsProfile = sspList.Items[0].Spec.TLSSecurityProfile
+	}
+
+	if tlsProfile == nil {
+		cfg.MinVersion = crypto.DefaultTLSVersion()
+		cfg.CipherSuites = nil
+		return cfg, nil
+	}


Suggested change

var tlsProfile *ocpconfigv1.TLSSecurityProfile

if len(sspList.Items) != 0 {

tlsProfile = sspList.Items[0].Spec.TLSSecurityProfile

}

if tlsProfile == nil {

cfg.MinVersion = crypto.DefaultTLSVersion()

cfg.CipherSuites = nil

return cfg, nil

}

if len(sspList.Items) == 0 {

cfg.MinVersion = crypto.DefaultTLSVersion()

cfg.CipherSuites = nil

return cfg, nil

}

tlsProfile := sspList.Items[0].Spec.TLSSecurityProfile

Do it like this?

We still need to check if spec.TLSSecurityProfile is nil or not, so we would need to do it like this

if len(sspList.Items) == 0 || sspList.Items[0].Spec.TLSSecurityProfile == nil { cfg.MinVersion = crypto.DefaultTLSVersion() cfg.CipherSuites = nil return cfg, nil } tlsProfile := sspList.Items[0].Spec.TLSSecurityProfile

0xFelix · 2023-08-21T08:38:53Z

main.go

+		return cfg, nil
+	}
+
+	cipherNames := ocpconfigv1.TLSProfiles[tlsProfile.Type].Ciphers


Maybe inline this?

0xFelix · 2023-08-21T08:38:57Z

main.go

+	cfg.CipherSuites = common.CipherIDs(cipherNames, &setupLog)
+	cfg.MinVersion, err = crypto.TLSVersion(string(minTypedTLSVersion))
+	if err != nil {
+		return nil, err


0xFelix · 2023-08-21T08:39:21Z

main.go

+	cipherNames := ocpconfigv1.TLSProfiles[tlsProfile.Type].Ciphers
+	cfg.CipherSuites = common.CipherIDs(cipherNames, &ctrl.Log)
+
+	minTypedTLSVersion := ocpconfigv1.TLSProfiles[tlsProfile.Type].MinTLSVersion


0xFelix · 2023-08-21T08:40:05Z

main.go

 	setupLog.Info("Starting Prometheus metrics endpoint server with TLS")
 	metrics.Registry.MustRegister(common_templates.CommonTemplatesRestored)
 	metrics.Registry.MustRegister(common.SSPOperatorReconcileSucceeded)
 	handler := promhttp.HandlerFor(metrics.Registry, promhttp.HandlerOpts{})
 	mux := http.NewServeMux()
 	mux.Handle("/metrics", handler)

-	minTlsVersion, err := tlsOptions.MinTLSVersionId()
+	s.server.Handler = mux


Merge it with L138?

It is not possible to call s.server.Handler.Handle() function, so we probably can not do it like that

0xFelix · 2023-08-21T08:49:10Z

tests/crypto_policy_test.go

+			Namespace: strategy.GetSSPDeploymentNameSpace(),
+		}, deployment)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(deployment.Status.ReadyReplicas).To(BeNumerically(">=", 1))


This should assert that we have the amount of ready and updated replicaes specified in spec.Replicas. Not just >= 1.

jcanocan · 2023-08-21T09:46:11Z

@0xFelix @akrejcir @opokornyy I have two questions:

As it is now, as far as I was able to test, the crypto suite is not reverting TLS profiles to its original state. Should we fix this as part of this PR or as a follow-up? Could @opokornyy confirm if it's still happening with your code?
IIUC, when this PR lands, TLS configuration changes do not require restarting pods or even to update the pods. Am I right? I'm raising this because in case pods do not need to be updated, it's very likely that line Expect(deployment.Status.UpdatedReplicas).To(BeNumerically(">=", 1)) will never be satisfied.

Thanks! :)

0xFelix

One more question and one nit.

@jcanocan

IMO this should be fixed in a follow up PR. It has nothing to do with SSP restarting.
Good point!

0xFelix · 2023-08-22T08:21:45Z

main.go

+	}
+
+	tlsProfile := sspList.Items[0].Spec.TLSSecurityProfile
+


nit: newline

0xFelix · 2023-08-22T08:24:36Z

tests/crypto_policy_test.go

-			g.Expect(err).ToNot(HaveOccurred())
-			g.Expect(deployment.Status.ReadyReplicas).To(BeNumerically(">=", 1))
-		}, env.Timeout(), time.Second).Should(Succeed())
+		deployment := &apps.Deployment{}


Do we still need Eventually here, or does strategy.RevertToOriginalSspCr wait?

It should wait, based on the previous comment the waitting was needed just because of the bug

ssp-operator/tests/crypto_policy_test.go

Lines 110 to 112 in 96302e1

// Because of bug[1], the SSP operator will move to CrashLoopBackOff state,

// so we need to wait until it is running.

// [1] - https://bugzilla.redhat.com/show_bug.cgi?id=2151248

ssp-operator/tests/tests_suite_test.go

Lines 360 to 364 in 96302e1

func (s *existingSspStrategy) RevertToOriginalSspCr() {

waitForSspDeletionIfNeeded(s.ssp)

createOrUpdateSsp(s.ssp)

waitUntilDeployed()

}

0xFelix · 2023-08-22T12:30:10Z

tests/crypto_policy_test.go

+		deployment := &apps.Deployment{}
+		err := apiClient.Get(ctx, client.ObjectKey{
+			Name:      strategy.GetSSPDeploymentName(),
+			Namespace: strategy.GetSSPDeploymentNameSpace(),
+		}, deployment)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(deployment.Status.ReadyReplicas).To(Equal(*deployment.Spec.Replicas))


Thinking about it, can't we drop all of this?

IMO we can drop it

0xFelix

Thank you! It looks good to me.

/approve
/retest-required

kubevirt-bot · 2023-08-22T15:00:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: 0xFelix

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [0xFelix]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

main.go

akrejcir · 2023-08-25T11:34:53Z

main.go

-		TLSConfig: &tlsConfig,
-	}
+	go func() {
+		if err := certWatcher.Start(ctx); err != nil {


Ideally we should use a different context for the certWatcher than ctx, so that it can be closed when this function returns on error.

We can do it in a follow-up PR.

main.go

akrejcir · 2023-08-25T11:36:54Z

main.go

+	server.TLSConfig = s.getPrometheusTLSConfig(ctx, certWatcher)
+
+	if err := server.ListenAndServeTLS(s.certPath, s.keyPath); err != nil {
+		setupLog.Error(err, "Failed to start Prometheus metrics endpoint server")


The error should be returned from this function.

The documentation for ListenAndServeTLS() says that it always returns a non-nil error:
https://github.com/golang/go/blob/0ae54ddd37302bdd2a8c775135bf5f076a18eeb3/src/net/http/server.go#L3281-L3283

https://github.com/golang/go/blob/0ae54ddd37302bdd2a8c775135bf5f076a18eeb3/src/net/http/server.go#L3026-L3028

These links are for go 1.19 but the same comments are in master.

main.go

akrejcir · 2023-08-28T09:00:25Z

main.go

-		err := server.ListenAndServeTLS(path.Join(sdkTLSDir, sdkTLSCrt), path.Join(sdkTLSDir, sdkTLSKey))
-		if err != nil {
-			setupLog.Error(err, "Failed to start Prometheus metrics endpoint server")
+		<-ctx.Done()


When below server.ListenAndServeTLS() returns because of an error, and this Start() function returns, this goroutine will still be waiting until context is closed.
Can you change the code, so that before the Start() returns, it makes sure that this goroutine has finished?

I've reconsidered this.
In this PR, we are already letting the above goroutine leak, from the Start() function, so this one can be left as is.

Can you add a // TODO comment to the code next to both goroutines, and open a new github issue, so that we don't forget to open a followup PR?

This commit adds the usage of callbacks to Prometheus and Webhook server to fetch TLS config on every request. Signed-off-by: Ondrej Pokorny <opokorny@redhat.com>

sonarcloud · 2023-08-28T09:44:34Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

No Coverage information
0.0% Duplication

akrejcir · 2023-08-28T11:56:20Z

/lgtm

opokornyy · 2023-08-28T14:13:31Z

/cherry-pick release-v0.18

kubevirt-bot · 2023-08-28T14:15:06Z

@opokornyy: new pull request created: #668

In response to this:

/cherry-pick release-v0.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kubevirt-bot added release-note-none Denotes a PR that doesn't merit a release note. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Aug 1, 2023

kubevirt-bot requested review from codingben and ksimon1 August 1, 2023 08:53

kubevirt-bot added the size/L label Aug 1, 2023

kubevirt-bot requested review from 0xFelix and akrejcir August 1, 2023 08:54

opokornyy marked this pull request as ready for review August 1, 2023 08:59

kubevirt-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 1, 2023

opokornyy changed the title ~~fix: SSP operator moving to CrashLoopBackOff after tlsProfile change~~ WIP: fix: SSP operator moving to CrashLoopBackOff after tlsProfile change Aug 1, 2023

kubevirt-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 1, 2023

0xFelix reviewed Aug 1, 2023

View reviewed changes

codingben reviewed Aug 1, 2023

View reviewed changes

jcanocan reviewed Aug 1, 2023

View reviewed changes

opokornyy changed the title ~~WIP: fix: SSP operator moving to CrashLoopBackOff after tlsProfile change~~ fix: CrashLoopBackOff once tlsProfile changed Aug 2, 2023

kubevirt-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 2, 2023

opokornyy force-pushed the fix-tls-bug branch from d44e8dc to ad89b79 Compare August 3, 2023 15:12

0xFelix reviewed Aug 4, 2023

View reviewed changes

opokornyy force-pushed the fix-tls-bug branch 2 times, most recently from f886ca7 to 82e898e Compare August 8, 2023 09:37

jcanocan reviewed Aug 8, 2023

View reviewed changes

0xFelix reviewed Aug 8, 2023

View reviewed changes

kubevirt-bot requested a review from lyarwood August 16, 2023 11:39

akrejcir reviewed Aug 16, 2023

View reviewed changes

opokornyy force-pushed the fix-tls-bug branch from d074058 to 58be93e Compare August 17, 2023 10:09

0xFelix reviewed Aug 21, 2023

View reviewed changes

opokornyy force-pushed the fix-tls-bug branch from 58be93e to 79fd45f Compare August 22, 2023 07:49

0xFelix reviewed Aug 22, 2023

View reviewed changes

opokornyy force-pushed the fix-tls-bug branch from 79fd45f to e0fcf15 Compare August 22, 2023 08:42

0xFelix reviewed Aug 22, 2023

View reviewed changes

opokornyy force-pushed the fix-tls-bug branch from e0fcf15 to 1206495 Compare August 22, 2023 12:54

0xFelix reviewed Aug 22, 2023

View reviewed changes

kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 22, 2023

akrejcir reviewed Aug 24, 2023

View reviewed changes

main.go Show resolved Hide resolved

main.go Outdated Show resolved Hide resolved

main.go Outdated Show resolved Hide resolved

main.go Show resolved Hide resolved

main.go Outdated Show resolved Hide resolved

opokornyy force-pushed the fix-tls-bug branch from 1206495 to 7cd8761 Compare August 25, 2023 07:19

akrejcir reviewed Aug 25, 2023

View reviewed changes

opokornyy force-pushed the fix-tls-bug branch from 7cd8761 to 01503df Compare August 28, 2023 08:10

akrejcir reviewed Aug 28, 2023

View reviewed changes

fix: SSP operator moving to CrashLoopBackOff after tlsProfile change

4fa9732

This commit adds the usage of callbacks to Prometheus and Webhook server to fetch TLS config on every request. Signed-off-by: Ondrej Pokorny <opokorny@redhat.com>

opokornyy force-pushed the fix-tls-bug branch from 01503df to 4fa9732 Compare August 28, 2023 09:44

opokornyy mentioned this pull request Aug 28, 2023

Add goroutine completion on error #667

Open

kubevirt-bot assigned akrejcir Aug 28, 2023

kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Aug 28, 2023

kubevirt-bot merged commit 4570a61 into kubevirt:main Aug 28, 2023
4 checks passed

kubevirt-bot mentioned this pull request Aug 28, 2023

[release-v0.18] fix: CrashLoopBackOff once tlsProfile changed #668

Merged

		@@ -239,9 +239,6 @@ func (r sspReconciler) isRestartNeeded(sspObj ssp.SSP) bool {
		if reflect.DeepEqual(r.lastSspSpec, ssp.SSPSpec{}) {

	func CipherSuitesOrDie(cipherNames []string) []uint16 {
	if len(cipherNames) == 0 {
	return DefaultCiphers()
	}
	cipherValues := []uint16{}
	for _, cipherName := range cipherNames {
	cipher, err := CipherSuite(cipherName)
	if err != nil {
	panic(err)
	}
	cipherValues = append(cipherValues, cipher)
	}
	return cipherValues

	var versions = map[string]uint16{
	"VersionTLS10": tls.VersionTLS10,
	"VersionTLS11": tls.VersionTLS11,
	"VersionTLS12": tls.VersionTLS12,
	"VersionTLS13": tls.VersionTLS13,
	}

	func tlsVersionToHumanReadable(version ocpv1.TLSProtocolVersion) (string, error) {
	switch version {
	case "":
	return "", nil
	case ocpv1.VersionTLS10:
	return "1.0", nil
	case ocpv1.VersionTLS11:
	return "1.1", nil
	case ocpv1.VersionTLS12:
	return "1.2", nil
	case ocpv1.VersionTLS13:
	return "1.3", nil
	default:
	return "", fmt.Errorf("invalid ocpv1.VersionTLS %v", version)
	}
	}

		"github.com/prometheus/client_golang/prometheus/promhttp"
		ssp "kubevirt.io/ssp-operator/api/v1beta2"

	// Because of bug[1], the SSP operator will move to CrashLoopBackOff state,
	// so we need to wait until it is running.
	// [1] - https://bugzilla.redhat.com/show_bug.cgi?id=2151248

	func (s *existingSspStrategy) RevertToOriginalSspCr() {
	waitForSspDeletionIfNeeded(s.ssp)
	createOrUpdateSsp(s.ssp)
	waitUntilDeployed()
	}

fix: CrashLoopBackOff once tlsProfile changed #640

fix: CrashLoopBackOff once tlsProfile changed #640

Conversation

opokornyy commented Aug 1, 2023 • edited Loading

openshift-ci bot commented Aug 1, 2023

opokornyy commented Aug 1, 2023

codingben commented Aug 1, 2023

0xFelix left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akrejcir Aug 2, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codingben commented Aug 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lyarwood commented Aug 16, 2023

akrejcir left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xFelix left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jcanocan commented Aug 21, 2023

0xFelix left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

0xFelix left a comment

Choose a reason for hiding this comment

kubevirt-bot commented Aug 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

opokornyy commented Aug 1, 2023 •

edited

Loading

akrejcir Aug 2, 2023 •

edited

Loading

codingben commented Aug 1, 2023 •

edited

Loading