Use static labels in pods #791
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
May 2, 2024 09:20
roypaulin
reviewed
May 2, 2024
| @@ -159,15 +159,15 @@ func (d *DBRemoveSubclusterReconciler) resetDefaultSubcluster(ctx context.Contex | |||
| scFinder := iter.MakeSubclusterFinder(d.VRec.Client, d.Vdb) | |||
| // We use the FindServices() API to get subclusters that already exist. | |||
| // We can only change the default subcluster to one of those. | |||
| svcs, err := scFinder.FindServices(ctx, iter.FindInVdb, vapi.MainCluster) | |||
| stss, err := scFinder.FindStatefulSets(ctx, iter.FindInVdb, vapi.MainCluster) | |||
| if err != nil { | |||
| return err | |||
| } | |||
| // If we don't find a service object we don't fail. The attempt to | |||
There was a problem hiding this comment.
Should we update this comment now that we find sts instead of svc?
| } | ||
| return sts.Labels[vmeta.SubclusterTypeLabel] == scType && !isTransient, nil | ||
| return sts.Labels[vmeta.SubclusterNameLabel] != transientName |
There was a problem hiding this comment.
I know it is been a while but why dowe need to return false for transient sc?
There was a problem hiding this comment.
When online upgrade iterates over subclusters (i.e. to upgrade, add client routing, etc), we always want to skip transient. Those aren't ever upgraded. They are meant to only serve read requests during the upgrade.
pkg/iter/sc_finder.go
Outdated
| @@ -207,12 +211,16 @@ func (m *SubclusterFinder) buildObjList(ctx context.Context, list client.ObjectL | |||
| // shouldSkipBasedOnSandboxState returns true if the object whose labels | |||
| // is passed does not belong to the given sandbox or main cluster | |||
| func shouldSkipBasedOnSandboxState(l map[string]string, sandbox string) bool { | |||
| // Note, for a pod, the labels passed in will be from the statefulset. So, | |||
There was a problem hiding this comment.
Is it better to add this to the function's description?
There was a problem hiding this comment.
yeah, that makes sense. There are 3 functions that have this comment. So, I'll change all 3.
pkg/meta/labels.go
Outdated
| // This label is added to a statefulset to indicate the sandbox it belongs | ||
| // to. This will allow the operator to filter these objects if it is looking | ||
| // for objects from the main cluster or a sandbox. Moreover, the sandbox | ||
| // controller will be watching statefulsets witGh this label and will trigger |
There was a problem hiding this comment.
Suggested change
| // controller will be watching statefulsets witGh this label and will trigger | |
| // controller will be watching statefulsets with this label and will trigger |
roypaulin
reviewed
May 3, 2024
| // statefulsets for other subclusters in the database. It may appear like it | ||
| // is derived from a subcluster name, but is does not get updated for | ||
| // subcluster rename. So, it shouldn't be treated as such. | ||
| SubclusterSelectorLabel = "vertica.com/subcluster-selector-name" |
There was a problem hiding this comment.
Does that mean the sts name no longer changes when the subcluster is renamed?
There was a problem hiding this comment.
Yes, this will stay static. This is tied to the statefulset name, which we'll keep static for the rename too.
roypaulin
approved these changes
May 3, 2024
Yeah, no problem. I'll wait. |
cchen-vertica
pushed a commit
that referenced
this pull request
Jul 17, 2024
During a replicated upgrade we will promote a sandbox to the main cluster, which cause it to drop its sandbox state, and we need to rename subclusters to match the original names. Prior to this change, the pods deployed by the operator had labels for the sandbox and subcluster name. And the labels were set in the statefulset pod template. Now, in order to change those, the k8s controller will actually do a rollout. This means it needs to restart each of the pods. But we require sandbox promotion and subcluster rename to happen without a pod restart. So, the change here to use different labeling to ensure we don't need to restart the pods. In some cases, if we have a pod we have to refer back to the statefulset in order to figure certain metadata. Since we are changing the statefulset selector values, the statefulset from old operator need to be recreated. We had done this in the past, so I repurposed the upgradeoperator120_reconciler.go for this purpose. A summary of all of the labels that are set is as follows. - All objects will have these labels set: ``` app.kubernetes.io/instance: <CR name> app.kubernetes.io/managed-by: verticadb-operator app.kubernetes.io/component: database app.kubernetes.io/name: vertica app.kubernetes.io/version: 2.2.0 vertica.com/database: <vertica db name> ``` - StatefulSets and Service objects will have the following labels: ``` vertica.com/subcluster-name: <subcluster-name> vertica.com/subcluster-type: <primary|secondary> vertica.com/sandbox: <sandbox name if applicable> ``` - Further Service objects will have this additional label: ``` vertica.com/svc-type: <external|headless> ``` - Pods will have the following labels: ``` vertica.com/subcluster-svc: <name of service object> vertica.com/client-routing: true # Only set if not doing a drain or pending removal vertica.com/subcluster-selector-name: <full name of the pod's statefulset> ``` - ConfigMaps used by the sandbox controller will have this label: ``` vertica.com/watched-by-sandbox-controller: true ``` - When the statefulset sets up the pod selector, it will use these labels: ``` app.kurbernetes.io/instance vertica.com/subcluster-selector-name ``` - When the service objects sets up its pod selector, it will use these labels: ``` app.kurbernetes.io/instance vertica.com/subcluster-selector-name vertica.com/client-routing ```
cchen-vertica
pushed a commit
that referenced
this pull request
Jul 17, 2024
During a replicated upgrade we will promote a sandbox to the main cluster, which cause it to drop its sandbox state, and we need to rename subclusters to match the original names. Prior to this change, the pods deployed by the operator had labels for the sandbox and subcluster name. And the labels were set in the statefulset pod template. Now, in order to change those, the k8s controller will actually do a rollout. This means it needs to restart each of the pods. But we require sandbox promotion and subcluster rename to happen without a pod restart. So, the change here to use different labeling to ensure we don't need to restart the pods. In some cases, if we have a pod we have to refer back to the statefulset in order to figure certain metadata. Since we are changing the statefulset selector values, the statefulset from old operator need to be recreated. We had done this in the past, so I repurposed the upgradeoperator120_reconciler.go for this purpose. A summary of all of the labels that are set is as follows. - All objects will have these labels set: ``` app.kubernetes.io/instance: <CR name> app.kubernetes.io/managed-by: verticadb-operator app.kubernetes.io/component: database app.kubernetes.io/name: vertica app.kubernetes.io/version: 2.2.0 vertica.com/database: <vertica db name> ``` - StatefulSets and Service objects will have the following labels: ``` vertica.com/subcluster-name: <subcluster-name> vertica.com/subcluster-type: <primary|secondary> vertica.com/sandbox: <sandbox name if applicable> ``` - Further Service objects will have this additional label: ``` vertica.com/svc-type: <external|headless> ``` - Pods will have the following labels: ``` vertica.com/subcluster-svc: <name of service object> vertica.com/client-routing: true # Only set if not doing a drain or pending removal vertica.com/subcluster-selector-name: <full name of the pod's statefulset> ``` - ConfigMaps used by the sandbox controller will have this label: ``` vertica.com/watched-by-sandbox-controller: true ``` - When the statefulset sets up the pod selector, it will use these labels: ``` app.kurbernetes.io/instance vertica.com/subcluster-selector-name ``` - When the service objects sets up its pod selector, it will use these labels: ``` app.kurbernetes.io/instance vertica.com/subcluster-selector-name vertica.com/client-routing ```
cchen-vertica
pushed a commit
that referenced
this pull request
Jul 24, 2024
During a replicated upgrade we will promote a sandbox to the main cluster, which cause it to drop its sandbox state, and we need to rename subclusters to match the original names. Prior to this change, the pods deployed by the operator had labels for the sandbox and subcluster name. And the labels were set in the statefulset pod template. Now, in order to change those, the k8s controller will actually do a rollout. This means it needs to restart each of the pods. But we require sandbox promotion and subcluster rename to happen without a pod restart. So, the change here to use different labeling to ensure we don't need to restart the pods. In some cases, if we have a pod we have to refer back to the statefulset in order to figure certain metadata. Since we are changing the statefulset selector values, the statefulset from old operator need to be recreated. We had done this in the past, so I repurposed the upgradeoperator120_reconciler.go for this purpose. A summary of all of the labels that are set is as follows. - All objects will have these labels set: ``` app.kubernetes.io/instance: <CR name> app.kubernetes.io/managed-by: verticadb-operator app.kubernetes.io/component: database app.kubernetes.io/name: vertica app.kubernetes.io/version: 2.2.0 vertica.com/database: <vertica db name> ``` - StatefulSets and Service objects will have the following labels: ``` vertica.com/subcluster-name: <subcluster-name> vertica.com/subcluster-type: <primary|secondary> vertica.com/sandbox: <sandbox name if applicable> ``` - Further Service objects will have this additional label: ``` vertica.com/svc-type: <external|headless> ``` - Pods will have the following labels: ``` vertica.com/subcluster-svc: <name of service object> vertica.com/client-routing: true # Only set if not doing a drain or pending removal vertica.com/subcluster-selector-name: <full name of the pod's statefulset> ``` - ConfigMaps used by the sandbox controller will have this label: ``` vertica.com/watched-by-sandbox-controller: true ``` - When the statefulset sets up the pod selector, it will use these labels: ``` app.kurbernetes.io/instance vertica.com/subcluster-selector-name ``` - When the service objects sets up its pod selector, it will use these labels: ``` app.kurbernetes.io/instance vertica.com/subcluster-selector-name vertica.com/client-routing ```
cchen-vertica
pushed a commit
that referenced
this pull request
Jul 24, 2024
During a replicated upgrade we will promote a sandbox to the main cluster, which cause it to drop its sandbox state, and we need to rename subclusters to match the original names. Prior to this change, the pods deployed by the operator had labels for the sandbox and subcluster name. And the labels were set in the statefulset pod template. Now, in order to change those, the k8s controller will actually do a rollout. This means it needs to restart each of the pods. But we require sandbox promotion and subcluster rename to happen without a pod restart. So, the change here to use different labeling to ensure we don't need to restart the pods. In some cases, if we have a pod we have to refer back to the statefulset in order to figure certain metadata. Since we are changing the statefulset selector values, the statefulset from old operator need to be recreated. We had done this in the past, so I repurposed the upgradeoperator120_reconciler.go for this purpose. A summary of all of the labels that are set is as follows. - All objects will have these labels set: ``` app.kubernetes.io/instance: <CR name> app.kubernetes.io/managed-by: verticadb-operator app.kubernetes.io/component: database app.kubernetes.io/name: vertica app.kubernetes.io/version: 2.2.0 vertica.com/database: <vertica db name> ``` - StatefulSets and Service objects will have the following labels: ``` vertica.com/subcluster-name: <subcluster-name> vertica.com/subcluster-type: <primary|secondary> vertica.com/sandbox: <sandbox name if applicable> ``` - Further Service objects will have this additional label: ``` vertica.com/svc-type: <external|headless> ``` - Pods will have the following labels: ``` vertica.com/subcluster-svc: <name of service object> vertica.com/client-routing: true # Only set if not doing a drain or pending removal vertica.com/subcluster-selector-name: <full name of the pod's statefulset> ``` - ConfigMaps used by the sandbox controller will have this label: ``` vertica.com/watched-by-sandbox-controller: true ``` - When the statefulset sets up the pod selector, it will use these labels: ``` app.kurbernetes.io/instance vertica.com/subcluster-selector-name ``` - When the service objects sets up its pod selector, it will use these labels: ``` app.kurbernetes.io/instance vertica.com/subcluster-selector-name vertica.com/client-routing ```
cchen-vertica
pushed a commit
that referenced
this pull request
Jul 24, 2024
During a replicated upgrade we will promote a sandbox to the main cluster, which cause it to drop its sandbox state, and we need to rename subclusters to match the original names. Prior to this change, the pods deployed by the operator had labels for the sandbox and subcluster name. And the labels were set in the statefulset pod template. Now, in order to change those, the k8s controller will actually do a rollout. This means it needs to restart each of the pods. But we require sandbox promotion and subcluster rename to happen without a pod restart. So, the change here to use different labeling to ensure we don't need to restart the pods. In some cases, if we have a pod we have to refer back to the statefulset in order to figure certain metadata. Since we are changing the statefulset selector values, the statefulset from old operator need to be recreated. We had done this in the past, so I repurposed the upgradeoperator120_reconciler.go for this purpose. A summary of all of the labels that are set is as follows. - All objects will have these labels set: ``` app.kubernetes.io/instance: <CR name> app.kubernetes.io/managed-by: verticadb-operator app.kubernetes.io/component: database app.kubernetes.io/name: vertica app.kubernetes.io/version: 2.2.0 vertica.com/database: <vertica db name> ``` - StatefulSets and Service objects will have the following labels: ``` vertica.com/subcluster-name: <subcluster-name> vertica.com/subcluster-type: <primary|secondary> vertica.com/sandbox: <sandbox name if applicable> ``` - Further Service objects will have this additional label: ``` vertica.com/svc-type: <external|headless> ``` - Pods will have the following labels: ``` vertica.com/subcluster-svc: <name of service object> vertica.com/client-routing: true # Only set if not doing a drain or pending removal vertica.com/subcluster-selector-name: <full name of the pod's statefulset> ``` - ConfigMaps used by the sandbox controller will have this label: ``` vertica.com/watched-by-sandbox-controller: true ``` - When the statefulset sets up the pod selector, it will use these labels: ``` app.kurbernetes.io/instance vertica.com/subcluster-selector-name ``` - When the service objects sets up its pod selector, it will use these labels: ``` app.kurbernetes.io/instance vertica.com/subcluster-selector-name vertica.com/client-routing ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
During a replicated upgrade we will promote a sandbox to the main cluster, which cause it to drop its sandbox state, and we need to rename subclusters to match the original names. Prior to this change, the pods deployed by the operator had labels for the sandbox and subcluster name. And the labels were set in the statefulset pod template. Now, in order to change those, the k8s controller will actually do a rollout. This means it needs to restart each of the pods. But we require sandbox promotion and subcluster rename to happen without a pod restart. So, the change here to use different labeling to ensure we don't need to restart the pods. In some cases, if we have a pod we have to refer back to the statefulset in order to figure certain metadata.
Since we are changing the statefulset selector values, the statefulset from old operator need to be recreated. We had done this in the past, so I repurposed the upgradeoperator120_reconciler.go for this purpose.
A summary of all of the labels that are set is as follows.