Skip to content

Commit

Permalink
Restart task status.Succeeded fixes (k8ssandra#440)
Browse files Browse the repository at this point in the history
* Restart task completion fixes.

* Add test for success tracking.

* Juggle types in test so that they match.

* More precise logic for tracking number of pods restarted for all STSs.

* Bring back simple int for restarted pods calculation.

* Ensure taskConfig.Completed is updated when the restart is completed.

* Changelog.
  • Loading branch information
Miles-Garnsey authored and emerkle826 committed Dec 1, 2022
1 parent c0e044b commit 4ff5485
Show file tree
Hide file tree
Showing 4 changed files with 12 additions and 3 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ Changelog for Cass Operator, new PRs should update the `main / unreleased` secti
## v1.13.1

* [CHANGE] [#291](https://github.com/k8ssandra/cass-operator/issues/291) Update Ginkgo to v2 (maintain current features, nothing additional from v2)
* [BUGFIX] [#431](https://github.com/k8ssandra/cass-operator/issues/431) Fix a bug where the restartTask would not provide success counts for restarted pods.
* [BUGFIX] [#444](https://github.com/k8ssandra/cass-operator/issues/444) Update cass-config-builder to 1.0.5. Update the target tag of cass-config-builder to :1.0 to allow future updates in 1.0.x without rolling restarts.
* [BUGFIX] [#437](https://github.com/k8ssandra/cass-operator/issues/437) Fix startOneNodeRack to not loop forever in case of StS with size 0 (such as decommission of DC)

Expand Down
4 changes: 4 additions & 0 deletions controllers/control/cassandratask_controller.go
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,9 @@ type TaskConfiguration struct {
// Functions not targeting the pod
ValidateFunc ValidatorFunc
PreProcessFunc ProcessFunc

// Status tracking
Completed int
}

func (t *TaskConfiguration) Validate() error {
Expand Down Expand Up @@ -285,6 +288,7 @@ JobDefinition:
if err != nil {
return ctrl.Result{}, err
}
completed = taskConfig.Completed
break JobDefinition
case api.CommandReplaceNode:
r.replace(taskConfig)
Expand Down
2 changes: 1 addition & 1 deletion controllers/control/cassandratask_controller_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -522,7 +522,7 @@ var _ = Describe("CassandraTask controller tests", func() {

// Set StatefulSet properties here so that the task completes.. verify first that there's been a change (but only to r1)
_ = waitForTaskCompletion(taskKey)

Expect(task.Status.Succeeded).Should(Equal(int(sts.Status.Replicas))) // Ensure that Succeeded field is updated.
// Verify other racks haven't been modified
var stsAll appsv1.StatefulSetList
Expect(k8sClient.List(context.TODO(), &stsAll, client.MatchingLabels(map[string]string{cassdcapi.DatacenterLabel: testDc.Name}), client.InNamespace(testNamespaceName))).To(Succeed())
Expand Down
8 changes: 6 additions & 2 deletions controllers/control/jobs.go
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ func (r *CassandraTaskReconciler) restartSts(ctx context.Context, sts []appsv1.S
}
}
}

restartedPods := 0
for _, st := range sts {
if st.Spec.Template.ObjectMeta.Annotations == nil {
st.Spec.Template.ObjectMeta.Annotations = make(map[string]string)
Expand All @@ -84,9 +84,13 @@ func (r *CassandraTaskReconciler) restartSts(ctx context.Context, sts []appsv1.S
status.ReadyReplicas == status.Replicas &&
status.ObservedGeneration == st.GetObjectMeta().GetGeneration() {
// This one has been updated, move on to the next one

restartedPods += int(status.UpdatedReplicas)
taskConfig.Completed = restartedPods
continue
}

restartedPods += int(status.UpdatedReplicas)
taskConfig.Completed = restartedPods
// This is still restarting
return ctrl.Result{RequeueAfter: jobRunningRequeue}, nil
}
Expand Down

0 comments on commit 4ff5485

Please sign in to comment.