Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dont skip drain for unhealthy nodes #839

Merged
merged 11 commits into from
Sep 22, 2023
Merged

dont skip drain for unhealthy nodes #839

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
6fdbc9b
dont skip drain for unhealthy nodes
elankath Aug 1, 2023
9454fad
Removed lastStateTransitionTime - undeeded for fix
elankath Aug 22, 2023
746e3ac
removed debug message
elankath Sep 12, 2023
581780f
corrected stage name
elankath Sep 12, 2023
98e21eb
reverted trivial changes to drain
elankath Sep 12, 2023
bed9967
Addressed review comments and documentation
elankath Sep 21, 2023
51cd01c
fixed anchor link for new q in faq
elankath Sep 21, 2023
3d34d1f
changed FAQ q to explain forceful deletion of unhealthy nodes
elankath Sep 21, 2023
bcb0d0e
removed excessive whitespace in faq
elankath Sep 21, 2023
b07a41f
del of vol attachments log machine name, minor faq update
elankath Sep 22, 2023
632e299
fixed unit tests after lastOperation.description adjust
elankath Sep 22, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions docs/FAQ.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ this document. Few of the answers assume that the MCM being used is in conjuctio
* [What health checks are performed on a machine?](#what-health-checks-are-performed-on-a-machine)
* [How does rate limiting replacement of machine work in MCM ? How is it related to meltdown protection?](#how-does-rate-limiting-replacement-of-machine-work-in-mcm-how-is-it-related-to-meltdown-protection)
* [How MCM responds when scale-out/scale-in is done during rolling update of a machinedeployment?](#how-mcm-responds-when-scale-outscale-in-is-done-during-rolling-update-of-a-machinedeployment)
* [How some unhealthy machines are drained quickly?](#how-some-unhealthy-machines-are-drained-quickly-)

* [Troubleshooting](#troubleshooting)
* [My machine is stuck in deletion for 1 hr, why?](#My-machine-is-stuck-in-deletion-for-1-hr-why)
Expand Down Expand Up @@ -256,6 +257,24 @@ During update for scaling event, a machineSet is updated if any of the below is

Once scaling is achieved, rollout continues.

## How some unhealthy machines are drained quickly ?

If a node is unhealthy for more than the `machine-health-timeout` specified for the `machine-controller`, the controller
health-check moves the machine phase to `Failed`. By default, the `machine-health-timeout` is 10` minutes.

`Failed` machines have their deletion timestamp set and the machine then moves to the `Terminating` phase. The node
drain process is initiated. The drain process is invoked either *gracefully* or *forcefully*.

The usual drain process is graceful. Pods are evicted from the node and the drain process waits until any existing
attached volumes are mounted on new node. However, if the node `Ready` is `False` or the `ReadonlyFilesystem` is `True`
for greater than `5` minutes (non-configurable), then a forceful drain is initiated. In a forceful drain, pods are deleted
and `VolumeAttachment` objects associated with the old node are also marked for deletion. This is followed by the deletion of the
cloud provider VM associated with the `Machine` and then finally ending with the `Node` object deletion.

During the deletion of the VM we only delete the local data disks and boot disks associated with the VM. The disks associated
with persistent volumes are left un-touched as their attach/de-detach, mount/unmount processes are handled by k8s
attach-detach controller in conjunction with the CSI driver.

# Troubleshooting
### My machine is stuck in deletion for 1 hr, why?

Expand Down
2 changes: 1 addition & 1 deletion pkg/apis/machine/v1alpha1/machine_types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,7 @@ const (
// MachineOperationHealthCheck indicates that the operation was a create
MachineOperationHealthCheck MachineOperationType = "HealthCheck"

// MachineOperationDelete indicates that the operation was a create
// MachineOperationDelete indicates that the operation was a delete
MachineOperationDelete MachineOperationType = "Delete"
)

Expand Down
18 changes: 11 additions & 7 deletions pkg/util/provider/machinecontroller/machine.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,6 @@ import (
"strings"
"time"

machineapi "github.com/gardener/machine-controller-manager/pkg/apis/machine"
"github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1"
"github.com/gardener/machine-controller-manager/pkg/apis/machine/validation"
"github.com/gardener/machine-controller-manager/pkg/util/provider/driver"
"github.com/gardener/machine-controller-manager/pkg/util/provider/machinecodes/codes"
"github.com/gardener/machine-controller-manager/pkg/util/provider/machinecodes/status"
"github.com/gardener/machine-controller-manager/pkg/util/provider/machineutils"
corev1 "k8s.io/api/core/v1"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
Expand All @@ -39,6 +32,14 @@ import (
"k8s.io/apimachinery/pkg/util/sets"
"k8s.io/client-go/tools/cache"
"k8s.io/klog/v2"

machineapi "github.com/gardener/machine-controller-manager/pkg/apis/machine"
"github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1"
"github.com/gardener/machine-controller-manager/pkg/apis/machine/validation"
"github.com/gardener/machine-controller-manager/pkg/util/provider/driver"
"github.com/gardener/machine-controller-manager/pkg/util/provider/machinecodes/codes"
"github.com/gardener/machine-controller-manager/pkg/util/provider/machinecodes/status"
"github.com/gardener/machine-controller-manager/pkg/util/provider/machineutils"
)

/*
Expand Down Expand Up @@ -592,6 +593,9 @@ func (c *controller) triggerDeletionFlow(ctx context.Context, deleteMachineReque
case strings.Contains(machine.Status.LastOperation.Description, machineutils.InitiateDrain):
return c.drainNode(ctx, deleteMachineRequest)

case strings.Contains(machine.Status.LastOperation.Description, machineutils.DelVolumesAttachments):
return c.deleteNodeVolAttachments(ctx, deleteMachineRequest)

case strings.Contains(machine.Status.LastOperation.Description, machineutils.InitiateVMDeletion):
return c.deleteVM(ctx, deleteMachineRequest)

Expand Down
47 changes: 24 additions & 23 deletions pkg/util/provider/machinecontroller/machine_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,6 @@ import (
"math"
"time"

machineapi "github.com/gardener/machine-controller-manager/pkg/apis/machine"
"github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1"
"github.com/gardener/machine-controller-manager/pkg/apis/machine/validation"
fakemachineapi "github.com/gardener/machine-controller-manager/pkg/client/clientset/versioned/typed/machine/v1alpha1/fake"
customfake "github.com/gardener/machine-controller-manager/pkg/fakeclient"
"github.com/gardener/machine-controller-manager/pkg/util/provider/driver"
"github.com/gardener/machine-controller-manager/pkg/util/provider/machinecodes/codes"
"github.com/gardener/machine-controller-manager/pkg/util/provider/machinecodes/status"
"github.com/gardener/machine-controller-manager/pkg/util/provider/machineutils"
. "github.com/onsi/ginkgo"
. "github.com/onsi/ginkgo/extensions/table"
. "github.com/onsi/gomega"
Expand All @@ -39,6 +30,16 @@ import (
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/util/validation/field"
k8stesting "k8s.io/client-go/testing"

machineapi "github.com/gardener/machine-controller-manager/pkg/apis/machine"
"github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1"
"github.com/gardener/machine-controller-manager/pkg/apis/machine/validation"
fakemachineapi "github.com/gardener/machine-controller-manager/pkg/client/clientset/versioned/typed/machine/v1alpha1/fake"
customfake "github.com/gardener/machine-controller-manager/pkg/fakeclient"
"github.com/gardener/machine-controller-manager/pkg/util/provider/driver"
"github.com/gardener/machine-controller-manager/pkg/util/provider/machinecodes/codes"
"github.com/gardener/machine-controller-manager/pkg/util/provider/machinecodes/status"
"github.com/gardener/machine-controller-manager/pkg/util/provider/machineutils"
)

const testNamespace = "test"
Expand Down Expand Up @@ -1491,7 +1492,7 @@ var _ = Describe("machine", func() {
},
},
expect: expect{
err: fmt.Errorf("Machine deletion in process. Drain successful. %s", machineutils.InitiateVMDeletion),
err: fmt.Errorf("Drain successful. %s", machineutils.InitiateVMDeletion),
retry: machineutils.ShortRetry,
nodeTerminationConditionIsSet: true,
machine: newMachine(
Expand Down Expand Up @@ -1631,7 +1632,7 @@ var _ = Describe("machine", func() {
),
},
}),
Entry("Drain skipping as machine is NotReady for a long time (5 minutes)", &data{
Entry("Force Drain as machine is NotReady for a long time (5 minutes)", &data{
setup: setup{
secrets: []*corev1.Secret{
{
Expand Down Expand Up @@ -1696,7 +1697,7 @@ var _ = Describe("machine", func() {
},
},
expect: expect{
err: fmt.Errorf("Skipping drain as machine is NotReady for over 5minutes. %s", machineutils.InitiateVMDeletion),
err: fmt.Errorf(fmt.Sprintf("Force Drain successful. %s", machineutils.DelVolumesAttachments)),
retry: machineutils.ShortRetry,
machine: newMachine(
&v1alpha1.MachineTemplateSpec{
Expand All @@ -1715,7 +1716,7 @@ var _ = Describe("machine", func() {
LastUpdateTime: metav1.Now(),
},
LastOperation: v1alpha1.LastOperation{
Description: fmt.Sprintf("Skipping drain as machine is NotReady for over 5minutes. %s", machineutils.InitiateVMDeletion),
Description: fmt.Sprintf("Force Drain successful. %s", machineutils.DelVolumesAttachments),
State: v1alpha1.MachineStateProcessing,
Type: v1alpha1.MachineOperationDelete,
LastUpdateTime: metav1.Now(),
Expand All @@ -1733,7 +1734,7 @@ var _ = Describe("machine", func() {
),
},
}),
Entry("Drain skipping as machine is in ReadonlyFilesystem for a long time (5 minutes)", &data{
Entry("Force Drain as machine is in ReadonlyFilesystem for a long time (5 minutes)", &data{
setup: setup{
secrets: []*corev1.Secret{
{
Expand Down Expand Up @@ -1798,7 +1799,7 @@ var _ = Describe("machine", func() {
},
},
expect: expect{
err: fmt.Errorf("Skipping drain as machine is in ReadonlyFilesystem for over 5minutes. %s", machineutils.InitiateVMDeletion),
err: fmt.Errorf(fmt.Sprintf("Force Drain successful. %s", machineutils.DelVolumesAttachments)),
retry: machineutils.ShortRetry,
machine: newMachine(
&v1alpha1.MachineTemplateSpec{
Expand All @@ -1817,7 +1818,7 @@ var _ = Describe("machine", func() {
LastUpdateTime: metav1.Now(),
},
LastOperation: v1alpha1.LastOperation{
Description: fmt.Sprintf("Skipping drain as machine is in ReadonlyFilesystem for over 5minutes. %s", machineutils.InitiateVMDeletion),
Description: fmt.Sprintf("Force Drain successful. %s", machineutils.DelVolumesAttachments),
State: v1alpha1.MachineStateProcessing,
Type: v1alpha1.MachineOperationDelete,
LastUpdateTime: metav1.Now(),
Expand All @@ -1835,7 +1836,7 @@ var _ = Describe("machine", func() {
),
},
}),
Entry("Drain skipping as machine is NotReady for a long time(5 min) ,also ReadonlyFilesystem is true for a long time (5 minutes)", &data{
Entry("Force Drain as machine is NotReady for a long time(5 min) ,also ReadonlyFilesystem is true for a long time (5 minutes)", &data{
setup: setup{
secrets: []*corev1.Secret{
{
Expand Down Expand Up @@ -1905,7 +1906,7 @@ var _ = Describe("machine", func() {
},
},
expect: expect{
err: fmt.Errorf("Skipping drain as machine is NotReady for over 5minutes. %s", machineutils.InitiateVMDeletion),
err: fmt.Errorf(fmt.Sprintf("Force Drain successful. %s", machineutils.DelVolumesAttachments)),
retry: machineutils.ShortRetry,
machine: newMachine(
&v1alpha1.MachineTemplateSpec{
Expand All @@ -1924,7 +1925,7 @@ var _ = Describe("machine", func() {
LastUpdateTime: metav1.Now(),
},
LastOperation: v1alpha1.LastOperation{
Description: fmt.Sprintf("Skipping drain as machine is NotReady for over 5minutes. %s", machineutils.InitiateVMDeletion),
Description: fmt.Sprintf("Force Drain successful. %s", machineutils.DelVolumesAttachments),
State: v1alpha1.MachineStateProcessing,
Type: v1alpha1.MachineOperationDelete,
LastUpdateTime: metav1.Now(),
Expand Down Expand Up @@ -2007,7 +2008,7 @@ var _ = Describe("machine", func() {
},
},
expect: expect{
err: fmt.Errorf("Machine deletion in process. Drain successful. %s", machineutils.InitiateVMDeletion),
err: fmt.Errorf("Drain successful. %s", machineutils.InitiateVMDeletion),
retry: machineutils.ShortRetry,
machine: newMachine(
&v1alpha1.MachineTemplateSpec{
Expand Down Expand Up @@ -2109,7 +2110,7 @@ var _ = Describe("machine", func() {
},
},
expect: expect{
err: fmt.Errorf("Machine deletion in process. Drain successful. %s", machineutils.InitiateVMDeletion),
err: fmt.Errorf("Drain successful. %s", machineutils.InitiateVMDeletion),
retry: machineutils.ShortRetry,
machine: newMachine(
&v1alpha1.MachineTemplateSpec{
Expand Down Expand Up @@ -2236,7 +2237,7 @@ var _ = Describe("machine", func() {
LastUpdateTime: metav1.Now(),
},
LastOperation: v1alpha1.LastOperation{
Description: fmt.Sprintf("Drain failed due to - Failed to update node. However, since it's a force deletion shall continue deletion of VM. %s", machineutils.InitiateVMDeletion),
Description: fmt.Sprintf("Drain failed due to - Failed to update node. However, since it's a force deletion shall continue deletion of VM. %s", machineutils.DelVolumesAttachments),
State: v1alpha1.MachineStateProcessing,
Type: v1alpha1.MachineOperationDelete,
LastUpdateTime: metav1.Now(),
Expand Down Expand Up @@ -2450,7 +2451,7 @@ var _ = Describe("machine", func() {
LastUpdateTime: metav1.Now(),
},
LastOperation: v1alpha1.LastOperation{
Description: fmt.Sprintf("Drain failed due to - Failed to update node. However, since it's a force deletion shall continue deletion of VM. %s", machineutils.InitiateVMDeletion),
Description: fmt.Sprintf("Drain failed due to - Failed to update node. However, since it's a force deletion shall continue deletion of VM. %s", machineutils.DelVolumesAttachments),
State: v1alpha1.MachineStateProcessing,
Type: v1alpha1.MachineOperationDelete,
LastUpdateTime: metav1.Now(),
Expand Down
Loading