[ISSUE-1051] Handle Kubelet's Wrong CSI Call Inconsistent with Real Volume Status #1050
Conversation
…ed volume Signed-off-by: Shi, Crane <crane.shi@emc.com>
Signed-off-by: Shi, Crane <crane.shi@emc.com>
CraneShiEMC
changed the title
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## master #1050 +/- ##
==========================================
+ Coverage 72.76% 72.81% +0.04%
==========================================
Files 63 63
Lines 8949 8994 +45
==========================================
+ Hits 6512 6549 +37
- Misses 2147 2153 +6
- Partials 290 292 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
CraneShiEMC
changed the title
CraneShiEMC
changed the title
Signed-off-by: Shi, Crane <crane.shi@emc.com>
Signed-off-by: Shi, Crane <crane.shi@emc.com>
| @@ -488,6 +494,27 @@ func (s *CSINodeService) NodePublishVolume(ctx context.Context, req *csi.NodePub | |||
| resp, errToReturn = nil, fmt.Errorf("failed to publish volume: fake attach error %s", err.Error()) | |||
| } | |||
| } else { | |||
| // will check whether srcPath is mounted, if not, need to redo NodeStageVolume | |||
There was a problem hiding this comment.
I am a little concerned about this process, as some mount logic(like global path setup) is done in kubelet. there redo nodeStage may not help in such case. do we have a test can prove that this will help in the failure case?
There was a problem hiding this comment.
The kubelet issued wrong CSI call NodePublishVolume in this case assumed that the volume's device global path's mountpoint has been successfully setup in some successful previous CSI call NodeStageVolume. But acutally the volume's device global path has been unmounted in the possible forceful node-removal and kubelet cannot successfully sync the volume's real status because of its continually-failed "orphan" pod's volume cleanup by calling CSI NodeUnpublish when CSI pods have not been intialized yet.
In spite of this, the device global path still exists there if the volume still exists in the most cases. Even though the device global path has also been removed in the worst case, CSI current logic can still create the device global path itself.
I will do the functional test on this code change to verify it.
There was a problem hiding this comment.
I've similuated the scenario of volume's k8s global device mountpoint missing in my standalone test. This CSI defensive enhancement can work well as expected in the test.
CraneShiEMC
changed the title
CraneShiEMC
changed the title
Signed-off-by: Shi, Crane <crane.shi@emc.com>
…e failed volume status Signed-off-by: Shi, Crane <crane.shi@emc.com>
Signed-off-by: Shi, Crane <crane.shi@emc.com>
Signed-off-by: Shi, Crane <crane.shi@emc.com>
|
it seems the UT in PR validation failed. please fix it. |
| @@ -602,6 +649,9 @@ func (s *CSINodeService) NodeUnpublishVolume(ctx context.Context, req *csi.NodeU | |||
| volumeCR.Spec.Owners = owners | |||
| if len(volumeCR.Spec.Owners) == 0 { | |||
| volumeCR.Spec.CSIStatus = apiV1.VolumeReady | |||
| } else { | |||
There was a problem hiding this comment.
it seems this is not needed, the default state is Published in this function.
There was a problem hiding this comment.
because we will proceed on the failed volume now, here the set of status to Published is for Failed volume successfully unpublished and used by multiple pods.
Signed-off-by: Shi, Crane <crane.shi@emc.com>
Purpose
Resolves #1051
PR checklist
Testing
I've similuated the scenario of volume's k8s global device mountpoint missing in solely kubelet-issued NodePublishVolume in my standalone test. This CSI defensive enhancement can work well as expected in the test.
custom-ci passed: https://asd-ecs-jenkins.isus.emc.com/job/csi-custom-ci/1562/
custom-acceptance passed:
Atlantic (rke2): https://asd-ecs-jenkins.isus.emc.com/job/csi-custom-acceptance-tar_b_ona/39/
Openshift: https://asd-ecs-jenkins.isus.emc.com/job/csi-custom-acceptance-oil_bd/255/