Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ISSUE-1051] Handle Kubelet's Wrong CSI Call Inconsistent with Real Volume Status for OBS Release 1.3 #1053

Open
wants to merge 18 commits into
base: rel/objectscale-1.3
Choose a base branch
from
Open

[ISSUE-1051] Handle Kubelet's Wrong CSI Call Inconsistent with Real Volume Status for OBS Release 1.3 #1053

wants to merge 18 commits into from

Conversation

CraneShiEMC
Copy link
Collaborator

@CraneShiEMC CraneShiEMC commented Aug 27, 2023 •
edited
Loading

Purpose

Resolves #1051

  1. Handle Kubelet's Wrong CSI Call Inconsistent with Real Volume Status
  2. Proceed Kubelet's CSI call also on Failed Volume

PR checklist

  • Add link to the issue
  • Choose Project
  • Choose PR label
  • New unit tests added
  • Modified code has meaningful comments
  • All TODOs are linked with the issues
  • All comments are resolved

Testing

I've similuated the scenario of volume's k8s global device mountpoint missing in my standalone test. This CSI defensive enhancement can work well as expected in the test.

custom ci passed: https://asd-ecs-jenkins.isus.emc.com/job/csi-custom-ci/1562/

custom-acceptance passed:
Atlantic (rke2): https://asd-ecs-jenkins.isus.emc.com/job/csi-custom-acceptance-tar_b_ona/39/
Openshift: https://asd-ecs-jenkins.isus.emc.com/job/csi-custom-acceptance-oil_bd/255/

yimingwangdell and others added 17 commits July 18, 2023 14:10
@yimingwangdell
[ISSUE-1038] 1.3.0 branched out. Change version to 1.4.0 for new rele…
b5ffc6c 
…ase. (#1039)

Signed-off-by: yimingwangdell <yiming.wang@dell.com>
@yimingwangdell
[ISSUE-1040] fix pip3 install PyYAML failed. (#1041)
57095d3 
Signed-off-by: yimingwangdell <yiming.wang@dell.com>
@CraneShiEMC
[ISSUE-1018] Refine Storage Group Feature (#1031)
8085a0a 
* add StorageGroupStatus

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* support StorageGroupStatus in current workflows

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine log

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine error handling

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* trigger storage group resync if applicable in drive removal

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* A drive whose Usage is REMOVED will not be selected in any storage group and its existing sg label takes no effect

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* sg feature will not apply to drive physically removed

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* handle the drive removal case of drive with manual sg label

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix go lint error

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add UT case for drive-removal-triggered sg sync

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* improve UT coverage

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine sg annotation for drive removal

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* handle case of invalid sg for drive removal

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* also exclude removing sg for trigger sg resync in drive removal

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine sg removal status handling

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* Revert "refine error handling"

This reverts commit 06607e7.

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine log and some code logic

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* try to add immutability validation rule to storagegroup spec

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* upgrade controller-gen version to v0.9.2

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add storagegroupcontroller UT initial

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* Revert "add storagegroupcontroller UT initial"

This reverts commit 1ea8660.

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refactor and add UT of storagegroupcontroller

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix storagegroupcontroller UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine the logic of sg deletion

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix bug

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix go-lint err

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix go-lint error

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add drive IsClean support, decrease k8s api call, remove manual sg labeling support

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine corner case handling

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine and add UT to storagegroupcontroller

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine storagegroupcontroller and add UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* make controller svc's k8scache also sync sg and lvg objs'

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* use k8s cache, re-support sg label manual change and refine in sg ctrl

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix lint err

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add storagegroupcontroller UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* storagegroup controller will not reconcile on drive delete event

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* not support Health, Status, Usage and IsClean as DriveSelector's MatchFields

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* in storagegroupcontroller's reconcile, only sync drive when reqName is uuid

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine the logic to avoid nil pointer error

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* revert the usage of k8scache

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add custom storage group proposal draft

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine custom storage group proposal

Signed-off-by: Shi, Crane <crane.shi@emc.com>

---------

Signed-off-by: Shi, Crane <crane.shi@emc.com>
@CraneShiEMC
[ISSUE-1045] Fake Attach Feature for Non-LVM-Block-Mode Volumes (#1046)
4c3fd07 
* fix pr validation startup failure

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* directly use generated file for fake attach block mode

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* still use loopback device wrap

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine code for creating fake device

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix go lint error

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add mock func implementation

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix go lint

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine func

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* support fake attach block mode with fake device, add removeLoopDevice,

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix go lint

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* support clean fake device in fake-attach block-mode

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* support non-existing current fake device case

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* change fake device dir on host

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine log

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* support the case of get fake device info failure

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* clean fake device also in removal of fake-attach block-mode vol

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* if fake-device ann is invalid, re-create the fake device and update ann

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* enhance

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add comment

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* add UT

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* check loop device err shouldn't block subsequent op; clean fake device should also check loop device first

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* update fake-attach doc accordingly

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* fix typo

Signed-off-by: Shi, Crane <crane.shi@emc.com>

* refine doc

Signed-off-by: Shi, Crane <crane.shi@emc.com>

---------

Signed-off-by: Shi, Crane <crane.shi@emc.com>
@CraneShiEMC
for NodeUnstage/NodePublish/NodeUnpublish, also need to retry on Fail…
5f071d6 
…ed volume

Signed-off-by: Shi, Crane <crane.shi@emc.com>
@CraneShiEMC
fix lint error
2e6ab7f 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
@CraneShiEMC
fix UT
fdc5970 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
@CraneShiEMC
add UT
a50a7f1 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
@CraneShiEMC
add UT
bfb1d98 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
@CraneShiEMC
Redo NodeStageVolume if srcPath is not mounted
1a77d43 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
@CraneShiEMC
correct stagingTargetPath
d36e785 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
@CraneShiEMC
add warning log for redo NodeStageVolume
4376b10 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
@CraneShiEMC
fix that, in multi-pod case, successful unpublish volume cannot resum…
c835b96 
…e failed volume status

Signed-off-by: Shi, Crane <crane.shi@emc.com>
@CraneShiEMC
unlock volume to redo NodeStageVolume if applicable in NodePublishVolume
a3abfb9 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
@CraneShiEMC
fix issue of outdated volume
af81d9a 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
@CraneShiEMC
Revert "[ISSUE-1038] 1.3.0 branched out. Change version to 1.4.0 for …
aee7d03 
…new release. (#1039)"

This reverts commit b5ffc6c.

Signed-off-by: Shi, Crane <crane.shi@emc.com>
@CraneShiEMC
Merge branch 'rel/objectscale-1.3' of https://github.com/dell/csi-bar…
e65eb59 
…emetal into bugfix-handle-kubelet-wrong-call-obs-1.3

Signed-off-by: Shi, Crane <crane.shi@emc.com>
libzhang
libzhang approved these changes Aug 27, 2023
View reviewed changes
Copy link
Collaborator

@libzhang libzhang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please fix UT failure in PR validation before merge.

@CraneShiEMC
Copy link
Collaborator Author

please fix UT failure in PR validation before merge.

OK

@CraneShiEMC
fix and add UT
11d8f7e 
Signed-off-by: Shi, Crane <crane.shi@emc.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lordjun lordjun lordjun approved these changes

@libzhang libzhang libzhang approved these changes

@yanxiucai yanxiucai Awaiting requested review from yanxiucai

@yimingwangdell yimingwangdell Awaiting requested review from yimingwangdell

@aiqiEMC aiqiEMC Awaiting requested review from aiqiEMC

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@CraneShiEMC @libzhang @lordjun @yimingwangdell