Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(semaphore): ensure holderKeys carry all information needed. Fixes #8684 #13553

Merged
merged 16 commits into from
Sep 18, 2024
Merged

fix(semaphore): ensure holderKeys carry all information needed. Fixes #8684 #13553

merged 16 commits into from
Sep 18, 2024

Conversation

isubasinghe
Copy link
Member

@isubasinghe isubasinghe commented Sep 3, 2024 •
edited
Loading

Fixes #8684

Motivation

The Holder keys in the status field of a workflow do not carry enough information to work as expected.
This change ensures a single function (getHolderKey) is used for both acquiring and releasing keys.
This in turn deprecates(getResourceKey).

Modifications

Store the entire holderKey in the status. Simple but effective change to fix #8684.
Allows controllers to support V2 holder names even if the holder was first populated by a v1 holder name.

Verification

The unit tests were run with HOLDER_KEY_VERSION=v2.
NOTE: E2E tests HAVE NOT BEEN RUN WITH THIS ENV VAR.
Largely due to being unable to run the E2E tests in a reliable way on my machine.

@isubasinghe isubasinghe changed the title fix: ensure Holding keys carry all information needed fix: ensure Holding keys carry all information needed. Fixes #8684 Sep 4, 2024
@isubasinghe isubasinghe marked this pull request as ready for review September 4, 2024 05:02
isubasinghe
isubasinghe commented Sep 4, 2024
View reviewed changes
Copy link
Member Author

@isubasinghe isubasinghe left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks alright, a few minor issues

pkg/apis/workflow/v1alpha1/workflow_types.go Outdated Show resolved Hide resolved
pkg/apis/workflow/v1alpha1/workflow_types.go Outdated Show resolved Hide resolved
workflow/controller/operator_concurrency_test.go Outdated Show resolved Hide resolved
workflow/sync/sync_manager.go Outdated Show resolved Hide resolved
@agilgur5 agilgur5 self-requested a review September 4, 2024 16:49
@agilgur5 agilgur5 mentioned this pull request Sep 4, 2024
Closed
8 tasks
agilgur5
agilgur5 reviewed Sep 4, 2024
View reviewed changes
Copy link
Contributor

@agilgur5 agilgur5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In #8684 (comment), you said this would be a breaking change? Then this should have a ! in the title and the appropriate milestone.

Or are you making this backward compatible with the env var?

NOTE: E2E tests HAVE NOT BEEN RUN WITH THIS ENV VAR.
Largely due to being unable to run the E2E tests in a reliable way on my machine.

If this is intended to be required in the future, we should probably enable it during testing similar to #12413 (comment)

@isubasinghe
Copy link
Member Author

isubasinghe commented Sep 4, 2024 •
edited
Loading

In #8684 (comment), you said this would be a breaking change? Then this should have a ! in the title and the appropriate milestone.

Or are you making this backward compatible with the env var?

Yeah this should mean that you can upgrade workflows to use the new holder key name without any issue, no longer a breaking change.

NOTE: E2E tests HAVE NOT BEEN RUN WITH THIS ENV VAR.
Largely due to being unable to run the E2E tests in a reliable way on my machine.

If this is intended to be required in the future, we should probably enable it during testing similar to #12413 (comment)

Yeah that sounds good to me.

@isubasinghe isubasinghe mentioned this pull request Sep 4, 2024
Closed
3 tasks
@agilgur5 agilgur5 changed the title fix: ensure Holding keys carry all information needed. Fixes #8684 fix(semaphore): ensure holderKeys carry all information needed. Fixes #8684 Sep 5, 2024
agilgur5
agilgur5 reviewed Sep 5, 2024
View reviewed changes
pkg/apis/workflow/v1alpha1/workflow_types.go Outdated Show resolved Hide resolved
workflow/sync/mutex_test.go Outdated Show resolved Hide resolved
workflow/controller/operator_concurrency_test.go Outdated Show resolved Hide resolved
pkg/apis/workflow/v1alpha1/workflow_types.go Outdated Show resolved Hide resolved
@agilgur5 agilgur5 changed the title fix(semaphore): ensure holderKeys carry all information needed. Fixes #8684 fix(semaphore): ensure holderKeys carry all information needed. Fixes #8684 Sep 5, 2024
@isubasinghe isubasinghe mentioned this pull request Sep 10, 2024
Open
Joibel
Joibel requested changes Sep 13, 2024
View reviewed changes
Copy link
Member

@Joibel Joibel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At minimum we need duplicates of the tests that are V1 only as V2 tests.

My preference is to drop V1 completely, agreeing with @agilgur5

docs/environment-variables.md Outdated Show resolved Hide resolved
workflow/sync/sync_manager.go Outdated Show resolved Hide resolved
pkg/apis/workflow/v1alpha1/workflow_types.go Outdated Show resolved Hide resolved
pkg/apis/workflow/v1alpha1/workflow_types.go Outdated Show resolved Hide resolved
@isubasinghe
Copy link
Member Author

legacy holding keys have been eliminated, now only v2 style keys exist. Meaning existing keys now test v2 directly.
We upgrade v1 keys to v2. There is a test for this upgrade process in place.

isubasinghe
isubasinghe commented Sep 16, 2024
View reviewed changes
Copy link
Member Author

@isubasinghe isubasinghe left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Joibel opinions? wait please, adding more tests

@isubasinghe isubasinghe marked this pull request as draft September 16, 2024 09:48
@isubasinghe isubasinghe marked this pull request as ready for review September 17, 2024 06:19
@isubasinghe
Copy link
Member Author

isubasinghe commented Sep 17, 2024 •
edited
Loading

This PR is now ready, I believe I have addressed the edge cases. Unfortunately there is some ambiguity (as expected really) if a sync exist at both the template level and workflow level.

We just choose to go with workflow level in this case of ambiguity.

P.S. I hate this bug

Joibel
Joibel requested changes Sep 17, 2024
View reviewed changes
workflow/sync/sync_manager.go Outdated Show resolved Hide resolved
workflow/sync/sync_manager.go Outdated Show resolved Hide resolved
workflow/sync/sync_manager.go Outdated Show resolved Hide resolved
workflow/sync/sync_manager.go Outdated Show resolved Hide resolved
workflow/sync/sync_manager.go Outdated Show resolved Hide resolved
workflow/sync/sync_manager.go Outdated Show resolved Hide resolved
pkg/apis/workflow/v1alpha1/workflow_types.go Outdated Show resolved Hide resolved
workflow/sync/sync_manager_test.go Outdated Show resolved Hide resolved
@isubasinghe
fix: ensure Holding keys carry all information needed
74895c0 
Signed-off-by: isubasinghe <isitha@pipekit.io>
@isubasinghe
fix: added tests for edge cases
bd53d0f 
Signed-off-by: isubasinghe <isitha@pipekit.io>
@isubasinghe
fix: lint issue addressed
335342a 
Signed-off-by: isubasinghe <isitha@pipekit.io>
@isubasinghe
fix: addressed comments
193320e 
Signed-off-by: isubasinghe <isitha@pipekit.io>
@isubasinghe
fix: address feedback
7872148 
Signed-off-by: isubasinghe <isitha@pipekit.io>
@isubasinghe
fix: replace os.Setenv with t.Setenv
37a251c 
Signed-off-by: isubasinghe <isitha@pipekit.io>
@isubasinghe
fix: added godoc comment
16d01e3 
Signed-off-by: isubasinghe <isitha@pipekit.io>
@isubasinghe
fix: remove v1 support
2b21394 
Signed-off-by: isubasinghe <isitha@pipekit.io>
@isubasinghe
docs: reset envrionment variable docs
c56d7ac 
Signed-off-by: isubasinghe <isitha@pipekit.io>
@isubasinghe
fix: removed unnecessary v1 holder key check
2fd00eb 
Signed-off-by: isubasinghe <isitha@pipekit.io>
@isubasinghe
chore: general code cleanup for holding key changes
854907e 
Signed-off-by: isubasinghe <isitha@pipekit.io>
@isubasinghe
chore: remove unnecessary import
2771425 
Signed-off-by: isubasinghe <isitha@pipekit.io>
@isubasinghe
fix: add template level migration
3e7a825 
Signed-off-by: isubasinghe <isitha@pipekit.io>
@isubasinghe
fix: check level first
94828e9 
Signed-off-by: isubasinghe <isitha@pipekit.io>
@isubasinghe
fix: address feedback
6d4da2c 
Signed-off-by: isubasinghe <isitha@pipekit.io>
@isubasinghe
fix: removed logging
e445525 
Signed-off-by: isubasinghe <isitha@pipekit.io>
@isubasinghe
Copy link
Member Author

/retest

1 similar comment
@isubasinghe
Copy link
Member Author

/retest

@isubasinghe
Copy link
Member Author

Irrelevant tests seem to be broken since a rebase.

@Joibel Joibel merged commit 440f1cc into argoproj:main Sep 18, 2024
27 checks passed
Joibel pushed a commit to pipekit/argo-workflows that referenced this pull request Sep 19, 2024
@isubasinghe @Joibel
fix(semaphore): ensure holderKeys carry all information needed. Fixes
97981b0 
argoproj#8684 (argoproj#13553)

Signed-off-by: isubasinghe <isitha@pipekit.io>
@Joibel Joibel mentioned this pull request Sep 20, 2024
Open
Joibel pushed a commit that referenced this pull request Sep 20, 2024
@isubasinghe @Joibel
fix(semaphore): ensure holderKeys carry all information needed. Fixes
a84edd9 
#8684 (#13553)

Signed-off-by: isubasinghe <isitha@pipekit.io>
@agilgur5 agilgur5 added this to the v3.5.x patches milestone Sep 20, 2024
@isubasinghe isubasinghe mentioned this pull request Oct 8, 2024
Closed
@Joibel Joibel deleted the fix-deadlock-mutex branch November 7, 2024 10:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@agilgur5 agilgur5 agilgur5 left review comments

@Joibel Joibel Joibel approved these changes

@sarabala1979 sarabala1979 Awaiting requested review from sarabala1979

Assignees
No one assigned
Labels
area/mutex-semaphore
Projects
None yet
Milestone
v3.5.x patches
Development

Successfully merging this pull request may close these issues.

Workflow deadlocks on mutex in steps template if controller is restarted
4 participants
@isubasinghe @Joibel @agilgur5 @sarabala1979