Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent multiple active instances with the same id #288

Merged
merged 17 commits into from
Nov 18, 2023

Conversation

cschleiden
Copy link
Owner

@cschleiden cschleiden commented Nov 17, 2023

With this change there can only be one active workflow instance with the same ID. If one attempts to create either a workflow or sub-workflow instance with the same ID a backend.ErrInstanceAlreadyExists error will be returned.

MySql and SQLite backends both make an additional query when creating a new workflow or sub-workflow instance.

The redis change is a bit bigger: checking for top-level instances with the same ID is relatively easy, but checking for existing subworkflow instances within the larger CompleteWorkflowTask transaction is not 100% safe. In practice it would probably be fine, preventing sub-workflow id collisions is relatively easy but I decided to try to move the full CompleteWorkflowTask transaction to a .lua script instead. That way no other commands are executed while this is being active.

The script requires quite a bit of data, so every potential key needed and every argument needs to be passed into from the calling Go code. In the script a couple helper functions getKey() and getArgv() return the next key/argument. This requires careful matching between Go and script and might be optimized by passing in a json struct instead that is then unmarshaled via cjson in the script but this works for now.

@cschleiden
Copy link
Owner Author

!bench !large !skipmysql !skipsqlite

@cschleiden
Copy link
Owner Author

!bench !large !skipmysql !skipsqlite

Copy link

Redis run

Command Mean [ms] Min [ms] Max [ms] Relative
redis-main 161.4 ± 19.3 124.4 194.2 1.15 ± 0.19
redis-pr 140.2 ± 16.4 112.3 172.0 1.00

Large Redis payload run (1MB)

Command Mean [s] Min [s] Max [s] Relative
redis-main 21.719 ± 0.625 20.614 22.596 1.00 ± 0.04
redis-pr 21.628 ± 0.737 20.663 22.775 1.00

@cschleiden cschleiden marked this pull request as ready for review November 17, 2023 18:28
@cschleiden cschleiden changed the title Draft: prevent instances with duplicate ids Prevent multiple active instances with the same id Nov 17, 2023
backend/redis/queue.go Outdated Show resolved Hide resolved
@cschleiden cschleiden merged commit 4277a8a into main Nov 18, 2023
4 checks passed
@cschleiden cschleiden deleted the duplicate-instances branch November 18, 2023 05:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant