[occ] Add scheduler logic for validation #336
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
stevenlanders
changed the title
udpatil
reviewed
Oct 19, 2023
| tasks := toTasks(reqs) | ||
| toExecute := tasks | ||
| for len(toExecute) > 0 { | ||
| for !allValidated(tasks) { |
There was a problem hiding this comment.
is it possible for a race condition to occur where all tasks are validated? for example, if TX 3 is valid, and TX 1 finishes execution, and needs to trigger revalidation for TX 3, but TX 1 finishes validation slightly faster, so momentarily theyr all in a valid state? would this be a possible situation in which TX 3 is truly invalid based on the retriggered validation, but the race allows exiting the loop? maybe validateAll is blocking for the loop, so the allValidated won't be evaluated then? not sure
There was a problem hiding this comment.
There's no race because this validateAll is blocking.
tasks/scheduler.go
Outdated
|
|
||
| // validated tasks can become unvalidated if an earlier re-run task now conflicts | ||
| case statusExecuted, statusValidated: | ||
| conflicts := s.findConflicts(tasks[i]) |
There was a problem hiding this comment.
isnt this just resulting in all the validations being performed sequentially as opposed to concurrently?
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## occ-main #336 +/- ##
============================================
+ Coverage 56.05% 56.09% +0.04%
============================================
Files 627 627
Lines 52638 52753 +115
============================================
+ Hits 29505 29591 +86
- Misses 21032 21060 +28
- Partials 2101 2102 +1
|
udpatil
approved these changes
Oct 19, 2023
| panic("not implemented") | ||
| } | ||
|
|
||
| func (store *VersionIndexedStore) UpdateIterateSet(iterationTracker iterationTracker) { |
There was a problem hiding this comment.
wasnt this already present from occ-main, or is the diff highlight just weird because of merge conflict resolution
There was a problem hiding this comment.
This line doesn't seem to show up in the overall diff, just in the commit itself (a merge from occ-main).
| tasks[i].Status = statusWaiting | ||
| } | ||
| } else { | ||
| // no conflicts means this task is validated | ||
| } else if len(conflicts) == 0 { | ||
| tasks[i].Status = statusValidated | ||
| } |
There was a problem hiding this comment.
in the case where we are valid but have conflicts, does the task states stay as executed then? (just want to make sure my understanding is correct
There was a problem hiding this comment.
that's right, it remains as executed and not returned for re-execution, and therefore is eligible for re-validation
udpatil
pushed a commit
that referenced
this pull request
Jan 2, 2024
- This was copied from #332 which became unwieldy due to commit history (merges/rebases) - Adds scheduler logic for validation - In this initial version it completes all executions then performs validations (which feed retries) - Once we start benchmarking we can make performance improvements to this - Retries tasks that fail validation and have no dependencies - Scheduler Test verifies multi-worker with conflicts
udpatil
pushed a commit
that referenced
this pull request
Jan 8, 2024
- This was copied from #332 which became unwieldy due to commit history (merges/rebases) - Adds scheduler logic for validation - In this initial version it completes all executions then performs validations (which feed retries) - Once we start benchmarking we can make performance improvements to this - Retries tasks that fail validation and have no dependencies - Scheduler Test verifies multi-worker with conflicts
udpatil
pushed a commit
that referenced
this pull request
Jan 18, 2024
- This was copied from #332 which became unwieldy due to commit history (merges/rebases) - Adds scheduler logic for validation - In this initial version it completes all executions then performs validations (which feed retries) - Once we start benchmarking we can make performance improvements to this - Retries tasks that fail validation and have no dependencies - Scheduler Test verifies multi-worker with conflicts
udpatil
pushed a commit
that referenced
this pull request
Jan 18, 2024
- This was copied from #332 which became unwieldy due to commit history (merges/rebases) - Adds scheduler logic for validation - In this initial version it completes all executions then performs validations (which feed retries) - Once we start benchmarking we can make performance improvements to this - Retries tasks that fail validation and have no dependencies - Scheduler Test verifies multi-worker with conflicts
udpatil
pushed a commit
that referenced
this pull request
Jan 25, 2024
- This was copied from #332 which became unwieldy due to commit history (merges/rebases) - Adds scheduler logic for validation - In this initial version it completes all executions then performs validations (which feed retries) - Once we start benchmarking we can make performance improvements to this - Retries tasks that fail validation and have no dependencies - Scheduler Test verifies multi-worker with conflicts
udpatil
pushed a commit
that referenced
this pull request
Jan 31, 2024
- This was copied from #332 which became unwieldy due to commit history (merges/rebases) - Adds scheduler logic for validation - In this initial version it completes all executions then performs validations (which feed retries) - Once we start benchmarking we can make performance improvements to this - Retries tasks that fail validation and have no dependencies - Scheduler Test verifies multi-worker with conflicts
codchen
pushed a commit
that referenced
this pull request
Feb 6, 2024
- This was copied from #332 which became unwieldy due to commit history (merges/rebases) - Adds scheduler logic for validation - In this initial version it completes all executions then performs validations (which feed retries) - Once we start benchmarking we can make performance improvements to this - Retries tasks that fail validation and have no dependencies - Scheduler Test verifies multi-worker with conflicts
udpatil
pushed a commit
that referenced
this pull request
Feb 27, 2024
- This was copied from #332 which became unwieldy due to commit history (merges/rebases) - Adds scheduler logic for validation - In this initial version it completes all executions then performs validations (which feed retries) - Once we start benchmarking we can make performance improvements to this - Retries tasks that fail validation and have no dependencies - Scheduler Test verifies multi-worker with conflicts
udpatil
pushed a commit
that referenced
this pull request
Mar 1, 2024
- This was copied from #332 which became unwieldy due to commit history (merges/rebases) - Adds scheduler logic for validation - In this initial version it completes all executions then performs validations (which feed retries) - Once we start benchmarking we can make performance improvements to this - Retries tasks that fail validation and have no dependencies - Scheduler Test verifies multi-worker with conflicts
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Describe your changes and provide context
Testing performed to validate your change