[occ] Add validation logic to scheduler #332

stevenlanders · 2023-10-16T23:03:11Z

Describe your changes and provide context

This adds a versioned store for each incarnation of a transaction
Scheduler re-executes based on validation output

Testing performed to validate your change

Unit test for scheduler (confirms expected tx output for conflicting transactions)

## Describe your changes and provide context This adds some comments with some useful code pointers for existing logic and discussing future OCC work ## Testing performed to validate your change NA

## Describe your changes and provide context Add multiversion store data structures file, and implement the multiversioned item ## Testing performed to validate your change Added unit tests to verify behavior

## Describe your changes and provide context This adds the incarnation field to the multiversion item data structure. ## Testing performed to validate your change updated unit tests

## Describe your changes and provide context This implements the multiversion with basic functionality, but still needs additional work to implement the iterator functionality and/or persisting readsets for validation ## Testing performed to validate your change Added unit tests for basic multiversion store

## Describe your changes and provide context - `ConcurrencyWorkers` represents the number of workers to use for concurrent transactions - since concurrrency-workers is a baseapp-level setting, implementations (like sei-chain) shouldn't have to pass it (but can) - it defaults to 10 if not set (via cli default value) - it defaults to 10 in app.toml only if that file is being created (and doesn't exist) - if explicitly set to zero on command line, it will override with the default (for safety) - cli takes precedence over the config file - no one has to do anything to get it to be 10 (no config changes no sei-chain changes required (aside from new cosmos version)) ## Testing performed to validate your change - Unit Tests for setting the value - Manually testing scenarios with sei-chain

## Describe your changes and provide context This adds in functionality to write the latest multiversion values to another store (to be used for writing to parent after transaction execution), and also adds in helpers for writeset management such as setting, invalidating, and setting estimated writesets. ## Testing performed to validate your change Unit testing for added functionality

## Describe your changes and provide context - `sei-cosmos` will receive a list of transactions, so that sei-chain does not need to hold the logic for OCC - This will make the logic easier to test, as sei-cosmos will be fairly self-contained - Types can be extended within a tx and within request/response Example interaction: <img src="https://github.com/sei-protocol/sei-cosmos/assets/6051744/58c9a263-7bc6-4ede-83ab-5e34794510b1" width=50% height=50%> ## Testing performed to validate your change - This is a skeleton for a batch interface

## Describe your changes and provide context This implements an mvkv store that will manage access from a transaction execution to the underlying multiversion store and underlying parent store if the multiversion store doesn't have that key. It will first serve any reads from its own writeset and readset, but if it does have to fall through to multiversion store or parent store, it will add those values to the readset. ## Testing performed to validate your change Unit tests

…ore (#330) ## Describe your changes and provide context This adds in validation for transaction state to multiversion store, and implements readset validation for it as well. ## Testing performed to validate your change Unit Test

## Describe your changes and provide context This implements Iterator and ReverseIterator for mvkv for the KVStore interface. The memiterator will be composed of versionindexedstore and multiversionstore, and will yield values in a cascading fashion firstly from the writeset, and then second from the multiversion store. This still needs optimization to persisted sorted keys instead of reconstructing sorted keys each time. ## Testing performed to validate your change Unit test to verify basic functionality

stevenlanders · 2023-10-18T14:24:27Z

Closing in favor of #336 (to avoid messy PR commit history after merges/rebases)

## Describe your changes and provide context - This was copied from #332 which became unwieldy due to commit history (merges/rebases) - Adds scheduler logic for validation - In this initial version it completes all executions then performs validations (which feed retries) - Once we start benchmarking we can make performance improvements to this - Retries tasks that fail validation and have no dependencies ## Testing performed to validate your change - Scheduler Test verifies multi-worker with conflicts

- This was copied from #332 which became unwieldy due to commit history (merges/rebases) - Adds scheduler logic for validation - In this initial version it completes all executions then performs validations (which feed retries) - Once we start benchmarking we can make performance improvements to this - Retries tasks that fail validation and have no dependencies - Scheduler Test verifies multi-worker with conflicts

udpatil and others added 27 commits September 13, 2023 10:04

Add occ todos / comments (#317)

e488091

## Describe your changes and provide context This adds some comments with some useful code pointers for existing logic and discussing future OCC work ## Testing performed to validate your change NA

Multiversion Item Implementation and Tests (#318)

6c69ad6

## Describe your changes and provide context Add multiversion store data structures file, and implement the multiversioned item ## Testing performed to validate your change Added unit tests to verify behavior

[occ] Add incarnation field (#321)

de307d0

## Describe your changes and provide context This adds the incarnation field to the multiversion item data structure. ## Testing performed to validate your change updated unit tests

add basic worker task and scheduler shell

a6a9d4a

remove ok to avoid a white-hot loop

edf17c5

fix comment spelling

8ed5ba9

add more skeleton

459ccd5

clarify comment

b06373e

init multi-version store

244f21d

updating comment

6f18c42

fix import cycle

7260454

remove context item for now (import cycle), needs refactor

a4b4a7f

add basic test

e8be7d5

move packages

37bb06e

remove excess go func

588f1f6

cleanup

aea0cf9

add clarifying comments

a16fbc2

Merge branch 'occ-main' into add-worker-task-and-shell-scheduler

c1a37b7

inject multiversion and versioned stores

93d508d

cleanup

e2ce65e

cleanup import

c5f3dca

stevenlanders changed the title ~~Add versioned store to each tx context~~ [occ] Add versioned store to each tx context Oct 16, 2023

fix imports

68a6ccc

stevenlanders marked this pull request as ready for review October 16, 2023 23:23

stevenlanders and others added 4 commits October 17, 2023 13:34

invalidate all mv stores on conflict

27adcba

Merge branch 'occ-main' into add-validation-to-scheduler

920228f

fix readset

36b4e64

udpatil force-pushed the occ-main branch from a81f520 to 663716a Compare October 17, 2023 19:03

stevenlanders added 7 commits October 18, 2023 08:11

working version

faeb7ac

cleanup

60d24b3

lowercase transaction index

0392b79

merge occ-main

319b45a

cleanup

0d966ba

remove mux init

307d5f8

fix signature

bf18ea7

stevenlanders changed the title ~~[occ] Add versioned store to each tx context~~ [occ] Add validation logic to scheduler Oct 18, 2023

stevenlanders added 2 commits October 18, 2023 09:17

remove debug log line

e6513d7

add validated to validatable statuses

93ad441

stevenlanders requested a review from udpatil October 18, 2023 13:41

remove todos

fcac0ba

stevenlanders closed this Oct 18, 2023

stevenlanders mentioned this pull request Oct 18, 2023

[occ] Add scheduler logic for validation #336

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[occ] Add validation logic to scheduler #332

[occ] Add validation logic to scheduler #332

stevenlanders commented Oct 16, 2023 •

edited

Loading

stevenlanders commented Oct 18, 2023

[occ] Add validation logic to scheduler #332

[occ] Add validation logic to scheduler #332

Conversation

stevenlanders commented Oct 16, 2023 • edited Loading

Describe your changes and provide context

Testing performed to validate your change

stevenlanders commented Oct 18, 2023

stevenlanders commented Oct 16, 2023 •

edited

Loading