Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Persistence] First Implementation of the StateHash #284

Closed
10 tasks
Olshansk opened this issue Oct 5, 2022 · 0 comments · Fixed by #285
Closed
10 tasks

[Persistence] First Implementation of the StateHash #284

Olshansk opened this issue Oct 5, 2022 · 0 comments · Fixed by #285
Assignees
@Olshansk
Labels
core Core infrastructure - protocol related persistence Persistence specific changes utility Utility specific changes
Milestone
M1: Pocket PoS (P...

Comments

@Olshansk
Copy link
Member

Olshansk commented Oct 5, 2022 •
edited
Loading

Objective

Implement the first fully functional, tested state hash implementation

Origin Document

Implement the interface and flow agreed on in #251. This replaces the original ticket (#147) whose scope was too large, whose implementation & discussion started in this PR #152 and whose further exploration was done in #237.

Goals

Deliverable

  • A PR with the first implementation of the state hash
  • Unit tests for verifying the business logic used to implement state hash computation

Non-goals / Non-deliverables

  • Modify any module interfaces
  • E2E tests for cross-module state hash verification
  • Provide the complete implementation of the state hash that will be used in production

General issue deliverables

  • Update the appropriate CHANGELOG
  • Update any relevant READMEs (local and/or global)
  • Update any relevant global documentation & references
  • If applicable, update the source code tree explanation
  • If applicable, add or update a state, sequence or flowchart diagram using mermaid

Testing Methodology

  • Add: Must add a suite of new unit tests
  • All tests: make develop_test
  • LocalNet: verify a LocalNet is still functioning correctly by following the instructions at docs/development/README.md

Creator: @Olshansk
@Olshansk Olshansk added core Core infrastructure - protocol related utility Utility specific changes persistence Persistence specific changes labels Oct 5, 2022
@Olshansk Olshansk self-assigned this Oct 5, 2022
This was referenced Oct 5, 2022
Closed
Closed
@Olshansk Olshansk moved this to In Progress in V1 Dashboard Oct 5, 2022
@Olshansk Olshansk mentioned this issue Oct 5, 2022
Merged
19 tasks
@jessicadaugherty jessicadaugherty moved this from In Progress to Up Next in V1 Dashboard Oct 10, 2022
@jessicadaugherty jessicadaugherty moved this from Up Next to In Progress in V1 Dashboard Oct 19, 2022
@Olshansk Olshansk mentioned this issue Oct 20, 2022
Merged
16 tasks
@Olshansk Olshansk mentioned this issue Nov 9, 2022
Closed
6 tasks
@Olshansk Olshansk moved this from In Progress to In Review in V1 Dashboard Nov 10, 2022
This was referenced Nov 27, 2022
Open
Merged
@andrewnguyen22 andrewnguyen22 removed their assignment Nov 30, 2022
Olshansk added a commit that referenced this issue Nov 30, 2022
@Olshansk @deblasis
@iajrz @Jasonyou1995 @okdas
[Persistence] First Implementation of the StateHash (#284) (#285)
9a2f643 
## Description

The first implementation of the 

## Issue

Fixes #284 with follow up work in #361.

## Type of change

Please mark the relevant option(s):

- [x] New feature, functionality or library
- [x] Bug fix
- [x] Code health or cleanup
- [x] Major breaking change
- [x] Documentation
- [ ] Other <!-- add details here if it a different type of change -->

## List of changes

### Persistence - Core Changes for SMT
* Introduced & defined for `block_persistence.proto`
    * A persistence specific protobuf for the Block stored in the BlockStore
* On `Commit`, prepare and store a persistence block in the KV Store, SQL Store
* Replace `IndexTransactions` (plural) to `IndexTransaction` (singular)
* Maintaining a list of StateTrees using Celestia’s SMT and badger as the KV store to compute the state hash
* Implemented `ComputeStateHash` to update the global state based on:
    * Validators
    * Applications
    * Servicers
    * Fisherman
    * Accounts
    * Pools
    * Transactions
    * Added a placeholder for `params` and `flags`
* Added a benchmarking and a determinism test suite to validate this

### Persistence - General module changes
* Implemented `GetAccountsUpdated`, `GetPoolsUpdated` and `GetActorsUpdated` functions
* Removed `GetPrevAppHash` and `indexTransactions` functions
* Removed `blockProtoBytes` and `txResults` from the local state and added `quorumCert`
* Consolidate all `resetContext` related operations into a single function
* Implemented `ReleaseWriteContext`
* Implemented ability to `ClearAllState` and `ResetToGenesis` for debugging & testing purposes
* Added unit tests for all of the supporting SQL functions implemented
* Some improvements in unit test preparation & cleanup (limited to this PR's functionality)

### Persistence - KVStore changes
* Renamed `Put` to `Set`
* Embedded `smt.MapStore` in the interface containing `Get`, `Set` and `Delete`
* Implemented `Delete`
* Modified `GetAll` to return both `keys` and `values`
* Turned off badger logging options since it’s noisy

### Persistence - Module Interface changes
* Removed `GetPrevHash` and just using `GetBlockHash` instead
* Removed `blockProtoBz` from `SetProposalBlock` interface
* Removed `GetLatestBlockTxs` and `SetLatestTxResults` in exchange for `IndexTransaction`
* Removed `SetTxResults`
* Renamed `UpdateAppHash` to `ComputeStateHash`
* Removed some getters related to the proposal block (`GetBlockTxs`, `GetBlockHash`, etc…)

### Consensus
* Propagate `highPrepareQC` if available to the block being created
* Remove `blockProtoBytes` from propagation in `SetProposalBlock`
* Guarantee that `writeContext` is released when refreshing the `utilityContext`
* Use `GetBlockHash(height)` instead of `GetPrevAppHash` to be more explicit
* Use the real `quorumCert` when preparing a new block

### Configs
* Updated the test generator to produce deterministic keys
* Added `trees_store_dir` to persistence configs
* Updated `LocalNet` configs to have an empty `tx_indexer_path` and `trees_store_dir`

### Makefile changes
* Added `db_cli_node`
* Added `db_show_schemas`
* Added `test_persistence_state_hash`
* Added `benchmark_persistence_state_hash`

### Debug
* `ResetToGenesis` - Added the ability to reset the state to genesis
* `ClearState` - Added the ability to clear the state completely (height 0 without genesis data)
## Testing

**New:**
- [x] `make benchmark_persistence_state_hash`
- [x] `make test_persistence_state_hash`
- [x] Iteration 3 demo was done using this PR: https://drive.google.com/file/d/1IOrzq-XJP04BJjyqPPpPu873aSfwrnur/view?usp=sharing

**Existing:**
- [x] `make develop_test`
- [x] [LocalNet](https://github.com/pokt-network/pocket/blob/main/docs/development/README.md) w/ all of the steps outlined in the `README`


## Required Checklist

- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have tested my changes using the available tooling
- [x] I have updated the corresponding CHANGELOG

### If Applicable Checklist
- [ ] I have updated the corresponding README(s); local and/or global
- [x] I have added tests that prove my fix is effective or that my feature works
- [ ] I have added, or updated, [mermaid.js](https://mermaid-js.github.io) diagrams in the corresponding README(s)
- [ ] I have added, or updated, documentation and [mermaid.js](https://mermaid-js.github.io) diagrams in `shared/docs/*` if I updated `shared/*`README(s)

--- 

Co-authored-by: Alessandro De Blasis <alex@deblasis.net>
Co-authored-by: Irving A.J. Rivas Z. <axel.rivas@gmail.com>
Co-authored-by: Andrew Nguyen <amnguyen1@mail.usf.edu>
Co-authored-by: Daniel Olshansky <olshansky.daniel@gmail.com>
Co-authored-by: Jason You <Jasonyou1995@users.noreply.github.com>
Co-authored-by: Dmitry Knyazev <okdas@users.noreply.github.com>
Repository owner moved this from In Review to Done in V1 Dashboard Nov 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Olshansk Olshansk

Labels
core Core infrastructure - protocol related persistence Persistence specific changes utility Utility specific changes
Projects
V1 Dashboard
Status: Done
Milestone
M1: Pocket PoS (Proof of Stake)
Development

Successfully merging a pull request may close this issue.

[Persistence] First Implementation of the StateHash (#284) pokt-network/pocket
2 participants
@Olshansk @andrewnguyen22