Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(backport): Backport Tenderdash 0.7 to 0.8 #246

Merged
merged 72 commits into from
Feb 15, 2022
Merged

Conversation

lklimek
Copy link
Collaborator

@lklimek lklimek commented Jan 11, 2022

Issue being fixed or feature implemented

Backport changes from Tenderdash 0.7 (based on Tendermint 0.34) to Tenderdash 0.8 (based on Tendermint 0.35).

What was done?

  1. Use new router infrastructure from Tendermint 0.35 inside Validator Conn Executor.
  2. Moved dash/types to types/ to avoid dependency loops
  3. Backport of changes

How Has This Been Tested?

  • Unit tests passed*
  • E2E tests

Notes:

  1. Unit test TestWDRRQueue_DecreasingWeights fails locally (both this branch and upstream 0.35.0), but it passes in the GitHub pipeline
  2. Unit tests for test/e2e/* fail for now, to be done as part of e2e tests task

Breaking Changes

none

Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added or updated relevant unit/integration/functional/e2e tests
  • I have made corresponding changes to the documentation

For repository code-owners and collaborators only

  • I have assigned this pull request to a milestone

lklimek and others added 30 commits December 16, 2021 14:54
Unified logging of height and round, fixed some bugs:

* consensus: improve logging of vote messages to get better reporting

* fix(pex): use Logger instead of Printf

This fixes issues with parsing of logs for elastic

* fix(scripts): consensus.CommitMessage not defined in wal2json

* refactor(consensus,log): Unify logging of object metadata

* fix(types): invalid vote sign bytes put into error msg

* refactor(p2p): log ping-pong on P2PDebug level

* refactor(log): revert addition of LogableObject

* refactor(consensus): apply code review feedback
* feat(consensus): InitChain can set initial core height

* test(consensus): initial core height unit test

* test(e2e): allow setting of core height in resp to InitApp

* refactor(e2e): unify cfg name init_app_core_chain_locked_height

* test(consensus): remove not needed InitChain response

* fix(consensus): fix last core chain lock detection tests

* refactor(abci): remove last_core_chain_locked_height from ResponseInfo

* refactor(statesync): fix lint warnings

* feat(maverick): support initial core height from InitChain

* test(e2e): revert chainlock update height to 1000 in rotate.toml
Bumps [github.com/rs/cors](https://github.com/rs/cors) from 1.8.0 to 1.8.2.
- [Release notes](https://github.com/rs/cors/releases)
- [Commits](rs/cors@v1.8.0...v1.8.2)

---
updated-dependencies:
- dependency-name: github.com/rs/cors
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
* feat(consensus): add empty block on h-1 and h-2 apphash change

Before this change, empty block was created when apphash was changed since previous (height-1) block. This change changes this behavior to also check block (height-2).

* test(consensus): empty block on h-1 and h-2 apphash change

* rebase(consensus): apply code review feedback
This commit adds the capability to establish direct communication between multiple validators that are members of the same validator set. See [ADR D001](https://github.com/dashevo/tenderdash/blob/bb69b12ef3fb45a38e85e05662bd93f907f153fc/docs/architecture/adr-d001-inter-validator-set-messaging.md) for more details.

* feat: Add address to ABCI ValidatorUpdate message

* feat:ipaddress parsing logic added

* refactor: reorganize protobuf dependency tree

* test: ipaddress unit tests added + some fixes

* refactor: avoid large changes in *.proto to make backporting easier

* refactor: move lines around to avoid future conflicts

* refactor: format .proto files using `make proto-format`

* refactor: use factory pattern for IPAddress

* doc: fix comment of IPAddress.MustParseIP

* refactor: iptables.Copy() improvements

* refactor: move mustParseIP() to ipaddress_test.go

* refactor: use String URI for validator address

* feat: update code to work with abci.ValidatorUpdate.Address

* test: updated tests to support abci ValidatorUpdate.Address

* test: validator update address - unit tests green

* refactor: add ValidatorAddress type

* e2e/tests: fix validator address test

* refactor: make linter happy

* dash:  validator connection executor, WIP

* feat: Add address to ABCI ValidatorUpdate message

* feat:ipaddress parsing logic added

* refactor: reorganize protobuf dependency tree

* test: ipaddress unit tests added + some fixes

* refactor: avoid large changes in *.proto to make backporting easier

* refactor: move lines around to avoid future conflicts

* refactor: format .proto files using `make proto-format`

* refactor: use factory pattern for IPAddress

* doc: fix comment of IPAddress.MustParseIP

* refactor: iptables.Copy() improvements

* refactor: move mustParseIP() to ipaddress_test.go

* refactor: use String URI for validator address

* feat: update code to work with abci.ValidatorUpdate.Address

* test: updated tests to support abci ValidatorUpdate.Address

* test: validator update address - unit tests green

* refactor: add ValidatorAddress type

* e2e/tests: fix validator address test

* refactor: make linter happy

* refactor: rename file with ValidatorConnExecutor

* feat:inter validator set - work in progress

* e2e: enable p2p_debug and add more validator updates

* feat: inter validator set messaging

* test: inter validator set communication

* doc: update adr-d001 to reflect current implementation

* p2p: test TestTransportHandshake - wait for server to start

* refactor: code cleanup of inter validator set

* refactor: replace types.ValidatorAddress with p2p.NodeAddress

* p2p: allow empty validator address

* state: remove unused code

* test(p2p): TestTransportHandshake - add some sync

* test: improve TestValidatorConnExecutor tests

* refactor: rneame validator(update).Address to NodeAddress

* typo: fix linter issue

* refactor: apply code review feedback (WIP)

* refactor(dash): apply code review feedback, continued

* refactor(dash): improve validatorMap

* state: Add QuorumHash to EventValidatorSetUpdates

* feat: select validators to connect directly according to DIP-6 (WIP)

* fix: fix rebase conflicts

* feat: algorithm to select validators according to DIP-6

* refactor: rename select_nodes.go to select_validators.go

* feat: Use DIP-6 to select Validators for inter-validator set comm

* dash: add some code comments

* refactor: move files to separate packages

* test: fix e2e tests for single and simple networks

Validators in validator set contain address only after first
quorum rotation.

* refactor: apply code review feedback

* refactor: store validator as value (not pointer) internally

* refactor: change package names in dash/

* refactor(selectpeers): rename SelectValidatorsDIP6 to DIP6

* refactor: minor code formatting fixes

* refactor: Apply suggestions from code review

Co-authored-by: Dmitrii Golubev <dmitrii.golubev@deliveryhero.com>

* refactor: minor code improvements

* refactor: minor code improvements

* test: revert handshake timeout

* refactor: apply peer review feedback

* refactor: fix comment

Co-authored-by: Dmitrii Golubev <dmitrii.golubev@deliveryhero.com>

* doc(adr-d001): update ValidatorUpdate struct

* fix(selectpeers): add fallback for less than 5 validators

* refactor(selectpeers): Add ValidatorSelector interface

* test(p2p): tune sleep time when waiting for port to get opened

* feat: use validators from init chain response or genesis to establish connection (#228)

* feat: use validator list from init-chain response or genesis file to establish connection with them in inter-validator component

* refactor: move the logger from required argument into optional parameter, by defailt is used NopLogger

* fix: check a size of persisted peers before deleting

* feat: add "island" manifest file where every validator is isolated from each other

* feat: add a task into Makefile to run "island" e2e test

* chore: remove debug logging

* feat: autodetect NodeID in ValdatorUpdate (TD-40) (#213)

* feat: autodetect NodeID in ValdatorUpdate

* feat(dash/quorum): handle validators without address

* feat(quorum): Autodetect node ID when connecting to quorum members

* test(p2p/conn): remove unused privKeyWithNilPubKey

* doc(adr-d001): inter-validator set node id lookup

* refactor(quorum): apply code review feedback

* refactor(dash/types): rename package dashtypes to types

* refactor(p2p): make ValidateID exportable

* test(dash/types): increase validator address test coverage

* test(e2e): no node id in validator update address

* fix: node id not retrieved when processing val updates

* feat(dash/quorum): lookup node id in address book

* fix(dash/quorum): log validators list as single string

Convert validators map to string before logging. Without this
change, elasticsearch maps each validator as a separate field,
like:
`json.validators.1AD56AAACE6385A240B3A85EC231ACDE3972CD21A09678A7924B28116314A29C.address.Hostname`

* refactor: code comments, improved strings

* doc(adr-d001): update doc after implementation

* doc(adr-d001): apply feedback

Co-authored-by: shotonoff <shotonoff@gmail.com>
Fixed: Validator disconnected outside of Validator Set was not found in the Switch but never removed from ValidatorConnExecutor.connectedValidators.
Added create_proof_block_range configuration option to control the proof block creation algorithm.
@lklimek lklimek removed the Stale label Jan 25, 2022
@lklimek lklimek added this to the v0.8.0 milestone Jan 25, 2022
@lklimek lklimek marked this pull request as ready for review January 25, 2022 12:50
@lklimek lklimek changed the title Backport Tenderdash 0.7 to 0.8 chore(backport): Backport Tenderdash 0.7 to 0.8 Jan 25, 2022
@github-actions
Copy link

github-actions bot commented Feb 5, 2022

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the Stale label Feb 5, 2022
@github-actions github-actions bot closed this Feb 9, 2022
@shotonoff shotonoff reopened this Feb 9, 2022
Base automatically changed from fix-unit-test-fails to v0.8-dev February 9, 2022 08:26
@github-actions github-actions bot closed this Feb 14, 2022
@lklimek lklimek removed the Stale label Feb 14, 2022
@lklimek
Copy link
Collaborator Author

lklimek commented Feb 14, 2022

This needs to be merged into 0.8

@lklimek lklimek reopened this Feb 14, 2022
@codecov-commenter
Copy link

Codecov Report

Merging #246 (c041150) into v0.8-dev (3b6f545) will decrease coverage by 14.67%.
The diff coverage is 51.89%.

❗ Current head c041150 differs from pull request most recent head 34d231f. Consider uploading reports for the commit 34d231f to get more accurate results

Impacted file tree graph

@@              Coverage Diff              @@
##           v0.8-dev     #246       +/-   ##
=============================================
- Coverage     59.95%   45.28%   -14.68%     
=============================================
  Files           323      288       -35     
  Lines         44615    53012     +8397     
=============================================
- Hits          26748    24004     -2744     
- Misses        15885    27049    +11164     
+ Partials       1982     1959       -23     
Impacted Files Coverage Δ
abci/types/pubkey.go 0.00% <0.00%> (ø)
abci/types/types.pb.go 3.07% <0.00%> (ø)
config/toml.go 73.33% <ø> (+22.64%) ⬆️
dash/types/nodeid_resolver.go 0.00% <0.00%> (ø)
p2p/pex/addrbook.go 70.30% <0.00%> (ø)
p2p/test_util.go 65.60% <0.00%> (ø)
statesync/syncer.go 76.27% <ø> (ø)
test/e2e/generator/generate.go 0.00% <0.00%> (-82.72%) ⬇️
types/events.go 100.00% <ø> (ø)
p2p/switch.go 65.60% <14.28%> (ø)
... and 412 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1dd7819...34d231f. Read the comment docs.

Copy link
Collaborator

@shotonoff shotonoff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lklimek lklimek merged commit 874b784 into v0.8-dev Feb 15, 2022
@lklimek lklimek deleted the backport-0.7-to-0.8 branch February 15, 2022 12:21
shotonoff pushed a commit that referenced this pull request May 10, 2022
Co-authored-by: Erik Grinaker <erik@interchain.berlin>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants