Skip to content

Conversation

@knst
Copy link
Collaborator

@knst knst commented Dec 28, 2023

Issue being fixed or feature implemented

llmq/utils has simple util code that used all over code base and also have too heavy code for calculation quorums such as: GetAllQuorumMembers, EnsureQuorumConnections and other.

These helpers for calculation quorums are used only by evo/deterministicmns, evo/simplifiedmns and llmq/* modules, but llmq/utils is included in many other modules for various trivial helpers.

What was done?

Prior work:

This PR remove all non-quorum calculation code from llmq/utils. Eventually it happens that easier to take everything out rather than move Quorum Calculation to new place atm:

  • new module llmq/options have a code related to various params, command line options, spork-related etc
  • llmq/utils is not included in various files which do not use any llmq/utils code
  • helper BuildCommitmentHash goes to llmq/commitment
  • helper BuildSignHash goes to llmq/signing
  • helper GetLLMQParam inlined since it's trivial (it has not been trivial when introduced ages ago)
  • removed dependency of IsQuorumEnabled on CQuorumManager which means quorumManager deglobalization is done for 90%

How Has This Been Tested?

  • Run unit functional tests
  • updated circular dependencies test/lint/lint-circular-dependencies.sh
  • check that llmq/utils is not included without needs to calculate Quorums Members
$ grep -r include src/ 2> /dev/null | grep -v .Po: | grep -vE 'llmq/utils.(h|cpp)': | grep llmq/utils  
src/evo/mnauth.cpp:#include <llmq/utils.h>
src/evo/deterministicmns.cpp:#include <llmq/utils.h>
src/llmq/quorums.cpp:#include <llmq/utils.h>
src/llmq/blockprocessor.cpp:#include <llmq/utils.h>
src/llmq/commitment.cpp:#include <llmq/utils.h>
src/llmq/debug.cpp:#include <llmq/utils.h>
src/llmq/dkgsessionhandler.cpp:#include <llmq/utils.h>
src/llmq/dkgsession.cpp:#include <llmq/utils.h>
src/llmq/dkgsessionmgr.cpp:#include <llmq/utils.h>
src/rpc/quorums.cpp:#include <llmq/utils.h>

Breaking Changes

N/A

Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added or updated relevant unit/integration/functional/e2e tests
  • I have made corresponding changes to the documentation
  • I have assigned this pull request to a milestone

@knst knst added this to the 20.1 milestone Dec 28, 2023
@knst knst changed the title refactor: keep in llmq/utils quorum calculations only new module llmq/options refactor: split llmq/utils to Quorum Calculation and llmq/options Dec 28, 2023
@knst knst force-pushed the refactor-llmq_utils branch from 6229ce8 to 5d265ac Compare December 29, 2023 17:16
@knst knst requested a review from UdjinM6 December 29, 2023 17:16
UdjinM6
UdjinM6 previously approved these changes Dec 29, 2023
Copy link

@UdjinM6 UdjinM6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK

@PastaPastaPasta
Copy link
Member

to get merge after #5782 (due to its conflicts)

@PastaPastaPasta PastaPastaPasta marked this pull request as draft January 7, 2024 01:38
@github-actions
Copy link

This pull request has conflicts, please rebase.

@github-actions
Copy link

This pull request has conflicts, please rebase.

@knst knst force-pushed the refactor-llmq_utils branch from 87c5163 to 9002f39 Compare January 10, 2024 21:57
@knst knst marked this pull request as ready for review January 10, 2024 21:58
@knst knst requested a review from UdjinM6 January 10, 2024 22:01
UdjinM6
UdjinM6 previously approved these changes Jan 10, 2024
Copy link

@UdjinM6 UdjinM6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-utACK

@knst
Copy link
Collaborator Author

knst commented Jan 11, 2024

need guix image to test a theory with crash:

Thread 1 "dashd" received signal SIGSEGV, Segmentation fault.
....
#0  0x0000555555c45198 in llmq::CQuorumManager::ScanQuorums (this=this@entry=0x55555694a110, llmqType=Consensus::LLMQType::LLMQ_TEST_INSTANTSEND, 
    pindexStart=pindexStart@entry=0x555556d66880, nCountRequested=nCountRequested@entry=1) at llmq/quorums.cpp:495
#1  0x0000555555809089 in llmq::utils::IsQuorumTypeEnabledInternal (llmqType=<optimized out>, qman=..., pindexPrev=..., optDIP0024IsActive=..., optDIP0024IsActive@entry=..., 
    optHaveDIP0024Quorums=...) at llmq/utils.cpp:942
#2  0x00005555558091d6 in llmq::utils::IsQuorumTypeEnabled (llmqType=<optimized out>, qman=..., pindexPrev=...) at llmq/utils.cpp:923
#3  0x0000555555c45211 in llmq::CQuorumManager::ScanQuorums (this=this@entry=0x55555694a110, llmqType=Consensus::LLMQType::LLMQ_TEST_INSTANTSEND, 
    pindexStart=pindexStart@entry=0x555556d66880, nCountRequested=nCountRequested@entry=1) at llmq/quorums.cpp:496
#4  0x0000555555809089 in llmq::utils::IsQuorumTypeEnabledInternal (llmqType=<optimized out>, qman=..., pindexPrev=..., optDIP0024IsActive=..., optDIP0024IsActive@entry=..., 
    optHaveDIP0024Quorums=...) at llmq/utils.cpp:942

run as dashd -regtest -llmqtestinstantsenddip0024=llmq_test_instantsend

@PastaPastaPasta
Copy link
Member

Guix Automation has began to build this PR tagged as v20.1.0-devpr5790.9002f39c. A new comment will be made when the image is pushed.

@knst knst removed the guix-build label Jan 11, 2024
@PastaPastaPasta
Copy link
Member

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v20.1.0-devpr5790.9002f39c. The image should be on dockerhub soon.

@PastaPastaPasta
Copy link
Member

Guix Automation has began to build this PR tagged as v20.1.0-devpr5790.98457039. A new comment will be made when the image is pushed.

@PastaPastaPasta
Copy link
Member

Guix Automation has completed; a release should be present here: https://github.com/dashpay/dash-dev-branches/releases/tag/v20.1.0-devpr5790.98457039. The image should be on dockerhub soon.

@github-actions
Copy link

This pull request has conflicts, please rebase.

knst added 15 commits January 12, 2024 19:20
Also this commit add llmq/utils in modules where it is included indirectly
but it requires to be used such as rpc/quorums
In the past implementation has been non-trivial:
```cpp
    return Params().GetConsensus().llmqs.at(llmqType);
```
But over time it was simplified and now does nothing but hide usage of
global variable Params():
```cpp
    return Params().GetLLMQ(llmqType);
```

Also it adds missing [[nodiscard]] for Params().GetLLMQ()

Eventually it caused some circular dependencies to disapear.
newly introduced circular dependencies are just longer pathes through
same modules that has been in past short-circuited
It seems as it is an artefact from non-deterministic IS.
It used only for LLMQ_TEST_INSTANTSEND quorum and doesn't look useful.

Removing this dependency helps to:
 - simplify implementation of many classes significantly so far as
   they don't need to pass quorumManager to low-level helpers
 - achieved significant progress of de-globalization quorumManager.
   Only one step left (evo/assertlocks, evo/creditpool, evo/mnhftx)
   and we can drop global variable ::quorumManager
 - disapeared 2 circular dependencies through llmq/quorum
@knst
Copy link
Collaborator Author

knst commented Jan 12, 2024

@knst knst force-pushed the refactor-llmq_utils branch from 9845703 to 45957d5

rebase to top of develop + resolved conflicts

@knst knst removed the guix-build label Jan 12, 2024
Copy link

@UdjinM6 UdjinM6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-utACK

Copy link
Member

@PastaPastaPasta PastaPastaPasta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK for squash merge

@PastaPastaPasta PastaPastaPasta merged commit 852adb5 into dashpay:develop Jan 18, 2024
@knst knst deleted the refactor-llmq_utils branch January 20, 2024 12:06
PastaPastaPasta pushed a commit that referenced this pull request Jan 28, 2024
## Issue being fixed or feature implemented
`develop` can't sync from genesis on mainnet,
b8a086d
broke it.

#5790 follow-up

## What was done?
Revive the old logic but using hardcoded block heights instead of
scanning via quorum manager.

## How Has This Been Tested?
Synced on mainnet/testnet, CI is happy
https://gitlab.com/UdjinM6/dash/-/pipelines/1148980046.

## Breaking Changes
n/a

## Checklist:
- [x] I have performed a self-review of my own code
- [x] I have commented my code, particularly in hard-to-understand areas
- [x] I have added or updated relevant unit/integration/functional/e2e
tests
- [ ] I have made corresponding changes to the documentation
- [x] I have assigned this pull request to a milestone _(for repository
code-owners and collaborators only)_
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants