Releases/v21 for tagging #2817
Merged
Releases/v21 for tagging #2817
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Bounded memory and redesign in the confirmation height processor * Disable frontiers confirmation for test, block doesn't exist during callback in fork resolution * Fix rpc.confirmation_height_currently_processing test * Cement blocks below receives not above * Fixes gcc build (hopefully) * Store start and end hash in pending write to remove extraneous IO * Optimise for the case where the top hash is 2 above cemented frontier * Fix TSAN issues * Serg comments * Use cached genesis_hash in CLI --confirmation_height_clear option * Set accounts_confirmed_info_size to 0 when clearing * Remove const for prepare_iterated_blocks_for_cementing as confusing
This changes the epoch byte in the sideband to also store these flags in the 3 most significant bits. These can be used to avoid grabbing the previous block to check its balance in some situations after ledger processing. The upgrade is done in-place, but takes a long time due to having to a random read to retrieve the previous block. A vacuum is done at the end. During the upgrade the ledger grows to almost 2x the ledger size, and up to 3x is required to vacuum as well. Due to using the most significant bits, the upgrade can safely be stopped and restarted (from the beginning), and no additional versioning was needed.
* Handle legacy confirm_req using the aggregator * Fix test node.local_votes_cache * Adjust test to make sure the cache is used * Use system polls instead
… with endpoint (#2521) * Poll all nodes and remove some metrics from bounds when consolidating * Update out of date comment * Formatting * Fix clang build on actions with long std::tuple * Allow square brackets in ipv6 address in RPC * Merge with develop * Fix ASAN issue * Gui review comments * Make parse_address accept v4 and v6 ip addresses * Incorrect order of arguments * Use new cached genesis hash * Move last_telemetry_req to bootstrap_server
* Extract stream.hpp from blocks.hpp * Add a probabilistic filter based on direct mapped caches, keyed by 128-bit SipHash. Co-Authored-By: Colin LeMahieu <clemahieu@gmail.com> The filter receives an array of bytes representing a network packet, and is checked for duplicity of the packet. The probability of a false duplicate is marginal, but not zero, and decreases with the size of the filter. The probability of a false non-duplicate is the infinitesimal probability of a 128-bit SipHash collision. Items are not normally erased. Instead, if a new item is different from the one at the insertion index (digest % capacity), the old item is replaced. There is also a function to erase an element from the filter, if the digest matches it. The filter state is protected by a mutex, whereas hashing is performed while not holding it. Uses 1MB of memory for every 64k elements. * Remove explicit instantiations * Remove alias * Optionally set digest in ::apply and add ::clear method directly from a digest
Updating relevant tests.
* Modifying update_difficulty test so it doesn't directly manipulate the election and instead relies on active_transactions updating itself on receipt of a higher difficulty block. * Using election attached to iterator. * Updating election fields inside election class. * Removing redundant code section.
…2548) * Add wallet-processed node to work watcher via the block processor * Remove active size check as the request loop may drop one before the check
Encapsulating election::confirmed so its implementation can vary.
* guard policy setting by version checks greater or equal short explainations of policies being set
* Restore max block processor signature verification batch size Removed in #2279 * Use assert to validate if else condition
* [ASAN] Access node through a weak_ptr on distributed_work dtor Since there is nothing to force a distributed_work to get destroyed before the node stops, this is a safer way to access the node on the destructor. Fixes some ASAN issues. * Add a comment explaining the use of node_w * [ASAN] fix issue in rpc.work_peer_many
* This changes the signature of active_transactions::start to match STL container ::insert signatures by returning a pointer to the newly created election and also a bool whether the election creation took place. This allows the call site to retrieve the election that was inserted and determine if it was the initiator of the election creation. Also updated tests that were taking extra steps to retrieve the newly created election. * Renaming active_transactions::start/add to insert/insert_impl to more closely match the behavior that's expected.
* Separate inactive votes cache from gap cache * Move functional changes to new PR
* Adding separate confirmation status / bootstrap status for inactive votes cache (it can be different with disabled lazy bootstrap using config bootstrap_fraction_numerator) * Apply reviews * Inactive vote cache item can be confirmed with single vote in tests * Add test to prevent confirmation without quorum
This client keeps connections alive and runs in a separate thread
Possibility to add or remove accounts in an existing subscription. This is useful for external wallets that can't use the all_local_accounts flag.
When insert_inactive_votes_cache confirms the election it deletes itself from the active roots, causing the next confirm_if_quorum call to access freed memory.
* Update --debug_profile_process CLI test * fix blocks signatures * support daemon launch options for profiling * use block count cache * Apply reviews changes
* Do not request UPnP for UDP when disabled * (unrelated) print endline on flag errors
Causing access of the counted_completion mutex after it goes out of scope.
… minimal (#2790) fix gcc dockerfile
- Incorrectly formed boost format - Work and difficulty not logged as hex
* Add ledger::backtrack limited to 128 jumps * Bisect election dependencies when activating * Addressing special case where the winner may not have a loaded sideband * Simplify ledger::backtrack by specifying desired number of jumps instead. Limit is placed on the caller * Perform DB transactions without holding the active mutex by batch processing * Add a debug_assert on getting the previous block since it happens in the same tx * Add a pessimistic fallback by also starting an election for the first unconfirmed block * Set empty sideband, required by recent change (thanks Serg!) * Check if first unconfirmed block is not being processed by conf height * Already in active_transactions * Use std::min<uint64_t> to avoid overflow (Wes review) * Use owns_lock from mutex (Wes)
@wezrule noticed a failure during a test related to a block not being found in the ledger when activating dependencies. There are in fact two issues: 1. In between adding the dependency and activating, the block could be rolled back, so we need to check if it exists 2. Due to potential simultaneous sideband changes in ledger processing, we need to retrieve a new copy of the block instead of simply checking if it exists Added `store::block_account_calculated` that is used when we already have a sideband-loaded block in memory.
To output a descriptive error message rather than: ``` terminate called after throwing an instance of 'boost::process::process_error' what(): execve failed: No such file or directory ```
cmake 3.13.0 added support for 1.69 borrowed findBoost.cmake and updated conditional for using it
* Some operations need to be processed after transaction commit, specifically when needing to process blocks that have just been inserted in to the ledger. Create the block_post_events class who's destructor will execute queued events. * There are multiple cases where we want to iterate over dependent blocks for a given block. This creates a uniform way to access block dependencies and re implements ledger::could_fit in terms of it. * Removing internal not-a-rep account and also logging vote's sequence number. * Holding reference to nano::ledger instead of nano::block_store so we can access more high-level methods. * Removing vector-of-hashes overload of vote_generator:add. * Triggering election confirmation directly for the setup portion of tests before votes are actually sent out. * Adding the function active_transactions::activate to activate an election for the first unconfirmed block on an account. It's possible we have a stale branch and won't be able to confirm the frontier so this gives us definite progress. * Adding can_vote function which checks block dependencies for confirmation. Sequentially voting for blocks decreases the state receivers need to maintain for dependent unconfirmed blocks. * Removing unused parameter and header * Use can_vote in all paths of request_aggregator New stat `requests_cannot_vote`, tests updated and a new test added * Fix tests affected by can_vote enforcement in request aggregator * Generate new election winner votes after processing * Fix intermittent failure in wallet.send_async This appears to be completely unrelated and only hitting on TSAN, likely a tiny change in thread timings * Re-enable test for windows CI * Small updates from reviews * Check block exists in Release builds too before cementing * Restrict generating votes on activation to prioritized elections And new elections already get votes generated in the constructor * Add activation support for legacy send * Add a debug_assert in ledger::is_send as it assumes the previous block exists * Save one disk read for blocks with a loaded sideband * Actually use the destination in activation (thanks Serg) A fix for c581d1a * Fix two intermittent failures in node.local_votes_cache 1. Due to the node wallet creating a block, leading to an unintended fork with the test block 2. Due to the delay between the vote being inserted in the store and in the cache * In replying to requests, check active elections for node winner before the ledger * Refactor request_aggregator::aggregate , getting rid of redundancy * Fix hanging test active_transactions.confirmation_consistency due to polling while holding a mutex * Activate the next block in an account only if both dependencies are confirmed * Add test node.dependency_graph ensuring account activation works accross more complex configurations * Reduce load_test to 5 nodes from 10 to avoid hanging * Async rpc requests in load_test Co-authored-by: Guilherme Lawless <guilherme@nano.org> Co-authored-by: Wesley Shillingford <wezrule@hotmail.com>
* Slow test fixes * Increase a deadline timer inside many_accounts_send_receive_self test * Inserting to wallet should be unnecessary * Fix node_telemetry.under_load test slowdown from sequential elections * Fix write_database_queue stopping bug (thanks Gui for investigation)
* Fix open blocks not getting activated through dependencies This was a regression introduced by #2778 Adds two tests to ensure proper behavior: - `election.dependencies_open_link` which covers the fix by itself - `dependency_graph_frontier`, a more complex test ensuring we can walk down a more complex graph before confirming it from the bottom up * Use transaction instead of read_transaction (Wes review)
Building off the multi-level election controls added in #2785, this PR changes the block processing behavior to only start an election if the block's ancestors are confirmed. - There is a superlinear usage of resources the more elections are active for the same chain, which is now prevented. - On confirming an election, the successors (in the same chain and destination chain for sends) are activated (functionality from #2785). - Frontier confirmation works as the fallback behavior, for bootstraps and nodes restarting after getting blocks. Backtracking (#2778) later starts the bottom-most election if unable to confirm the frontier.
* Adding epoch2_signing key.
DB was PRE_RELEASE 1-16
* Output ledger initialization message on start-up * Moving to node daemon and changing text (Gui review)
* shared boost for artifacts enable shared boost for windows tests * .deb support via cmake reorganized nix packages
zhyatt
approved these changes
Jun 16, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.