Mining Rule Engine #634

coderofstuff · 2025-02-12T06:42:13Z

Mining Rule Engine

This is the full implementation the of the mining rule engine discussed in https://research.kas.pa/t/sync-monitor-mining-rule-engine/294

Concepts

Sync Rate - observed_number_of_blocks_received_per_interval / expected_number_of_blocks_to_receive_per_interval
Mining Rule - some rule that affect how and when a node should mine based on the conditions of the node

Changes

Outline

Create a new Mining Rule Engine that currently tracks the sync rate of the node
Move all is_nearly_synced logic to the rule engine area
Introduce "Mining Rules" which are checked every 10s and depending on the conditions of the node triggers rules to be applied.

Mining Rule Engine

1. Implements a BlueOnlyParents strategy

Trigger: virtual processing time average is above 500ms
Recovery:
- virtual processing time is back below 100ms AND
- a merge bound amount of blocks has passed since the trigger

2. Implements a NoTransactions (empty blocks) strategy

Trigger: BadMerkleRoot count > valid block count in the interval
Recovery: 2 intervals have passed since the rule was triggered

3. Implements a SyncRateRule strategy

This is the implementation of the Sync Monitor discussed in the thread and doc above.

Create a new Mining Rule Engine that currently tracks the sync rate of the node
Introduce the Sync Rate Rule - allow mining if the sync rate of the node is lower than the sync rate threshold and the finality point is recent, even if the node is not "nearly synced"
Move all is_nearly_synced logic to the rule engine area

- monitor sync state and update anything that cares about is_sycned to use the updated rules - sync rate rule: allow mining if sync rate is below threshold and finality point is recent Move most is_nearly_synced logic to mining_rule_engine - instantiate Hub in daemon - Reverse the dependency of flow context and MRE - remove the async_is_nearly_synced consensus api - add get_sink_daa_score_timestamp consensus API Move is_nearly_synced logic from params to rule engine

- Overrides TX selection to instead automatically return an empty vec

- Includes only the logic that uses the rule - Make all reds "bad reds" when selected parents - Select 100% top blue work parents - Trigger logic not present yet

- Add BlueParentsOnly Rule - Add stub for NoTransactionsRule

- Makes it look like any other mining rule

michaelsutton · 2025-03-12T13:07:39Z

virtual processing time does not cover things such as calc_block_parents which are only called as part of build_block_template_from_virtual_state. Perhaps that method should be added to the same timing counter.

In general I also think that virtual processing time needs to be normalized by the amount of UTXO-validated blocks during the resolve. This can be solved by changing self.counters.virtual_resolve_counts.fetch_add(1, Ordering::Relaxed) to fetch add such a count (and renaming the field). See chain_block_counts for a reference of such counting.

Other option (which I prefer): ignore UTXO validation and time only the parts of resolve virtual which prep for block template building: from before pick_virtual_parents and until after calculate_and_commit_virtual_state. This segment includes all sensitive header composing areas and, added with the time of build_block_template_from_virtual_state, should suffice for our cause.

One more comment re blue parents only rule: once triggered, this rule remains for a long period. So it seems wise to activate it only after a few consecutive slow rounds and not only after 10 seconds (or extend the monitor interval to 30 secs?)

michaelsutton · 2025-03-12T13:12:17Z

The general structuring of the rules and the rule engine in terms of code design and style is top-notch btw

consensus/src/pipeline/body_processor/processor.rs

protocol/mining/src/rules/no_transactions_rule.rs

michaelsutton · 2025-03-12T14:03:18Z

protocol/mining/src/rule_engine.rs

+            // enter the DAA window of fully-synced nodes and thus contribute to overall network difficulty
+            //
+            // [Crescendo]: both durations are nearly equal so this decision is negligible
+            unix_now()


This duration is too long for 10BPS (~44 minutes = 26,400 block depth). To me, part of the motivation for identifying slow sync rate was to allow the definition of nearly synced to be tightened (as in testnets below). It's safer now bcs the network will still mine on a slowdown due to the new rule even if nearly synced = false.

Should we set it to as tight a bound as the one we have for testnet below? So, change this fn to behave the same way in all networks?

If it used to be 44 minutes at 1bps, something like 4.4mins at 10bps is an overwhelming amount of time for blocks not to pass. Perhaps even tighter?

If we consider 1min as the nearly synced bound, what are the consequences of that?

consensus/src/pipeline/body_processor/processor.rs

someone235 · 2025-03-16T09:48:21Z

consensus/src/pipeline/virtual_processor/processor.rs

@@ -798,25 +817,32 @@ impl VirtualStateProcessor {
        let mut ghostdag_data = self.ghostdag_manager.ghostdag(&virtual_parents);
        let merge_depth_root = self.depth_manager.calc_merge_depth_root(&ghostdag_data, current_pruning_point);
        let mut kosherizing_blues: Option<Vec<Hash>> = None;
-        let mut bad_reds = Vec::new();
+        let bad_reds = if self.mining_rules.blue_parents_only.load(Ordering::Relaxed) {


nit, but I think it'll be nicer to have some wrapper methods for MiningRules around the atomic operations (and then we should probably make the members private)

protocol/mining/src/rules/blue_parents_only_rule.rs

someone235 · 2025-03-16T13:01:51Z

protocol/mining/src/rules/blue_parents_only_rule.rs

+/// Attempt to recover from high virtual processing times possibly caused merging red blocks.
+/// by only pointing to blue parents.
+///
+/// Trigger: virtual processing average time is above threshold


I think slowness in build_block_template should trigger it. @hashdag and I also suggested slowness in header validation to trigger it

protocol/mining/src/rules/sync_rate_rule.rs

rpc/service/src/service.rs

Trigger condition: - if submit_block_bad_merkle_root_count > 0 && submit_block_success_count == 0 (no submit block calls rpcs succeeded in this 10s period) Recovery conditions: - if submit_block_success_count > 0 (any submit block call succeeded, even if there are any that errored out), OR - two cooldown periods have passed

coderofstuff · 2025-03-20T02:35:52Z

Closing this as it has been split in #654 #655 #656

coderofstuff mentioned this pull request Feb 12, 2025

Mining Rule Engine - Sync Monitor component #628

Closed

coderofstuff marked this pull request as ready for review February 14, 2025 06:45

coderofstuff requested review from michaelsutton and someone235 February 14, 2025 06:46

coderofstuff added 7 commits March 11, 2025 22:58

Add MiningRules and wire it up

75d96ce

Apply NoTransactions rule

dbdd9be

- Overrides TX selection to instead automatically return an empty vec

Apply BlueParentsOnly rule

307b870

- Includes only the logic that uses the rule - Make all reds "bad reds" when selected parents - Select 100% top blue work parents - Trigger logic not present yet

Implement MiningRule trait in a submodule

3edcf8b

- Add BlueParentsOnly Rule - Add stub for NoTransactionsRule

Refactor Sync Rate Rule

527035f

- Makes it look like any other mining rule

Implement trigger and recovery for NoTransactionsRule

74284ee

coderofstuff force-pushed the rule-engine branch from 88080d3 to b43e97a Compare March 12, 2025 05:03

coderofstuff added 3 commits March 11, 2025 23:08

Add merge depth check for BlueParentsOnlyRule recovery

de7d1ad

Add documentation for rules

c9e8302

Use virtual resolve count to calc processing average

b59e332

coderofstuff force-pushed the rule-engine branch from b43e97a to b59e332 Compare March 12, 2025 05:08

michaelsutton requested changes Mar 12, 2025

View reviewed changes

consensus/src/pipeline/body_processor/processor.rs Outdated Show resolved Hide resolved

protocol/mining/src/rules/no_transactions_rule.rs Outdated Show resolved Hide resolved

michaelsutton reviewed Mar 12, 2025

View reviewed changes

someone235 requested changes Mar 16, 2025

View reviewed changes

coderofstuff added 4 commits March 16, 2025 17:29

Merge branch 'master' into rule-engine

53269c5

BlueParentsOnlyRule comment fixes

ea3047a

Windowed sampling for sync rate rule

a087772

coderofstuff force-pushed the rule-engine branch from e0510cc to c3c950f Compare March 18, 2025 06:08

BlueParentOnlyRule: focus trigger and recovery on build block template

de10c18

This was referenced Mar 19, 2025

Mining Rule Engine Scaffolding and Sync Rate Rule implementation #654

Merged

Mining Rule Engine: No Transactions Rule #655

Open

Mining Rule Engine: Blue Only Mergeset Rule #656

Draft

coderofstuff closed this Mar 20, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mining Rule Engine #634

Mining Rule Engine #634

coderofstuff commented Feb 12, 2025 •

edited

Loading

michaelsutton commented Mar 12, 2025

michaelsutton commented Mar 12, 2025

michaelsutton Mar 12, 2025

coderofstuff Mar 13, 2025

someone235 Mar 16, 2025

someone235 Mar 16, 2025

coderofstuff commented Mar 20, 2025

Mining Rule Engine #634

Mining Rule Engine #634

Conversation

coderofstuff commented Feb 12, 2025 • edited Loading

Mining Rule Engine

Concepts

Changes

Outline

Mining Rule Engine

1. Implements a BlueOnlyParents strategy

2. Implements a NoTransactions (empty blocks) strategy

3. Implements a SyncRateRule strategy

michaelsutton commented Mar 12, 2025

michaelsutton commented Mar 12, 2025

michaelsutton Mar 12, 2025

Choose a reason for hiding this comment

coderofstuff Mar 13, 2025

Choose a reason for hiding this comment

someone235 Mar 16, 2025

Choose a reason for hiding this comment

someone235 Mar 16, 2025

Choose a reason for hiding this comment

coderofstuff commented Mar 20, 2025

coderofstuff commented Feb 12, 2025 •

edited

Loading