Feature: New milestone stage and solidification pipeline #1844

DyrellC · 2020-04-29T17:14:09Z

Description of change

Merges the new milestone solidification logic. This includes the addition of the MilestoneStage in the TransactionProcessingPipeline for handling transactions immediately on receiving them, as well as the consolidation of the LatestMilestoneTracker, MilestoneSolidifier and LatestSolidMilestoneTracker into one efficient MilesotneSolidifier class. These changes streamline the milestone solidification process and no longer requires continuous pulling of transactions from the db, reducing the memory consumption and general CPU usage of the node.

Type of change

Enhancement (a non-breaking change which adds functionality)

How the change has been tested

These changes have been put through every test at our disposal:

Synchronisation tests
Internal network testing
Regression testing
Jmeter bench-marking
Spam load testing

Change checklist

My code follows the contribution guidelines for this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes

* Move validation to validation service package * Add Transaction Solidifier * Update Transaction Validator * Update Unit Tests * Remove tip field from solidify stage * Move broadcast queue retreival to solidify stage * Undo auto-formatting * More autoformatting * Re-remove refillBroadcast * Move propagation logic to inner class * Remove unused imports * Add comment to propagator * Remove unused txSolidifier private field * Remove separate thread logic, check solidity from milestone solidifier * Add solidification queue to transaction solidifier * LinkedHashSet -> ArrayDeque in checkSolidity * Update maxProcessedTransactions value determination * Make Transaction Propagator private * Add milestone stage * add heartbeat message (#1825) * add heartbeat message * added javadocs * print exception message * Regression: Update pruning test (#1775) * Update pruning test for before and after pruning status * m6 -> m1 Co-authored-by: Gal Rogozinski <gal.rogozinski@iota.org> * Fix: Update Broadcast Stage logic (#1827) * Update broadcast fetch logic * Comment * Feature: Add transaction solidifier queue (#1821) * Move validation to validation service package * Add Transaction Solidifier * Update Transaction Validator * Update Unit Tests * Remove tip field from solidify stage * Move broadcast queue retreival to solidify stage * Undo auto-formatting * More autoformatting * Re-remove refillBroadcast * Move propagation logic to inner class * Remove unused imports * Add comment to propagator * Remove unused txSolidifier private field * Remove separate thread logic, check solidity from milestone solidifier * Add solidification queue to transaction solidifier * LinkedHashSet -> ArrayDeque in checkSolidity * Update maxProcessedTransactions value determination * Make Transaction Propagator private * Fix test conflicts * Update src/main/java/com/iota/iri/service/validation/impl/TransactionSolidifierImpl.java Co-Authored-By: Gal Rogozinski <galrogogit@gmail.com> * Update src/main/java/com/iota/iri/service/validation/impl/TransactionSolidifierImpl.java Co-Authored-By: Gal Rogozinski <galrogogit@gmail.com> * Rename Co-authored-by: Gal Rogozinski <gal.rogozinski@iota.org> Co-authored-by: Gal Rogozinski <galrogogit@gmail.com> * Unit test for milestone stage Co-authored-by: Gal Rogozinski <gal.rogozinski@iota.org> Co-authored-by: Acha Bill <57879913+acha-bill@users.noreply.github.com> Co-authored-by: Gal Rogozinski <galrogogit@gmail.com>

* Move validation to validation service package * Add Transaction Solidifier * Update Transaction Validator * Update Unit Tests * Remove tip field from solidify stage * Move broadcast queue retreival to solidify stage * Undo auto-formatting * More autoformatting * Re-remove refillBroadcast * Move propagation logic to inner class * Remove unused imports * Add comment to propagator * Remove unused txSolidifier private field * Remove separate thread logic, check solidity from milestone solidifier * Add solidification queue to transaction solidifier * LinkedHashSet -> ArrayDeque in checkSolidity * Update maxProcessedTransactions value determination * Make Transaction Propagator private * Add milestone stage * add heartbeat message (#1825) * add heartbeat message * added javadocs * print exception message * Regression: Update pruning test (#1775) * Update pruning test for before and after pruning status * m6 -> m1 Co-authored-by: Gal Rogozinski <gal.rogozinski@iota.org> * Fix: Update Broadcast Stage logic (#1827) * Update broadcast fetch logic * Comment * Feature: Add transaction solidifier queue (#1821) * Move validation to validation service package * Add Transaction Solidifier * Update Transaction Validator * Update Unit Tests * Remove tip field from solidify stage * Move broadcast queue retreival to solidify stage * Undo auto-formatting * More autoformatting * Re-remove refillBroadcast * Move propagation logic to inner class * Remove unused imports * Add comment to propagator * Remove unused txSolidifier private field * Remove separate thread logic, check solidity from milestone solidifier * Add solidification queue to transaction solidifier * LinkedHashSet -> ArrayDeque in checkSolidity * Update maxProcessedTransactions value determination * Make Transaction Propagator private * Fix test conflicts * Update src/main/java/com/iota/iri/service/validation/impl/TransactionSolidifierImpl.java Co-Authored-By: Gal Rogozinski <galrogogit@gmail.com> * Update src/main/java/com/iota/iri/service/validation/impl/TransactionSolidifierImpl.java Co-Authored-By: Gal Rogozinski <galrogogit@gmail.com> * Rename Co-authored-by: Gal Rogozinski <gal.rogozinski@iota.org> Co-authored-by: Gal Rogozinski <galrogogit@gmail.com> * Unit test for milestone stage * Consolidate milestone solidification logic * Update milestone stage tests * update bootstrap * Minor refactor * Log correct milestones, add switch for processing solidification queue * Bootstrap milestone hash and index properly * set index and hash directly instead of through method * Place all milestone tails into solidifier from milestone stage * setLatestMilestone -> logNewMilestone * remove second LSM update in update milestones * Stop repeats and solid entry points in broadcast queue * Don't abort milestone stage for second transaction in milestone bundle * Shorten regression wait times * Clean up, syntax updates, comment clarification * Remove unecessary method, syntax * Atomic variables and syntax changes * Update scanMilestones in queue * Extract repair logic * Update src/main/java/com/iota/iri/service/milestone/MilestoneSolidifier.java Co-Authored-By: Gal Rogozinski <galrogogit@gmail.com> * Typo, check solidity in milestone stage * syntax and minor logic change * Rename logNewMilestone * log -> register in milestone stage * Update milestone payload to pass forward transaction, fix tests Co-authored-by: Gal Rogozinski <gal.rogozinski@iota.org> Co-authored-by: Acha Bill <57879913+acha-bill@users.noreply.github.com> Co-authored-by: Gal Rogozinski <galrogogit@gmail.com>

* Typo, if to while loop * Don't register new milestone unless a milestone makes it through validation in the solidifier * Add logs for bootstrap and thread interrupt, thread.sleep(), remove milestones below solid milestone * new line * Remove logging check from test * Set latest milestone on incomplete milestone objects * Only log higher milestones, set oldestMilestoneInQueue to null in scans

acha-bill

Just a few comments.

acha-bill · 2020-04-30T12:16:32Z

src/main/java/com/iota/iri/network/pipeline/MilestoneStage.java

+
+            TransactionViewModel milestone = payload.getMilestoneTransaction();
+            int newMilestoneIndex = payload.getMilestoneIndex();
+            boolean isTail = (milestone.getCurrentIndex() == 0);


Are the braces necessary?

I suppose not huh?

acha-bill · 2020-04-30T12:17:38Z

src/main/java/com/iota/iri/network/pipeline/MilestoneStage.java

+                milestoneSolidifier.addMilestoneCandidate(milestone.getHash(), newMilestoneIndex);
+            }
+
+            if (milestone.isSolid()) {


The second part of the comment above should be here? No?
if transaction is solid, add to the propagation queue
Besides, I don't feel strongly that we need these comments.

acha-bill · 2020-04-30T12:19:49Z

src/main/java/com/iota/iri/network/pipeline/ReceivedStage.java

     * @param tangle           The {@link Tangle} database used to store/update the transaction
-     * @param txSolidifier      The {@link TransactionSolidifier} used to store/update the transaction
+     * @param txSolidifier      The {@link TransactionValidator} used to store/update the transaction


Must have happened during the conflict resolution while merging dev in, i'll touch that up.

acha-bill · 2020-04-30T12:23:46Z

src/main/java/com/iota/iri/service/milestone/MilestoneRepairer.java

+
+import com.iota.iri.controllers.MilestoneViewModel;
+
+public interface MilestoneRepairer {


acha-bill · 2020-04-30T12:24:50Z

src/main/java/com/iota/iri/service/milestone/MilestoneSolidifier.java

@@ -14,7 +18,7 @@
     * @param milestoneHash Hash of the milestone that shall be solidified
     * @param milestoneIndex index of the milestone that shall be solidified
     */
-    void add(Hash milestoneHash, int milestoneIndex);
+    void addMilestoneCandidate(Hash milestoneHash, int milestoneIndex);


acha-bill · 2020-04-30T12:30:08Z

src/test/java/com/iota/iri/network/pipeline/TransactionProcessingPipelineTest.java

@@ -180,11 +194,68 @@ public void processingAValidNewTransactionFlowsThroughTheEntirePipeline() throws
        Mockito.verify(broadcastStage).process(Mockito.any());
    }

+    @Test
+    public void processingAValidMilestone() throws InterruptedException {


DyrellC and others added 4 commits April 21, 2020 14:14

Merge branch 'dev' into Dyrell-Milestone-Stage-Draft

63a78c7

DyrellC requested a review from GalRogozinski April 29, 2020 17:20

acha-bill reviewed Apr 30, 2020

View reviewed changes

Fix: max propagate up to 10 transactions for solidification (#1845)

bf34410

GalRogozinski approved these changes Apr 30, 2020

View reviewed changes

GalRogozinski changed the title ~~Merge new milestone logic~~ Feature: New milestone stage and solidification pipeline Apr 30, 2020

GalRogozinski merged commit adb57b9 into dev Apr 30, 2020

GalRogozinski mentioned this pull request May 6, 2020

Release 1.8.6 #1851

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: New milestone stage and solidification pipeline #1844

Feature: New milestone stage and solidification pipeline #1844

DyrellC commented Apr 29, 2020

acha-bill left a comment

acha-bill Apr 30, 2020

DyrellC Apr 30, 2020

acha-bill Apr 30, 2020

acha-bill Apr 30, 2020

DyrellC Apr 30, 2020

acha-bill Apr 30, 2020

acha-bill Apr 30, 2020

acha-bill Apr 30, 2020


		import com.iota.iri.controllers.MilestoneViewModel;

		public interface MilestoneRepairer {

Feature: New milestone stage and solidification pipeline #1844

Feature: New milestone stage and solidification pipeline #1844

Conversation

DyrellC commented Apr 29, 2020

Description of change

Type of change

How the change has been tested

Change checklist

acha-bill left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment