Multithreaded booking and new data flow with solidification of transa…

…ctions (#2151)

* Refactor: started to comment utils

* Feat: started refactoring models

* Feat: commented ChildBranch

* Fix: fixed bug with default options

* Refactor: refactored Conflict

* Refactor: removed unused code

* Feat: finished branchdag refactor

* Refactor: refactored some code

* Add multithreaded test for solidification and propagation (#2147)

* Feat: commented Ledger (models missing)

* Add tests for edge cases

* remove unused code

* Refactor: refactored code

* Add tests for childbranches when asserting branch DAG references

* Refactor: refactored code

* Fix: fixed some refactor bugs

* Fix: fixed refactor errors in devnetvm

* Feat: go mod tidy

* Feat: commented Outputs

* Feat: commented more code

* Feat: commented models

* Refactor: reviewed the code

* Refactor: changed more code

* Add comments to testframework

* Add locks to testframework

* Feat: added comments for properties in TestFramework

* Start implementing new tangle data flow

* Fix: fixed some random things

* Feat: started implementing error handling

* Fix: fixed some random stuff

* Feat: added comment

* Feat: added Shutdown calls

* Feat: added context to all final events Booked/Invalid

* change to improved data flow and embed payloadbooker into booker

* Rename ledgerstate to ledgerstateOLD in preparation to get rid of it. Add new ledger to Tangle

* WIP

* WIP

* fix more errors

* Feat: added UndefinedBranchID

* fix more errors

* Feat: refactored comment

* Feat: use EventLoop for gossip manager

* Feat: fixed some bugs

* Fix: fixed some random code

* fix more stuff

* Feat: more generic events

* Feat: started implementing indexer

* Feat: more generic events

* Feat: more generic events

* Feat: more generic events

* Feat: more generic events

* Feat: added an Indexer for the devnetvm

* Fix: fixed more random things

* Fix: fixed approvalweightmanager

* fix more and more stuff

* Feat: more generic events

* Fix: fixed more stuff

* Fix: fixed more bugs

* Feat: more generic events

* Feat: upgrade hive.go

* Fix: change workerpool in tangle_test

* Fix: fixed workerpool

* Feat: more generic events

* Feat: fixed wallet + client

* Fix: fixed some stuff in otv test

* Fix: fixed more stuff in otv.test

* Feat: more generic events

* Fix: remove replace in go.mod

* Fix: fixed otv_test

* Fix: fixed more stuff

* Fix: fixed more stuff

* Fix: fixed more ledgerstate references

* Fix: removed all references to ledgerstate package

* Feat: started adding snapshots

* Fix more errors

* Fix more stuff

* Fix more compile errors, only snapshot left now

* Feat: Implemented TransactionConfirmed event (#2168)

* Feat: Implemented TransactionConfirmed event

* Feat: started implementing mana snapshit

* Feat: node builds

* Refactor: simplified code by removing some wrappers

* Fix: fixed refactor related bugs

* Refactor: removed Transaction wrapping

* Fix: fixed broken mocks

* Update to latest hive.go version and fix out of order event error on startup

* Fix permissions in Docker file when mounting volume

* Feat: started reworking snapshot logic

* Feat: added OutputMetadata snapshot

* Feat: added more snapshot stuff

* Feat: fixed create snapshot tool

* Fix snapshit generation

* Fix uninitialized events

* Fix: fixed problem in booker

* Use eventloop instead of goroutines

* Add snapshot to Docker network and index outputs from snapshot when loading it

* Fix: fixed waiting in tests

* Feat: debug

* Fix: fixed some stuff

* Fix: removed unused code

* debug tests and stuff

* Feat: added debug outputs

* Fix one issue with out of order test

* Hook forking events

* Fix: fixed stuff

* Fix: fixed some stuff

* Feat: refactored solidifier

* Fix: fix bug

* Fix: booker_test TestScenario_1

* Fix: fixed test

* Fix: fixed test

* Fix: fixed tests

* Fix: fixed otv test

* Fix: fixed broken plugins

* Fix: fixed go.mod

* Fix: fixed some minor things

* Fix: fixed go.mod

* Fix: fixed broken test

* Feat: TSA remove past-cone checks and unconfirmed weak references

* Feat: upgraded to latest hive.go

* Feat: implemented commitment and proof systems for state commitments

* Fix: fixed order in booking

* tangle booker test

* Fix TestMultiThreadedBookingAndForking

* Fix: debug.Enabled race condition

* Fix: make tests more solid

* Feat: generic branchdag (#2204)

* Bump hive version

* Remove debugging prints

* Bump hive.go to fix serializer/v2 nested module

* Fix: TestMultiThreadedBookingAndForking test

* Refactor: renamed BranchDAG to ConflictDAG

* Refactor: started cleaning up stuff

* Fix: code warnings

* Feat: removed MasterBranch

* Feat: TestMultiThreadedBookingAndForkingNested

* Fix: BranchDAG -> ConflictDAG renaming

* Faster TestMultiThreadedBookingAndForkingNested

* Fix snapshot generation with new snapshotcreator tool

* Fix: fixed snapshit

* Feat: feed outputs to the messagelayer Indexer

* Fix: make sure ConflictIDs are empty sets

* Fix snapshot generation for integration tests

* Fix common integration tests

* Fix autopeering integration test

* Remove diagnostics integration tests

* Remove drng integration tests

* Fix: dashboard

* Fix: fixed some tests

* Fix: dashboard again

* Fix: fixed some stuff

* Fix: fixed test

* Fix: fixed stuff

* Fix: some unit tests

* Fix mana integration tests

* Fix faucet integration tests

* Feat: renamed branchdag

* Refactor: moved conflictdag package

* Refactor: refactored more conflictdag models

* Fix faucet integration tests

* Fix value integration tests

* Fix nil pointer in conflicts dashboard

* Fix reference creation for shallow dislike and TestConflictSpamAndMergeToMaster integration test

* Fix consensus integration tests

* Run integration tests for multithreaded tangle (#2217)

* Adjust runTests.sh

* Add missing context import

* Fix: dashboard

* Refactor: refactored code

* Fix: critical - return strongparent as-is if branch liked

* Refactor: cleaned up stuff

* Merge develop

* Revert "Fix: critical - return strongparent as-is if branch liked"

This reverts commit 74288b2.

* Fix issue where liked branch is not returned when LikedConflictMember is called with the liked branch of the conflict set (#2174)

* Add Feature network identities in snapshot creation tool

* WIP

* Make code compile

* Fix output unmarshallimg

* Fix: do not evaluate time difference if message approves Genesis

* Fix: Faucet properly attach async events

* Feat: cleaned up first model

* Adjust models in conflict dag

* Fix: fixed conflictdag

* Fix: fixed some more stuff

* fixed test

* Ledger tests pass

* Ledger tests and devnetvm tests pass again

* Fix indexer

* Fix objectstorage interface for some objects

* Use serix for model.Model serialization

* Fix more stuff

* Fix: fixed stuff

* Refactor: removed unnecessary helper

* Feat: cleaned up more code

* Refactor: changed to new models

* Fix more stuff

* Fix snapshit again

* Fix: fixed bug with models

* Bump hive.go

* Few fixes and prints

* Feat: added test for outputcommitment

* Add manual serialization to Ouput types to enable correct interface (de)serialization

* go mod tidy

* Fix value test hopefully

* Fix unsanitized user input print

* Fix: removed unused identifier

* Update CHANGELOG.md

* Update pkger

Co-authored-by: Hans Moog <3293976+hmoog@users.noreply.github.com>
Co-authored-by: Andrea V <1577639+karimodm@users.noreply.github.com>

.github/workflows/integration-tests.yml

@@ -156,55 +156,6 @@ jobs:
           name: ${{ env.TEST_NAME }}
           path: tools/integration-tests/logs
 
-  drng:
-    name: drng
-    env:
-      TEST_NAME: drng
-    runs-on: ubuntu-latest
-    steps:
-
-      - name: Checkout repository
-        uses: actions/checkout@v2
-
-      - name: Cache Go modules for tester container
-        uses: actions/cache@v2
-        with:
-          path: |
-            ~/go
-          # make sure concurrent runs (not really supported) do not match the same key but instead fall back to a reasonable cache
-          key: ${{ runner.os }}-it-go-${{ hashFiles('tools/integration-tests/tester/go.sum') }}-${{ github.job }}-${{ github.sha }}
-          restore-keys: |
-            ${{ runner.os }}-it-go-${{ hashFiles('tools/integration-tests/tester/go.sum') }}-
-            ${{ runner.os }}-it-go-
-
-      - name: Build GoShimmer image
-        run: |
-          docker build \
-            --build-arg DOWNLOAD_SNAPSHOT=0 \
-            -t iotaledger/goshimmer .
-
-      - name: Pull additional Docker images
-        run: |
-          docker pull angelocapossele/drand:v1.1.4
-
-      - name: Run integration tests
-        run: |
-          export CURRENT_UID=$(id -u):$(id -g)
-          docker-compose -f tools/integration-tests/tester/docker-compose.ci.yml up --abort-on-container-exit --exit-code-from tester
-
-      - name: Create logs from tester
-        if: always()
-        run: |
-          docker logs tester &> tools/integration-tests/logs/tester.log
-
-      - name: Save logs as artifacts
-        if: always()
-        uses: actions/upload-artifact@v1
-        with:
-          name: ${{ env.TEST_NAME }}
-          path: tools/integration-tests/logs
-
-
   value:
     name: value
     env:
@@ -349,51 +300,3 @@ jobs:
         with:
           name: ${{ env.TEST_NAME }}
           path: tools/integration-tests/logs
-
-  diagnostics:
-    name: diagnostics
-    env:
-      TEST_NAME: diagnostics
-    runs-on: ubuntu-latest
-    steps:
-
-      - name: Check out code
-        uses: actions/checkout@v2
-
-      - name: Cache Go modules for tester container
-        uses: actions/cache@v2
-        with:
-          path: |
-            ~/go
-          # make sure concurrent runs (not really supported) do not match the same key but instead fall back to a reasonable cache
-          key: ${{ runner.os }}-it-go-${{ hashFiles('tools/integration-tests/tester/go.sum') }}-${{ github.job }}-${{ github.sha }}
-          restore-keys: |
-            ${{ runner.os }}-it-go-${{ hashFiles('tools/integration-tests/tester/go.sum') }}-
-            ${{ runner.os }}-it-go-
-
-      - name: Build GoShimmer image
-        run: |
-          docker build \
-            --build-arg DOWNLOAD_SNAPSHOT=0 \
-            -t iotaledger/goshimmer .
-
-      - name: Pull additional Docker images
-        run: |
-          docker pull alpine/socat:1.7.4.3-r0
-
-      - name: Run integration tests
-        run: |
-          export CURRENT_UID=$(id -u):$(id -g)
-          docker-compose -f tools/integration-tests/tester/docker-compose.ci.yml up --abort-on-container-exit --exit-code-from tester
-
-      - name: Create logs from tester
-        if: always()
-        run: |
-          docker logs tester &> tools/integration-tests/logs/tester.log
-
-      - name: Save logs as artifacts
-        if: always()
-        uses: actions/upload-artifact@v1
-        with:
-          name: ${{ env.TEST_NAME }}
-          path: tools/integration-tests/logs

Dockerfile

@@ -135,7 +135,7 @@ FROM prepare-runtime as debugger-enabled-0
 
 ENTRYPOINT ["/app/goshimmer", "--config=/app/config.json"]
 
-# We execute this stage only if debugging is enabled, i.e REMOTE_DEBUGGIN==1
+# We execute this stage only if debugging is enabled, i.e REMOTE_DEBUGGING==1
 FROM prepare-runtime as debugger-enabled-1
 EXPOSE 40000
 

client/diagnostics.go

@@ -46,7 +46,7 @@ func (api *GoShimmerAPI) GetDiagnosticsMessagesByRank(rank uint64) (*csv.Reader,
 // GetDiagnosticsUtxoDag runs diagnostics over utxo dag.
 // Returns csv with the following fields:
 //
-//	ID,IssuanceTime,SolidTime,AccessManaPledgeID,ConsensusManaPledgeID,Inputs,Outputs,Attachments,
+//	ID,IssuanceTime,BookingTime,AccessManaPledgeID,ConsensusManaPledgeID,Inputs,Outputs,Attachments,
 //	BranchID,Conflicting,LazyBooked,GradeOfFinality,GradeOfFinalityTime
 func (api *GoShimmerAPI) GetDiagnosticsUtxoDag() (*csv.Reader, error) {
 	return api.diagnose(RouteDiagnosticsUtxoDag)
@@ -55,23 +55,23 @@ func (api *GoShimmerAPI) GetDiagnosticsUtxoDag() (*csv.Reader, error) {
 // GetDiagnosticsBranches runs diagnostics over branches.
 // Returns csv with the following fields:
 //
-//	ID,ConflictSet,IssuanceTime,SolidTime,LazyBooked,GradeOfFinality
+//	ID,ConflictSet,IssuanceTime,BookingTime,LazyBooked,GradeOfFinality
 func (api *GoShimmerAPI) GetDiagnosticsBranches() (*csv.Reader, error) {
 	return api.diagnose(RouteDiagnosticsBranches)
 }
 
 // GetDiagnosticsLazyBookedBranches runs diagnostics over lazy booked branches.
 // Returns csv with the following fields:
 //
-//	ID,ConflictSet,IssuanceTime,SolidTime,LazyBooked,GradeOfFinality
+//	ID,ConflictSet,IssuanceTime,BookingTime,LazyBooked,GradeOfFinality
 func (api *GoShimmerAPI) GetDiagnosticsLazyBookedBranches() (*csv.Reader, error) {
 	return api.diagnose(RouteDiagnosticsLazyBookedBranches)
 }
 
 // GetDiagnosticsInvalidBranches runs diagnostics over invalid branches.
 // Returns csv with the following fields:
 //
-//	ID,ConflictSet,IssuanceTime,SolidTime,LazyBooked,GradeOfFinality
+//	ID,ConflictSet,IssuanceTime,BookingTime,LazyBooked,GradeOfFinality
 func (api *GoShimmerAPI) GetDiagnosticsInvalidBranches() (*csv.Reader, error) {
 	return api.diagnose(RouteDiagnosticsInvalidBranches)
 }
@@ -84,7 +84,7 @@ func (api *GoShimmerAPI) GetDiagnosticsTips() (*csv.Reader, error) {
 // GetDiagnosticsDRNG runs diagnostics for DRNG
 // Returns csv with the following fields:
 //
-// 	ID,IssuerID,IssuerPublicKey,IssuanceTime,ArrivalTime,SolidTime,ScheduledTime,BookedTime,
+// 	ID,IssuerID,IssuerPublicKey,IssuanceTime,ArrivalTime,BookingTime,ScheduledTime,BookedTime,
 //	dRNGPayloadType,InstanceID,Round,PreviousSignature,Signature,DistributedPK
 func (api *GoShimmerAPI) GetDiagnosticsDRNG() (*csv.Reader, error) {
 	return api.diagnose(RouteDiagnosticsDRNG)

client/evilspammer/spammer.go

@@ -6,7 +6,9 @@ import (
 	"github.com/cockroachdb/errors"
 
 	"github.com/iotaledger/goshimmer/client/evilwallet"
-	"github.com/iotaledger/goshimmer/packages/ledgerstate"
+	"github.com/iotaledger/goshimmer/packages/ledger/utxo"
+	"github.com/iotaledger/goshimmer/packages/ledger/vm/devnetvm"
+
 	"github.com/iotaledger/hive.go/configuration"
 	"github.com/iotaledger/hive.go/logger"
 	"github.com/iotaledger/hive.go/types"
@@ -184,7 +186,7 @@ func (s *Spammer) StopSpamming() {
 
 // PostTransaction use provided client to issue a transaction. It chooses API method based on Spammer options. Counts errors,
 // counts transactions and provides debug logs.
-func (s *Spammer) PostTransaction(tx *ledgerstate.Transaction, clt evilwallet.Client) {
+func (s *Spammer) PostTransaction(tx *devnetvm.Transaction, clt evilwallet.Client) {
 	if tx == nil {
 		s.log.Debug(ErrTransactionIsNil)
 		s.ErrCounter.CountError(ErrTransactionIsNil)
@@ -197,7 +199,7 @@ func (s *Spammer) PostTransaction(tx *ledgerstate.Transaction, clt evilwallet.Cl
 	}
 
 	var err error
-	var txID ledgerstate.TransactionID
+	var txID utxo.TransactionID
 	txID, err = clt.PostTransaction(tx)
 	if err != nil {
 		s.log.Debug(ErrFailPostTransaction)
@@ -213,7 +215,7 @@ func (s *Spammer) PostTransaction(tx *ledgerstate.Transaction, clt evilwallet.Cl
 	return
 }
 
-func (s *Spammer) handleSolidityForReuseOutputs(clt evilwallet.Client, tx *ledgerstate.Transaction) (ok bool) {
+func (s *Spammer) handleSolidityForReuseOutputs(clt evilwallet.Client, tx *devnetvm.Transaction) (ok bool) {
 	ok = true
 	ok = s.EvilWallet.AwaitInputsSolidity(tx.Essence().Inputs(), clt)
 	if s.EvilScenario.OutputWallet.Type() == evilwallet.Reuse {

client/evilspammer/spamming_functions.go

@@ -5,8 +5,9 @@ import (
 	"sync"
 
 	"github.com/cockroachdb/errors"
+
 	"github.com/iotaledger/goshimmer/client/evilwallet"
-	"github.com/iotaledger/goshimmer/packages/ledgerstate"
+	"github.com/iotaledger/goshimmer/packages/ledger/vm/devnetvm"
 )
 
 func DataSpammingFunction(s *Spammer) {
@@ -41,7 +42,7 @@ func CustomConflictSpammingFunc(s *Spammer) {
 		wg := sync.WaitGroup{}
 		for i, tx := range txs {
 			wg.Add(1)
-			go func(clt evilwallet.Client, tx *ledgerstate.Transaction) {
+			go func(clt evilwallet.Client, tx *devnetvm.Transaction) {
 				defer wg.Done()
 				s.PostTransaction(tx, clt)
 			}(clients[i], tx)

client/evilwallet/aliasmanager.go

@@ -3,18 +3,18 @@ package evilwallet
 import (
 	"sync"
 
+	"github.com/cockroachdb/errors"
 	"go.uber.org/atomic"
 
-	"github.com/cockroachdb/errors"
-	"github.com/iotaledger/goshimmer/packages/ledgerstate"
+	"github.com/iotaledger/goshimmer/packages/ledger/vm/devnetvm"
 )
 
 // region AliasManager /////////////////////////////////////////////////////////////////////////////////////////////////
 
 // AliasManager is the manager for output aliases.
 type AliasManager struct {
-	outputMap map[string]ledgerstate.Output
-	inputMap  map[string]ledgerstate.Input
+	outputMap map[string]devnetvm.Output
+	inputMap  map[string]devnetvm.Input
 
 	outputAliasCount *atomic.Uint64
 	mu               sync.RWMutex
@@ -23,14 +23,14 @@ type AliasManager struct {
 // NewAliasManager creates and returns a new AliasManager.
 func NewAliasManager() *AliasManager {
 	return &AliasManager{
-		outputMap:        make(map[string]ledgerstate.Output),
-		inputMap:         make(map[string]ledgerstate.Input),
+		outputMap:        make(map[string]devnetvm.Output),
+		inputMap:         make(map[string]devnetvm.Input),
 		outputAliasCount: atomic.NewUint64(0),
 	}
 }
 
 // AddOutputAlias maps the given outputAliasName to output, if there's duplicate outputAliasName, it will be overwritten.
-func (a *AliasManager) AddOutputAlias(output ledgerstate.Output, aliasName string) {
+func (a *AliasManager) AddOutputAlias(output devnetvm.Output, aliasName string) {
 	a.mu.Lock()
 	defer a.mu.Unlock()
 
@@ -39,7 +39,7 @@ func (a *AliasManager) AddOutputAlias(output ledgerstate.Output, aliasName strin
 }
 
 // AddInputAlias adds an input alias.
-func (a *AliasManager) AddInputAlias(input ledgerstate.Input, aliasName string) {
+func (a *AliasManager) AddInputAlias(input devnetvm.Input, aliasName string) {
 	a.mu.Lock()
 	defer a.mu.Unlock()
 
@@ -48,15 +48,15 @@ func (a *AliasManager) AddInputAlias(input ledgerstate.Input, aliasName string)
 }
 
 // GetInput returns the input for the alias specified.
-func (a *AliasManager) GetInput(aliasName string) (ledgerstate.Input, bool) {
+func (a *AliasManager) GetInput(aliasName string) (devnetvm.Input, bool) {
 	a.mu.RLock()
 	defer a.mu.RUnlock()
 	in, ok := a.inputMap[aliasName]
 	return in, ok
 }
 
 // GetOutput returns the output for the alias specified.
-func (a *AliasManager) GetOutput(aliasName string) ledgerstate.Output {
+func (a *AliasManager) GetOutput(aliasName string) devnetvm.Output {
 	a.mu.RLock()
 	defer a.mu.RUnlock()
 
@@ -68,8 +68,8 @@ func (a *AliasManager) ClearAllAliases() {
 	a.mu.Lock()
 	defer a.mu.Unlock()
 
-	a.inputMap = make(map[string]ledgerstate.Input)
-	a.outputMap = make(map[string]ledgerstate.Output)
+	a.inputMap = make(map[string]devnetvm.Input)
+	a.outputMap = make(map[string]devnetvm.Output)
 }
 
 // ClearAliases clears provided aliases.
@@ -86,7 +86,7 @@ func (a *AliasManager) ClearAliases(aliases ScenarioAlias) {
 }
 
 // AddOutputAliases batch adds the outputs their respective aliases.
-func (a *AliasManager) AddOutputAliases(outputs []ledgerstate.Output, aliases []string) error {
+func (a *AliasManager) AddOutputAliases(outputs []devnetvm.Output, aliases []string) error {
 	if len(outputs) != len(aliases) {
 		return errors.New("mismatch outputs and aliases length")
 	}
@@ -102,7 +102,7 @@ func (a *AliasManager) AddInputAliases(inputs []*Output, aliases []string) error
 		return errors.New("mismatch outputs and aliases length")
 	}
 	for i, out := range inputs {
-		input := ledgerstate.NewUTXOInput(out.OutputID)
+		input := devnetvm.NewUTXOInput(out.OutputID)
 		a.AddInputAlias(input, aliases[i])
 	}
 	return nil