Add Plugin for UTXO tracking by account. #52

jsolman · 2019-03-11T20:11:02Z

This plugin facilitates sending NEO or GAS for arbitrary addresses by providing an RPC method getunspents for retrieving the unspent UTXOs. The unspent UTXOs are stored indexed by account script hash in an additional LevelDB database whose path is configurable in the plugin settings. This plugin was migrated from my initial PR here: neo-project/neo/pull/569

The format of the response data is compatible with the format returned from the neoscan v1 API get_balance` call.

jsolman · 2019-03-11T20:12:58Z

I'll be testing this by re-syncing MainNet from the latest chain file and will report here with the results from my testing.

vncoelho · 2019-03-11T20:20:12Z

Building and testing it now.

vncoelho · 2019-03-11T20:25:53Z

I got this error but need double check, maybe it is my configuration.

[ERROR][03/11/2019 20:24:58][Thread 0004][akka://NeoSystem/user/$a] Error while creating actor instance of type Neo.Ledger.Blockchain with 2 args: (Neo.NeoSystem,Neo.Persistence.LevelDB.LevelDBStore)
Cause: [akka://NeoSystem/user/$a#360886005]: Akka.Actor.ActorInitializationException: Exception during creation ---> System.TypeLoadException: Error while creating actor instance of type Neo.Ledger.Blockchain with 2 args: (Neo.NeoSystem,Neo.Persistence.LevelDB.LevelDBStore) ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.AggregateException: One or more errors occurred. (Object reference not set to an instance of an object.) ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Neo.Plugins.RpcSystemAssetPlugin.OnCommit(Snapshot snapshot)
   at Neo.Ledger.Blockchain.Persist(Block block)
   --- End of inner exception stack trace ---
   at Neo.Ledger.Blockchain.Persist(Block block)
   at Neo.Ledger.Blockchain..ctor(NeoSystem system, Store store)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at Akka.Actor.Props.ActivatorProducer.Produce()
   at Akka.Actor.Props.NewActor()
   --- End of inner exception stack trace ---
   at Akka.Actor.Props.NewActor()
   at Akka.Actor.ActorCell.CreateNewActorInstance()
   at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<NewActor>b__0()
   at Akka.Actor.ActorCell.UseThreadContext(Action action)
   at Akka.Actor.ActorCell.NewActor()
   at Akka.Actor.ActorCell.Create(Exception failure)
   --- End of inner exception stack trace ---
   at Akka.Actor.ActorCell.Create(Exception failure)
   at Akka.Actor.ActorCell.SysMsgInvokeAll(EarliestFirstSystemMessageList messages, Int32 currentState)

EDIT: I just tested removing the plugin .ddl and error disappeared.

jsolman · 2019-03-11T20:38:50Z

@vncoelho Maybe it is trying to persist the genesis block before the plugin is loaded. I'll check into it.

~~Actually, must be something else.~~

vncoelho

Maybe the folder can be just RpcSystemAsset.

vncoelho · 2019-03-11T20:41:19Z

Sorry for not checking it in detail, @jsolman. I will also try to check it more tomorrow.

jsolman · 2019-03-11T20:42:04Z

Maybe the folder can be just RpcSystemAsset.

Maybe RpcSystemAssetTracker

vncoelho

[ERROR][03/11/2019 20:42:34][Thread 0006][akka://NeoSystem/user/$a] Error while creating actor instance of type Neo.Ledger.Blockchain with 2 args: (Neo.NeoSystem,Neo.Persistence.LevelDB.LevelDBStore)
Cause: [akka://NeoSystem/user/$a#123821120]: Akka.Actor.ActorInitializationException: Exception during creation ---> System.TypeLoadException: Error while creating actor instance of type Neo.Ledger.Blockchain with 2 args: (Neo.NeoSystem,Neo.Persistence.LevelDB.LevelDBStore) ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: Collection was modified; enumeration operation may not execute.
   at System.Collections.Generic.List`1.Enumerator.MoveNextRare()
   at Neo.Ledger.Blockchain.Persist(Block block)
   at Neo.Ledger.Blockchain..ctor(NeoSystem system, Store store)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at Akka.Actor.Props.ActivatorProducer.Produce()
   at Akka.Actor.Props.NewActor()
   --- End of inner exception stack trace ---
   at Akka.Actor.Props.NewActor()
   at Akka.Actor.ActorCell.CreateNewActorInstance()
   at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<NewActor>b__0()
   at Akka.Actor.ActorCell.UseThreadContext(Action action)
   at Akka.Actor.ActorCell.NewActor()
   at Akka.Actor.ActorCell.Create(Exception failure)
   --- End of inner exception stack trace ---
   at Akka.Actor.ActorCell.Create(Exception failure)
   at Akka.Actor.ActorCell.SysMsgInvokeAll(EarliestFirstSystemMessageList messages, Int32 currentState)

This error is a little different, maybe this can help to track.

vncoelho · 2019-03-11T21:07:50Z

@jsolman, it works for scripthash, right? Not base58?

vncoelho · 2019-03-11T21:17:45Z

[ERROR][03/11/2019 21:16:49][Thread 0003][akka://NeoSystem/user/$a] Error while creating actor instance of type Neo.Ledger.Blockchain with 2 args: (Neo.NeoSystem,Neo.Persistence.LevelDB.LevelDBStore)
Cause: [akka://NeoSystem/user/$a#1123270816]: Akka.Actor.ActorInitializationException: Exception during creation ---> System.TypeLoadException: Error while creating actor instance of type Neo.Ledger.Blockchain with 2 args: (Neo.NeoSystem,Neo.Persistence.LevelDB.LevelDBStore) ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Neo.Plugins.RpcSystemAssetTrackerPlugin.ResetBatch()
   at Neo.Plugins.RpcSystemAssetTrackerPlugin.OnPersist(Snapshot snapshot, IReadOnlyList`1 applicationExecutedList)
   at Neo.Ledger.Blockchain.Persist(Block block)
   at Neo.Ledger.Blockchain..ctor(NeoSystem system, Store store)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at Akka.Actor.Props.ActivatorProducer.Produce()
   at Akka.Actor.Props.NewActor()
   --- End of inner exception stack trace ---
   at Akka.Actor.Props.NewActor()
   at Akka.Actor.ActorCell.CreateNewActorInstance()
   at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<NewActor>b__0()
   at Akka.Actor.ActorCell.UseThreadContext(Action action)
   at Akka.Actor.ActorCell.NewActor()
   at Akka.Actor.ActorCell.Create(Exception failure)
   --- End of inner exception stack trace ---
   at Akka.Actor.ActorCell.Create(Exception failure)
   at Akka.Actor.ActorCell.SysMsgInvokeAll(EarliestFirstSystemMessageList messages, Int32 currentState)

This log is more precise now, @jsolman

RpcSystemAssetTracker/UserUnspentCoinOutputsKey.cs

RpcSystemAssetTracker/RpcSystemAssetTrackerPlugin.cs

jsolman · 2019-03-11T21:51:16Z

@jsolman, it works for scripthash, right? Not base58?

I'll make it work for both.

jsolman · 2019-03-11T22:17:20Z

I need to change the plugins so they can override a method that will be called when all plugins have finished loading (ImportBlocks needs to start its work after all plugins are loaded for example). Also, the genesis block needs to not be persisted until plugins are loaded.

jsolman · 2019-03-11T23:45:46Z

Blocks are importing now and I'll report back on how testing goes.
This change is required: neo-project/neo/pull/633

jsolman · 2019-03-12T16:41:26Z

Testing after syncing with MainNet looks good. Keep in mind that to use this you need:

To be using Neo 2.10.1 (currently master at or above commit neo-project/neo@432a3f6)
The updated ImportBlocks plugin: Import blocks after plugins are loaded. #53

Also, to be safe currently, do not stop the import blocks process when importing from chain file by using the exit command or control+c. I am going to improve shutdown to capture control+c and shutdown cleanly, but currently exit will hang while importing blocks, and interrupting could possibly stop after a block had been committed to the main LevelDB database and before the UTXOs are committed to this LevelDB database, leaving them in an inconsistent state.

As an improvement, I could likely have it persist the block number in LevelDB when this plugin commits so that it can check if it didn't persist the latest block on start-up and handle it that way.

RpcSystemAssetTracker/RpcSystemAssetTrackerPlugin.cs

jsolman · 2019-03-13T05:56:10Z

I have not done any testing yet after adding the last commit that adds the RPC method to get the claimable spent transactions. I will test it tomorrow and also likely add a method for getting the total unclaimed gas available for an address to fill the last gap.

jsolman · 2019-03-13T17:16:07Z

The API for getting all the claimable transactions is working as expected from my testing with MainNet. Now all that remains is implementing an operation to get total claimable amount, and it should be done.

jsolman · 2019-03-13T20:23:37Z

Ok, support for all operations needed for claiming GAS now are in and seem to be working as expected.

vncoelho

I also tested here and it is working brilliantly. If there are some errors they would be minors and we can fix in the next steps.
Thanks and congratulation for this game changing PR and dedication, @jsolman.

jsolman · 2019-03-13T20:47:33Z

@vncoelho
Maybe we can create a 2.10.1 branch as you suggested elsewhere; I will not merge this as it references a CI NuGet build.

CI NuGet reference

Add Plugin for UTXO tracking by account.

96f6bab

jsolman requested a review from erikzhang March 11, 2019 20:11

jsolman mentioned this pull request Mar 11, 2019

Add UTXO Tracking by Account neo-project/neo#569

Closed

vncoelho reviewed Mar 11, 2019

View reviewed changes

Rename RpcSystemAssetPlugin -> RpcSystemAssetTracker.

0a80ceb

vncoelho reviewed Mar 11, 2019

View reviewed changes

RpcSystemAssetTracker/UserUnspentCoinOutputsKey.cs Outdated Show resolved Hide resolved

RpcSystemAssetTracker/RpcSystemAssetTrackerPlugin.cs Outdated Show resolved Hide resolved

Accept address or script hash.

87aea4f

Remove unnecessary white space.

fc297be

jsolman mentioned this pull request Mar 11, 2019

Load plugins before initializing the system and call OnPluginsLoaded afterward. neo-project/neo#633

Merged

jsolman mentioned this pull request Mar 12, 2019

Fix application logs after adding EventStream usage and OnPluginsLoaded. #55

Merged

jsolman and others added 2 commits March 11, 2019 21:34

Merge branch 'master' into RpcSystemAssetPlugin

0f12ea0

Update project ref to Neo 2.10.1-CI00002.

1b9870a

jsolman added 5 commits March 12, 2019 12:35

Ensure consistency in case of crash or kill.

44d358a

Fix setting lastPersistedBlock after each block is persisted.

f21450a

Performance optimization: Don't write block number for empty blocks.

c7e0922

Optimize.

0777eee

Add support for tracking unclaimed spent outputs.

ab8a4d7

jsolman commented Mar 12, 2019

View reviewed changes

RpcSystemAssetTracker/RpcSystemAssetTrackerPlugin.cs Outdated Show resolved Hide resolved

Provide getclaimable RPC call for getting available claim spent outputs.

1544902

Merge branch 'master' into RpcSystemAssetPlugin

d206f94

vncoelho mentioned this pull request Mar 13, 2019

[feature] support for neo-cli rpc utxo tracking CityOfZion/neon-js#395

Closed

jsolman added 3 commits March 13, 2019 10:50

Clean-up.

d2722ad

Don't leak memory.

ee6ff9c

Implement getclaimable to get get total unclaimed amount for an address.

008087e

vncoelho mentioned this pull request Mar 13, 2019

UTXO and Claims from c# neo-cli instead of neoscan NeoResearch/neocompiler-eco#50

Merged

vncoelho previously approved these changes Mar 13, 2019

View reviewed changes

Merge branch 'master' into RpcSystemAssetPlugin

dfa8f7d

vncoelho mentioned this pull request Mar 13, 2019

[feature] Support to neo-cli UTXO and Claims tracking from new plugin CityOfZion/neon-wallet#1815

Closed

Remove unnecessary comment.

01fb48e

jsolman changed the base branch from master to plugins-2.10.1 March 14, 2019 18:23

vncoelho approved these changes Mar 14, 2019

View reviewed changes

jsolman merged commit 4a37c19 into plugins-2.10.1 Mar 14, 2019

jsolman mentioned this pull request Mar 14, 2019

Aggregate - Plugin Improvements for Neo 2.10.1 #61

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Plugin for UTXO tracking by account. #52

Add Plugin for UTXO tracking by account. #52

jsolman commented Mar 11, 2019

jsolman commented Mar 11, 2019

vncoelho commented Mar 11, 2019

vncoelho commented Mar 11, 2019 •

edited

Loading

jsolman commented Mar 11, 2019 •

edited

Loading

vncoelho left a comment

vncoelho commented Mar 11, 2019

jsolman commented Mar 11, 2019

vncoelho left a comment

vncoelho commented Mar 11, 2019

vncoelho commented Mar 11, 2019

jsolman commented Mar 11, 2019

jsolman commented Mar 11, 2019 •

edited

Loading

jsolman commented Mar 11, 2019

jsolman commented Mar 12, 2019 •

edited

Loading

jsolman commented Mar 13, 2019

jsolman commented Mar 13, 2019

jsolman commented Mar 13, 2019

vncoelho left a comment •

edited

Loading

jsolman commented Mar 13, 2019 •

edited

Loading

Add Plugin for UTXO tracking by account. #52

Add Plugin for UTXO tracking by account. #52

Conversation

jsolman commented Mar 11, 2019

jsolman commented Mar 11, 2019

vncoelho commented Mar 11, 2019

vncoelho commented Mar 11, 2019 • edited Loading

jsolman commented Mar 11, 2019 • edited Loading

vncoelho left a comment

Choose a reason for hiding this comment

vncoelho commented Mar 11, 2019

jsolman commented Mar 11, 2019

vncoelho left a comment

Choose a reason for hiding this comment

vncoelho commented Mar 11, 2019

vncoelho commented Mar 11, 2019

jsolman commented Mar 11, 2019

jsolman commented Mar 11, 2019 • edited Loading

jsolman commented Mar 11, 2019

jsolman commented Mar 12, 2019 • edited Loading

jsolman commented Mar 13, 2019

jsolman commented Mar 13, 2019

jsolman commented Mar 13, 2019

vncoelho left a comment • edited Loading

Choose a reason for hiding this comment

jsolman commented Mar 13, 2019 • edited Loading

vncoelho commented Mar 11, 2019 •

edited

Loading

jsolman commented Mar 11, 2019 •

edited

Loading

jsolman commented Mar 11, 2019 •

edited

Loading

jsolman commented Mar 12, 2019 •

edited

Loading

vncoelho left a comment •

edited

Loading

jsolman commented Mar 13, 2019 •

edited

Loading