Segmentation fault when starting neo-cli on Linux

[Crypto2]

Using neo-cli with application logging enabled (dotnet neo-cli.dll --rpc --log) I keep getting a Segmentation fault during startup. I then have to delete my Chain/Index/ApplicationLog folders and resync with chain.acc.zip in order to start NEO up again.

[Crypto2]

Any update on this. Even when it is running it's running really poorly compared to older versions, being slow and RPC requests timing out with no reply. Not sure if this is just because of --log which we just had to add recently or what???

[vncoelho]

Hi @Crypto2. Trying to solve this issue. It looks like that the problem was just some configuration file that had changed.

Apparently, it something related to the plugins configurations. By copying the last simplepolicy config you may solve the issue.

Which release are you using?
The error that I receive is:

[ERROR][02/09/2019 22:28:14][Thread 0003][akka://NeoSystem/user/$a] Object reference not set to an instance of an object.
Cause: System.NullReferenceException: Object reference not set to an instance of an object.
   at Neo.Plugins.SimplePolicyPlugin.get_MaxTxPerBlock()
   at Neo.Ledger.MemoryPool.LoadMaxTxLimitsFromPolicyPlugins()
   at Neo.Ledger.MemoryPool.UpdatePoolForBlockPersisted(Block block, Snapshot snapshot)
   at Neo.Ledger.Blockchain.OnPersistCompleted(Block block)
   at Neo.Ledger.Blockchain.Persist(Block block)
   at Neo.Ledger.Blockchain.OnImport(IEnumerable`1 blocks)
   at Akka.Actor.UntypedActor.Receive(Object message)
   at Akka.Actor.ActorBase.AroundReceive(Receive receive, Object message)
   at Akka.Actor.ActorCell.ReceiveMessage(Object message)
   at Akka.Actor.ActorCell.Invoke(Envelope envelope)
[ERROR][02/09/2019 22:28:14][Thread 0004][akka://NeoSystem/user/$a] Error while creating actor instance of type Neo.Ledger.Blockchain with 2 args: (Neo.NeoSystem,Neo.Persistence.LevelDB.LevelDBStore)
Cause: [akka://NeoSystem/user/$a#184477984]: Akka.Actor.PostRestartException: Exception post restart (System.NullReferenceException) ---> System.TypeLoadException: Error while creating actor instance of type Neo.Ledger.Blockchain with 2 args: (Neo.NeoSystem,Neo.Persistence.LevelDB.LevelDBStore) ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: Operation is not valid due to the current state of the object.
   at Neo.Ledger.Blockchain..ctor(NeoSystem system, Store store)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at Akka.Actor.Props.ActivatorProducer.Produce()
   at Akka.Actor.Props.NewActor()
   --- End of inner exception stack trace ---
   at Akka.Actor.Props.NewActor()
   at Akka.Actor.ActorCell.CreateNewActorInstance()
   at Akka.Actor.ActorCell.<>c__DisplayClass109_0.<NewActor>b__0()
   at Akka.Actor.ActorCell.UseThreadContext(Action action)
   at Akka.Actor.ActorCell.NewActor()
   at Akka.Actor.ActorCell.FinishRecreate(Exception cause, ActorBase failedActor)
   --- End of inner exception stack trace ---

[jsolman]

That could happen if you are using master of Neo GitHub and trying to use the 2.9.4 released plugin of SimplePlugin. It needs this Commit that hasn’t been included in a release yet to avoid the error above:
neo-project/neo-modules@a7fd659

If you are using the master version of Neo you would also need to be using this PR for neo-cli: #288

[vncoelho]

Perhaps was it, Jeff, I tested with last masters as well, but sometimes we get confused with all different versions.
I will double check.

[vncoelho]

Checked again. Not sure yet, even with last commits this errors persisted.
Let leave this thread open until further investigation.

[vncoelho]

I just had another problem, after closing the neo-cli and reopening it gave these errors. Thus, it might be also related to something when restarting a node.

[vncoelho]

@jsolman, the strange think is that sometimes it syncs without problems and sometimes it crashes...

[Crypto2]

I was using the 2.9.4 tagged release and 2.9.4 plugins. It didn't print out any error messages just Segmentation fault. I've gone back to 2.9.2 for now since it runs better. 2.9.4 even when running was so laggy/slow to be unusable :(

[jsolman]

@Crypto2 were you using the setting in 2.9.4 to limit your connection count?

[jsolman]

I have a PR out for a more graceful shutdown but I need to do a little more testing on it and try one more thing before it will be merged.

[Crypto2]

@jsolman Nope, the only non-default item in my config is the RPC bind address and only command line parms are --rpc and --log

[Crypto2]

Is this and the slow running fixed in 2.10.0?

[jsolman]

Try it and let me know. I don’t usually use the application log plugin. I am not getting the segmentation fault or performance issue.

[Crypto2]

You do know the application log plugin is essentially required to support token deposits (at least as far as I can tell, docs aren't exactly great here.) You'd think it would be well tested by the core devs and major bugs like this fixed ASAP. Unfortunately I have to try 2.10.0 now since 2.9.2 won't sync past 3473781 now. :(

[jsolman]

I cannot speak for all the other core devs. Personally, I have tracked NEP5 balances using my own code for the last 9 months rather than the application log. I also wrote an efficient way to track the NEP5 balances as a Neo plugin that works with 2.10.0.

https://github.com/neo-project/neo-plugins/tree/master/RpcNep5Tracker

[jsolman]

Were the problems you were having related to syncing using the ImportBlocks plugin? I found an issue with the way the plugins load where it could start importing blocks potentially before the the plugin loaded; a fix for that will be in 2.10.1.

[jsolman]

I just took a look at the plugin source again, and I see something I can do to improve its performance. Also, I see an issue with it if neo-cli is killed at an inopportune time since it isn’t doing its writes with batching. I’ll make a PR to improve the plugin.

[Crypto2]

We did sync using ImportBlocks but not sure if it caused the problem or not, just once the program was closed we'd get a segfault on the next start unless we deleted the Chain_* folder and resynced. Since we have so many addresses we end up walking the blockchain for deposit processing (which is how a lot of our coins end up working so don't mind just need working wallet that doesn't need to resync every time you restart it.)

We do always close the wallet by typing 'exit', never kill or kill -9 or anything like that.

[jsolman]

The wallet does not stop properly when typing exit while in the processes of syncing using the ImportBlocks plugin. I am planning to fix this.

[jsolman]

I opened an issue to track the exit issue while syncing blocks using the ImportBlocks plugin: neo-project/neo-modules#59

[jsolman]

I opened an issue to track application log plugin improvement also: neo-project/neo-modules#60

[jsolman]

just once the program was closed we'd get a segfault on the next start unless we deleted the Chain_* folder

What version of leveldb were you using for my reference? Also, what OS were you using?

I will make changes to the application log plugin today which should hopefully resolve your problems.

[Crypto2]

Using Ubuntu 16.04 LTS 64-bit (since it says newer versions aren't supported) using the stock leveldb 1.18-5

[jsolman]

I've started a PR for batching writes in the Application Log plugin: neo-project/neo-modules/pull/62

[jsolman]

I fixed the ImportBlocks plugin. I have a PR out here: neo-project/neo-modules/pull/66

[jsolman]

With these 2 PRs, this issue should be resolved.

[vncoelho]

Are you getting this error @Crypto2?

Unhandled Exception: System.ArgumentException: Unable to resolve the target Provider
Parameter name: self
   at Akka.Actor.Futures.Ask[T](ICanTell self, Func`2 messageFactory, Nullable`1 timeout, CancellationToken cancellationToken)
   at Neo.Plugins.ImportBlocks.OnImport()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
/opt/start_node.sh: line 14:    38 Aborted                 (core dumped) dotnet neo-cli.dll --rpc --log

[vncoelho]

I will soon test with your new PR, @jsolman.

[superboyiii]

@vncoelho Crypto2's issue is like this, not that one. Yours seems like the param [self] in settings.cs hasn't been set into ImportBlocks.cs.

[shoxive]

@vncoelho
Hi,I met this error after finish downloading the Full offline package. Any suggest of this problem(It looks like ImportBlocks error)?
Btw,I am using Neo CLI v2.10.1.

[vncoelho]

Hi @shoxive, thanks for replying it.

Could you please print the message here and, if possible, try with 2.10.2? :)
Thanks,

Since it is not a critical issue we are not really focusing on it. But let's try to cover all these exceptions.

[superboyiii]

@Crypto2 @shoxive Seems this issue is fixed in neo-cli-2.10.2, could you make a test? It seems work in mine.

[lock9]

Closing issue due to inactivity. @Crypto2 @shoxive please open a new issue if the problem persists. Thanks