Unable to start main loop

[banky]

Description

Seeing an issue with besu not being able to start up due to the following error

java.util.concurrent.CompletionException: org.hyperledger.besu.plugin.services.exception.StorageException: org.rocksdb.RocksDBException: While pread offset 20119552 len 32738: /ethclient/besu/database/131812.sst: Input/output error
rocketpool_eth1  | 	at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)
rocketpool_eth1  | 	at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)
rocketpool_eth1  | 	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:649)
rocketpool_eth1  | 	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
rocketpool_eth1  | 	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
rocketpool_eth1  | 	at org.hyperledger.besu.ethereum.p2p.discovery.VertxPeerDiscoveryAgent.handleListenerSetup(VertxPeerDiscoveryAgent.java:159)
rocketpool_eth1  | 	at org.hyperledger.besu.ethereum.p2p.discovery.VertxPeerDiscoveryAgent.lambda$listenForConnections$3(VertxPeerDiscoveryAgent.java:119)
rocketpool_eth1  | 	at io.vertx.core.impl.future.FutureImpl$3.onSuccess(FutureImpl.java:141)
rocketpool_eth1  | 	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:60)
rocketpool_eth1  | 	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211)
rocketpool_eth1  | 	at io.vertx.core.impl.future.FixedMapping.onSuccess(FixedMapping.java:31)
rocketpool_eth1  | 	at io.vertx.core.impl.future.FutureBase.lambda$emitSuccess$0(FutureBase.java:54)
rocketpool_eth1  | 	at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:81)
rocketpool_eth1  | 	at io.vertx.core.impl.ContextImpl.execute(ContextImpl.java:260)
rocketpool_eth1  | 	at io.vertx.core.impl.EventLoopContext.execute(EventLoopContext.java:22)
rocketpool_eth1  | 	at io.vertx.core.impl.future.FutureBase.emitSuccess(FutureBase.java:51)
rocketpool_eth1  | 	at io.vertx.core.impl.future.FutureImpl.tryComplete(FutureImpl.java:211)
rocketpool_eth1  | 	at io.vertx.core.impl.future.PromiseImpl.tryComplete(PromiseImpl.java:23)
rocketpool_eth1  | 	at io.vertx.core.Promise.complete(Promise.java:66)
rocketpool_eth1  | 	at io.vertx.core.impl.future.PromiseImpl.operationComplete(PromiseImpl.java:65)
rocketpool_eth1  | 	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
rocketpool_eth1  | 	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552)
rocketpool_eth1  | 	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
rocketpool_eth1  | 	at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184)
rocketpool_eth1  | 	at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95)
rocketpool_eth1  | 	at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30)
rocketpool_eth1  | 	at io.vertx.core.datagram.impl.DatagramSocketImpl.lambda$listen$1(DatagramSocketImpl.java:316)
rocketpool_eth1  | 	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
rocketpool_eth1  | 	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552)
rocketpool_eth1  | 	at io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35)
rocketpool_eth1  | 	at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:502)
rocketpool_eth1  | 	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
rocketpool_eth1  | 	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
rocketpool_eth1  | 	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
rocketpool_eth1  | 	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:391)
rocketpool_eth1  | 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
rocketpool_eth1  | 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
rocketpool_eth1  | 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
rocketpool_eth1  | 	at java.base/java.lang.Thread.run(Thread.java:833)
rocketpool_eth1  | Caused by: org.hyperledger.besu.plugin.services.exception.StorageException: org.rocksdb.RocksDBException: While pread offset 20119552 len 32738: /ethclient/besu/database/131812.sst: Input/output error

I want to find out how I can resolve this. Please can I get some help?

Acceptance Criteria

• Besu client should start up as expected

Steps to Reproduce (Bug)

1. Start up service
2. Service fails with this error

Expected behavior: [What you expect to happen]

1. Service start up process should not fail

Actual behavior: [What actually happens]

1. Service start up process fails

Frequency: [What percentage of the time does it occur?]

1. This occurs every time I start besu

Versions (Add all that apply)

• Software version: hyperledger/besu:22.10.0-openjdk-latest
• Java version: [java -version]
• OS Name & Version: Ubuntu 22.04
• Kernel Version: Linux k2 5.15.0-52-generic Add default SECURITY policy #58-Ubuntu SMP Thu Oct 13 08:03:55 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
• Virtual Machine software & version: N/A
• Docker Version: 20.10.21
• Cloud VM, type, size: N/A

[iamhsk]

Currently @non-fungible-nelson is getting more information regarding this error

[garyschulte]

This looks like a corrupted sst file (database file) and/or a hardware failure. I doubt this is a recoverable error. You can delete (or move and save off) the besu database directory and re-sync, but I would be looking hard at the disk in question

[banky]

I resolved this by resyncing my client, thanks for the helpful info!

[pozyx]

This happened to me after updating to v22.10.2.

[Alkia]

I have the problem every day since moving to the latest version. Is there some system parameters that would help reducing the issue?