Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error during fresh sync #7423

Closed
yorickdowne opened this issue Aug 2, 2024 · 7 comments
Closed

Error during fresh sync #7423

yorickdowne opened this issue Aug 2, 2024 · 7 comments
Assignees
@fab-10
Labels
P2 High (ex: Degrading performance issues, unexpected behavior of core features (DevP2P, syncing, etc))

Comments

@yorickdowne
Copy link

yorickdowne commented Aug 2, 2024 •
edited
Loading

Description

Fresh sync on Ethereum mainnet with Besu 24.7.1 and --receipt-compaction-enabled=true --Xbonsai-parallel-tx-processing-enabled=true shows an error:

execution-1  | 2024-08-02 20:23:30.946+00:00 | EthScheduler-Services-3 (batchPersistAccountData) | INFO  | SnapSyncMetricsManager | Worldstate download progress: 3.00%, Peer count: 7
execution-1  | 2024-08-02 20:24:00.240+00:00 | EthScheduler-Services-37 (importBlock) | INFO  | ImportBlocksStep | Block import progress: 378539 of 20443107 (1%), Peer count: 9
execution-1  | 2024-08-02 20:25:34.992+00:00 | vert.x-worker-thread-4 | ERROR | BaseJsonRpcProcessor | Error processing method: eth_gasPrice null
execution-1  | java.lang.IllegalStateException: Could not retrieve block #390035
execution-1  | 	at org.hyperledger.besu.ethereum.api.query.BlockchainQueries.lambda$gasPrice$25(BlockchainQueries.java:994)
execution-1  | 	at java.base/java.util.Optional.orElseThrow(Optional.java:403)
execution-1  | 	at org.hyperledger.besu.ethereum.api.query.BlockchainQueries.lambda$gasPrice$26(BlockchainQueries.java:993)
execution-1  | 	at java.base/java.util.stream.LongPipeline$1$1.accept(LongPipeline.java:177)
execution-1  | 	at java.base/java.util.stream.Streams$RangeLongSpliterator.forEachRemaining(Streams.java:232)
execution-1  | 	at java.base/java.util.Spliterator$OfLong.forEachRemaining(Spliterator.java:777)
execution-1  | 	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
execution-1  | 	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
execution-1  | 	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
execution-1  | 	at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
execution-1  | 	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
execution-1  | 	at org.hyperledger.besu.ethereum.api.query.BlockchainQueries.gasPrice(BlockchainQueries.java:999)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.EthGasPrice.calculateGasPrice(EthGasPrice.java:49)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.EthGasPrice.response(EthGasPrice.java:45)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:44)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:54)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.TimedJsonRpcProcessor.process(TimedJsonRpcProcessor.java:45)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
execution-1  | 	at org.hyperledger.besu.ethereum.api.handlers.AbstractJsonRpcExecutor.executeRequest(AbstractJsonRpcExecutor.java:87)
execution-1  | 	at org.hyperledger.besu.ethereum.api.handlers.JsonRpcObjectExecutor.execute(JsonRpcObjectExecutor.java:57)
execution-1  | 	at org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$0(JsonRpcExecutorHandler.java:57)
execution-1  | 	at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196)
execution-1  | 	at org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$2(JsonRpcExecutorHandler.java:54)
execution-1  | 	at io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
execution-1  | 	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:191)
execution-1  | 	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
execution-1  | 	at io.vertx.core.impl.ContextImpl.lambda$internalExecuteBlocking$2(ContextImpl.java:210)
execution-1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
execution-1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
execution-1  | 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
execution-1  | 	at java.base/java.lang.Thread.run(Thread.java:1583)
execution-1  | 2024-08-02 20:25:38.168+00:00 | EthScheduler-Services-3 (batchPersistAccountData) | INFO  | SnapSyncMetricsManager | Worldstate download progress: 3.49%, Peer count: 10

Steps to Reproduce (Bug)

SNAP sync Ethereum mainnet with --Xbonsai-parallel-tx-processing-enabled=true. Syncing without this parameter gets past the problematic block without error messages, but then shows the message a little later. So maybe parallel just makes it happen more often?

Versions (Add all that apply)

  • Software version: 24.7.1
  • OS Name & Version: Debian 12
  • Docker Version: 27.1.1
@yorickdowne
Copy link
Author

Same on Holesky:

execution-1  | 2024-08-02 20:44:18.986+00:00 | vert.x-worker-thread-13 | ERROR | BaseJsonRpcProcessor | Error processing method: eth_gasPrice null
execution-1  | java.lang.IllegalStateException: Could not retrieve block #903907
execution-1  | 	at org.hyperledger.besu.ethereum.api.query.BlockchainQueries.lambda$gasPrice$25(BlockchainQueries.java:994)
execution-1  | 	at java.base/java.util.Optional.orElseThrow(Optional.java:403)
execution-1  | 	at org.hyperledger.besu.ethereum.api.query.BlockchainQueries.lambda$gasPrice$26(BlockchainQueries.java:993)
execution-1  | 	at java.base/java.util.stream.LongPipeline$1$1.accept(LongPipeline.java:177)
execution-1  | 	at java.base/java.util.stream.Streams$RangeLongSpliterator.forEachRemaining(Streams.java:232)
execution-1  | 	at java.base/java.util.Spliterator$OfLong.forEachRemaining(Spliterator.java:777)
execution-1  | 	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
execution-1  | 	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
execution-1  | 	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
execution-1  | 	at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
execution-1  | 	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
execution-1  | 	at org.hyperledger.besu.ethereum.api.query.BlockchainQueries.gasPrice(BlockchainQueries.java:999)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.EthGasPrice.calculateGasPrice(EthGasPrice.java:49)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.EthGasPrice.response(EthGasPrice.java:45)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:44)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:54)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.TimedJsonRpcProcessor.process(TimedJsonRpcProcessor.java:45)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
execution-1  | 	at org.hyperledger.besu.ethereum.api.handlers.AbstractJsonRpcExecutor.executeRequest(AbstractJsonRpcExecutor.java:87)
execution-1  | 	at org.hyperledger.besu.ethereum.api.handlers.JsonRpcObjectExecutor.execute(JsonRpcObjectExecutor.java:57)
execution-1  | 	at org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$0(JsonRpcExecutorHandler.java:57)
execution-1  | 	at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196)
execution-1  | 	at org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$2(JsonRpcExecutorHandler.java:54)
execution-1  | 	at io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
execution-1  | 	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:191)
execution-1  | 	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
execution-1  | 	at io.vertx.core.impl.ContextImpl.lambda$internalExecuteBlocking$2(ContextImpl.java:210)
execution-1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
execution-1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
execution-1  | 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
execution-1  | 	at java.base/java.lang.Thread.run(Thread.java:1583)

@yorickdowne
Copy link
Author

Full parameters:

--data-path /var/lib/besu --p2p-port 30303 --rpc-http-enabled --rpc-http-host 0.0.0.0 --rpc-http-port 8545 --rpc-http-cors-origins=* --rpc-http-max-active-connections=65536 --rpc-max-logs-range=65536 --rpc-ws-enabled --rpc-ws-host 0.0.0.0 --rpc-ws-port 8546 --max-peers 25 --host-allowlist=* --engine-host-allowlist=* --engine-jwt-secret=/var/lib/besu/ee-secret/jwtsecret --engine-rpc-port=8551 --logging info --metrics-enabled --metrics-host 0.0.0.0 --metrics-port 6060 --nat-method=DOCKER --network holesky --rpc-http-api WEB3,ETH,NET --data-storage-format=BONSAI --sync-mode=SNAP --receipt-compaction-enabled=true --Xbonsai-parallel-tx-processing-enabled=true

@yorickdowne
Copy link
Author

Without parallel processing on Ethereum mainnet, this still happens:

execution-1  | 2024-08-02 20:53:50.832+00:00 | vert.x-worker-thread-12 | ERROR | BaseJsonRpcProcessor | Error processing method: eth_gasPrice null
execution-1  | java.lang.IllegalStateException: Could not retrieve block #986679
execution-1  | 	at org.hyperledger.besu.ethereum.api.query.BlockchainQueries.lambda$gasPrice$25(BlockchainQueries.java:994)
execution-1  | 	at java.base/java.util.Optional.orElseThrow(Optional.java:403)
execution-1  | 	at org.hyperledger.besu.ethereum.api.query.BlockchainQueries.lambda$gasPrice$26(BlockchainQueries.java:993)
execution-1  | 	at java.base/java.util.stream.LongPipeline$1$1.accept(LongPipeline.java:177)
execution-1  | 	at java.base/java.util.stream.Streams$RangeLongSpliterator.forEachRemaining(Streams.java:232)
execution-1  | 	at java.base/java.util.Spliterator$OfLong.forEachRemaining(Spliterator.java:777)
execution-1  | 	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
execution-1  | 	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
execution-1  | 	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
execution-1  | 	at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
execution-1  | 	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
execution-1  | 	at org.hyperledger.besu.ethereum.api.query.BlockchainQueries.gasPrice(BlockchainQueries.java:999)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.EthGasPrice.calculateGasPrice(EthGasPrice.java:49)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.EthGasPrice.response(EthGasPrice.java:45)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:44)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:54)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.TimedJsonRpcProcessor.process(TimedJsonRpcProcessor.java:45)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
execution-1  | 	at org.hyperledger.besu.ethereum.api.handlers.AbstractJsonRpcExecutor.executeRequest(AbstractJsonRpcExecutor.java:87)
execution-1  | 	at org.hyperledger.besu.ethereum.api.handlers.JsonRpcObjectExecutor.execute(JsonRpcObjectExecutor.java:57)
execution-1  | 	at org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$0(JsonRpcExecutorHandler.java:57)
execution-1  | 	at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196)
execution-1  | 	at org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$2(JsonRpcExecutorHandler.java:54)
execution-1  | 	at io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
execution-1  | 	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:191)
execution-1  | 	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
execution-1  | 	at io.vertx.core.impl.ContextImpl.lambda$internalExecuteBlocking$2(ContextImpl.java:210)
execution-1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
execution-1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
execution-1  | 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
execution-1  | 	at java.base/java.lang.Thread.run(Thread.java:1583)

Just later in the sync process

@yorickdowne
Copy link
Author

Vanilla, neither parallel processing nor receipts compaction, I still see this error - just a lot less frequently

execution-1  | 2024-08-02 21:24:36.669+00:00 | vert.x-worker-thread-0 | ERROR | BaseJsonRpcProcessor | Error processing method: eth_gasPrice null
execution-1  | java.lang.IllegalStateException: Could not retrieve block #695838
execution-1  | 	at org.hyperledger.besu.ethereum.api.query.BlockchainQueries.lambda$gasPrice$25(BlockchainQueries.java:994)
execution-1  | 	at java.base/java.util.Optional.orElseThrow(Optional.java:403)
execution-1  | 	at org.hyperledger.besu.ethereum.api.query.BlockchainQueries.lambda$gasPrice$26(BlockchainQueries.java:993)
execution-1  | 	at java.base/java.util.stream.LongPipeline$1$1.accept(LongPipeline.java:177)
execution-1  | 	at java.base/java.util.stream.Streams$RangeLongSpliterator.forEachRemaining(Streams.java:232)
execution-1  | 	at java.base/java.util.Spliterator$OfLong.forEachRemaining(Spliterator.java:777)
execution-1  | 	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
execution-1  | 	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
execution-1  | 	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575)
execution-1  | 	at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
execution-1  | 	at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616)
execution-1  | 	at org.hyperledger.besu.ethereum.api.query.BlockchainQueries.gasPrice(BlockchainQueries.java:999)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.EthGasPrice.calculateGasPrice(EthGasPrice.java:49)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.EthGasPrice.response(EthGasPrice.java:45)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:44)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.TracedJsonRpcProcessor.process(TracedJsonRpcProcessor.java:54)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.TimedJsonRpcProcessor.process(TimedJsonRpcProcessor.java:45)
execution-1  | 	at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
execution-1  | 	at org.hyperledger.besu.ethereum.api.handlers.AbstractJsonRpcExecutor.executeRequest(AbstractJsonRpcExecutor.java:87)
execution-1  | 	at org.hyperledger.besu.ethereum.api.handlers.JsonRpcObjectExecutor.execute(JsonRpcObjectExecutor.java:57)
execution-1  | 	at org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$0(JsonRpcExecutorHandler.java:57)
execution-1  | 	at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196)
execution-1  | 	at org.hyperledger.besu.ethereum.api.handlers.JsonRpcExecutorHandler.lambda$handler$2(JsonRpcExecutorHandler.java:54)
execution-1  | 	at io.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)
execution-1  | 	at io.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:191)
execution-1  | 	at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:279)
execution-1  | 	at io.vertx.core.impl.ContextImpl.lambda$internalExecuteBlocking$2(ContextImpl.java:210)
execution-1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
execution-1  | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
execution-1  | 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
execution-1  | 	at java.base/java.lang.Thread.run(Thread.java:1583)

@yorickdowne yorickdowne changed the title Error during fresh sync with parallel processing Error during fresh sync Aug 2, 2024
@jflo jflo added the P2 High (ex: Degrading performance issues, unexpected behavior of core features (DevP2P, syncing, etc)) label Aug 13, 2024
@fab-10
Copy link
Contributor

fab-10 commented Aug 13, 2024 •
edited
Loading

duplicate of #7287

@yorickdowne yorickdowne reopened this Aug 13, 2024
@yorickdowne
Copy link
Author

Sorry, confused. Duplicate of which issue? This seems to say it duplicates itself 😅

@fab-10
Copy link
Contributor

fab-10 commented Aug 13, 2024

sorry, linked the correct issue now

@fab-10 fab-10 closed this as completed Aug 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fab-10 fab-10

Labels
P2 High (ex: Degrading performance issues, unexpected behavior of core features (DevP2P, syncing, etc))
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jflo @yorickdowne @fab-10