Under heavy load eth_estimateGas returns INTERNAL_ERROR

[matthew1001]

Description

Running a moderately high workload using caliper, after ~ 2000 transactions eth_estimateGas returns INTERNAL_ERROR to the caliper WS client

Command that was run to start Besu (single validator):

besu --data-path=data --genesis-file=../genesis.json --min-gas-price=0 --sync-mode=FULL --rpc-http-enabled --rpc-ws-enabled --rpc-http-apis=DEBUG,TXPOOL,WEB3,ETH,TRACE,QBFT,ADMIN --tx-pool-limit-by-account-percentage=0.20 --rpc-ws-max-active-connections=300 --tx-pool=sequenced -l TRACE

Logs

I've pinned it down to this code block:

final BlockHeader blockHeader = blockHeader();
if (blockHeader == null) {
    return errorResponse(requestContext, RpcErrorType.INTERNAL_ERROR);
}

besu/ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/EthEstimateGas.java

Line 53 in e0cd89f

return errorResponse(requestContext, RpcErrorType.INTERNAL_ERROR);

It was difficult to pin down the offending line using TRACE but adding a new Exception().printStackTrace() to the JsonRpcErrorResponse(...) constructor gave the stack trace:

java.lang.Exception
        at org.hyperledger.besu.ethereum.api.jsonrpc.internal.response.JsonRpcErrorResponse.<init>(JsonRpcErrorResponse.java:47)
        at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.AbstractEstimateGas.errorResponse(AbstractEstimateGas.java:135)
        at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.AbstractEstimateGas.errorResponse(AbstractEstimateGas.java:130)
        at org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.EthEstimateGas.response(EthEstimateGas.java:53)
        at org.hyperledger.besu.ethereum.api.jsonrpc.execution.BaseJsonRpcProcessor.process(BaseJsonRpcProcessor.java:44)
        at org.hyperledger.besu.ethereum.api.jsonrpc.execution.JsonRpcExecutor.execute(JsonRpcExecutor.java:92)
        at org.hyperledger.besu.ethereum.api.jsonrpc.websocket.WebSocketMessageHandler.lambda$handle$1(WebSocketMessageHandler.java:90)
        at io.vertx.core.impl.ContextBase.lambda$null$0(ContextBase.java:137)
        at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
        at io.vertx.core.impl.ContextBase.lambda$executeBlocking$1(ContextBase.java:135)
        at io.vertx.core.impl.TaskQueue.run(TaskQueue.java:76)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:833)

Once I've got a fix in place I might add some additional TRACE logs in EthEstimateGas wherever an INTERNAL_ERROR is returned, to make diagnosing similar issues a little quicker.

[matthew1001]

Looks to be the same issue as #6143