Intermittent 400 Bad Request when sending a transaction

[samsondav]

We are seeing intermittent errors trying to send transactions on Besu (consensus client - teku). It only occurs on some RPC nodes and not all the time.

We are using eth_sendRawTransaction call.

Occasionally, we see this response:

400 Bad Request: {\n \"jsonrpc\" : \"2.0\",\n \"id\" : 60246,\n \"error\" : {\n \"code\" : -32602,\n \"message\" : \"Invalid params\"\n }\n}

This is a blocker for us to run Chainlink on Besu. We need reliable transaction sending.

Could be related to this issue? #2251

[jflo]

Can you provide an example call please? A curl statement would be fine.

[yorickdowne]

I'm going to play go-between here. "Literally just send any transaction" ... I assume if you just send enough tx, do include those that call contracts, you'll see the issue.

[lyxer123]

In response to the above situation, this happens to my program from time to time, but after I use the metamask wallet to send data once, my original program can work again. (my program works the same as metamask)

[lyxer123]

I use 4G wireless module to post the eth_sendRawTransaction：this error {\n "jsonrpc" : "2.0",\n "id" : 60246,\n "error" : {\n "code" : -32602,\n "message" : "Invalid params"\n }\n} occured many times。

when error occured，I use metamask to send a Transaction using same wallet address,then the 4G wireless module can send data to besu with the same wallet address.

[macfarla]

Hi @samsondav we are taking a look at RPC issues this sprint. Are you able to provide any more context re the intermittency of this issue - anything unusual about how you're sending the transactions in general, or the ones that fail?

We're also working on adding more info to the "invalid params" error logging so hopefully that may assist #4326

[samsondav]

Don't have any more context, it seems completely random, but if you send enough tx you will see it.

[baxterjfinch]

I am also seeing this behavior intermittently on our private 4 node QBFT network. We load test ~10,000 transactions of the same contract interaction type, and we see about 4 of these 400 Invalid params error returned.

"rawTx": {
    "type": "0x0",
    "nonce": "0x5f32",
    "gasPrice": "0x3e8",
    "maxPriorityFeePerGas": null,
    "maxFeePerGas": null,
    "gas": "0x271b8",
    "value": "0x0",
    "input": "0xb660892c00000000000000000000000076113240ed0f339d9552357d10270b7247c452050000000000000000000000000000000000000000000000000000000000000060000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000002463656562313766622d653466302d343838642d616261622d616538636265353139356464000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
    "v": "0x0",
    "r": "0x0",
    "s": "0x0",
    "to": "0x68c30c87759db789f798cbd7689f6107250a572a",
    "hash": "0xec5832268822547cd2de314240fae26be88951754b63f03e06cd02e7d609fba5"
  },

  "error": "400 Bad Request: {\n  \"jsonrpc\" : \"2.0\",\n  \"id\" : 8240,\n  \"error\" : {\n    \"code\" : -32602,\n    \"message\" : \"Invalid params\"\n  }\n}",
  

We are running Besu v10.22.0

[samsondav]

This is still a blocker for Chainlink. We cannot use Besu because of this bug. To reproduce, send literally any transaction with a contract call and do it over and over again. Eventually you will hit this.

[macfarla]

@samsondav how are you sending the multiple transactions?

I tried to reproduce with a simple get/set contract, sending set transaction repeatedly, incrementing the nonce and signing, sending via RPC batch request. Somewhere around 600-800 I hit the "Transaction nonce is too distant from current sender nonce" - which admittedly we only introduced more recently.

[macfarla]

Also - in #4510 we added more detail at DEBUG level for Invalid params. If you can turn logging up to debug (even just for jsonrpc) and send us the log - that would help us diagnose.

[macfarla]

so you would get a message with the invalid call eg
2023-01-18 12:17:15.406+10:00 | vert.x-worker-thread-15 | DEBUG | JsonRpcExecutor | JSON-RPC request -> eth_call [{"data":"0x6d4ce63clkj","to":"0x9a3DBCa554e9f6b9257aAa24010DA8377C57c17e","gas":"0x2faf080"},"latest",null]

[macfarla]

please note with #4965 the default max batch size changes to 1, you can increase this using the CLI option if required.

[non-fungible-nelson]

@macfarla - ok to close?

[macfarla]

@samsondav Could you try to reproduce it with the latest release that's coming out soon? V.23.1.0.?

[nano-adhara]

Same here. The problem we noticed is that is intermitent.. so its really hard to reproduce.

It happens maybe 1 tx very 5000 txs. We cannot identify the root issue.. but sending the same exact transaction after it failed works, so the data itself doesnt looks like the issue.

Example of a transaction that failed:
curl -X POST http://localhost:18545 -d '{"jsonrpc":"2.0","method":"eth_sendRawTransaction", "params":["0xf869824501808309ba62944be396d29bcd5af6e43ce1c5155693973f3ca8b080840d52b4168303059ea095b845c2f3d4c02e4249013e6ea7e4caa30b41f7bd6daaa0360ff9f1604b5ad8a05bfccbb6898b354fe950179ec331755e7ad826255ba1034f6974da07197d4dbb"], "id": "1"}'

NOTE: we updated from 22.4.2 into 22.7.7 when we started to obtain this errors randomly.

[dmitridr]

Also seeing this same behavior on a private network, been seeing this for a while now.
Upgraded to 23.1.1 just now and can confirm the behavior is still the same:

INVALID_PARAMS is returned a small percentage of the time at high load for eth_sendRawTransaction, most of the time, around the time a new block is cut, but not always (I suspect a race condition). Retrying the request without any modification to the request (but with some small ~100ms delay) typically results in success.

We have been seeing these errors since 22.4.4.
Hope that helps.

[macfarla]

recent AT fail with invalid params - might be a clue https://app.circleci.com/pipelines/github/hyperledger/besu/22285/workflows/dad5b738-a624-4db5-8a90-1a7c8c304019/jobs/138045/tests - possibly related to exception handling, with invalid params used as default error typoe

[non-fungible-nelson]

Closed as completed via #5506 (might not solve the initial problem, but will give more info about the cause of the errors). Curious to see any results on this load level with the new approach here.

[macfarla]

Further improved by #6143