Polygon-mainnet: debug_traceBlockByHash Returns wrong data for some newly created block and may or may not get fixed after a while.

[edwinchen12]

Our support team has aggregated some common issues and their solutions from past which are faced while running or interacting with a bor client. In order to prevent redundant efforts, we would encourage you to have a look at the FAQ's section of our documentation mentioning the same, before filing an issue here. In case of additional support, you can also join our discord server

System information

Bor client version: [e.g. v0.2.16]
I am not entirely sure the Bor client version, we use QuickNode and Achemy as our node provider and it depends on which client version they are running.

Heimdall client version: [e.g. v0.2.10]

OS & Version: Linux

Environment: Polygon Mainnet

Type of node: Archive

Additional Information:

Overview of the problem

Please describe the issue you experiencing.

debug_traceBlockByHash can return wrong data when compare against polygon block explorer. Example for blockHash=0x1a050fb2bac702638c53ee9f49aaf448d4ceb3c4bd4df13e478b777a23792325

Block Explorer: https://polygonscan.com/block/0x1a050fb2bac702638c53ee9f49aaf448d4ceb3c4bd4df13e478b777a23792325

The gas and gasUsed field can be wrong for multiple traces in the block, one of the example https://polygonscan.com/tx/0x4092ef3d113e813755031d87017676a42609f6f9821ef4272f48a1a4f6467e9c

1. QuickNode initially returns wrong data and the data gets corrected after a few minutes.
2. Achemy data stays wrong for an extensive period of time.

This issue started happening on 02/26/2023 and is currently actively happening on 03/02/2023 started around 15:30pm PST and paused around 16:15pm PST

Reproduction Steps

Please mention the steps required to reproduce this issue.
This problem is dynamic and a bit difficult to reproduce, you can query an achemy cluster and it should return the wrong data right now.

1. query curl -s https://achemy_end_point -H "Content-Type: application/json" -d '{"jsonrpc": "2.0", "id": 1, "method": "debug_traceBlockByHash", "params": ["0x1a050fb2bac702638c53ee9f49aaf448d4ceb3c4bd4df13e478b777a23792325", {"tracer": "callTracer"}]}' | jq

Logs / Traces / Output / Error Messages

Please post any logs/traces/output/error messages (as text and not screenshots) which you believe may have caused the issue. If the log is longer than a few dozen lines, please include the URL to the gist of the log instead of posting it in the issue.

Additional Information

In order to debug the issue faster, we would stongly encourage if you can provide some of the details mentioned below (whichever seems relevant to your issue)

Wrong trace data being returned:
WrongData.csv

Correct trace data being returned after a while;
CorrectData.csv

[Raneet10]

Hey @edwinchen12 , thanks for reporting the issue! We'll have a look and get back on this.

[edwinchen12]

@Raneet10 thanks a lot for your response, I just wanted to add one more piece of information, the QuickNode trace data eventually changed back to be WrongData.csv and stayed there, which makes me think this is the actual correct data. But I am having trouble to reconcile the gasUsed returned by the eth_getTransactionReceipt call and the debug_traceTransaction call for this particular transaction 0x22ab5ddd8917086df8f55a4cf50a16cd6e2ba0582e6fdabc4bf81a740775a60e. If I can compute the gasUsed from transaction receipt from the gasUsed from the traces, then I will be confident that WrongData.csv actually contains the correct data.

[edwinchen12]

Some detail numbers here

gasUsed from transaction receipt: 0x1632bc(1454780)

gasUsed from traces: 0x15ead8(1436376)

Diff = 1454780 - 1436376 = 18404

18404 is less than the fix transaction fee of 21000 let along we need to pay for the input data. I am a bit confused here how to match the numbers

[pratikspatil024]

Hi, @edwinchen12 we looked into this and found out that geth also had the same issue, and it was fixed in this PR.

We will be doing an upstream merge in the near future, and hence this will be resolved. Thanks.