incorrect gasUsed with debug_traceCallMany/debug_traceTransaction/debug_traceBlockByNumber and callTracer #4262

0xSt1ng3R · 2023-08-18T10:41:41Z

Describe the bug

While simulating a tx/block with debug_traceCallMany/debug_traceTransaction/debug_traceBlockByNumber and callTracer, I'm getting incorrect gasUsed.

Steps to reproduce

block #17933760 correct gasUsed: 11,808,302
tx 0x28ef70e4edcd6ce7b7ec80bf516adcaf73da5c1be5fef1e9a37aa600a8e80a69 correct gasUsed: 51,214

def sum_gas_used(data):
    total = int(data['gasUsed'], 16)
    if 'calls' in data:
        for call in data['calls']:
            total += sum_gas_used(call)
    return total

# RPC request to -> eth_getBlockByNumber([
#     hex(17933760),
#     True
# ])
int(response["gasUsed"], 16)
# 11,808,302, correct

# RPC request to -> eth_getTransactionReceipt("0x28ef70e4edcd6ce7b7ec80bf516adcaf73da5c1be5fef1e9a37aa600a8e80a69")
int(response['gasUsed'], 16)
# 51,214, correct

# RPC request to -> debug_traceBlockByNumber([
#     hex(17933760), 
#     {
#         "tracer": "callTracer"
#     }
# ])
sum([sum_gas_used(tx['result']) for tx in response])
# 20,284,310, incorrect

# RPC request to -> debug_traceTransaction([
#     "0x28ef70e4edcd6ce7b7ec80bf516adcaf73da5c1be5fef1e9a37aa600a8e80a69",
#     {
#         "tracer": "callTracer"
#     }
# ])
sum_gas_used(response)
# 54,359, incorrect

# RPC request to -> debug_traceCallMany([
#     [
#         {
#             "transactions": [...], # response of eth_getBlockByNumber([hex(17933760), True])['transactions']
#             "blockOverride": {
#                 "number": hex(17933760),
#             }
#         }
#     ], 
#     {
#         "blockNumber": hex(17933760),
#         "transactionIndex": 0,
#     },
#     {
#         "tracer": "callTracer",
#         "tracerConfig": {
#             "onlyTopCall": False,
#             "withLog": False
#         }
#     }
# ])
sum([sum_gas_used(x) for x in response])
# 20,284,310, incorrect

Node logs

No response

Platform(s)

Linux (x86)

What version/commit are you on?

reth Version: 0.1.0-alpha.6
Commit SHA: 639a6eac
Build Timestamp: 2023-08-17T10:45:03.473434751Z
Build Features: default,jemalloc
Build Profile: maxperf

What database version are you on?

Current database version: 1
Local database version: 1

If you've built Reth from source, provide the full command you used

LD_PRELOAD=/usr/local/lib/libjemalloc.so RUSTFLAGS="-C target-cpu=native" cargo build --profile maxperf --features jemalloc

Code of Conduct

I agree to follow the Code of Conduct

The text was updated successfully, but these errors were encountered:

0xSt1ng3R · 2023-08-18T11:21:32Z

while using erigon's js call_tracer with debug_traceTransaction (on reth), I got 23,859 gasUsed... so idk what is going on here?

0xSt1ng3R · 2023-08-20T08:25:58Z

any update on this one? all gas estimation tests that we do fails... :'(

mattsse · 2023-08-20T13:14:58Z

while using erigon's js call_tracer with debug_traceTransaction (on reth), I got 23,859 gasUsed

can you elaborate on this?

0xSt1ng3R · 2023-08-20T15:05:56Z

while using erigon's js call_tracer with debug_traceTransaction (on reth), I got 23,859 gasUsed

can you elaborate on this?

I attempted to simulate the tx and block in question using the JS tracer from erigon, instead of the native one from reth. However, I received completely different and incorrect gasUsed results.
So it seems like a possible bug inside the tracer?

mattsse · 2023-08-20T15:18:57Z

can you please add some examples

0xSt1ng3R · 2023-08-20T17:37:14Z

can you please add some examples

for the js tracing, or for the regular reth callTracer?

libevm · 2023-08-21T05:43:02Z

I'm getting the same issue, here is a concrete example using a transaction from mainnet

const { ethers } = require("ethers");
const { parseUnits } = require("ethers/lib/utils");

const provider = new ethers.providers.JsonRpcProvider();

const main = async () => {
  // Transaction is first tx at 17961011, so we will get block state at 17961010 and replay all txs
  const txHash =
    "0x4dd0d2ad98583a2a6081b6be8023029356c085d6e448a704d576787db977ee29";
  const tx = {
    from: "0xda68D25D47f353f5852138adBcb7d0b260c8c35e",
    to: "0x7a250d5630B4cF539739dF2C5dAcb4c659F2488D",
    data: "0xb6f9de9500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000080000000000000000000000000da68d25d47f353f5852138adbcb7d0b260c8c35e0000000000000000000000000000000000000000000000000000000064e2f6170000000000000000000000000000000000000000000000000000000000000002000000000000000000000000c02aaa39b223fe8d0a0e5c4f27ead9083c756cc200000000000000000000000081ce643f52c7e24c781cf45b5ade83560515f18f",
    value: parseUnits("0.1").toHexString(),
  };
  const blockNumberHex = "0x1121032"; // 17961010

  const txRecp = await provider.getTransactionReceipt(txHash);
  const traceTxRecp = await provider.send("debug_traceTransaction", [
    txHash,
    { tracer: "callTracer" },
  ]);
  const traceCallRecp = await provider.send("debug_traceCall", [
    tx,
    { blockNumber: blockNumberHex },
    { tracer: "callTracer" },
  ]);
  const estimateGasRecp = await provider.send("eth_estimateGas", [
    tx,
    blockNumberHex,
  ]);

  console.log("gas used (tx receipt)", parseInt(txRecp.gasUsed.toString(), 16));
  console.log(
    "gas used (debug_traceTransaction recp)",
    parseInt(traceTxRecp.gasUsed, 16)
  );
  console.log("gas used (debug_traceCall)", parseInt(traceCallRecp.gasUsed, 16));
  console.log("estimate gas", parseInt(estimateGasRecp, 16));
};

main();

output

> node index.js
gas used (tx receipt) 1413734
gas used (debug_traceTransaction recp) 136514
gas used (debug_traceCall) 126098
estimate gas 173833

mattsse · 2023-08-21T13:10:05Z

thanks for flagging, for calltracer, this should be fixed by #4293

0xSt1ng3R · 2023-08-21T13:45:11Z

Thank you!

LooterAI · 2023-08-30T17:35:22Z

@0xSt1ng3R were you able to verify the fix? it's still giving incorrect gasUsed for me.

0xSt1ng3R · 2023-08-30T20:31:58Z

@0xSt1ng3R were you able to verify the fix? it's still giving incorrect gasUsed for me.

from tests that I've made on the same txs/blocks from the first comment, everything works fine (on alpha 7).
tomorrow/day after I'll do tests on a much larger set (on last commit) and will update here.

mattsse · 2023-08-30T22:34:12Z

@LooterAI if you encounter discrepancies please include the version and the rpc method

LooterAI · 2023-08-31T14:17:54Z

@mattsse

const main = async () => {
  // Transaction is first tx at 17961011, so we will get block state at 17961010 and replay all txs
  const txHash =
    '0x4dd0d2ad98583a2a6081b6be8023029356c085d6e448a704d576787db977ee29';
  const tx = {
    from: '0xda68D25D47f353f5852138adBcb7d0b260c8c35e',
    to: '0x7a250d5630B4cF539739dF2C5dAcb4c659F2488D',
    data: '0xb6f9de9500000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000080000000000000000000000000da68d25d47f353f5852138adbcb7d0b260c8c35e0000000000000000000000000000000000000000000000000000000064e2f6170000000000000000000000000000000000000000000000000000000000000002000000000000000000000000c02aaa39b223fe8d0a0e5c4f27ead9083c756cc200000000000000000000000081ce643f52c7e24c781cf45b5ade83560515f18f',
    value: toHex(parseUnits('0.1')),
  };
  const blockNumberHex = '0x1121032'; // 17961010

  const txRecp = await provider.getTransactionReceipt(txHash);
  const traceTxRecp = await provider.send('debug_traceTransaction', [
    txHash,
    { tracer: 'callTracer' },
  ]);
  const debugTraceCallRecp = await provider.send('debug_traceCall', [
    tx,
    { blockNumber: blockNumberHex },
    { tracer: 'callTracer' },
  ]);
  const traceCallRecp = await provider.send('trace_call', [
    tx,
    ['trace'],
    blockNumberHex,
  ]);
  const estimateGasRecp = await provider.send('eth_estimateGas', [
    tx,
    blockNumberHex,
  ]);

  console.log('gas used (tx receipt)', parseInt(txRecp.gasUsed.toString(), 16));
  console.log(
    'gas used (debug_traceTransaction recp)',
    parseInt(traceTxRecp.gasUsed, 16),
  );
  console.log(
    'gas used (debug_traceCall)',
    parseInt(debugTraceCallRecp.gasUsed, 16),
  );
  console.log(
    'gas used (trace_call)',
    parseInt(traceCallRecp.trace[0].result.gasUsed, 16),
  );
  console.log('estimate gas', parseInt(estimateGasRecp, 16));
};

main();

Added trace_call to @libevm 's example.

gas used (tx receipt) 1413734
gas used (debug_traceTransaction recp) 159266
gas used (debug_traceCall) 148850
gas used (trace_call) 126098
estimate gas 173833

reth Version: 0.1.0-alpha.8
Commit SHA: 0f14ec40
Build Timestamp: 2023-08-29T14:48:42.653130017Z
Build Features: default,jemalloc
Build Profile: release

LooterAI · 2023-09-05T13:00:17Z

@mattsse do you know if this is going to be fixed soon? I am just trying to estimate gas usage of a tx bundle

0xSt1ng3R · 2023-09-05T16:44:41Z

@LooterAI do you get the same result with debug_traceCallMany?

mattsse · 2023-09-05T16:47:33Z

@mattsse do you know if this is going to be fixed soon? I am just trying to estimate gas usage of a tx bundle

can you please elaborate on the discrepancies, code snippet is not useful to me, json request object would be helpful.

0xSt1ng3R · 2023-09-06T15:16:27Z

another example, from latest build:

from web3 import Web3

WEB3 = Web3(Web3.HTTPProvider("http://127.0.0.1:8545", request_kwargs={'timeout': 15}))

block_number = 17678889
tx_hash = "0x9a03b506e1168bffa6595f41b0c0688ff12ee53d2f9cfe66b7ab4b57200be9a2"

data0 = WEB3.manager.request_blocking("trace_block", [hex(block_number)])
tx_to_check = [x for x in data0 if x['transactionHash'] == tx_hash][0]
gas_limit = int(tx_to_check['action']['gas'], 16) # 304,473, which is correct
gas_used = int(tx_to_check['result']['gasUsed'], 16) # 155,155, which is incorrect

data1 = WEB3.manager.request_blocking("trace_replayBlockTransactions", [hex(block_number), ["trace"]])
tx_to_check = [x for x in data1 if x['transactionHash'] == tx_hash][0]
gas_limit = int(tx_to_check['trace'][0]['action']['gas'], 16) # 304,473, which is correct
gas_used = int(tx_to_check['trace'][0]['result']['gasUsed'], 16) # 155,155, which is incorrect

(not posting here the json output because it's massive)

reth Version: 0.1.0-alpha.8
Commit SHA: 422d9309
Build Timestamp: 2023-09-06T15:01:03.365687185Z
Build Features: default,jemalloc
Build Profile: maxperf

mattsse · 2023-09-07T09:52:03Z

(not posting here the json output because it's massive)

it shouldn't be, I just need the request,
having something I can use for debugging would help a lot so I don't have to convert the python code into json manually.

0xSt1ng3R · 2023-09-07T10:27:29Z

understandable!

here are the json requests for the previous example:

curl -s http://127.0.0.1:8545 \
-X POST \
-H "Content-Type: application/json" \
--data '{"method":"trace_block","params":["0x10dc229"],"id":1,"jsonrpc":"2.0"}' \
| jq -s 'map(select(.id == 1)
           | .result[]
           | select(.transactionHash == "0x9a03b506e1168bffa6595f41b0c0688ff12ee53d2f9cfe66b7ab4b57200be9a2")
           | {gas_limit: .action.gas, gas_used: .result.gasUsed}
         ) | first'

curl -s http://127.0.0.1:8545 \
-X POST \
-H "Content-Type: application/json" \
--data '{"method":"trace_replayBlockTransactions","params":["0x10dc229", ["trace"]],"id":1,"jsonrpc":"2.0"}' \
| jq -s 'map(
           select(.id == 1)
           | .result[]
           | select(.transactionHash == "0x9a03b506e1168bffa6595f41b0c0688ff12ee53d2f9cfe66b7ab4b57200be9a2")
           | { gas_limit: .trace[0].action.gas, gas_used: .trace[0].result.gasUsed }
         ) | first'

0xSt1ng3R · 2023-09-07T10:36:50Z

thanks for the fix!
should we reindex everything, so the data about the gas in the mdbx.dat file will be correct?

mattsse · 2023-09-07T10:56:09Z

the tracing fix does not require any db changes

it only changes what values are used when re-executing the transaction to populate the trace

0xSt1ng3R · 2023-09-10T09:24:44Z

reth Version: 0.1.0-alpha.8
Commit SHA: e1d66868
Build Timestamp: 2023-09-10T09:10:59.345616204Z
Build Features: default,jemalloc
Build Profile: maxperf

@mattsse, i've tried the latest version with the tests from my previous comment:

curl -s http://127.0.0.1:8545 \
-X POST \
-H "Content-Type: application/json" \
--data '{"method":"trace_block","params":["0x10dc229"],"id":1,"jsonrpc":"2.0"}' \
| jq -s 'map(select(.id == 1)
           | .result[]
           | select(.transactionHash == "0x9a03b506e1168bffa6595f41b0c0688ff12ee53d2f9cfe66b7ab4b57200be9a2")
           | {gas_limit: .action.gas, gas_used: .result.gasUsed}
         ) | first'

here i got 155,155 gas used - which is incorrect.

on the second test:

curl -s http://127.0.0.1:8545 \
-X POST \
-H "Content-Type: application/json" \
--data '{"method":"trace_replayBlockTransactions","params":["0x10dc229", ["trace"]],"id":1,"jsonrpc":"2.0"}' \
| jq -s 'map(
           select(.id == 1)
           | .result[]
           | select(.transactionHash == "0x9a03b506e1168bffa6595f41b0c0688ff12ee53d2f9cfe66b7ab4b57200be9a2")
           | { gas_limit: .trace[0].action.gas, gas_used: .trace[0].result.gasUsed }
         ) | first'

i got 177,907, which is the correct gas used... so it looks like a partial fix?

i also tried on a newer block on the latest commit - and still got the same issue, that trace_replayBlockTransactions returns the valid gas used and trace_block not...

0xSt1ng3R · 2023-09-12T11:08:12Z

works now! thank you!

0xSt1ng3R added C-bug An unexpected or incorrect behavior S-needs-triage This issue needs to be labelled labels Aug 18, 2023

github-project-automation bot added this to Reth Tracker Aug 18, 2023

github-project-automation bot moved this to Todo in Reth Tracker Aug 18, 2023

0xSt1ng3R changed the title ~~wrong gasUsed with debug_traceCallMany/debug_traceTransaction/debug_traceBlockByNumber~~ incorrect gasUsed with debug_traceCallMany/debug_traceTransaction/debug_traceBlockByNumber Aug 18, 2023

0xSt1ng3R changed the title ~~incorrect gasUsed with debug_traceCallMany/debug_traceTransaction/debug_traceBlockByNumber~~ incorrect gasUsed with debug_traceCallMany/debug_traceTransaction/debug_traceBlockByNumber and callTracer Aug 18, 2023

mattsse added A-rpc Related to the RPC implementation and removed S-needs-triage This issue needs to be labelled labels Aug 18, 2023

mattsse self-assigned this Aug 18, 2023

This was referenced Aug 21, 2023

chore: add additional docs about root call gas limit #4292

Merged

fix: use gas_used from execution result #4293

Merged

mattsse mentioned this issue Sep 7, 2023

fix: set trace results root trace's gas used to execution results gas #4506

Merged

mattsse mentioned this issue Sep 11, 2023

fix: set trace results root trace's gas used to execution result gas #4556

Merged

mattsse closed this as completed Sep 12, 2023

github-project-automation bot moved this from Todo to Done in Reth Tracker Sep 12, 2023

Robert-MacWha mentioned this issue Apr 22, 2024

trace_call/trace_callMany report different values for gas, gasUsed, and balance compared to reference Erigon node #7785

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

incorrect gasUsed with debug_traceCallMany/debug_traceTransaction/debug_traceBlockByNumber and callTracer #4262

incorrect gasUsed with debug_traceCallMany/debug_traceTransaction/debug_traceBlockByNumber and callTracer #4262

0xSt1ng3R commented Aug 18, 2023 •

edited

Loading

0xSt1ng3R commented Aug 18, 2023 •

edited

Loading

0xSt1ng3R commented Aug 20, 2023 •

edited

Loading

mattsse commented Aug 20, 2023

0xSt1ng3R commented Aug 20, 2023 •

edited

Loading

mattsse commented Aug 20, 2023

0xSt1ng3R commented Aug 20, 2023

libevm commented Aug 21, 2023

mattsse commented Aug 21, 2023

0xSt1ng3R commented Aug 21, 2023

LooterAI commented Aug 30, 2023

0xSt1ng3R commented Aug 30, 2023

mattsse commented Aug 30, 2023

LooterAI commented Aug 31, 2023 •

edited

Loading

LooterAI commented Sep 5, 2023

0xSt1ng3R commented Sep 5, 2023

mattsse commented Sep 5, 2023 •

edited

Loading

0xSt1ng3R commented Sep 6, 2023 •

edited

Loading

mattsse commented Sep 7, 2023

0xSt1ng3R commented Sep 7, 2023

0xSt1ng3R commented Sep 7, 2023

mattsse commented Sep 7, 2023

0xSt1ng3R commented Sep 10, 2023 •

edited

Loading

0xSt1ng3R commented Sep 12, 2023

incorrect gasUsed with debug_traceCallMany/debug_traceTransaction/debug_traceBlockByNumber and callTracer #4262

incorrect gasUsed with debug_traceCallMany/debug_traceTransaction/debug_traceBlockByNumber and callTracer #4262

Comments

0xSt1ng3R commented Aug 18, 2023 • edited Loading

Describe the bug

Steps to reproduce

Node logs

Platform(s)

What version/commit are you on?

What database version are you on?

If you've built Reth from source, provide the full command you used

Code of Conduct

0xSt1ng3R commented Aug 18, 2023 • edited Loading

0xSt1ng3R commented Aug 20, 2023 • edited Loading

mattsse commented Aug 20, 2023

0xSt1ng3R commented Aug 20, 2023 • edited Loading

mattsse commented Aug 20, 2023

0xSt1ng3R commented Aug 20, 2023

libevm commented Aug 21, 2023

mattsse commented Aug 21, 2023

0xSt1ng3R commented Aug 21, 2023

LooterAI commented Aug 30, 2023

0xSt1ng3R commented Aug 30, 2023

mattsse commented Aug 30, 2023

LooterAI commented Aug 31, 2023 • edited Loading

LooterAI commented Sep 5, 2023

0xSt1ng3R commented Sep 5, 2023

mattsse commented Sep 5, 2023 • edited Loading

0xSt1ng3R commented Sep 6, 2023 • edited Loading

mattsse commented Sep 7, 2023

0xSt1ng3R commented Sep 7, 2023

0xSt1ng3R commented Sep 7, 2023

mattsse commented Sep 7, 2023

0xSt1ng3R commented Sep 10, 2023 • edited Loading

0xSt1ng3R commented Sep 12, 2023

0xSt1ng3R commented Aug 18, 2023 •

edited

Loading

0xSt1ng3R commented Aug 18, 2023 •

edited

Loading

0xSt1ng3R commented Aug 20, 2023 •

edited

Loading

0xSt1ng3R commented Aug 20, 2023 •

edited

Loading

LooterAI commented Aug 31, 2023 •

edited

Loading

mattsse commented Sep 5, 2023 •

edited

Loading

0xSt1ng3R commented Sep 6, 2023 •

edited

Loading

0xSt1ng3R commented Sep 10, 2023 •

edited

Loading