Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

failed to get logs for block #7950923 (0xff68a7..396b6d) on long running node #30470

Open
shammahalchemy opened this issue Sep 20, 2024 · 11 comments
Labels

Comments

@shammahalchemy
Copy link

shammahalchemy commented Sep 20, 2024

System information

Geth version: ethereum/client-go:v1.14.5
CL client & version: prysmaticlabs/prysm-beacon-chain:v5.0.4
OS & Version: Linux

Expected behaviour

Output block logs.

Actual behaviour

{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"failed to get logs for block #7950923 (0xff68a7..396b6d)"}}

Node logs output the following:

ERROR[09-20|00:08:54.573] Invalid receipt array RLP                hash=ff68a7..396b6d err="rlp: expected input list for []*types.Log, decoding into ([]*rawdb.receiptLogs)[0](rawdb.storedReceiptRLP).Logs"
WARN [09-20|00:08:54.573] Served eth_getLogs                       conn=127.0.0.1:37962 reqid=1 duration="333.35µs" err="failed to get logs for block #7950923 (0xff68a7..396b6d)"

It seems to be a similar issue to this: #25809

Steps to reproduce the behaviour

curl --location 'localhost:8545' --header 'Content-Type: application/json' --data '{
    "method": "eth_getLogs",
    "params": [
        {
            "blockHash": "0xff68a7848d2ca566f2f3770001bc97858a747595952b26b962f37a3fe9396b6d"
        }
    ],
    "id": 1,
    "jsonrpc": "2.0"
}'
@s1na
Copy link
Contributor

s1na commented Oct 1, 2024

Can you please try eth_getBlockReceipts(hash) and report the result?

@shammahalchemy
Copy link
Author

Sure thing.

curl localhost:8545 --header 'Content-Type: application/json' --data '{
    "method": "eth_getBlockReceipts",
    "params": [
        {
            "blockHash": "0x2acd4a70f418203eb43b3731a4fe96927e1c109a6a8e2c3d8d21d6d9af8f66db"
        }
    ],
    "id": 1,
    "jsonrpc": "2.0"
}'
{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"receipts length mismatch: 3 vs 0"}}

@s1na
Copy link
Contributor

s1na commented Oct 3, 2024

It would be helpful to see what is exactly persisted. Since it is an old block it is stored in the freezer and not kvdb.

Even though we have a method debug_dbGet for reading values from the kvdb we don't have something for reading freezer values. Once we have this method we can make progress on debugging the issue.

@s1na
Copy link
Contributor

s1na commented Oct 3, 2024

Ah pardon turns out we do have a method for that. Can you please try calling this? you will have to enable the debug namespace (via --http.api "eth,debug"):

debug_dbAncient("receipts", 1022389)

This is the value I'm getting on my node:

0xf872e5a0ceabd1a5ecc5ddda016deb638f0609dd7ad3254b2ff4117cfeb062ff7b6e01fb825208c0e5a0b5ff8dd2a923204fe1bc9c9875518f139acca756a79139cae6907d0582b40a7f82a410c0e5a03530d12c12f5b965546f432d4fe9ec388ecd22e161bb84089e81e4f0a6b9f52182f618c0

@MariusVanDerWijden
Copy link
Member

I think the issue is that we might have at some point changed the way we store receipts in the database.
What was the first version that you ran this node on? Or how long have you been running this node?

@shammahalchemy
Copy link
Author

These nodes have been running for quite a long time. I guess we'll need to do a resync?

@karalabe
Copy link
Member

karalabe commented Oct 8, 2024

@shammahalchemy Could you please check this comment: #30470 (comment) ?

@fjl
Copy link
Contributor

fjl commented Oct 8, 2024

@shammahalchemy Please run this to retrieve the raw value:

curl localhost:8545 --header 'Content-Type: application/json' --data '{
    "method": "debug_dbAncient",
    "params": [ "receipts", 1022389 ],
    "id": 1,
    "jsonrpc": "2.0"
}'

@shammahalchemy
Copy link
Author

@fjl sure thing. Thank you for looking.

{"jsonrpc":"2.0","id":1,"result":"0xf9042cf90161a0ceabd1a5ecc5ddda016deb638f0609dd7ad3254b2ff4117cfeb062ff7b6e01fb825208b9010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0785afabffc4683bc75cd796dcd00468c451dbacdaf9abaff9d2ada054c6f7c2c940000000000000000000000000000000000000000c0825208f90161a0b5ff8dd2a923204fe1bc9c9875518f139acca756a79139cae6907d0582b40a7f82a410b9010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0b3366c8a993f2bd20ebcc5c0337ae39cce2e06b92cd208480cccd32e5ea4ecd1940000000000000000000000000000000000000000c0825208f90161a03530d12c12f5b965546f432d4fe9ec388ecd22e161bb84089e81e4f0a6b9f52182f618b9010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a007a3187b343af8d2bdfcdbed2cbe095daeb43eeae2cc4d3172aa678e1d220fa5940000000000000000000000000000000000000000c0825208"}

@s1na
Copy link
Contributor

s1na commented Oct 15, 2024

rlpdump:

[
  [
    ceabd1a5ecc5ddda016deb638f0609dd7ad3254b2ff4117cfeb062ff7b6e01fb,
    5208,
    00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000,
    785afabffc4683bc75cd796dcd00468c451dbacdaf9abaff9d2ada054c6f7c2c,
    0000000000000000000000000000000000000000,
    [],
    5208,
  ],
  [
    b5ff8dd2a923204fe1bc9c9875518f139acca756a79139cae6907d0582b40a7f,
    a410,
    00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000,
    b3366c8a993f2bd20ebcc5c0337ae39cce2e06b92cd208480cccd32e5ea4ecd1,
    0000000000000000000000000000000000000000,
    [],
    5208,
  ],
  [
    3530d12c12f5b965546f432d4fe9ec388ecd22e161bb84089e81e4f0a6b9f521,
    f618,
    00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000,
    07a3187b343af8d2bdfcdbed2cbe095daeb43eeae2cc4d3172aa678e1d220fa5,
    0000000000000000000000000000000000000000,
    [],
    5208,
  ],
]

Vs the correct dump:

[
  [
    ceabd1a5ecc5ddda016deb638f0609dd7ad3254b2ff4117cfeb062ff7b6e01fb,
    5208,
    [],
  ],
  [
    b5ff8dd2a923204fe1bc9c9875518f139acca756a79139cae6907d0582b40a7f,
    a410,
    [],
  ],
  [
    3530d12c12f5b965546f432d4fe9ec388ecd22e161bb84089e81e4f0a6b9f521,
    f618,
    [],
  ],
]

Yeah Marius was right this is a legacy receipt type. @shammahalchemy were you using --ignore-legacy-receipts? We even dropped that flag. I'm not sure how geth was starting for you 2 years ago when we were urging users to migrate from the legacy format (context: #26225). I am really puzzled how there is still a node with that format.

@fjl fjl removed the status:triage label Oct 24, 2024
@fjl fjl closed this as completed Oct 24, 2024
@fjl fjl reopened this Oct 24, 2024
@fjl
Copy link
Contributor

fjl commented Oct 24, 2024

There is no conversion path in the code for legacy receipts anymore. So the only option is resync without ancients.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants
@fjl @karalabe @s1na @MariusVanDerWijden @shammahalchemy and others