Node Issue: view_account query returns INTERNAL_ERROR : MissingTrieValue

[ella-quicknode]

Node type

RPC

Which network are you running?

mainnet

What happened?

request payload:

{
  "id": "dontcare",
  "jsonrpc": "2.0",
  "method": "query",
  "params": {
    "account_id": "b542afaa8a53b4605ca3f6ed94eb333db3372a105d6811f30561a4c897012ae4",
    "block_id": 116401567,
    "request_type": "view_account"
  }
}

This account_id became only available on a higher block number 116401580 , so instead of giving UNKNOWN_ACCOUNT it's returning below error.

{
  "jsonrpc": "2.0",
  "error": {
    "name": "INTERNAL_ERROR",
    "cause": {
      "info": {
        "error_message": "MissingTrieValue(TrieStorage, Cxcr2sH2BRqm3m6rt3kqfkmjJSkd4geCjYH39MKqsBAp)"
      },
      "name": "INTERNAL_ERROR"
    },
    "code": -32000,
    "message": "Server error",
    "data": "The node reached its limits. Try again later. More details: MissingTrieValue(TrieStorage, Cxcr2sH2BRqm3m6rt3kqfkmjJSkd4geCjYH39MKqsBAp)"
  },
  "id": "dontcare"
}

Version

neard (release 2.2.1) (build 2.2.1) (rustc 1.79.0) (protocol 71) (db 40)
features: [default, json_rpc, rosetta_rpc]

Node upgrade history

upgraded to nearcore v2.2.1 on 2024-09-17 from older v2.2.0

[telezhnaya]

Which RPC endpoint do you use? Do you run your own node?

[telezhnaya]

This account_id became only available on a higher block number 116401580

This is fine, the receipt with transfer (and creating the account) was executed only at block 116401580, it was a delayed receipt.

instead of giving UNKNOWN_ACCOUNT it's returning [MissingTrieValue]

I think it can be a consequence of the delayed receipt

[telezhnaya]

From the user:

I downloaded the latest archive snapshot (2024-10-18T23:17:22Z) but still got MissingTrieValue error while the public archive endpoint can return the result.

They've also tried the instruction https://near.github.io/nearcore/misc/archival_data_recovery.html
And it does not help as well

[qnduncan]

Another error from an archive node spawned with the latest archive snapshot (2024-10-18T23:17:22Z).

Req:

{
  "id": "dontcare", 
  "jsonrpc": "2.0", 
  "method": "query", 
  "params": {
    "account_id": "01.near", 
    "block_id": 115514400, 
    "request_type": "view_account"}
}

Response:

{
  "jsonrpc": "2.0",
  "error": {
    "name": "INTERNAL_ERROR",
    "cause": {
      "info": {
        "error_message": "MissingTrieValue(TrieStorage, 4gbW1Y55ApFiPWzGWxvmLZwFkSsfJ7WRUEMTkyyBRa8S)"
      },
      "name": "INTERNAL_ERROR"
    },
    "code": -32000,
    "message": "Server error",
    "data": "The node reached its limits. Try again later. More details: MissingTrieValue(TrieStorage, 4gbW1Y55ApFiPWzGWxvmLZwFkSsfJ7WRUEMTkyyBRa8S)"
  },
  "id": "dontcare"
}

[andrei-near]

Hello,

We will need to adjust a bit the documentation to reflect correct archival download links.
Correct commands to run for Archival Split Storage Data Download:

chain="mainnet"  # or "testnet"
kind="archive"       
rclone copy --no-check-certificate near_cf://near-protocol-public/backups/${chain:?}/${kind:?}/latest_split_storage ./
latest=$(cat latest_split_storage)
rclone copy --no-check-certificate --progress --transfers=6 --checkers=6 \
  near_cf://near-protocol-public/backups/${chain:?}/${kind:?}/${latest:?}/hot-data ~/.near/hot-data
rclone copy --no-check-certificate --progress --transfers=6 --checkers=6 \
  near_cf://near-protocol-public/backups/${chain:?}/${kind:?}/${latest:?}/cold-data ~/.near/cold-data

The difference is that archival DB is now split into 2 databases to allow usage of cheaper disks for cold data.

P.S cold storage size is about 50TB at the moment so it will take a while to download

[qnduncan]

Hi @andrei-near,

I indeed used the split storage setting and downloaded cold data and hot data (2024-10-18T23:17:22Z). This is the doc I followed. The MissingTrieValue error persists.

Could you please double-check if the error can be reproduced on the machine used to create archive snapshots?

[andrei-near]

So yes, it is a known issue which we already fixed. Recovered DB is ready and successfully ran your queries there:

curl -X POST -H "Content-Type: application/json" -d '{
  "id": "dontcare",
  "jsonrpc": "2.0",
  "method": "query",
  "params": {
    "account_id": "b542afaa8a53b4605ca3f6ed94eb333db3372a105d6811f30561a4c897012ae4",
    "block_id": 116401567,
    "request_type": "view_account"
  }
}' recovery_db_node:3030
{"jsonrpc":"2.0","error":{"name":"HANDLER_ERROR","cause":{"info":{"block_hash":"AGBGuPwgFRZUQpuNCAFkBh5zG8Njr1ERsRLFGiRyVC53","block_height":116401567,"requested_account_id":"b542afaa8a53b4605ca3f6ed94eb333db3372a105d6811f30561a4c897012ae4"},"name":"UNKNOWN_ACCOUNT"},"code":-32000,"message":"Server error","data":"account b542afaa8a53b4605ca3f6ed94eb333db3372a105d6811f30561a4c897012ae4 does not exist while viewing"},"id":"dontcare"

and

curl -X POST -H "Content-Type: application/json" -d '{
  "id": "dontcare",
  "jsonrpc": "2.0",
  "method": "query",
  "params": {
    "account_id": "01.near",
    "block_id": 115514400,
    "request_type": "view_account"}
}' recovery_db_node:3030
{"jsonrpc":"2.0","result":{"amount":"4931591222539575464438358","block_hash":"G5wFX4GpZabeu7vUeDtiEf4gDsU7ufFjRE4ubQyC6L87","block_height":115514400,"code_hash":"11111111111111111111111111111111","locked":"0","storage_paid_at":0,"storage_usage":400},"id":"dontcare"}

Snapshots update with this data will be done after 2.3.0 is released on mainnet.

[qnduncan]

Thanks for the info. I'll follow up #11895. 🙏