[Data Indexing] Known issue relating to indexing of internal transaction traces ("Awaiting internal transactions for status")

[arthurgousset]

Update - This incident is resolved ✅

TLDR

Note

• A temporary issue is affecting data services of explorer.celo.org such as the website, HTTP API, and GraphQL API.
• The Celo blockchain is working correctly, your funds are safe, and transactions are confirmed on the blockchain as expected.
• The issue relates to data indexing on our side and is restricted to explorer.celo.org services. As such it does not affect other data infrastructure providers such as celoscan.io, QuickNode, Infura, TheGraph, etc.
• During the incident, we recommend you use celoscan.io to double-check recent transactions and blocks on Celo mainnet and Alfajores testnet.

We apologise for the inconvenience and are working hard to bring the service back. Thank you for your patience! 🙏

More details

On Friday June 9th, the indexing engine that powers the explorer.celo.org website, HTTP API, and GraphQL API started experiencing performance issues.

Specifically, our indexing engine started falling behind on internal transaction traces, which are expensive to compute, but not ordinary transactions, which are cheaper to compute.

This means that:

1. ordinary transactions are indexed in time, specifically, as soon as the latest block is confirmed on the blockchain, and the respective transaction data available via the explorer.celo.org website and APIs is up-to-date.
2. internal transaction traces currently take longer to compute, which means that internal transaction data is delayed and temporarily missing in the explorer.celo.org website and APIs while our indexing engine is experiencing performance issues.

The symptoms of this issue manifest the following way:

• Transaction result message: (Awaiting internal transactions for status)
  • Example: explorer.celo.org/baklava/tx/0x62e25558aa6574ce...
  • Notice that the transaction is confirmed, but the transaction result is pending due to the internal transaction indexing issue.
    

You can follow the fix for this issue in this pull request:

• Resolve Blockscout live issues #918

This issue is being tracked here:

• [Live Updates] Known ongoing incident with Blockscout (Resolved) #917

[arthurgousset]

Bug reports by the community (raw)

Replicating a few bugs reported on Slack

Contract Call (Awaiting internal transactions for status)

Reported by: @ericnakagawa and @nestorbonilla

Also, is “Contract Call (Awaiting internal transactions for status)” an acceptable status?

https://explorer.celo.org/baklava/tx/0x62e25558aa6574ce6d66ce6b0250844addbd0c33d454b5d4be9f98dda694cc33

Contract Call (Awaiting internal transactions for status)

Reported by: @martinvol

i’m seeing a couple of tx in Baklava that show (Awaiting internal transactions for status), the tx also took quite a while to show up as confirmed in the explorer.

Balance inaccurate in blockchain explorer

Reported by: @ericnakagawa and @nestorbonilla

hey all Nestor Bonilla and I are encountering issues with the blockchain explorer being inaccurate.

https://explorer.celo.org/baklava/address/0x60e5d9041620C7728fF0970454F09A95d2C44dCc
balance in drop down shows 22000 cGLD, but balance on main section shows 11989… CELO

[yerdua]

Update - incident resolved

As of today (Tuesday, July 12), indexing has caught up with internal transaction tracing and is once again operating normally. We consider this issue resolved, but will continue to monitor its performance.