What conditions would cause getLogs requests to be "too broad" and "may get dropped"?

[shanefontaine]

The ethers provider.getLogs() documentation mentions the following:

Keep in mind that many backends will discard old events, and that requests which are too broad may get dropped as they require too many resources to execute the query.

I have two questions about that statement:

1. Are there any specific conditions that make a request "too broad"
2. Would the requests be dropped silently? Would an error be thrown? If n logs are returned from a single request, would n-1 logs be returned in the case of a log "drop"?

I recognize these may be questions for Infura (or another provider), so feel free to close this and I will ask there. Thank you!

[zemse]

Heya!

The "too broad" means that:

1. You specified a huge block range (imagine fromBlock: 0 and toBlock: 'latest' in a blockchain with 12 million blocks). To serve your query, the backend has to do a lot of IO. Generally, backends will want to prevent you from making them scan for lot of blocks in a single query. So they "drop" your request (or give an rpc error) if your range is over than a limit, e.g. 10k blocks.
2. Now though you specified a small range, your query results in a lot of log results (imagine DAI transfer events which I have seen avg. 50 each block, so 1000 blocks about 50k transfer events). This puts a strain on network bandwidth for the backend service. Generally, backends prevent you from making them serve more than 5k results by simply dropping the request with an rpc error.

It's great when backend gives an rpc error, at least you know it. But sometimes (#1389), the backend just "discards" some entries and returns only the latest 100 entries or so.

Does that help?

[shanefontaine]

Hi @zemse thank you for the answer!

All of that makes sense. I would like to provide more information and specifics because I think what we are dealing with, however, is not a request that is too large but may be a more underlying issue.

We run a watcher that uses ethers.js and Infura to look for an event on a specific contract on mainnet. We do it with the following ethers.js code using a JSON RPC endpoint from Infura:

    myContract.on(
        myContract.filters.MyFilter(),
        () => console.log('test')
      )

We have noticed that this code sometimes misses events in real time. We verified this by running two instances of this code -- one where myContract uses a JSON RPC provider (Infura) and one where myContract uses a WSS provider (Infura). During these tests, we would send a transaction that should fire off the transaction. We would watch that the process with the JSON RPC provider "misses" this event while the process with the WSS provider sees the event.

We are in contact with Infura and they mentioned the following:

This issue was brought up to the rest of the team and it seems like it is related to the method ethers uses for querying these events not being compatible with our system. Your method may miss some events depending on a race condition between when their “new block watcher” gets notified there is a new block vs when our event indexing system is able to provide a response to your query.

I'm not sure where the root issue may be, but I'd like to start a discussion to figure it out. I believe the fact that, according to this quote, ethers.js and Infura not "being compatible" seems like a pretty big issue. I am curious if you are aware of such a race condition mentioned in this quote.

I am in contact with Infura and would be happy to provide as much information as you may need!

[shanefontaine]

I am going to close this in favor of #1814 . Thank you!

[ricmoo]

That documentation was included purely on anecdotal evidence. I have witnessed, especially early on in the Ethereum days, that some backends would drop (prune) old state and not return some events.

Things are much better now, but still depend very much on your backend. INFURA has been very reliable in that area, but often running your own gets may result in timeouts if the range is too long. I also believe Alchemy now has better support for this, but at one point they would only honour getLogs with a range less than 1000 blocks.

But for recent block ranges, where there are a reasonable number of events (less than 10,000), I have not witnessed this.

[shanefontaine]

Thanks @ricmoo . I spoke with Infura earlier this week and they said they have identified an issue with the way that Infura interacts with ethers.js. They said they will have a PR in this repo by the end of this week.

Because it seems like they are handling it, I will now close this and #1814.

Thank you for being so responsive!

[ricmoo]

Thanks! Can you link me to that PR if you have it? I'm curious what the issue is and what their fix is. I thought ethers should handle consistency automatically. :)

[shanefontaine]

I do not believe they have submitted it yet. Here is their communication with me from an email:

We (Infura) are working with ethers to get a fix in place. We need to get a working prototype of ethers using the filter methods under the hood. We hope to get a PR opened this week.

This was in response to their previous comment:

This issue was brought up to the rest of the team and it seems like it is related to the method ethers uses for querying these events not being compatible with our system. Your method may miss some events depending on a race condition between when their “new block watcher” gets notified there is a new block vs when our event indexing system is able to provide a response to your query.

Happy to provide additional information. I'm communicating with them through a support ticket, so I don't have a super direct line of communication open with them at this time.

[ricmoo]

No worries. I can bug them too. :)

[kshinn]

@ricmoo I'm going to open a PR with a proposal for the fix. I'll reference #1814 in that PR.