Eth RPC: Address performance of EthGetTransactionCount

[maciejwitowski]

We identified EthGetTransactionCount as one of the hotspots on Glif production nodes. The goal of this issue is to investigate the reason and fix it.

[jennijuju]

@snissn i think we should look into MpoolGetNonce - which exchanges have been reporting a 6-8 sec slow response. If we can narrow down the issue there - it should improve eth get tx count too

[snissn]

i'm having trouble reproducing on master -- i'm running a lotus node and pummeling the node with the bash script below the response time is between 20 and 50 ms. Could there just be other slow endpoints causing their performance issues but eth_getTransactionCount is fast?

while true; do
        {
                time curl  http://localhost:1234/rpc/v1  -X POST -H "Content-Type: application/json" --data '{"method":"eth_getTransactionCount","params":["0x7C2540eE20773995a7304A2ab6fd785cBBef8354", "latest"],"id":1,"jsonrpc":"2.0"}'
        } 2>&1 |grep real|cut -f2 -dm |cut -f1 -ds
sleep 0.1
done

[jennijuju]

We might need to spin up a local net, then send L many messages and having them in mpool, and then call the api with the same account multiple times to repro

[snissn]

this is interesting - an LRU cache for nonce with only 256 slots in it https://github.com/filecoin-project/lotus/blob/master/chain/messagepool/messagepool.go#L374

[Stebalien]

So, the issue is that

lotus/chain/messagepool/messagepool.go

Line 1081 in 3f74840

act, err := mp.api.GetActorAfter(addr, ts)

computes the tipset under a lock.

The answer is to not call GetActorAfter. Instead, we need to look at the tipset iself:

1. Check if the nonce is cached.
2. If not, load the tipset messages, look for the relevent message, and fill the cache with all nonces found.
3. Check the state before the tipset (i.e., GetActor, not GetActorAfter).
4. Update the cache.

At the same time, we'll definitely want to increase the size of this cache. Also, for the current head, we could pre-cache in HeadChange. We already process all messages to update the message pool, so this operation should be basically free.

[maciejwitowski]

Thanks for coming up with this @Stebalien! ❤️

[Stebalien]

I've tested this with @snissn's test above and it pauses for 10 seconds every so often, so it's clearly related to tipset computation.

[maciejwitowski]

Deploying to Glif canary nodes today to get production data.