-
Notifications
You must be signed in to change notification settings - Fork 919
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FN/BN CPU BBQ #3630
Comments
The implementation for the dummy solution is quite straighforward. Here instead of doing the expensive checks, we simply return the data size, which in our case is constant. Intuitively, returning the completely arbritrary or approximate size there should work, but to minimize risks we could still return the factual sizes. There are two types of data we serve there. Leaf and inner nodes with their respective sizes defined here. Detecting which type we are requested with is tricky and might not be possible. Quic look into the old protocol made me realize that we don't have different CID codecs for leaf or inner node requests and we unified them to be under the same code. If we won't find a way to determine which type of request that is by simply looking at the CID, we should try returning the max size, which is the size of leaf or the share. |
Related issue that would simplify our life: ipfs/boxo#657 But the proposed implementation above is not blocked on that and we can adopt the change in boxo once it gets shipped |
Overview
Recent spikes in network activity increased resource consumption for FN/BN node operators to inadequate levels.
Problem Description
The flamegraph shown below is from the profile captured at the exact moment where such a spike happened.
The profile suggest that the most of the time spent happens during
GetSize
operation on theBlockstore
. FN/BN usesGetSize
to determine whether it has data or not. This check is triggered by remote Bitswap requests from LNs. Before requesting data itself, LNs check which peers have the data by broadcasting these requests to all connected peers(in worst case). Once they know who has the data, they request a particular peer for the data.As it turns out, checking existence of data is an expensive operation which is as expensive as getting the data itself. This comes from the inverted_index the main bottleneck of the celestia-node(at the moment of writing). This index is maintained within KVStore that is enormous and simple lookup operations are the source of the CPU overhead we seeing.
Overall, we have LNs broadcasting data checks for every sample operations, which, in fact, consists of multiple requests where each of those multiple requests get duplicated to every connected peer in the worst case, creating massive load on the network.
Potential Solutions
DataRoot/Hash
routing FN to respective EDS CAR file.Decision
After internal discussion we decided to go with "Dummy Solution". It solves the problem in a simple and fast way and tradeoffs it brings are negligible.
We may still do additionally the solution that removes inverted index entirely to win more time if Shwap gets delayed or if there is more pressure on achieving bigger blocks that inverted index is blocking as well
The text was updated successfully, but these errors were encountered: