core/bloombits, eth/filter: transformed bloom bitmap based log search #14631

zsfelfoldi · 2017-06-16T01:51:25Z

This PR is based on #14522 and supersedes #3749 which was the same thing but based on the old version of the chain processor. Both versions are kept until one of them is merged.

Further parts of the code may be moved into separate PRs to make the review process easier (suggestions are welcome).

This PR optimizes log searching by creating a data structure (BloomBits) that makes it cheaper to retrieve bloom filter data relevant to a specific filter. When searching in a long section of the block history, we are checking three specific bits of each bloom filter per address/topic. In order to do that, currently we read/retrieve a cca. 500 byte block header for each block. The implemented structure optimizes this by a "bitwise 90 degree rotation" of the bloom filters. Blocks are grouped into sections (SectionSize is 4096 blocks at the moment), BloomBits[bitIdx][sectionIdx] is a 4096 bit (512 byte) long bit vector that contains a single bit of each bloom filter from the block range [sectionIdx*SectionSize ... (sectionIdx+1)*SectionSize-1]. (Since bloom filters are usually sparse, a simple data compression makes this structure even more efficient, especially for ODR retrieval.) By reading and binary AND-ing three BloomBits sections, we can filter for an address/topic in 4096 blocks at once ("1" bits in the binary AND result mean bloom matches).

Implementation and design rationale of the matcher logic

Pipelined structure

The matcher was designed with the needs of both full and light nodes in mind. A simpler architecture would probably be satisfactory for full nodes (where the bit vectors are available in the local database) but the network retrieval bottleneck of light clients justifies a more sophisticated algorithm that tries to minimize the amount of retrieved data and return results as soon as possible. The current implementation is a pipelined structure based on input and output channels (receiving section indexes and sending potential matches). The matcher is built from sub-matchers, one for the addresses and one for each topic group. Since we are interested in matches that each sub-matcher signals as positive, they are daisy-chained in a way that subsequent sub-matchers are only retrieving and matching the bit vectors of sections where the previous matchers have found a potential match. The "1" bits of the output of the last sub-matcher are returned as bloom filter matches.
Sub-matchers use a set of fetchers to retrieve a stream of bit vectors belonging to certain bloom filter bit indexes. Fetchers are also pipelined, receiving a stream of section indexes and sending a stream of bit vectors. As soon as each fetcher returned the bit vector belonging to a certain section index, the sub-matcher performs the necessary binary AND and OR operations and outputs the resulting vector and its belonging section index if the vector is not made of zeroes only.

Prioritizing and batching requests

Light clients retrieve the bit vectors with merkle proofs, which makes it much more efficient to retrieve batches of vectors (whose merkle proofs share most of their trie nodes) in a single request. Also, it is preferable to prioritize requests based on their section index (regardless of bit index) in order to ensure that matches are found and returned as soon as possible (and in a sequential order). Prioritizing and batching are realized by a common request distributor that receives individual bit index/section index requests from fetchers and keeps an ordered list of section indexes to be requested, grouped by bit index. It does not call any retrieval backend function but it is called by a "server" process (see serveMatcher in filter.go). NextRequest returns the next batch to be requested, retrieved vectors are returned through the Deliver function. This method ensures that the bloombits package should only care about implementing the matching logic. The caller can retain full control over the resources (CPU/disk/network bandwidth) assigned to this task.

Arachnid

I don't think I fully understand the operation of the various goroutines here. Could we schedule a call to discuss and review interactively?

Arachnid · 2017-07-06T10:38:48Z

core/bloombits/matcher.go

+ reqLock sync.RWMutex
+}
+
+type req struct {


Can you decompress these identifiers a bit? bitIndex, requestMap, request, etc.

Arachnid · 2017-07-06T10:40:22Z

core/bloombits/matcher.go

+func (f *fetcher) fetch(sectionCh chan uint64, distCh chan distReq, stop chan struct{}, wg *sync.WaitGroup) chan []byte {
+ dataCh := make(chan []byte, channelCap)
+ returnCh := make(chan uint64, channelCap)
+ wg.Add(2)


Arachnid · 2017-07-06T10:44:55Z

core/bloombits/matcher.go

+ returnCh := make(chan uint64, channelCap)
+ wg.Add(2)
+
+ go func() {


Why two separate goroutines? Can't these be combined?

Arachnid · 2017-07-06T10:45:16Z

core/bloombits/matcher.go

+ for i, idx := range sectionIdxList {
+ r := f.reqMap[idx]
+ if r.data != nil {
+ panic("BloomBits section data delivered twice")


Is this really a cause for panic?

Arachnid · 2017-07-06T11:18:18Z

core/bloombits/matcher.go

+ // set up fetchers
+ fetchIdx := make([][3]chan uint64, len(idxs))
+ fetchData := make([][3]chan []byte, len(idxs))
+ for i, idx := range idxs {


Can you give 'idxs' and 'ii' more descriptive names?

Arachnid · 2017-07-06T11:19:37Z

core/bloombits/matcher.go

+ return
+ }
+
+ for _, ff := range fetchIdx {


These variable names are also pretty hard to follow.

Arachnid · 2017-07-06T11:20:33Z

core/bloombits/matcher.go

+
+ m.wg.Add(2)
+ // goroutine for starting retrievals
+ go func() {


It's unclear to me why there need to be two independent goroutines here, or how they work together. Can you elaborate?

Arachnid · 2017-07-06T11:23:29Z

core/bloombits/matcher.go

+func (m *Matcher) distributeRequests(stop chan struct{}) {
+ m.distWg.Add(1)
+ stopDist := make(chan struct{})
+ go func() {


Is this really necessary?

karalabe · 2017-09-04T09:40:37Z

I am now satisfied with this PR, though given that I added heavy reworks to it, I cannot mark it as reviewed. @zsfelfoldi Please double check that the code still makes sense from your perspective. @fjl or @Arachnid Please do a final review.

Arachnid

Generally looks good, but I'm concerned the code to retrieve data from disk is massively overengineered.

Arachnid · 2017-09-05T10:35:33Z

core/bloombits/generator.go

+
+// AddBloom takes a single bloom filter and sets the corresponding bit column
+// in memory accordingly.
+func (b *Generator) AddBloom(bloom types.Bloom) error {


I'm a little concerned that the implied state here (which bit is being set) could lead to something becoming invisibly out of sync. Could this take the bit being set as a parameter and error if it's not the one it expects?

you're right, done

Arachnid · 2017-09-05T10:36:01Z

core/bloombits/generator.go

+ bitMask := byte(1) << byte(7-b.nextBit%8)
+
+ for i := 0; i < types.BloomBitLength; i++ {
+ bloomByteMask := types.BloomByteLength - 1 - i/8


This is an index, not a mask, isn't it?

it is, and also byteMask is. fixed.

Arachnid · 2017-09-05T10:39:24Z

core/bloombits/matcher.go

+ b = crypto.Keccak256(b)
+
+ var idxs bloomIndexes
+ for i := 0; i < len(idxs); i++ {


There's a lot of magic numbers here. Could they be made into constants?

These are kind of speced by the yellow paper on how to construct the header bloom filters. I'm unsure if it makes sense to separate them out since there's not much room to reuse them elsewhere. But I'm open to suggestions.

Arachnid · 2017-09-05T10:42:58Z

core/bloombits/matcher.go

+type Matcher struct {
+ sectionSize uint64 // Size of the data batches to filter on
+
+ addresses []bloomIndexes // Addresses the system is filtering for


Wouldn't it be cleaner to just accept a list of byte slices, and leave the address/topic distinction to the caller?

API wise within NewMatcher I think it might be nice thing to support the filter by being explicitly catered for that use case. Internally, I'm unsure if we need to retain this distinction. @zsfelfoldi ?

Even in NewMatcher, this seems to me like it shouldn't care about what it's matching on - they should just be opaque hashes to it.

I changed it as @Arachnid suggested in the last commit, now we have to convert addresses and hashes to byte slices in filter.go but maybe it's still nicer this way. If you don't like it, we can drop the last commit.

Arachnid · 2017-09-05T10:45:46Z

core/bloombits/matcher.go

+ // Iterate over all the blocks in the section and return the matching ones
+ for i := first; i <= last; i++ {
+ // If the bitset is nil, we're a special match-all cornercase
+ if res.bitset == nil {


Why the corner case? Why not just return a fully set bitset?

Fair enough.

Because that's what comes from the source: https://github.com/zsfelfoldi/go-ethereum/blob/bloombits2/core/bloombits/matcher.go#L229
I don't want to do unnecessary binary ANDs every time in the first subMatcher so I'm sending nils at the source. But in the corner case there are no subMatchers at all (we are receiving the source at the end) so we have to do some special case handling anyway. We can do it some other way but I'm not sure it's going to be any nicer. But I'm open to suggestions.

I've added a commit to always send 0xff-s in the first instance. I don't think doing a binary and on a few bytes will matter given that the data comes from disk or the network.

Arachnid · 2017-09-05T10:55:11Z

core/bloombits/matcher.go

+
+// distributor receives requests from the schedulers and queues them into a set
+// of pending requests, which are assigned to retrievers wanting to fulfil them.
+func (m *Matcher) distributor(dist chan *request, session *MatcherSession) {


Does having multiple parallel retrievers help performance? I wonder if it wouldn't be far simpler to just have each filter fetch data as it needs it, with a cache to prevent duplicate reads?

I think the complexity is to aid the light client which needs to batch multiple requests together and send them out multiple batches to different light servers. Hence why the whole distribution "mess".

Ah, good point. Objection withdrawn, then.

karalabe · 2017-09-06T08:16:51Z

@Arachnid Issues should be addresses now, PTAL

zsfelfoldi requested review from fjl, Arachnid and karalabe June 16, 2017 01:52

zsfelfoldi added review and removed review labels Jun 16, 2017

zsfelfoldi force-pushed the bloombits2 branch from 738d0f2 to 47740ef Compare July 4, 2017 22:30

ethereum deleted a comment from GitCop Jul 4, 2017

zsfelfoldi added review and removed in progress labels Jul 4, 2017

Arachnid reviewed Jul 6, 2017

View reviewed changes

zsfelfoldi force-pushed the bloombits2 branch from c2e6acc to 69049b0 Compare August 4, 2017 02:23

zsfelfoldi added in progress and removed review labels Aug 8, 2017

zsfelfoldi force-pushed the bloombits2 branch 2 times, most recently from 7fae8bc to ef18689 Compare August 9, 2017 14:24

zsfelfoldi added review and removed in progress labels Aug 9, 2017

zsfelfoldi force-pushed the bloombits2 branch from 82bc7db to 1e27517 Compare August 18, 2017 20:34

ethereum deleted a comment from GitCop Aug 18, 2017

zsfelfoldi force-pushed the bloombits2 branch 3 times, most recently from 5f9be14 to 900d968 Compare August 19, 2017 17:44

karalabe force-pushed the bloombits2 branch 2 times, most recently from b37d46f to 06ca7f7 Compare September 4, 2017 09:24

karalabe force-pushed the bloombits2 branch from 06ca7f7 to b6b78f4 Compare September 4, 2017 09:44

Arachnid reviewed Sep 5, 2017

View reviewed changes

karalabe mentioned this pull request Sep 5, 2017

Event filtering is slow #15091

Closed

zsfelfoldi and others added 5 commits September 6, 2017 11:13

core, eth: add bloombit indexer, filter based on it

4ea4d2d

core, eth: clean up bloom filtering, add some tests

f585f9e

core/bloombits: AddBloom index parameter and fixes variable names

6ff2c02

core/bloombits: use general filters instead of addresses and topics

451ffdb

core/bloombits: drop nil-matcher special case

564c8f3

karalabe force-pushed the bloombits2 branch from fd53a1e to 564c8f3 Compare September 6, 2017 08:15

karalabe added this to the 1.7.0 milestone Sep 6, 2017

Arachnid approved these changes Sep 6, 2017

View reviewed changes

karalabe merged commit c4d21bc into ethereum:master Sep 6, 2017

sirnicolas21 mentioned this pull request Sep 8, 2017

Clique : Discarded bad propagated block#1 when syncing #14945

Closed

holiman mentioned this pull request Sep 21, 2022

core/bloombits: speed up windows-test #25844

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core/bloombits, eth/filter: transformed bloom bitmap based log search #14631

core/bloombits, eth/filter: transformed bloom bitmap based log search #14631

zsfelfoldi commented Jun 16, 2017

Arachnid left a comment

Arachnid Jul 6, 2017

Arachnid Jul 6, 2017

Arachnid Jul 6, 2017

Arachnid Jul 6, 2017

Arachnid Jul 6, 2017

Arachnid Jul 6, 2017

Arachnid Jul 6, 2017

Arachnid Jul 6, 2017

karalabe commented Sep 4, 2017

Arachnid left a comment

Arachnid Sep 5, 2017

zsfelfoldi Sep 6, 2017

Arachnid Sep 5, 2017

zsfelfoldi Sep 6, 2017

Arachnid Sep 5, 2017

karalabe Sep 5, 2017

Arachnid Sep 5, 2017

karalabe Sep 5, 2017

Arachnid Sep 5, 2017

zsfelfoldi Sep 6, 2017

Arachnid Sep 5, 2017

karalabe Sep 5, 2017

zsfelfoldi Sep 6, 2017

karalabe Sep 6, 2017

Arachnid Sep 5, 2017

karalabe Sep 5, 2017

Arachnid Sep 5, 2017

karalabe commented Sep 6, 2017

core/bloombits, eth/filter: transformed bloom bitmap based log search #14631

core/bloombits, eth/filter: transformed bloom bitmap based log search #14631

Conversation

zsfelfoldi commented Jun 16, 2017

Arachnid left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karalabe commented Sep 4, 2017

Arachnid left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karalabe commented Sep 6, 2017