Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for the async iterators #21

Merged
merged 6 commits into from
Sep 1, 2023

Conversation

nodech
Copy link
Member

@nodech nodech commented Aug 15, 2023

Current implementation of the iterators is not flexible. There are each, range, keys and values.
range, keys and values - all accumulate the results and return it like that. Given the use cases in the bcoin and hsd - this is never a good idea. Some entries in db if they can't be range filtered (gte/lte) properly and used w/o limits can result in huge memory usage and also double processing in most of them time (stored in array -> filter/modify -> etc.).

each allows to work on each while they are received from the database, but it's mostly still accumulated as you would need to setup callback chain in order to use it properly.

This implementation of the iterators instead uses for await () which allows for the chaining database results to the codebase. Instead of consuming the data from db at once and constructing array to use, we can chain asyng generators if we want to have transformations which can reduce the amount of information we store in memory.

Iterators, DB and bucket now have additional methods for the async generators:

  • entriesAsync - returns an async iterator that yields IteratorItems with .key and .value
  • valuesAsync - return an async iterator that yields values. NOTE: This needs values: true in the iterator configs.
  • keysAsync - return an async iterator that yields keys.

Examples:

Best translation of this concept is using each:

await iter.each(async (key, value) => {
      //work with key/values
});

Which can be now written as:

for await (const {key, value} of iter) {
   // work with key/values
}

In case where we want to chain multiple calls it becomes more complicated than just using arrays, but as I mentioned above - accumulating arrays first comes with huge cost for the big queries.

Example of chaining these calls:

const filter = async function*(iter) {
  for await (const item of iter) {
    if (item[0] === 0x8b)
      yield item;

    continue;
  }
};

const values = [];

for await (const value of filter(valuesIter))
  values.push(value);

@nodech nodech merged commit c4d4fad into bcoin-org:master Sep 1, 2023
12 checks passed
@nodech nodech deleted the async-iterator branch September 1, 2023 12:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant