Streams with parallel processing, lazy filtering and random sampling

[xandkar]

TL;DR

This is a prerequisite to #1334 which also adds optimal random sampling of RocksDB and other reusable stream operations.

Summary

1. generalizes the stream type as data_stream:t/1, initially used in streaming blocks from ledger snapshot;
2. implements general sequence operations on streams:

• eager, immediately consuming the whole stream:
  • foreach
  • fold
  • sample: implementing an optimal reservoir sampling algorithm - can be used in all cases where we need to pick a random record from RocksDB;
  • pmap_to_bag: parallel maps a stream into a list with arbitrary re-order - this was used to parallelize build_hash_chain for a ~6x speedup in Parallelize build_hash_chain #1334
• lazy, deferring stream consumption until access time:
  • map
  • filter (can be used in restrict iterator_move to returning valid pubkeys erlang-libp2p#437, see: blockchain_rocks_SUITE:t_sample_filtered/1)
  • append (allowing chaining N streams into one)

3. exposes RocksDB access as a stream ( blockchain_rocks), offering all of the above operations.

Pitch

Our data access patterns have the same broad shape - stream processing; and the same specific patterns, including, but not limited to:

• side-effecting iteration
• folding
• filtering
• random sampling
• parallel processing

While we've accumulated adhoc solutions, they:

• non-essentially differ in subtle ways
• often untested
• sometimes incomplete (at least one random sampling implementation isn't guaranteed to pick and requires retries)
• impose higher than necessary cognitive load to account for particular invariants (like the error-prone juggling of RocksDB iterator handles Fix unclosed rocks iterators #1397) when modifying or composing with new needs (like filtering a stream before picking a random element: restrict iterator_move to returning valid pubkeys erlang-libp2p#437).

This PR canonicalizes a stream abstraction adaptable to any of those needs, implements the common stream processing patterns and adapts them to RocksDB, testing at every step.

Reservations

• I'm not crazy about the data_ prefix;
• it should ideally live outside of core, so it can be used by libs deeper in the dependency tree, like erlang-libp2p.

Something like erlang-hel (standing for Helium Library) can be a good home.