Skip to content
This repository has been archived by the owner on Mar 5, 2024. It is now read-only.

Streams with parallel processing, lazy filtering and random sampling #1389

Open
wants to merge 25 commits into
base: master
Choose a base branch
from
Open

Streams with parallel processing, lazy filtering and random sampling #1389

wants to merge 25 commits into from

Commits on Jun 8, 2022

  1. Implement stream with a parallel map to a bag

    @xandkar
    xandkar committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    8e01fb7 View commit details
    Browse the repository at this point in the history

  2. Use more-intuitive field names

    @xandkar
    xandkar committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    d093d91 View commit details
    Browse the repository at this point in the history

  3. Note drawback of current scheduling strategy in pmap_to_bag

    @xandkar
    xandkar committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    014024e View commit details
    Browse the repository at this point in the history

  4. Complete the sequence operations on streams

    @xandkar
    xandkar committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    c5fff4f View commit details
    Browse the repository at this point in the history

  5. Switch snap bin list streaming to the general stream

    @xandkar
    xandkar committed Jun 8, 2022
    Configuration menu
    Copy the full SHA
    87906a3 View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2022

  1. Implement random elements selection from a stream 

    via the optimal reservoir sampling algorithm.
    @xandkar
    xandkar committed Jun 13, 2022
    Configuration menu
    Copy the full SHA
    55d3486 View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2022

  1. Rename random_elements/2 to sample/2

    @xandkar
    xandkar committed Jun 17, 2022
    Configuration menu
    Copy the full SHA
    bd11ffa View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2022

  1. Implement and test RocksDB facade with streams and random sampling

    @xandkar
    xandkar committed Jun 18, 2022
    Configuration menu
    Copy the full SHA
    1aa82af View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2022

  1. Extend and tidy-up rocks tests

    @xandkar
    xandkar committed Jun 19, 2022
    Configuration menu
    Copy the full SHA
    8acb284 View commit details
    Browse the repository at this point in the history

  2. Remove read_options() from blockchain_rocks API

    @xandkar
    xandkar committed Jun 19, 2022
    Configuration menu
    Copy the full SHA
    4977566 View commit details
    Browse the repository at this point in the history

  3. Implement blockchain_rocks:fold 

    in terms of data_stream:fold
    @xandkar
    xandkar committed Jun 19, 2022
    Configuration menu
    Copy the full SHA
    5797695 View commit details
    Browse the repository at this point in the history

  4. Rename data_stream:iter to foreach

    @xandkar
    xandkar committed Jun 19, 2022
    Configuration menu
    Copy the full SHA
    644450c View commit details
    Browse the repository at this point in the history

  5. Normalize t-first position for streams

    @xandkar
    xandkar committed Jun 19, 2022
    Configuration menu
    Copy the full SHA
    a34fa6c View commit details
    Browse the repository at this point in the history

  6. Create map and filter aliases without the lazy_ prefix

    @xandkar
    xandkar committed Jun 19, 2022
    Configuration menu
    Copy the full SHA
    3aa4ac4 View commit details
    Browse the repository at this point in the history

  7. Remove direct implementation of blockchain_rocks:foreach 

    and replace with data_stream:foreach
    @xandkar
    xandkar committed Jun 19, 2022
    Configuration menu
    Copy the full SHA
    7ff843d View commit details
    Browse the repository at this point in the history

  8. Fix blockchain_rocks:fold signature

    @xandkar
    xandkar committed Jun 19, 2022
    Configuration menu
    Copy the full SHA
    f25c8e6 View commit details
    Browse the repository at this point in the history

  9. Add the CF variant of blockchain_rocks:foreach

    @xandkar
    xandkar committed Jun 19, 2022
    Configuration menu
    Copy the full SHA
    6227eab View commit details
    Browse the repository at this point in the history

  10. Improve rocks tests: 

    - remove superfluous checks
- increase DB size
- use more-descriptive DB name
    @xandkar
    xandkar committed Jun 19, 2022
    Configuration menu
    Copy the full SHA
    5c5a33c View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2022

  1. Re-expose read_options()

    @xandkar
    xandkar committed Jun 20, 2022
    Configuration menu
    Copy the full SHA
    da86996 View commit details
    Browse the repository at this point in the history

  2. Assert sample size

    @xandkar
    xandkar committed Jun 20, 2022
    Configuration menu
    Copy the full SHA
    44296f3 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2022

  1. Fix typo and clarify

    @xandkar
    xandkar committed Jun 21, 2022
    Configuration menu
    Copy the full SHA
    6a2579e View commit details
    Browse the repository at this point in the history

  2. Fix extra Opts param in CF version and note to test it

    @xandkar
    xandkar committed Jun 21, 2022
    Configuration menu
    Copy the full SHA
    92ebb1d View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2022

  1. Support chaining streams

    @xandkar
    xandkar committed Jun 26, 2022
    Configuration menu
    Copy the full SHA
    c38973f View commit details
    Browse the repository at this point in the history

Commits on Jun 29, 2022

  1. Refine stream type to non-empty list of streams 

    - to closer reflect the reality.
    @xandkar
    xandkar committed Jun 29, 2022
    Configuration menu
    Copy the full SHA
    3ecf3e2 View commit details
    Browse the repository at this point in the history

  2. Note the reason for representation as a list

    @xandkar
    xandkar committed Jun 29, 2022
    Configuration menu
    Copy the full SHA
    e996286 View commit details
    Browse the repository at this point in the history