NAME

Concurrent::Iterator - Enables safe concurrent consumption of Iterable values

SYNOPSIS

use Concurrent::Iterator;

DESCRIPTION

A standard Raku Iterator should only be consumed from one thread at a time. Concurrent::Iterator allows any Iterable to be iterated concurrently.

-head1 SYNOPSIS

use Concurrent::Iterator;

# Make a concurrent iterator over an infinite Range
# of ascending integers.
my $ids = concurrent-iterator(0..Inf);

# Many concurrent workers can use it to obtain the IDs,
# being confident of no duplication.
do await for ^10 {
    start {
        say "I got $ids.pull-one()";
    }
}

Overview

The purpose of Concurrent::Iterator is to allow multiple threads to safely race to obtain values from a single iterable source of data. It:

• Uses locking to ensure that only one thread can be pulling a value from the underlying iterator at a time

• Will return IterationEnd to all future requests for values after it is returned by the underlying iterator (it is erroneous to call pull-one on a normal Iterator that has already produced IterationEnd)

• Will rethrow any exception thrown by the underlying iterator on all future requests

Together, these mean that it is possible to use a Concurrent::Iterator to have many workers compete for data items to process, and have them all terminate on end of sequence or exception. That might look something like this

my \to-process = concurrent-seq(@data);
my @results = flat await do for ^4 {
    start (compute-stuff($_) for to-process)
}

However, there's no reason to use Concurrent:Iterator for such a simple use case. It is far more simply expressed without this module as just:

my @results = @data.hyper.map(&compute-stuff);

Or, if you really wanted to enforce one-at-a-time and exactly 4 workers:

my @results = @data.hyper(:degree(4), :batch(1)).map(&compute-stuff);

Concurrent::Iterator

The Concurrent::Iterator class is constructed with a single positional argument, which must be of type Iterable:D:

my $ci = Concurrent::Iterator.new(1..Inf);

It implements the Raku standard Iterator interface.

Convenience subs

There is a convenience sub to form a Concurrent::Iterator:

my $ci = concurrent-iterator(1..Inf);

There is also one to have it wrapped in a Seq:

my $cs = concurrent-seq(1..Inf);

Which literally just passes the result of calling concurrent-iteratorr to Seq.new.

AUTHOR

Jonathan Worthington

COPYRIGHT AND LICENSE

Copyright 2016 - 2024 Jonathan Worthington

Copyright 2024 Raku Community

This library is free software; you can redistribute it and/or modify it under the Artistic License 2.0.