Reported length of zip of stateful iterators wrong

[jakobnissen]

A common idiom in the Python world to partition an iterator into chunks of length N is to turn it into a stateful iterator, then zip it with itself N times.

However, such a Zip iterator in Julia falsely report to have the same length as the original iterator.

Perhaps the takeaway is that stateful iterators cannot be HasLength or HasShape, as it's unknowable if another reference to the iterator iterates it.

[StefanKarpinski]

I don't think this is really a bug: one has to be careful to only be currently be using a stateful iterator in one place or it will not produce the desired results. This is similar to the fact that if you modify a collection while iterating it bad things can happen. What could be done, however, is to copy an iterator (and its state) when doing something like this. Can you give a more concrete example?

[jakobnissen]

Good point. On the other hand I would argue that the major use case for opting in to stateful iterators is exactly to use it multiple places. For example, one can make a Stateful(itr) to pass into two functions and in that way make sure that the second function starts the iterator where the first one stopped. Of course concurrently iterating stateful iterators is dangerous, but this is not what zip is supposed to do, I think.

In my particular example, I got an annoying type instability from using Iterators.partition, which tends to yield Vector{Any} when partitioning Generators. To get around it, I tried to use the old Python idiom for chunking an iterator:

julia> itr = (i for i in 1:9); # Base.eltype == Any

julia> first(Iterators.partition(itr, 3)) isa Vector{Any}
true

julia> collect(zip(repeat([Iterators.Stateful(itr)], 3)...))
9-element Vector{Tuple{Int64, Int64, Int64}}:
 (1, 2, 3)
 (4, 5, 6)
 (7, 8, 9)
 (139908521644592, 32433, 65537)
 (3, 4, 8)
 (3, 32, 1)
 (100, 1000, 250)
 (20, 0, 3)
 (32, 4, 0)

This produces NTuple{3, Int} which is much better for efficiency and stability, but because length is wrongly calculated, most of the resulting array is uninitialized.

It seems to me to be a tradeoff: Stateful iterators can only preserve shape information if the user promises to never have two active iterators at the same time. A safety/performance tradeoff.

[StefanKarpinski]

Aha, I see what you're trying to do: zipping an interator with itself so that the values are interleaved. If seems to me that either the contract has to be that the iterators are all unconnected or we cannot rely on predicting the length in advance. It would be nice if this worked though. Copying wouldn't help here since you want the same iterator used multiple times. One way to fix this would be for zip to only claim to be able to predict its length when its arguments are unentangled. We'd need a way to check for that though, which is challenging.

[martinholters]

We don't guarantee the order in which zip iterates its inputs. The first element could just as well be (3,2,1) or any other permutation, making this use case a bit questionable.