Stack overflow issues with many episodes

[wasowski]

We seem to be having two kinds of issues with stack overflow:

• We get a stack overflow even when running one epoch of a single episode training (see issue [Stack Overflow on one iteration] Car.isFinal should stop beyond 10 or beyond 10000? #48 where we are debugging why breaking car does not work).
• We get a stack overflow with many episodes

These two issues seem unrelated. In this issue we are concerned with the second problem. We collect info that might help to resolve it.

• It appears that stack overflow could result only from iterateUntilM or iterateWhileM from the scheduler Monad (and we use Randomized as a scheduler in all current examples.
• Both of these functions call tailRecM whose name would indicate tail-recursiveness, but see below
• The comment in this file indicates that tailRecM is only stack-safe if flatMap is stack safe.
• The flatMap we are using comes from LazyList or more precisely from IterableOnce in the standard library. This needs to be checked for stack safety.
• If it turns out not to be stack-safe, then there are hints that some trampolining can be done to work around this issue (see: https://stackoverflow.com/questions/33989157/stack-safety-of-scalaz-streamts-flatmap)

[wasowski]

Further investigation log:

• I was trying to reproduce some stack overflows on long issues, and I am no longer sure we have them. I am able to run 300000 episodes on simplemaze. I do not get a stack overflow, but I seem to be starting to have garbage collection issues. So perhaps the long stack overflow is fixed, and this issue is invalid, but we have a memory leak.
• I started to question Randomized.const which seems to multiply every constant indefinitely, which could lead to hogging memory. Shouldn't this return a singleton list?
• This is an indication that lists and streams are not stack safe in our sense, and that the effect is that the memory blows up: TailRecM does not make sense for lists / streams typelevel/cats#1329 (there is even a paper about this: https://functorial.com/stack-safety-for-free/index.pdf , supposedly not easily comprehensible)

[wasowski]

After fixing the issue (PR #54) with const (switching it to singleton and changing toGen to be able to deal with finite streams) I seem to be able to get 350K iterations with simple maze without GC issues. No stack overflows, but I feel 350K is still way too low - so we still have some memory leak issue.

The small number of episodes non-terminating case (#48) still remains, but I do not think we actually have stack overflows so I am closing this issue.

[wasowski]

For the record: after redesigning the Randomized schedule (and also realizing that our schedule needs to be a Monad and Foldable, I seem to be able to reach 400K episodes with simple maze with a handful of GC spikes, and using almost 100% of the 3GB heap. The new design does not suffer from stack-overflows either, but it does suffer from OutOfMemory errors, which we should track in a separate issue.