Question and Answers

Question: "Tasks failed with timeout"

Cause

It might that your invocation is running too much and hit maximal allowed running time.

Steps to resolve

Try to reduce input data to the invocation. For example, experimenting with chunk size, may trigger PyWren to generate more invocations against data from IBM COS, where each inviocation process a single chunk size. If input data is array, make less partitions. See example of stock prediction with Monte Carlo

Question: "Tasks failed with out of memory"

Question: "Tasks failed with ‘error’: ‘Too many concurrent requests in flight (count: X, allowed: Y).’}"

Cause

PyWren generates too many concurrent invocations and hit limitation of maximum concurrent invocations.

Steps to resolve

• Try to reduce number of invocations generated by PyWren. For example increasing chunk size when reading from IBM COS will generate less invocations where each invocation process the requested chunk size. This might be tricky, as increasing chunk size, may also need to increase overall memory required for the invocation. Increasing chunk size will also make an invocation to process more data which will increase running time.
• Increase retry times and numnber of retries. Limitation of this approach is that you may hit maximal running time. This will also might make entire flow slower, due to longer retry times. Follow 'retry_sleeps' and 'retires' from here.