Race condition in vmaf / vmafossexec (results in "problem generating pooled VMAF score")

[nbirkbeck]

I've observed some random failures / flakiness in the CLI tools when running in testing environments when threads >= 1 (not with threads = 0). Typically the failure looks like:

problem generating pooled VMAF score

The simplest example that I have to reproduce is as follows.

First, Create some sample input files (small):

k=0; w=32; cnt=2; 
dd bs=$((w*w*3)) count=${cnt} if=/dev/random  of=/tmp/ref.yuv 2>/dev/null
dd bs=$((w*w*3)) count=${cnt} if=/dev/random  of=/tmp/src.yuv 2>/dev/null

Run either vmafossexec:

vmaf=build/tools/vmafossexec
while ${vmaf} yuv444p $w $w /tmp/ref.yuv /tmp/src.yuv ../model/vmaf_v0.6.1.json --log /tmp/vmaf.log --log-fmt json --thread 1; do k=$((k+1)); echo $k; done; echo "Failed on iteration $k";

or vmaf and wait for failure

vmaf=build/tools/vmaf

while ${vmaf} -r /tmp/ref.yuv -d /tmp/src.yuv --width $w --height $w --pixel_format 444 -b 8 -m path=../model/vmaf_v0.6.1.json --json --threads 1 -o /tmp/vmaf.log ; do k=$((k+1)); echo $k; done; echo "Failed on iteration $k";

Failure may occur randomly after 100s or thousands of iterations:

[Trial1]
problem generating pooled VMAF score
Failed on iteration 255

[Trial2]
problem generating pooled VMAF score
Failed on iteration 151

[Trial3]	
problem generating pooled VMAF score
Failed on iteration 25

Occurs on any value of threads >= 1 (but doesn't happen with threads=0 as thread pools aren't used there).

I'm fairly sure the termination condition in vmaf_thread_pool_wait is not waiting for the queue to be drained. I'll send a pull request.