Run all regression tests on Jenkins

[snukky]

I have lightened most tests by decreasing mini/maxi-batch sizes and reducing the length of test inputs wherever appropriate, but at the moment, only a few CPU and GPU tests are run on Jenkins.

However, some tests generate wrong textual output for me. @hieuhoang Does all tests work for you?

[snukky]

Mini/maxi-batches have been reduced because on Jenkins machine there is usually only 4GB GPU memory available.

[hieuhoang]

2 tests seems to be failing ensemble.return-nematus-alignment ensemble.return-soft-alignment Hieu Hoang http://statmt.org/hieu

…

On Mon, 17 Sep 2018 at 22:12, Roman Grundkiewicz ***@***.***> wrote: Mini/maxi-batches have been reduced because on Jenkins machine there is usually only 4GB GPU memory available. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAqOFDN_1U4MzJTIrBwMD7FWM6aNLJqiks5ucBBUgaJpZM4WsuiU> .

[hieuhoang]

these 2 as well
test_en_de_gpu.return-nematus-alignment/
test_en_de_gpu.return-soft-alignment
There is textual difference

[hieuhoang]

Probably my fault.haven't checked reg tests in a while. Will take a look

[snukky]

OK. Jenkins jobs work now properly. @hieuhoang To add more tests to Jenkins (which we should do), just update TESTS_CPU and TESTS_GPU variables in the main Makefile.

[hieuhoang]

cheers. so only 3 GPU test pass on Jenkins? that's weird.

Is the GPU particularly old, with only 4GB? I don't have access/never used Jenkins.

Most tests pass on my GTX1080 (apart from 4 mentioned above), I'll make sure all tests pass when I have time. Then we will go from there

[snukky]

No, I added only 3 tests to be run on Jenkins to check if everything is configured correctly. I wasn't sure which tests are designed for GPU, and which for CPU as only some tests have gpu/cpu in their names. Maybe we should run all tests on both CUDA and CPU-only amun?

[snukky]

On the Jenkins machine these tests fail:

tests/ensemble.return-nematus-alignment/err:make[1]: [test] Error 1 (ignored)
tests/ensemble.return-soft-alignment/err:make[1]: [test] Error 1 (ignored)
tests/large-model/err:make[1]: [test] Error 1 (ignored)
tests/test_en_de_gpu.return-nematus-alignment/err:make[1]: [test] Error 1 (ignored)
tests/test_en_de_gpu.return-soft-alignment/err:make[1]: [test] Error 1 (ignored)

[hieuhoang]

Yep. They fail locally too. Will look at out next week