Port the old test suite to the new one

[D4N]

We need to retire the old test suite so that we can move forward with #180 and other patches.

Unfortunately, that is a lot of grind work. So, if you happen to have a little time at your hand, please help.

Progress

The main focus should be on the regression tests:

• GitHub issues: Port GitHub issues to new test suite #249
• Redmine issues 426-825 (bugfixes-test.sh) ([WIP] Port redmine issues #287)
• Redmine issues 1179 - 1305 (bugfixes-test.sh) (Port redmine issues 1179-1305 #333)
• Redmine issues 831-937 (bugfixes-test.sh) (Port redmine issues 831 to 937 to the new testsuite #353)
• Redmine issues 1024-1170 (bugfixes-test.sh) (Port redmine issues from 1024 to 1170 #367)
• Redmine issues 1043, 1074, 1137 (bugfixes-test.sh) (Testsuite improvements #419)

The remaining ones have lower priority:

• tiff-test.sh (already in master)
• webp-test.sh
• addmoddel.sh
• conversions.sh
• crw-test.sh
• curliotest.sh
• eps-preview-test.sh
• eps-test.sh
• exifdata-test.sh
• exiv2-test.sh
• geotag-test.sh
• httpiotest.sh
• http-test.sh
• icc-test.sh
• imagetest.sh
• iotest.sh
• iptctest.sh
• iso65k-test.sh
• modify-test.sh
• path-test.sh
• preview-test.sh
• sshiotest.sh
• stdin-test.sh
• stringto-test.sh (these should become unit tests)
• testMSVC.sh
• tiff-test.sh
• version-test.sh
• video-test.sh
• write2-test.sh
• write-test.sh
• write-video-test.sh
• xmpparser-test.sh

Porting guide

The actual porting process is not too hard. The old test suite is just a set of shell scripts in test/ that call exiv2 repeatedly, collect the output of the whole script and compare it to a binary output file (it is under test/data/ and has the same name as the script, but with the ending .out instead of .sh).

Example from test/bugfixes-test.sh:

num=426
filename=`prep_file $num`
runTest exiv2 -u -pi $filename

and the corresponding output from test/data/bugfixes-test.out

------> Bug 426 <-------
Iptc.Application2.RecordVersion              Short       1  2
Iptc.Application2.FixtureId                  String     22  2005 Oregon Coast Tour
Iptc.Application2.Byline                     String     14  Julie V. Early
Iptc.Application2.City                       String     21  159 km SW of Portland
Iptc.Application2.ProvinceState              String      6  Oregon
Iptc.Application2.CountryName                String     13  United States
Iptc.Application2.Contact                    String     16  Jeffrey J. Early

The new test suite on the other hand uses Python's unit test suite to perform the same task. In addition to that, it check the standard output & error separately and also checks the return value.

Each issue should have its own python test file, which has roughly this form:

# -*- coding: utf-8 -*-

import system_tests


class AnInformativeName(metaclass=system_tests.CaseMeta):

    filename = system_tests.path("$data_path/exiv2-reproducer-filename")
    commands = ["$exiv2 $filename"]
    stdout = [""]
    stderr = ["""$exiv2_exception_message $filename:
$kerCorruptedMetadata
"""]
    retval = [1]

This file should be located in an appropriate directory under tests/, in that directory must be an empty file called __init__.py and the file must have the prefix test_ (otherwise python's test collector will ignore it).

Most issues can be ported rather easily (using the excerpt from bugfixes-test.sh):

1. Create a new file under tests/ with an appropriate name. E.g. tests/bugfixes/redmine/test_issue_426.py
2. Insert the python class skeleton.
3. Identify the reproducer which is run, either its name is written in the script or it is created via prep_file. prep_file $num means that the file is called exiv2-bug$num.jpg. Write this name into the python class into the variable filename, prefixed with $data_path and wrap it in system_tests.path() (for Windows path support). For the example, you'll get: filename = system_tests.path("$data_path/exiv2-bug426.jpg").
4. Extract the command from the script, that is the part after runTest.
5. Run the reproducer, to obtain the stdout, stderr & return value and write them as python strings into the class.

For issue 426 we'll end up with something like this:

# -*- coding: utf-8 -*-

import system_tests


class ReadIPTfromJPEG(metaclass=system_tests.CaseMeta):

    url = "http://dev.exiv2.org/issues/426"

    filename = system_tests.path("$data_path/exiv2-bug426.jpg")
    commands = ["exiv2 -u -pi $filename"]
    stdout = ["""Iptc.Application2.RecordVersion              Short       1  2
Iptc.Application2.FixtureId                  String     22  2005 Oregon Coast Tour
Iptc.Application2.Byline                     String     14  Julie V. Early
Iptc.Application2.City                       String     21  159 km SW of Portland
Iptc.Application2.ProvinceState              String      6  Oregon
Iptc.Application2.CountryName                String     13  United States
Iptc.Application2.Contact                    String     16  Jeffrey J. Early
"""]
    stderr = [""]
    retval = [0]

Remarks

• Not all output from exiv2 is useful. Especially stderr is often cluttered with error messages about which we do not really care. In that case, leave stderr empty and instead set the following inside the python class (thereby the actual output will be ignored except for error messages from ASAN and UBSAN):

    compare_stderr = system_tests.check_no_ASAN_UBSAN_errors

• Some of the tests should not be system tests. They should be rewritten as unit tests instead.

[clanmills]

Dan: You've done a very good job of documenting this. I'll work on this after Easter when I get home from Wales.

[piponazo]

In #333 we ported redmine issues from 1179 to 1305.

[clanmills]

We can close this one now. The python test suite in tests/system_tests.py is working extremely well. The remaining bash tests have been implemented in python in tests/bash_tests/. @1div0 has agreed to take on the maintenance of the python code for v1.00. I think Peter is going to enjoy working on this valuable and useful code.