pvcheck is a tool that automates the testing of computer programs by checking the textual output of the program under testing with a set of user-defined test cases.
pvcheck has been developed to be used as automatic correction tool in the courses of Principles of Computer Programming at the University of Pavia.
pvcheck requires a Python interpreter and can be executed as a standalone application. The current version has been tested with Python 3.4.3.
To install pvcheck:
pip install pvcheck
To compile all the sample programs:
cd examples
make
pvcheck has 3 main arguments (run, info and export) .
To see the main help:
pvcheck -h
or
pvcheck --help
To see the help of a specific command:
pvcheck argument -h
or
pvcheck argument --help
The run argument, being the default argument, also works without explicit invocation. So
pvcheck run ./program
is the same as:
pvcheck ./program
If you have a test file with the default name pvcheck.test located in the pvcheck folder, you can test your program simply by using:
pvcheck ./program
If you want to test a program using a specific test file:
pvcheck -f testfile ./program
pvcheck --file testfile ./program
If you want to test a program using a specific test contained in a test suite, you can use the info argument to list all the available tests. Then you can use:
pvcheck -T testnumber ./program
pvcheck --test testnumber ./program
to use only the desired test.
pvcheck lets you choose the output format. The formats available are json, csv and html. To choose the ouput format:
pvcheck -F format ./program
pvcheck --format format ./program
You can set how many seconds it should be waited for the termination of the program:
pvcheck -t timeout ./program
pvcheck --timeout timeout ./program
The default is 10 seconds.
To report up to N errors per section:
pvcheck -e N ./program
pvcheck --errors N ./program
The default is 4.
To set the verbosity level, where the level must be an integer between 0 (minimum) and 4 (maximum):
pvcheck -v verbositylevel ./program
pvcheck --verbosity verbositylevel ./program
The default is 3.
To use Valgrind (if installed) to check memory usage:
pvcheck -V ./program
pvcheck --valgrind ./program
To specify the name of the file used for logging:
pvcheck -l logfile ./program
pvcheck --log logfile ./program
The default is ~/.pvcheck.log.
To cut the output of the program to a maximum of L lines:
pvcheck -L L ./program
pvcheck --output_limit L ./program
The default is 10000.
To use the specified configuration file:
pvcheck -c configurationfile ./program
pvcheck --config configurationfile ./program
To enable or disable colored output:
pvcheck -C option ./program
pvcheck --color option ./program
The options available are YES, NO and AUTO. The default value is AUTO.
The info argument lists all the available tests in a test file.
To see all the tests available in a test file:
pvcheck info testfile
The export argument allows you to export in a file the input arguments from the selected test.
To export the input argument from a test, you can use the info argument to list all the available tests. Then you can use:
pvcheck export testnumber testfile
The output file is saved into the current directory with the name testname.dat .
A test file is a file which contains one or more test cases.
Tests are divided in sections with names between square brackets.
Empty lines, and lines starting with '#' are ignored.
Non-empty lines are compared against those produced by the program in the corresponding section.
The format of test definition files is rather simple. The following examples include comments to concisely describe the various aspects of the format:
The special section [.SECTIONS] allows to specify additional options for the sections. Options are usually declared at the beginning of the test file, or in a separate configuration file.
For example:
...
[SECTION1]
...
[.SECTIONS]
SECTION2 unordered
[SECTION2]
...
indicates that the order of the lines in the SECTION2 section is not relevant.
In case of multiple tests, each test is introduced by the special section [.TEST] followed by the name of the test.
Example:
...
[.TEST]
Test1
[SECTION1]
...
[SECTION2]
...
[.TEST]
Test2
[SECTION1]
...
[SECTION2]
...
Common parts among all the test cases (for instance options in the [.SECTIONS] special section) can be specified before the first [.TEST] section. They will be prepended to all the tests.
The special section [.INPUT] allows to specify the text to be written on the program's standard input:
Example:
...
[.TEST]
Name of the test
[.INPUT]
input1
input2
...
[SECTION1]
...
[SECTION2]
...
[.TEST]
Thi is another test
[.INPUT]
input3
input4
...
[SECTION1]
...
[SECTION2]
...
The special section [.ARGS] allows to specify additional arguments to be passed on the command line, one extra argument per line.
Example:
...
[.TEST]
Test1
[.ARGS]
arg1
arg2
...
[SECTION1]
...
[SECTION2]
...
[.TEST]
Test2
[.ARGS]
arg3
arg4
...
[SECTION1]
...
[SECTION2]
...
The special section [.FILE] shall be used together with the special section [.ARGS]. When the special argument ".FILE" is present in the [.ARGS] section, a temporary file is automatically generated and filled with the content of the text in the [.FILE] section. The name of the temporary file is passed on the command line of the program under test.
Example:
...
[.ARGS]
.FILE
...
[.TEST]
Test1
[.FILE]
first line written in the temporary file used for Test1
second line written in the temporary file used for Test1
...
[SECTION1]
...
[SECTION2]
...
[.TEST]
Test2
[.FILE]
first line written in the temporary file used for Test2
second line written in the temporary file used for Test2
...
[SECTION1]
...
[SECTION2]
...
For more information, please visit the wiki of the project.