JasPer 'make test' output is not helpful

[bobfriesenhahn]

The output from 'make test' is entirely unhelpful. When the JasPer codecs have been intentionally trimmed, the tests show many failures since the test suite assumes that all possible codecs are available.

It is useful if the tests only test the codecs which were configured or else support a way to report expected failures.
It is useful if the test output either specify the tests which failed, or say where to look for the results.

% make test
Running tests...
Test project /home/bfriesen/src/jasper-build
    Start 1: run_test_imginfo
1/7 Test #1: run_test_imginfo .................***Failed    0.70 sec
    Start 2: run_test_imgcmp
2/7 Test #2: run_test_imgcmp ..................***Failed    0.19 sec
    Start 3: run_test_1
3/7 Test #3: run_test_1 .......................   Passed    0.71 sec
    Start 4: run_test_2
4/7 Test #4: run_test_2 .......................***Failed    0.02 sec
    Start 5: run_test_3
5/7 Test #5: run_test_3 .......................***Failed    0.96 sec
    Start 6: run_test_4
6/7 Test #6: run_test_4 .......................   Passed    0.04 sec
    Start 7: run_test_5
7/7 Test #7: run_test_5 .......................***Failed    0.03 sec

29% tests passed, 5 tests failed out of 7

Total Test time (real) =   2.65 sec

The following tests FAILED:
          1 - run_test_imginfo (Failed)
          2 - run_test_imgcmp (Failed)
          4 - run_test_2 (Failed)
          5 - run_test_3 (Failed)
          7 - run_test_5 (Failed)
Errors while running CTest
make: *** [Makefile:107: test] Error 8

[mdadams]

The test suite does not require all codecs. It does require all of the native codecs, however. This is unlikely to change, since most of the tests perform transcoding (i.e., encoding in one format and decoding in another format), and this cannot be done when native codecs are missing. Furthermore, since there are no external dependencies for the native codecs, there is really no reason not to include them in the first place. Which codecs have you included in the library?

[bobfriesenhahn]

On Wed, 22 Dec 2021, Michael Adams wrote: The test suite does not require all codecs. It does require all of the native codecs, however. This is unlikely to change, since most of the tests perform transcoding (i.e., encoding in one format and decoding in another format), and this cannot be done when native codecs are missing. Furthermore, since there are no external dependencies for the native codecs, there is really no reason not to include them in the first place. Which codecs have you included in the library?

In my latest build jasper says this: ``` The following formats are supported: jp2 JPEG-2000 JP2 File Format Syntax (ISO/IEC 15444-1) jpc JPEG-2000 Code Stream Syntax (ISO/IEC 15444-1) heic HEIC (ISO/IEC 23008-12) pgx JPEG-2000 VM Format (PGX) ``` I did not intend for heic to be supported but I recently added the library to my system and the request for supported formats is subtractive rather than additive and so it is now supported. It is not really necessary to strip support for the formats that GraphicsMagick does not need, but that is what I did in order to make sure that the result still works. This is what I did for oss-fuzz test builds as well. I agree that it may be difficult to design an test suite without a simple primordial non-lossy input format to store test images in. Clearly, a lot of work has gone into developing the test suite. It is useful if the output that the user sees provides a clue as to where to look to investigate the details of the test results. Bob -- Bob Friesenhahn ***@***.***, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt

[uvic-frodo]

I have added a warning message and disabled the test suite in the case that the test suite cannot function properly (due to excluded codec support). There is almost never a good reason to exclude native codec support since its inclusion does not result in additional external software dependencies. The only reason that I have added this ability is for users on embedded systems that are very memory constrained, where an extra 1 KiB of memory would break the system.

[uvic-frodo]

Incidentally, I also added comments to the CMake options for various codecs indicating if they are native.

[bobfriesenhahn]

On Thu, 23 Dec 2021, Michael Adams wrote: I have added a warning message and disabled the test suite in the case that the test suite cannot function properly (due to excluded codec support). There is almost never a good reason to exclude native codec support since its inclusion does not result in additional external software dependencies. The only reason that I have added this ability is for users on embedded systems that are very memory constrained, where an extra 1 KiB of memory would break the system.

The reason why the "native codec support" was disabled is that its implementation is not robust enough to support fuzz testing. Bob -- Bob Friesenhahn ***@***.***, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ Public Key, http://www.simplesystems.org/users/bfriesen/public-key.txt