FPE on unknown address when decoding JP2 format

[bobfriesenhahn]

oss-fuzz is detecting a floating point exception in the i386 build while decoding JP2 format.

       /mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_graphicsmagick_364babd4f1406e0e2b68256230a27a3911dd0072/revisions/coder_JP2_fuzzer: Running 1 inputs 100 time(s) each.
        Running: /mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases/crash-afbb7f3d27332481e61835c7a061d6cd4bf7afc4
        warning: trailing garbage in marker segment (30 bytes)
        warning: trailing garbage in marker segment (4 bytes)
        AddressSanitizer:DEADLYSIGNAL
        =================================================================
        ==15842==ERROR: AddressSanitizer: FPE on unknown address 0x08c7533f (pc 0x08c7533f bp 0xffd00548 sp 0xffd00480 T0)
        SCARINESS: 10 (signal)
            #0 0x8c7533f in jpc_pi_nextcprl jasper/src/libjasper/jpc/jpc_t2cod.c:487:34
            #1 0x8c7533f in jpc_pi_next jasper/src/libjasper/jpc/jpc_t2cod.c:136:10
            #2 0x8c78745 in jpc_dec_decodepkts jasper/src/libjasper/jpc/jpc_t2dec.c:470:14
            #3 0x8c35bc2 in jpc_dec_process_sod jasper/src/libjasper/jpc/jpc_dec.c:643:6
            #4 0x8c2d712 in jpc_dec_decode jasper/src/libjasper/jpc/jpc_dec.c:436:10
            #5 0x8c2d712 in jpc_decode jasper/src/libjasper/jpc/jpc_dec.c:270:6
            #6 0x8c1bc97 in jp2_decode jasper/src/libjasper/jp2/jp2_dec.c:231:21
            #7 0x874422b in ReadJP2Image graphicsmagick/coders/jp2.c:663:17
            #8 0x8307e19 in ReadImage graphicsmagick/magick/constitute.c:1630:13
            #9 0x82b283c in BlobToImage graphicsmagick/magick/blob.c:764:13
            #10 0x821d5a8 in Magick::Image::read(Magick::Blob const&) graphicsmagick/Magick++/lib/Image.cpp:1591:5
            #11 0x8211489 in LLVMFuzzerTestOneInput graphicsmagick/fuzzing/coder_fuzzer.cc:22:15
            #12 0x80fa2ee in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned int) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerLoop.cpp:611:15
            #13 0x80e540e in fuzzer::RunOneTest(fuzzer::Fuzzer*, char const*, unsigned int) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:324:6
            #14 0x80eb12c in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned int)) /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerDriver.cpp:860:9
            #15 0x81146b7 in main /src/llvm-project/compiler-rt/lib/fuzzer/FuzzerMain.cpp:20:10
            #16 0xf7ba4ee4 in __libc_start_main
            #17 0x80c1985 in _start
        
        AddressSanitizer can not provide additional info.
        SUMMARY: AddressSanitizer: FPE (/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-builds-i386_graphicsmagick_364babd4f1406e0e2b68256230a27a3911dd0072/revisions/coder_JP2_fuzzer+0x8c7533f)
        ==15842==ABORTING

This is the bit of code in jpc_t2cod.c which is apparently being executed:

		for (pi->y = pi->ystart; pi->y < pi->yend;
		  pi->y += pi->ystep - (pi->y % pi->ystep)) {
			for (pi->x = pi->xstart; pi->x < pi->xend;
			  pi->x += pi->xstep - (pi->x % pi->xstep)) {

The following is an input file which causes the problem:

clusterfuzz-testcase-minimized-coder_JP2_fuzzer-5048547093905408.gz

[uvic-frodo]

Fortunately, I was able to determine the problem without a reproducer being provided in this case. The bug is quite platform specific and depends on the size of various integer types. The problem is caused by integer overflow.