OpenJPEG 1.5.0 crashes on a ridiculously big file...

[gcode-importer]

Originally reported on Google Code with ID 151

What steps will reproduce the problem?
1. Download and unpack ftp://ftp.microimages.com/pub/tnt/data/jp2/gtopo30lossless.zip
(378MiByte) from http://www.microimages.com/gallery/jp2/
2. Execute gdb --args j2k_to_image -i g30mosLossless.jp2 -o test.raw
3. Run the tool inside gdb and obtain a backtrace

What is the expected output? What do you see instead?
I expect a decoded image and the relevant output from the tool.
Instead I see this and a segmentation fault:

[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (0) than number
of tile-parts (0)
[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (1) than number
of tile-parts (0)
[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (2) than number
of tile-parts (0)
[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (3) than number
of tile-parts (0)
[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (4) than number
of tile-parts (0)
[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (5) than number
of tile-parts (0)
[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (6) than number
of tile-parts (0)
[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (7) than number
of tile-parts (0)
[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (8) than number
of tile-parts (0)
[INFO] tile 1 of 1

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7bcc076 in t1_decode_cblks (t1=0x8816180, tilec=0x612b80, tccp=0x612320)
at /home/usr/src/openjpeg-1.5.0/libopenjpeg/t1.c:1560
1560                                                                    ((int*)tiledp)[(j
* tile_w) + i] = tmp / 2;
(gdb) bt
#0  0x00007ffff7bcc076 in t1_decode_cblks (t1=0x8816180, tilec=0x612b80, tccp=0x612320)
at /home/usr/src/openjpeg-1.5.0/libopenjpeg/t1.c:1560
#1  0x00007ffff7bd3b66 in tcd_decode_tile (tcd=<optimized out>, 
    src=0x7fffb9b9a010 "\303\371fp\357\230sV\323\353j\004VV\312\337O\207U\331&\363\177k\220\267\a\376V\032\240\311f\an\031j\221\267\246\027\070#+\323`\313È\237\253\241\344]q\240ԟ\230\374j\264\347zJQ\326n7\034^*\336\306\323q\346\307w\371)\322dȗܜ\207\216诜\303\344\212\002\377!W\257\254\246\250\017\304\300",
len=395528744, tileno=0, cstr_info=<optimized out>) at /home/usr/src/openjpeg-1.5.0/libopenjpeg/tcd.c:1383
#2  0x00007ffff7bc22b7 in j2k_read_eoc (j2k=0x60f0c0) at /home/usr/src/openjpeg-1.5.0/libopenjpeg/j2k.c:1566
#3  0x00007ffff7bc34d9 in j2k_decode (j2k=0x60f0c0, cio=0x6107d0, cstr_info=<optimized
out>)
    at /home/usr/src/openjpeg-1.5.0/libopenjpeg/j2k.c:1877
#4  0x00007ffff7bc5c70 in opj_jp2_decode (jp2=0x60f050, cio=0x6107d0, cstr_info=<optimized
out>)
    at /home/usr/src/openjpeg-1.5.0/libopenjpeg/jp2.c:749
#5  0x000000000040231b in main (argc=<optimized out>, argv=<optimized out>)
    at /home/usr/src/openjpeg-1.5.0/applications/codec/j2k_to_image.c:717

What version of the product are you using? On what operating system?
OpenJPEG-1.5.0 on Debian.

Please provide any additional information below.
Running the same command in valgrind gives these printouts:

==23383== Warning: silly arg (-2812252756) to malloc()
==23383== Invalid write of size 4
==23383==    at 0x4E42076: t1_decode_cblks (t1.c:1560)
==23383==    by 0x4E49B65: tcd_decode_tile (tcd.c:1383)
==23383==    by 0x4E382B6: j2k_read_eoc (j2k.c:1566)
==23383==    by 0x4E394D8: j2k_decode (j2k.c:1877)
==23383==    by 0x4E3BC6F: opj_jp2_decode (jp2.c:749)
==23383==    by 0x40231A: main (j2k_to_image.c:717)
==23383==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==23383== 
==23383== 
==23383== Process terminating with default action of signal 11 (SIGSEGV)
==23383==  Access not within mapped region at address 0x0
==23383==    at 0x4E42076: t1_decode_cblks (t1.c:1560)
==23383==    by 0x4E49B65: tcd_decode_tile (tcd.c:1383)
==23383==    by 0x4E382B6: j2k_read_eoc (j2k.c:1566)
==23383==    by 0x4E394D8: j2k_decode (j2k.c:1877)
==23383==    by 0x4E3BC6F: opj_jp2_decode (jp2.c:749)
==23383==    by 0x40231A: main (j2k_to_image.c:717)

Reported by sebras on 2012-06-16 02:05:09

[gcode-importer]

The image 'g30mosLossless.jp2' has a precision of 14.

Does libopenjpeg automatically extend to 16 bit?

Using openjpeg-branch-r1656, I get:

[INFO] tile 1 of 1
Segmentation fault

Using openjpeg-branch15-r1718, I get:

[INFO] tile 1 of 1
[ERROR] Out of memory
[ERROR] Failed to decode J2K image
ERROR -> j2k_to_image: failed to decode image!

winfried

Reported by szukw000 on 2012-06-18 22:30:28

[gcode-importer]

D'oh! I should of course have tried svn HEAD. And on your recommendation I did, but
r1718 built from scratch still segfaults for me:

gdb --args bin/j2k_to_image -i g30mosLossless.jp2  -o test.ppm
(gdb) r
[INFO] Start to read j2k main header (85).
[INFO] Main header has been correctly decoded.
[INFO] No decoded area parameters, set the decoded area to the whole image
[INFO] Header of tile 0 / 0 has been read.

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7bb8290 in t1_decode_cblks_v2 (t1=0x61d760, tilec=0x61aec0, 
    tccp=0x61ca00) at t1.c:1834
1834 ((OPJ_INT32*)tiledp)[(j * tile_w) + i] = tmp / 2;
(gdb) p tiledp
$1 = (void *) 0x7fffe5b41eb0
(gdb) p j
$2 = 60
(gdb) p tile_w
$3 = 166392
(gdb) p i
$4 = 0
(gdb) bt
#0  0x00007ffff7bb8290 in t1_decode_cblks_v2 (t1=0x61d760, tilec=0x61aec0, 
    tccp=0x61ca00) at t1.c:1834
#1  0x00007ffff7bcae05 in tcd_t1_decode (p_tcd=0x61ae50) at tcd.c:2933
#2  0x00007ffff7bca684 in tcd_decode_tile_v2 (p_tcd=0x61ae50, 
    p_src=0x7fffd0783010 "\303\371fp\357\230sV\323\353j\004VV\312\337O\207U\331&\363\177k\220\267\a\376V\032\240\311f\an\031j\221\267\246\027\070#+\323`\313È\237\253\241\344]q\240ԟ\230\374j\264\347zJQ\326n7\034^*\336\306\323q\346\307w\371)\322dȗܜ\207\216诜\303\344\212\002\377!W\257\254\246\250\017\304\300",

    p_max_length=395528744, p_tile_no=0, p_cstr_index=0x619f80) at tcd.c:2650
#3  0x00007ffff7b9a765 in j2k_decode_tile (p_j2k=0x618420, p_tile_index=0, 
    p_data=0x7ffecbe78010 "", p_data_size=2888840912, p_stream=0x618250, 
    p_manager=0x618328) at j2k.c:10047
#4  0x00007ffff7b9e6a1 in j2k_decode_tiles (p_j2k=0x618420, p_stream=0x618250, 
    p_manager=0x618328) at j2k.c:11670
#5  0x00007ffff7b99006 in j2k_exec (p_j2k=0x618420, p_procedure_list=0x61a9b0, 
    p_stream=0x618250, p_manager=0x618328) at j2k.c:9406
#6  0x00007ffff7b9eca1 in j2k_decode_v2 (p_j2k=0x618420, p_stream=0x618250, 
    p_image=0x61af00, p_manager=0x618328) at j2k.c:11856
#7  0x00007ffff7ba3c33 in jp2_decode_v2 (jp2=0x618380, cio=0x618250, 
    p_image=0x61af00, p_manager=0x618328) at jp2.c:1777
#8  0x00007ffff7baab28 in opj_decode_v2 (p_codec=0x6182d0, p_stream=0x618250, 
    p_image=0x61af00) at openjpeg.c:1214
#9  0x0000000000413ba4 in main (argc=5, argv=0x7fffffffe1e8)
    at j2k_to_image.c:821
(gdb)

I also convinced myself that j2k_to_image included the correct libopenjpeg.so, since
I already have one installed on my system. Oh and as you probably can tell from above
I'm on a 64-bit system (which ought to be able to handle the 7GByte raw image that
will be generated by this ridiculously big image...).

Reported by sebras on 2012-06-18 22:41:41

[gcode-importer]

Re-running valgrind confirms my problems:

valgrind ./bin/j2k_to_image -i ~g30mosLossless.jp2  -o test.ppm

==15719== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
[...]
[INFO] Start to read j2k main header (85).
[INFO] Main header has been correctly decoded.
[INFO] No decoded area parameters, set the decoded area to the whole image
[...]
[INFO] Header of tile 0 / 0 has been read.
==15719== Warning: set address range perms: large range [0x2563a4428, 0x3026a7710)
(undefined)
==15719== Invalid write of size 4
==15719==    at 0x4E70290: t1_decode_cblks_v2 (t1.c:1834)
==15719==    by 0x4E82E04: tcd_t1_decode (tcd.c:2933)
==15719==    by 0x4E82683: tcd_decode_tile_v2 (tcd.c:2650)
==15719==    by 0x4E52764: j2k_decode_tile (j2k.c:10047)
==15719==    by 0x4E566A0: j2k_decode_tiles (j2k.c:11670)
==15719==    by 0x4E51005: j2k_exec (j2k.c:9406)
==15719==    by 0x4E56CA0: j2k_decode_v2 (j2k.c:11856)
==15719==    by 0x4E5BC32: jp2_decode_v2 (jp2.c:1777)
==15719==    by 0x4E62B27: opj_decode_v2 (openjpeg.c:1214)
==15719==    by 0x413BA3: main (j2k_to_image.c:821)
==15719==  Address 0x91efa120 is not stack'd, malloc'd or (recently) free'd

Reported by sebras on 2012-06-18 22:56:32

[gcode-importer]

Using openjpeg-trunk-r1718 and valgrind-3.8.0.SVN, I get:

==14831== Invalid write of size 4
==14831==    at 0x4E6E8CB: t1_decode_cblks_v2 (t1.c:1834)
==14831==    by 0x4E816B0: tcd_t1_decode (tcd.c:2937)
==14831==    by 0x4E80EE4: tcd_decode_tile_v2 (tcd.c:2653)
==14831==    by 0x4E50B32: j2k_decode_tile (j2k.c:10047)
==14831==    by 0x4E54A40: j2k_decode_tiles (j2k.c:11670)
==14831==    by 0x4E4F4C4: j2k_exec (j2k.c:9406)
==14831==    by 0x4E55046: j2k_decode_v2 (j2k.c:11856)
==14831==    by 0x4E5A007: jp2_decode_v2 (jp2.c:1777)
==14831==    by 0x4E60F99: opj_decode_v2 (openjpeg.c:1214)
==14831==    by 0x413E0F: main (j2k_to_image.c:821)
==14831==  Address 0x920390a0 is 16 bytes before a block of size 64 alloc'd
==14831==    at 0x4C28271: malloc (vg_replace_malloc.c:263)
==14831==    by 0x4E804EA: tcd_init_decode_tile (in /usr/local/lib64/libopenjpeg.so.1.99.0)
==14831==    by 0x4E508AE: j2k_read_tile_header (j2k.c:9997)
==14831==    by 0x4E549A0: j2k_decode_tiles (j2k.c:11643)
==14831==    by 0x4E4F4C4: j2k_exec (j2k.c:9406)
==14831==    by 0x4E55046: j2k_decode_v2 (j2k.c:11856)
==14831==    by 0x4E5A007: jp2_decode_v2 (jp2.c:1777)
==14831==    by 0x4E60F99: opj_decode_v2 (openjpeg.c:1214)
==14831==    by 0x413E0F: main (j2k_to_image.c:821)
==14831==
==14831== Conditional jump or move depends on uninitialised value(s)
==14831==    at 0x4E6EB19: t1_decode_cblk_v2 (t1.c:1891)
==14831==    by 0x4E6E652: t1_decode_cblks_v2 (t1.c:1793)
==14831==    by 0x4E816B0: tcd_t1_decode (tcd.c:2937)
==14831==    by 0x4E80EE4: tcd_decode_tile_v2 (tcd.c:2653)
==14831==    by 0x4E50B32: j2k_decode_tile (j2k.c:10047)
==14831==    by 0x4E54A40: j2k_decode_tiles (j2k.c:11670)
==14831==    by 0x4E4F4C4: j2k_exec (j2k.c:9406)
==14831==    by 0x4E55046: j2k_decode_v2 (j2k.c:11856)
==14831==    by 0x4E5A007: jp2_decode_v2 (jp2.c:1777)
==14831==    by 0x4E60F99: opj_decode_v2 (openjpeg.c:1214)
==14831==    by 0x413E0F: main (j2k_to_image.c:821)
==14831==
==14831== Invalid write of size 4
==14831==    at 0x4C2AC6C: memset (mc_replace_strmem.c:966)
==14831==    by 0x4E6CF21: allocate_buffers (t1.c:1239)
==14831==    by 0x4E6EA37: t1_decode_cblk_v2 (t1.c:1869)
==14831==    by 0x4E6E652: t1_decode_cblks_v2 (t1.c:1793)
==14831==    by 0x4E816B0: tcd_t1_decode (tcd.c:2937)
==14831==    by 0x4E80EE4: tcd_decode_tile_v2 (tcd.c:2653)
==14831==    by 0x4E50B32: j2k_decode_tile (j2k.c:10047)
==14831==    by 0x4E54A40: j2k_decode_tiles (j2k.c:11670)
==14831==    by 0x4E4F4C4: j2k_exec (j2k.c:9406)
==14831==    by 0x4E55046: j2k_decode_v2 (j2k.c:11856)
==14831==    by 0x4E5A007: jp2_decode_v2 (jp2.c:1777)
==14831==    by 0x4E60F99: opj_decode_v2 (openjpeg.c:1214)
==14831==  Address 0x3033b5ab0 is 0 bytes after a block of size 16,384 alloc'd
==14831==    at 0x4C28271: malloc (vg_replace_malloc.c:263)
==14831==    by 0x4E6CED3: allocate_buffers (t1.c:1233)
==14831==    by 0x4E6EA37: t1_decode_cblk_v2 (t1.c:1869)
==14831==    by 0x4E6E652: t1_decode_cblks_v2 (t1.c:1793)
==14831==    by 0x4E816B0: tcd_t1_decode (tcd.c:2937)
==14831==    by 0x4E80EE4: tcd_decode_tile_v2 (tcd.c:2653)
==14831==    by 0x4E50B32: j2k_decode_tile (j2k.c:10047)
==14831==    by 0x4E54A40: j2k_decode_tiles (j2k.c:11670)
==14831==    by 0x4E4F4C4: j2k_exec (j2k.c:9406)
==14831==    by 0x4E55046: j2k_decode_v2 (j2k.c:11856)
==14831==    by 0x4E5A007: jp2_decode_v2 (jp2.c:1777)
==14831==    by 0x4E60F99: opj_decode_v2 (openjpeg.c:1214)
==14831==
==14831==
==14831== Process terminating with default action of signal 11 (SIGSEGV)
==14831==  Access not within mapped region at address 0x30355D000
==14831==    at 0x4C2AC6C: memset (mc_replace_strmem.c:966)
==14831==    by 0x4E6CF21: allocate_buffers (t1.c:1239)
==14831==    by 0x4E6EA37: t1_decode_cblk_v2 (t1.c:1869)
==14831==    by 0x4E6E652: t1_decode_cblks_v2 (t1.c:1793)
==14831==    by 0x4E816B0: tcd_t1_decode (tcd.c:2937)
==14831==    by 0x4E80EE4: tcd_decode_tile_v2 (tcd.c:2653)
==14831==    by 0x4E50B32: j2k_decode_tile (j2k.c:10047)
==14831==    by 0x4E54A40: j2k_decode_tiles (j2k.c:11670)
==14831==    by 0x4E4F4C4: j2k_exec (j2k.c:9406)
==14831==    by 0x4E55046: j2k_decode_v2 (j2k.c:11856)
==14831==    by 0x4E5A007: jp2_decode_v2 (jp2.c:1777)
==14831==    by 0x4E60F99: opj_decode_v2 (openjpeg.c:1214)

This test runs on Linux 3.3.6 x86_64 AMD with:
free -m

             total       used       free     shared    buffers     cached
Mem:         15966       1053      14913          0         66        737
-/+ buffers/cache:        248      15717
Swap:        32767          0      32767


winfried

Reported by szukw000 on 2012-06-19 08:37:35

[gcode-importer]

As seen on opj 1.5 branch:


==29174== Warning: set address range perms: large range [0x5a34040, 0x1d397866) (undefined)
==29174== Warning: set address range perms: large range [0x5a34040, 0x1d397040) (defined)
[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (0) than number
of tile-parts (0)
[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (1) than number
of tile-parts (0)
[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (2) than number
of tile-parts (0)
[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (3) than number
of tile-parts (0)
[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (4) than number
of tile-parts (0)
[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (5) than number
of tile-parts (0)
==29174== Warning: set address range perms: large range [0x1df74030, 0x301f2863) (noaccess)
[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (6) than number
of tile-parts (0)
==29174== Warning: set address range perms: large range [0x3959d030, 0x4ce0fd34) (noaccess)
[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (7) than number
of tile-parts (0)
==29174== Warning: set address range perms: large range [0x1df74030, 0x33103f93) (noaccess)
[WARNING] SOT marker inconsistency in tile 0: tile-part index greater (8) than number
of tile-parts (0)
==29174== Warning: set address range perms: large range [0x3959d030, 0x4f243927) (noaccess)
==29174== Warning: set address range perms: large range [0x1df74030, 0x33dc3e5a) (noaccess)
[INFO] tile 1 of 1
==29174== Warning: silly arg (-2812252756) to malloc()
==29174== Invalid write of size 4
==29174==    at 0x4E400B2: t1_decode_cblks (t1.c:1560)
==29174==    by 0x4E44605: tcd_decode_tile (tcd.c:1383)
==29174==    by 0x4E36856: j2k_read_eoc (j2k.c:1566)
==29174==    by 0x4E37E2D: j2k_decode (j2k.c:1877)
==29174==    by 0x4E3A91C: opj_jp2_decode (jp2.c:749)
==29174==    by 0x402F0A: main (j2k_to_image.c:717)
==29174==  Address 0x0 is not stack'd, malloc'd or (recently) free'd
==29174== 
==29174== 
==29174== Process terminating with default action of signal 11 (SIGSEGV)
==29174==  Access not within mapped region at address 0x0
==29174==    at 0x4E400B2: t1_decode_cblks (t1.c:1560)
==29174==    by 0x4E44605: tcd_decode_tile (tcd.c:1383)
==29174==    by 0x4E36856: j2k_read_eoc (j2k.c:1566)
==29174==    by 0x4E37E2D: j2k_decode (j2k.c:1877)
==29174==    by 0x4E3A91C: opj_jp2_decode (jp2.c:749)
==29174==    by 0x402F0A: main (j2k_to_image.c:717)
==29174==  If you believe this happened as a result of a stack
==29174==  overflow in your program's main thread (unlikely but
==29174==  possible), you can try to increase the size of the
==29174==  main thread stack using the --main-stacksize= flag.
==29174==  The main thread stack size used in this run was 8388608.
==29174== 
==29174== HEAP SUMMARY:
==29174==     in use at exit: 1,334,321,399 bytes in 2,661,256 blocks
==29174==   total heap usage: 2,792,096 allocs, 130,840 frees, 3,637,511,598 bytes
allocated
==29174== 
==29174== LEAK SUMMARY:
==29174==    definitely lost: 0 bytes in 0 blocks
==29174==    indirectly lost: 0 bytes in 0 blocks
==29174==      possibly lost: 0 bytes in 0 blocks
==29174==    still reachable: 1,334,321,399 bytes in 2,661,256 blocks
==29174==         suppressed: 0 bytes in 0 blocks
==29174== Rerun with --leak-check=full to see details of leaked memory
==29174== 
==29174== For counts of detected and suppressed errors, rerun with: -v
==29174== Use --track-origins=yes to see where uninitialised values come from
==29174== ERROR SUMMARY: 3 errors from 3 contexts (suppressed: 2 from 2)
[2]    29174 segmentation fault  valgrind j2k_to_image -i  -o test.raw

Reported by malaterre on 2012-07-11 15:21:05

[gcode-importer]

Reported by malaterre on 2014-02-25 16:09:32

• Labels added: Priority-High, Milestone-Release2.1
• Labels removed: Priority-Medium

[gcode-importer]

Same issue, single (large) tile image:

$ kdu_jp2info -i g30mosLossless.jp2

<JP2_family_file>
  <ftyp name="file-type box" header="8" body="12" pos="12">
    <brand> "jp2_" 0x6A703220 </brand>
    <minor_version> 0 </minor_version>
    <compatible_brand> "jp2_" 0x6A703220 </compatible_brand>
  </ftyp>
  <jp2h name="JP2-header box" header="8" body="37" pos="32">
    <ihdr name="image-header box" header="8" body="14" pos="40"></ihdr>
    <colr name="colour box" header="8" body="7" pos="62"></colr>
  </jp2h>
  <jp2c name="contiguous-codestream box" header="8" body="rubber" pos="77">
    <codestream>
      <width> 166392 </width>
      <height> 21587 </height>
      <components> 1 </components>
      <tiles> 1 </tiles>
    </codestream>
  </jp2c>
</JP2_family_file>

Reported by malaterre on 2014-02-27 13:32:28

• Status changed: Duplicate
• Merged into: Large single-tile image and memory: need low level streaming implementation #94