Bypasses all current error checking in opj_decompress and still triggers resource exhaustion

[pic4xiu]

When I was fuzzing, I found a file that can bypass all current error checks. This file can cause program denial of service, similar to cve-2019-6988.

Expected behavior and actual behavior.

The program finds hardware limitations and directly refuses to parse.

But the program took up my memory, causing resource exhaustion, my system is ubuntu20, but I also tested it on windows, the effect is the same

pic@pic-RESCUER-R720-15IKBN:~/Download/openjpeg/build/bin$ ./opj_decompress -i 2000 -o te.raw

===========================================
The extension of this file is incorrect.
FOUND 2000. SHOULD BE .j2k or .jpc or .j2c or .jhc
===========================================

[INFO] Start to read j2k main header (0).
[WARNING] Cannot take in charge mct data within multiple MCT records
[INFO] Main header has been correctly decoded.
[INFO] No decoded area parameters, set the decoded area to the whole image
Killed

pic@pic-RESCUER-R720-15IKBN:~/Download/openjpeg/build/bin$ dmesg | egrep -i -B100 'killed process'
[ 7139.855289] [   1293]   126  1293    80741      237   102400        0             0 gsd-housekeepin
[ 7139.855291] [   1294]   126  1294    87156      708   151552        0             0 gsd-power
[ 7139.855292] [   1295]   126  1295    43827      173    98304        0             0 ibus-engine-sim
...
[ 7139.855480] [   5572]  1000  5572  3831508  3752527 30732288    74122             0 opj_decompress
[ 7139.855482] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/user@1000.service,task=opj_decompress,pid=5572,uid=1000
[ 7139.855492] Out of memory: Killed process 5572 (opj_decompress) total-vm:15326032kB, anon-rss:15010108kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:30012kB oom_score_adj:0

Steps to reproduce the problem.

The poc is here

Run: opj_decompress -i poc -o te.raw

Operating system

ubuntu20/windows10

openjpeg version

OpenJPEG 2.5.0

[pedrohc]

CVE-2023-39328 was assigned for this issue. Please let me know if you wish to dipuste/reject it.