Stack overflow from jas_image_writecmpt()

[dgr37]

With jasper 2.0.23 I am having major problems with decompression of medium-sized jpeg2000 files, resulting in a stack overflow. These problems did not occur in jasper 2.0.19. For example, if I take the attached .jp2 file and do a decompression such as

jasper.exe --input "BACanonEos400D 003.jp2" --output test.jp2 --output-format jp2

stack memory is already exhausted and jasper crashes. Compiler is MSVC.

I noticed that the stack overflow arises from jas_image.c:534, jas_image_writecmpt(), where _alloca() is called explicitly to allocate memory on stack. Replacing this by a heap allocation, e.g. with jas_alloc2(), solves the problem. I attached a patch for this.

Is there a reason why stack is preferable here?

Thank you!

jas_image_writecmpt.zip

[jubalh]

Thank your for the report and investigation!

These problems did not occur in jasper 2.0.19

The change was done in 5017d3a so is available since jasper 2.0.20

I noticed that the stack overflow arises from jas_image.c:534, jas_image_writecmpt(), where _alloca() is called explicitly to allocate memory on stack. Replacing this by a heap allocation, e.g. with jas_alloc2(), solves the problem. I attached a patch for this.
Is there a reason why stack is preferable here?

@MaxKellermann probably you know more about this.

[MaxKellermann]

The problem is that alloca() is called for every iteration. That needs to be moved out of the loop.
The problem doesn't exist on C99 compliant systems because that stack allocation is only done once with C99.
Having the array declaration inside the loop may give the compiler more opportunities for optimization, because we tell him that he doesn't need to preserve the contents.

[jubalh]

Thanks @MaxKellermann !

@dgr37 please give feedback here once you have tested.

[dgr37]

Thank you @jubalh and @MaxKellermann !

The change works perfectly for me.