Processing of some input files causes segmentation fault on x86_64 Linux

[michaelrsweet]

Version: 8.15.1
CUPS.org User: bel

CUPS test page beeing processed with cms color correction tables,
resolution 150 dpi and above and device=bitcmyk causes segmentation fault
on x86_64 Linux (SuSE 9.3 and Mandriva 2005 LE).

Commands caused segmentation fault:

cat full_file.ps | gs -sOutputFile=- -sDEVICE=bitcmyk -r600x600 > res.bin
cat small_file.ps | gs -sOutputFile=- -sDEVICE=bitcmyk -r1200x600 > res.bin

Files full_file.ps and small_file.ps are attached

[michaelrsweet]

CUPS.org User: mike

What options did you use with the configure script?

What Linux distribution are you using?

What version of GCC are you using?

[michaelrsweet]

CUPS.org User: bel

I run configure script without options.
I use distributions SuSE 9.3 (x86_64) and Mandriva 2005 LE (x86_64).
I use default compilers of these distributions:
SuSE 9.3 (x86_64) -- gcc v. 3.3.5
Mandriva 2005 LE (x86_64) -- gcc v. 3.4.3

This bug was reported to AFPL Ghostscript v. 8.51 as Bug# 688252.

http://bugs.ghostscript.com/show_bug.cgi?id=688252

The bug is resolved in AFPL Ghostscript v. 8.52 and above.

I beleive the syncronization with their changes would be
solution of the problem.

[michaelrsweet]

CUPS.org User: mike

That's fine, except when I compile on an AMD64 system running RHEL4 using GCC 3.4.3, both of your test files work just fine...

Can you run your tests in a debugger and show me the backtrace ("where")?

[michaelrsweet]

CUPS.org User: bel

I have installed RHEL4 Workstation X86_64 and tested ESP Ghostscript 8.15.1
with that OS. There was fresh install, no foreign software or other artifacts.

There is the session transcript with some snips:

--- Session transcript begin ---

cat /etc/issue

Red Hat Enterprise Linux WS release 4 (Nahant)
Kernel \r on an \m

gcc -v

Reading specs from /usr/lib/gcc/x86_64-redhat-linux/3.4.3/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --disable-checking --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-languages=c,c++,objc,java,f77 --enable-java-awt=gtk --host=x86_64-redhat-linux
Thread model: posix
gcc version 3.4.3 20041212 (Red Hat 3.4.3-9.EL4)

md5sum espgs-8.15.1-source.tar.gz

80e86a300c2e3f3c9fc9a707c2212743 espgs-8.15.1-source.tar.gz

zcat espgs-8.15.1-source.tar.gz | tar -xf -

cd espgs-8.15.1

./configure

8-< --- snip --- snip --- snip --- snip --- snip --- snip --- snip --- >-8

make

8-< --- snip --- snip --- snip --- snip --- snip --- snip --- snip --- >-8

make install

8-< --- snip --- snip --- snip --- snip --- snip --- snip --- snip --- >-8

ls -l /usr/local/bin/gs

-rwxr-xr-x 1 root root 5245783 Apr 26 13:29 /usr/local/bin/gs

/usr/local/bin/gs -v

ESP Ghostscript 8.15.1 (2005-09-22)
Copyright (C) 2004 artofcode LLC, Benicia, CA. All rights reserved.

cd ..

md5sum full_file.ps

0d609d5324afefc20c2b125f25c73ea2 full_file.ps

cat full_file.ps | /usr/local/bin/gs -sOutputFile=- -sDEVICE=bitcmyk -r600x600 > res.bin

Segmentation fault

gdb /usr/local/bin/gs

GNU gdb Red Hat Linux (6.1post-1.20040607.62rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...(no debugging symbols found)...Using host libthread_db library "/lib64/tls/libthread_db.so.1".

(gdb) r -sOutputFile=- -sDEVICE=bitcmyk -r600x600 > res.bin < full_file.ps
Starting program: /usr/local/bin/gs -sOutputFile=- -sDEVICE=bitcmyk -r600x600 > res.bin < full_file.ps
(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...
Program received signal SIGSEGV, Segmentation fault.
0x00000000005ded51 in set_color_ht_le_4 ()
(gdb) backtrace
#0 0x00000000005ded51 in set_color_ht_le_4 ()
#1 0x00000000005dcd49 in gx_dc_ht_colored_fill_rectangle ()
#2 0x00000000005e7f1e in fill_loop_by_trapezoids ()
#3 0x00000000005e6a95 in gx_default_fill_path ()
#4 0x00000000005fabd0 in gx_stroke_path_only ()
#5 0x00000000004d093f in clist_playback_band ()
#6 0x00000000004d2837 in clist_playback_file_bands ()
#7 0x00000000004d2944 in clist_render_rectangle ()
#8 0x00000000004d2b8b in clist_rasterize_lines ()
#9 0x00000000004d2dd8 in clist_get_bits_rectangle ()
#10 0x0000000000602787 in gx_default_get_bits ()
#11 0x00000000004cb0e7 in gdev_prn_get_bits ()
#12 0x00000000004defc2 in bit_print_page ()
#13 0x00000000004c9edb in gdev_prn_output_page ()
#14 0x0000000000476366 in zoutputpage ()
#15 0x00000000004588b6 in gs_interpret ()
#16 0x000000000044ff61 in gs_main_interpret ()
#17 0x000000000045011e in gs_main_run_string_end ()
#18 0x0000000000450ca7 in run_string ()
#19 0x0000000000403bb3 in main ()
(gdb) quit
The program is running. Exit anyway? (y or n) y

--- Session transcript end ---

I have tried ESP Ghostscript 8.15.2, released 11:01 Apr 25, 2006 also.
It shows the same behaviour:

Unpack ... ./configure ... make ... make install -- as above.

/usr/local/bin/gs -v

ESP Ghostscript 8.15.2 (2006-04-19)
Copyright (C) 2004 artofcode LLC, Benicia, CA. All rights reserved.

cat full_file.ps | /usr/local/bin/gs -sOutputFile=- -sDEVICE=bitcmyk -r600x600 > res.bin

Segmentation fault

Backtrace for 8.15.2 follows:
#0 0x00000000005e00b1 in set_color_ht_le_4 ()
#1 0x00000000005de0a9 in gx_dc_ht_colored_fill_rectangle ()
#2 0x00000000005e927e in fill_loop_by_trapezoids ()
#3 0x00000000005e7df5 in gx_default_fill_path ()
#4 0x00000000005fbf30 in gx_stroke_path_only ()
#5 0x00000000004d0def in clist_playback_band ()
#6 0x00000000004d2ce7 in clist_playback_file_bands ()
#7 0x00000000004d2df4 in clist_render_rectangle ()
#8 0x00000000004d303b in clist_rasterize_lines ()
#9 0x00000000004d3288 in clist_get_bits_rectangle ()
#10 0x0000000000603ae7 in gx_default_get_bits ()
#11 0x00000000004cb597 in gdev_prn_get_bits ()
#12 0x00000000004df472 in bit_print_page ()
#13 0x00000000004ca38b in gdev_prn_output_page ()
#14 0x0000000000476806 in zoutputpage ()
#15 0x0000000000458d56 in gs_interpret ()
#16 0x0000000000450401 in gs_main_interpret ()
#17 0x00000000004505be in gs_main_run_string_end ()
#18 0x0000000000451147 in run_string ()
#19 0x0000000000403c43 in main ()

i.e. the same stack.

Please see description and resolution of this bug at:

http://bugs.ghostscript.com/show_bug.cgi?id=688642
http://bugs.ghostscript.com/show_bug.cgi?id=688252

[michaelrsweet]

CUPS.org User: mike

Unable to reproduce...