utf8 messes global match array context #2910

p5pRT · 2000-11-28T20:55:33Z

Migrated from rt.perl.org#4796 (status was 'resolved')

Searchable as RT4796$

p5pRT · 2000-11-28T20:55:33Z

From @jhi

Created by jhi@kosh.hut.fi

Using the /g match in array context seems to be the killer (it doesn't
have to be an array assignment). Compiling with system's malloc hides(?)
the bug.

The test is a stripped down t/op/pat.t #44. Works fine without the -Mutf8.

kosh:~/pp4/perl ; ./perl -Ilib -Mutf8 -e '@a=("b" =~ /(.)/g)'
assertion botched (chunk's tail overwrite?): *((char *)((caddr_t)ovp + nbytes - sizeof (unsigned int) + i)) == 0x55
zsh: 2481 abort (core dumped) ./perl -Ilib -Mutf8 -e @a=("b" =~ /(.)/g)
kosh:~/pp4/perl ; dbx ./perl core.perl.kosh.0
dbx version 3.11.10
Type 'help' for help.
Core file created by program "perl"

signal IOT/Abort trap at >*[__kill, 0x3ff800e9b78] beq r19, 0x3ff800e9b90
(dbx) where

0 __kill(0x3ff800dccdc, 0x3ffc0086c80, 0x3ffc0089540, 0x100000074, 0x3ff801766c0) [0x3ff800e9b78]
1 (unknown)() [0x3ff801a5218]
2 __tis_raise(0x3ff801766c0, 0x1400016d0, 0x3ff80112014, 0x6, 0x3ff8015f6c4) [0x3ff80112010]
3 raise(0x3ff80112014, 0x6, 0x3ff8015f6c4, 0x140012280, 0x3ff801766ec) [0x3ff8015f6c0]
4 abort(0x3ffbff01bac, 0x3fffffd1998, 0x0, 0x0, 0xfffffc0000000000) [0x3ff801766e8]
5 botch(diag = 0x3fffffd1c78 = "chunk's tail overwrite", s = 0x3fffffd1f48 = "*((char *)((caddr_t)ovp + nbytes - sizeof (unsigned int) + i)) == 0x55") ["malloc.c":997, 0x3ffbff01ba8]
6 free(mp = 0x1400016d8) ["malloc.c":1619, 0x3ffbff02eb8]
7 Perl_sv_clear(sv = 0x140001dd8) ["sv.c":3833, 0x3ffbff8a02c]
8 Perl_sv_free(sv = 0x140001dd8) ["sv.c":3961, 0x3ffbff8a464]
9 Perl_free_tmps() ["scope.c":193, 0x3ffbffacd64]
10 perl_run(my_perl = 0x140001408) ["perl.c":1381, 0x3ffbfefb08c]
11 main(argc = 5, argv = 0x11ffff468, env = 0x11ffff498) ["perlmain.c":52, 0x120001ff0]
(dbx)

Perl Info


Flags:
    category=core
    severity=high

This perlbug was built using Perl v5.7.0 - Wed Nov 29 05:15:56 EET 2000
It is being executed now by  Perl v5.7.0 - Wed Nov 29 06:15:13 EET 2000.

Site configuration information for perl v5.7.0:

Configured by jhi at Wed Nov 29 06:15:13 EET 2000.

Summary of my perl5 (revision 5.0 version 7 subversion 0) configuration:
  Platform:
    osname=dec_osf, osvers=4.0f, archname=alpha-dec_osf
    uname='osf1 kosh.hut.fi v4.0 1229 alpha '
    config_args='-des -Dusedevel -Doptimize=-g'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
    useperlio=undef d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=define use64bitall=define uselongdouble=undef
  Compiler:
    cc='cc', ccflags ='-std -DDEBUGGING -DLANGUAGE_C',
    optimize='-g',
    cppflags='-std -DDEBUGGING -DLANGUAGE_C'
    ccversion='V5.9-010', gccversion='', gccosandvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=8
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, usemymalloc=y, prototype=define
  Linker and Libraries:
    ld='ld', ldflags =''
    libpth=/usr/shlib /usr/ccs/lib /usr/lib/cmplrs/cc /usr/lib /var/shlib
    libs=-lgdbm -ldbm -ldb -lm -liconv -lutil
    perllibs=-lm -liconv -lutil
    libc=/usr/shlib/libc.so, so=so, useshrplib=true, libperl=libperl.so
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='  -Wl,-rpath,/usr/local/lib/perl5/5.7.0/alpha-dec_osf/CORE'
    cccdlflags=' ', lddlflags='-shared -expect_unresolved "*" -g -msym -std'

Locally applied patches:
    DEVEL7891


@INC for perl v5.7.0:
    lib
    /u/vieraat/vieraat/jhi/Perl/lib
    /usr/local/lib/perl5/5.7.0/alpha-dec_osf
    /usr/local/lib/perl5/5.7.0
    /usr/local/lib/perl5/site_perl/5.7.0/alpha-dec_osf
    /usr/local/lib/perl5/site_perl/5.7.0
    /usr/local/lib/perl5/site_perl
    .


Environment for perl v5.7.0:
    HOME=/u/vieraat/vieraat/jhi
    LANG=C
    LANGUAGE (unset)
    LC_ALL=fi_FI.ISO8859-1
    LC_CTYPE=fi_FI.ISO8859-1
    LD_LIBRARY_PATH=/u/vieraat/vieraat/jhi/pp4/perl
    LOGDIR (unset)
    PATH=/u/vieraat/vieraat/jhi/.s:/u/vieraat/vieraat/jhi/.b/OSF1:/c/bin:/p/bin:/p/adm/bin:/usr/bin:/usr/sbin:/sbin:/bin:/usr/ccs/bin:/usr/lib:/etc:/lib:/p/X6/bin:/p/X5/bin:/usr/bin/X11:/usr/lbin:/usr/sbin/acct:/usr/tcb/bin:/tcb/bin:/usr/field:/u/vieraat/vieraat/jhi
    PERLIO=perlio
    PERLLIB=/u/vieraat/vieraat/jhi/Perl/lib
    PERL_BADLANG (unset)
    SHELL=/bin/zsh

p5pRT · 2000-11-30T10:00:25Z

From @jhi

Just for the bug db record, Robin managed to whittle the core dump
inducer down to:

./perl -Ilib -Mutf8 -we 'map /()/, q()'

Dumps core e.g. in Solaris, with or without use64bitint, but seemingly
not in in (x86) Linux...

p5pRT · 2000-11-30T10:22:01Z

From @jhi

On Thu, Nov 30, 2000 at 12:00:11PM -0600, Jarkko Hietaniemi wrote:

Just for the bug db record, Robin managed to whittle the core dump
inducer down to:

./perl -Ilib -Mutf8 -we 'map /()/, q()'

Dumps core e.g. in Solaris, with or without use64bitint, but seemingly
not in in (x86) Linux...

Closing in on it... it's the sv_utf8_downgrade() call in the
pp_hot.c:pp_match() (line 1127 in my copy) that breaks things...
<diving back into debugger>

p5pRT · 2000-11-30T11:46:50Z

From @jhi

The below seems to fix the bug, just don't ask me why...

Change 7936 by jhi@alpha on 2000/11/30 20:41:39

Fix for 20001128.006, be more careful in Perl_sv_utf8_downgrade().
Why the different platforms behave so differently (core dump vs
no core dump) on this bug is a but of a mystery, but if I had to
guess I would mumble something like 'alignment'.

Affected files ...

... //depot/perl/sv.c#299 edit

Differences ...

==== //depot/perl/sv.c#299 (text) ====
Index: perl/sv.c

Inline Patch

--- perl/sv.c.~1~	Thu Nov 30 21:43:34 2000
+++ perl/sv.c	Thu Nov 30 21:43:34 2000
@@ -2447,22 +2447,26 @@
 Perl_sv_utf8_downgrade(pTHX_ register SV* sv, bool fail_ok)
 {
     if (SvPOK(sv) && SvUTF8(sv)) {
-        char *c = SvPVX(sv);
-	STRLEN len = SvCUR(sv) + 1;	/* include trailing NUL */
-        if (!utf8_to_bytes((U8*)c, &len)) {
-	    if (fail_ok)
-		return FALSE;
-	    else {
-		if (PL_op)
-		    Perl_croak(aTHX_ "Wide character in %s",
-			       PL_op_desc[PL_op->op_type]);
-		else
-		    Perl_croak(aTHX_ "Wide character");
+        if (SvCUR(sv)) {
+	    char *c = SvPVX(sv);
+	    STRLEN len = SvCUR(sv);
+
+	    if (!utf8_to_bytes((U8*)c, &len)) {
+	        if (fail_ok)
+		    return FALSE;
+		else {
+		    if (PL_op)
+		        Perl_croak(aTHX_ "Wide character in %s",
+				   PL_op_desc[PL_op->op_type]);
+		    else
+		        Perl_croak(aTHX_ "Wide character");
+		}
 	    }
+	    SvCUR(sv) = len;
 	}
-	SvCUR(sv) = len - 1;
 	SvUTF8_off(sv);
     }
+
     return TRUE;
 }
 
End of Patch.

p5pRT · 2000-12-05T13:50:41Z

From [Unknown Contact. See original ticket]

From perl5-porters-return-26698-rmb1=cise.npl.co.uk@perl.org Wed Nov 29 06:58:08 2000
Mailing-List: contact perl5-porters-help@perl.org; run by ezmlm
Precedence: bulk
list-help: <mailto:perl5-porters-help@perl.org>
list-unsubscribe: <mailto:perl5-porters-unsubscribe@perl.org>
list-post: <mailto:perl5-porters@perl.org>
Delivered-To: mailing list perl5-porters@perl.org
X-Authentication-Warning: ak-71.mind.de: k set sender to andreas.koenig@anima.de using -f
Sender: k@ak-71.mind.de
To: Jarkko Hietaniemi <jhi@iki.fi>
Cc: perl5-porters@perl.org
Subject: Re: [ID 20001128.006] utf8 messes global match array context
From: andreas.koenig@anima.de (Andreas J. Koenig)
Date: 29 Nov 2000 07:57:53 +0100
Lines: 8
User-Agent: Gnus/5.0807 (Gnus v5.8.7) Emacs/20.7
MIME-Version: 1.0

On Tue, 28 Nov 2000 23:11:10 -0600, Jarkko Hietaniemi <jhi@iki.fi> said:
  > /perl \-Ilib \-Mutf8 \-e '@&#8203;a=$"bc" =~ /\(\.$$\.$/\)'
Works for all perls I have on linux .

--
andreas

These often come from use64bitint, which the original bugee had.

Robin

p5pRT · 2000-12-05T13:51:03Z

From @jhi

On Tue, 28 Nov 2000 23:11:10 -0600, Jarkko Hietaniemi <jhi@iki.fi> said:
  > /perl \-Ilib \-Mutf8 \-e '@&#8203;a=$"bc" =~ /\(\.$$\.$/\)'
Works for all perls I have on linux .

--
andreas
These often come from use64bitint, which the original bugee had.

Doesn't seem to be a use64bitint problem as such, I now built a Linux
use64bitint perl, and no core dump. Could still be a 64-bit issue, of
course, for example some dormant assumption (that gets tickled by the
above test in Alpha) that pointers are 32 bits and therefore fits in
an int, or something equally devious.

p5pRT · 2000-12-05T13:54:06Z

From @jhi

On Wed, Nov 29, 2000 at 06:55:27AM +0200, Jarkko Hietaniemi wrote:

This is a bug report for perl from jhi@kosh.hut.fi,
generated with the help of perlbug 1.33 running under perl v5.7.0.

-----------------------------------------------------------------
[Please enter your report here]

Using the /g match in array context seems to be the killer (it doesn't
have to be an array assignment). Compiling with system's malloc hides(?)
the bug.

Actually, the /g is a red herring. Just array context for a match is enough:

/perl -Ilib -Mutf8 -e '@a=("bc" =~ /(.)(.)/)'
assertion botched (chunk's tail overwrite?): *((char *)((caddr_t)ovp + nbytes - sizeof (unsigned int) + i)) == 0x55
zsh: 18493 abort (core dumped) ./perl -Ilib -Mutf8 -e @a=("bc" =~ /(.)(.)/)
kosh:~/pp4/perl ;

Same stack dump.

p5pRT · 2000-12-05T13:54:07Z

From @andk

On Tue, 28 Nov 2000 23:11:10 -0600, Jarkko Hietaniemi <jhi@iki.fi> said:

> /perl -Ilib -Mutf8 -e '@a=("bc" =~ /(.)(.)/)'

Works for all perls I have on linux .

p5pRT closed this as completed Nov 28, 2003

p5pRT added Severity High distro-dec_osf type-library labels Oct 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

utf8 messes global match array context #2910

utf8 messes global match array context #2910

p5pRT commented Nov 28, 2000

p5pRT commented Nov 28, 2000

p5pRT commented Nov 30, 2000

p5pRT commented Nov 30, 2000

p5pRT commented Nov 30, 2000

p5pRT commented Dec 5, 2000

p5pRT commented Dec 5, 2000

p5pRT commented Dec 5, 2000

p5pRT commented Dec 5, 2000

utf8 messes global match array context #2910

utf8 messes global match array context #2910

Comments

p5pRT commented Nov 28, 2000

p5pRT commented Nov 28, 2000

From @jhi

Created by jhi@kosh.hut.fi

p5pRT commented Nov 30, 2000

From @jhi

p5pRT commented Nov 30, 2000

From @jhi

p5pRT commented Nov 30, 2000

From @jhi

p5pRT commented Dec 5, 2000

From [Unknown Contact. See original ticket]

p5pRT commented Dec 5, 2000

From @jhi

p5pRT commented Dec 5, 2000

From @jhi

p5pRT commented Dec 5, 2000

From @andk