Perl_yylex: Assertion `((((U8)*(((U8*)s)+1 )) & 0xC0) == 0x80)' failed (toke.c:4821)

[p5pRT]

Migrated from rt.perl.org#127262 (status was 'resolved')

Searchable as RT127262$

[p5pRT]

From @geeknik

The attached 'script' causes the following assertion failure in Perl v5.23.7 (v5.23.6-109-g388b516)​:

perl​: toke.c​:4821​: Perl_yylex​: Assertion `((((U8)*(((U8*)s)+1 )) & 0xC0) == 0x80)' failed.
Aborted

[p5pRT]

From @geeknik

test00

[p5pRT]

From @tonycoz

On Wed Jan 13 17​:41​:39 2016, brian.carpenter@​gmail.com wrote​:

The attached 'script' causes the following assertion failure in Perl
v5.23.7 (v5.23.6-109-g388b516)​:

perl​: toke.c​:4821​: Perl_yylex​: Assertion `((((U8)*(((U8*)s)+1 )) &
0xC0) == 0x80)' failed.
Aborted

Also reproducible with​:

$ hexdump -C ../127262b.pl
00000000 75 73 65 20 75 74 66 38 3b c2 |use utf8;.|
0000000a

Tony

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @khwilliamson

On Mon Jan 18 19​:20​:11 2016, tonyc wrote​:

On Wed Jan 13 17​:41​:39 2016, brian.carpenter@​gmail.com wrote​:

The attached 'script' causes the following assertion failure in Perl
v5.23.7 (v5.23.6-109-g388b516)​:

perl​: toke.c​:4821​: Perl_yylex​: Assertion `((((U8)*(((U8*)s)+1 )) &
0xC0) == 0x80)' failed.
Aborted

Also reproducible with​:

$ hexdump -C ../127262b.pl
00000000 75 73 65 20 75 74 66 38 3b c2 |use utf8;.|
0000000a

Tony

I'm unable to reproduce this. Could someone supply configuration details?
--
Karl Williamson

[p5pRT]

From @geeknik

On Wed Mar 02 09​:03​:30 2016, khw wrote​:

I'm unable to reproduce this. Could someone supply configuration details?

./Configure -des -Dusedevel -DDEBUGGING -Dcc=afl-gcc -Doptimize=-O2\ -g && AFL_HARDEN=1 make

Debian 8.2 x64, gcc 4.9.2, afl 2.05b, libc 2.19-18+deb8u3, etc.

[p5pRT]

From @hvds

On Wed Mar 02 09​:03​:30 2016, khw wrote​:

I'm unable to reproduce this. Could someone supply configuration details?

I can reproduce it on blead here​:

% git clean -dfx
% ./Configure -des -Dcc=gcc -Dprefix=/opt/scratch -Doptimize='-g -O6' -DDEBUGGING -Dusedevel -Uversiononly
% make perl
% perl -e 'print "use utf8;\x{c2}"' | ./perl -Ilib
perl​: toke.c​:4821​: Perl_yylex​: Assertion `((((U8)((*(((U8*)s)+1 )) | 0)) & 0xC0) == 0x80)' failed.
Aborted (core dumped)
% ./perl -Ilib -V
Summary of my perl5 (revision 5 version 23 subversion 9) configuration​:
  Commit id​: 451c6e0
  Platform​:
  osname=linux, osvers=3.13.0-37-generic, archname=x86_64-linux
  uname='linux shad2 3.13.0-37-generic #64-ubuntu smp mon sep 22 21​:28​:38 utc 2014 x86_64 x86_64 x86_64 gnulinux '
  config_args='-des -Dcc=gcc -Dprefix=/opt/scratch -Doptimize=-g -O6 -DDEBUGGING -Dusedevel -Uversiononly'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  use64bitint=define, use64bitall=define, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='gcc', ccflags ='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
  optimize='-g -O6',
  cppflags='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
  ccversion='', gccversion='4.8.4', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
  ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='gcc', ldflags =' -fstack-protector -L/usr/local/lib'
  libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /lib64 /usr/lib64
  libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  libc=libc-2.19.so, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.19'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -g -O6 -L/usr/local/lib -fstack-protector'

Characteristics of this binary (from libperl)​:
  Compile-time options​: DEBUGGING HAS_TIMES PERLIO_LAYERS PERL_COPY_ON_WRITE
  PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP
  PERL_PRESERVE_IVUV PERL_USE_DEVEL USE_64_BIT_ALL
  USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE
  USE_LOCALE_COLLATE USE_LOCALE_CTYPE
  USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO
  USE_PERL_ATOF
  Built under linux
  Compiled at Mar 2 2016 23​:34​:18
  @​INC​:
  lib
  /opt/scratch/lib/perl5/site_perl/5.23.9/x86_64-linux
  /opt/scratch/lib/perl5/site_perl/5.23.9
  /opt/scratch/lib/perl5/5.23.9/x86_64-linux
  /opt/scratch/lib/perl5/5.23.9
  .
%

Hugo

[p5pRT]

From @khwilliamson

Thanks for reporting this. Fixed by commit
710740a
--
Karl Williamson

[p5pRT]

@khwilliamson - Status changed from 'open' to 'pending release'

[p5pRT]

From @khwilliamson

Thank you for submitting this report. You have helped make Perl better.
 
With the release of Perl 5.24.0 on May 9, 2016, this and 149 other issues have been resolved.

Perl 5.24.0 may be downloaded via https://metacpan.org/release/RJBS/perl-5.24.0

[p5pRT]

@khwilliamson - Status changed from 'pending release' to 'resolved'