Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests spew unreferenced scalar warnings #14618

Closed
p5pRT opened this issue Mar 25, 2015 · 32 comments
Closed

Tests spew unreferenced scalar warnings #14618

p5pRT opened this issue Mar 25, 2015 · 32 comments

Comments

@p5pRT
Copy link

p5pRT commented Mar 25, 2015

Migrated from rt.perl.org#124181 (status was 'resolved')

Searchable as RT124181$

@p5pRT
Copy link
Author

p5pRT commented Mar 25, 2015

From @cpansprout

I just noticed this​:

ext/XS-Typemap/t/Typemap ...................................... Attempt to free unreferenced scalar​: SV 0x7fd68ba42750 during global destruction.
Attempt to free unreferenced scalar​: SV 0x7fd68b885f88 during global destruction.
Attempt to free unreferenced scalar​: SV 0x7fd68ba3fb98 during global destruction.
Attempt to free unreferenced scalar​: SV 0x7fd68ba42600 during global destruction.
ok

We need to fix it (or at least look into it) before 5.22.

$ perl5.21.11 -V
Summary of my perl5 (revision 5 version 21 subversion 11) configuration​:
  Snapshot of​: 9baf310
  Platform​:
  osname=darwin, osvers=12.5.0, archname=darwin-2level
  uname='darwin pint.local 12.5.0 darwin kernel version 12.5.0​: sun sep 29 13​:33​:47 pdt 2013; root​:xnu-2050.48.12~1release_x86_64 x86_64 '
  config_args='-de -Dusedevel -DDEBUGGING'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  use64bitint=define, use64bitall=define, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-fno-common -DPERL_DARWIN -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include',
  optimize='-O3 -g',
  cppflags='-fno-common -DPERL_DARWIN -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
  ccversion='', gccversion='4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.27)', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
  ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags =' -fstack-protector -L/usr/local/lib'
  libpth=/usr/local/lib /usr/bin/../lib/clang/4.2/lib /usr/lib
  libs=-lpthread -ldbm -ldl -lm -lutil -lc
  perllibs=-lpthread -ldl -lm -lutil -lc
  libc=, so=dylib, useshrplib=false, libperl=libperl.a
  gnulibc_version=''
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
  cccdlflags=' ', lddlflags=' -bundle -undefined dynamic_lookup -L/usr/local/lib -fstack-protector'

Characteristics of this binary (from libperl)​:
  Compile-time options​: DEBUGGING HAS_TIMES PERLIO_LAYERS
  PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP
  PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV
  PERL_USE_DEVEL USE_64_BIT_ALL USE_64_BIT_INT
  USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE
  USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME
  USE_PERLIO USE_PERL_ATOF
  Built under darwin
  Compiled at Mar 25 2015 08​:55​:47
  @​INC​:
  /usr/local/lib/perl5/site_perl/5.21.11/darwin-2level
  /usr/local/lib/perl5/site_perl/5.21.11
  /usr/local/lib/perl5/5.21.11/darwin-2level
  /usr/local/lib/perl5/5.21.11
  /usr/local/lib/perl5/site_perl
  .

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Mar 25, 2015

From @wolfsage

On Wed, Mar 25, 2015 at 12​:22 PM, Father Chrysostomos
<perlbug-followup@​perl.org> wrote​:

I just noticed this​:

ext/XS-Typemap/t/Typemap ...................................... Attempt to free unreferenced scalar​: SV 0x7fd68ba42750 during global destruction.
Attempt to free unreferenced scalar​: SV 0x7fd68b885f88 during global destruction.
Attempt to free unreferenced scalar​: SV 0x7fd68ba3fb98 during global destruction.
Attempt to free unreferenced scalar​: SV 0x7fd68ba42600 during global destruction.
ok

Hrm, I can't reproduce. I tried your configure args and also
./Configure -de -Dusedevel -DDEBUGGING -Doptimize='-O3 -g' (since I
get -O2 by default) and I get no such messages from any test.

./perl -Ilib -V
Summary of my perl5 (revision 5 version 21 subversion 11) configuration​:
  Commit id​: d8c1af4
  Platform​:
  osname=linux, osvers=3.13.0-46-generic, archname=x86_64-linux
  uname='linux dory 3.13.0-46-generic #79-ubuntu smp tue mar 10
20​:06​:50 utc 2015 x86_64 x86_64 x86_64 gnulinux '
  config_args='-de -Dusedevel -DDEBUGGING -Doptimize=-O3 -g'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  use64bitint=define, use64bitall=define, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe
-fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64',
  optimize='-O3 -g',
  cppflags='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe
-fstack-protector -I/usr/local/include'
  ccversion='', gccversion='4.8.2', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8,
byteorder=12345678, doublekind=3
  d_longlong=define, longlongsize=8, d_longdbl=define,
longdblsize=16, longdblkind=3
  ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
  libpth=/usr/local/lib
/usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed
/usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu
/lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
  libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  libc=libc-2.19.so, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.19'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -O3 -g -L/usr/local/lib
-fstack-protector'

Characteristics of this binary (from libperl)​:
  Compile-time options​: DEBUGGING HAS_TIMES PERLIO_LAYERS
  PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP
  PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV
  PERL_USE_DEVEL USE_64_BIT_ALL USE_64_BIT_INT
  USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE
  USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME
  USE_PERLIO USE_PERL_ATOF
  Built under linux
  Compiled at Mar 25 2015 14​:59​:18
  @​INC​:
  lib
  /usr/local/lib/perl5/site_perl/5.21.11/x86_64-linux
  /usr/local/lib/perl5/site_perl/5.21.11
  /usr/local/lib/perl5/5.21.11/x86_64-linux
  /usr/local/lib/perl5/5.21.11
  .

-- Matthew Horsfall (alh)

@p5pRT
Copy link
Author

p5pRT commented Mar 25, 2015

The RT System itself - Status changed from 'new' to 'open'

@p5pRT
Copy link
Author

p5pRT commented Mar 26, 2015

From @jkeenan

On Wed Mar 25 12​:09​:28 2015, alh wrote​:

On Wed, Mar 25, 2015 at 12​:22 PM, Father Chrysostomos
<perlbug-followup@​perl.org> wrote​:

I just noticed this​:

ext/XS-Typemap/t/Typemap ......................................
Attempt to free unreferenced scalar​: SV 0x7fd68ba42750 during global
destruction.
Attempt to free unreferenced scalar​: SV 0x7fd68b885f88 during global
destruction.
Attempt to free unreferenced scalar​: SV 0x7fd68ba3fb98 during global
destruction.
Attempt to free unreferenced scalar​: SV 0x7fd68ba42600 during global
destruction.
ok

Hrm, I can't reproduce. I tried your configure args and also
./Configure -de -Dusedevel -DDEBUGGING -Doptimize='-O3 -g' (since I
get -O2 by default) and I get no such messages from any test.

I tried both with and without -DDEBUGGING in an environment similar to alh's (Linux x86_64). I could not reproduce this warning, either.

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented Mar 26, 2015

From @cpansprout

On Wed Mar 25 18​:47​:06 2015, jkeenan wrote​:

On Wed Mar 25 12​:09​:28 2015, alh wrote​:

Hrm, I can't reproduce. I tried your configure args and also
./Configure -de -Dusedevel -DDEBUGGING -Doptimize='-O3 -g' (since I
get -O2 by default) and I get no such messages from any test.

I tried both with and without -DDEBUGGING in an environment similar to
alh's (Linux x86_64). I could not reproduce this warning, either.

That puts the onus on me to debug it, then. :-( I can reproduce this with multiple compilers and multiple configurations. Maybe it is Mac-specific.

--

Father Chrysostomos

@p5pRT
Copy link
Author

p5pRT commented Mar 30, 2015

From @tonycoz

On Wed Mar 25 22​:11​:21 2015, sprout wrote​:

On Wed Mar 25 18​:47​:06 2015, jkeenan wrote​:

On Wed Mar 25 12​:09​:28 2015, alh wrote​:

Hrm, I can't reproduce. I tried your configure args and also
./Configure -de -Dusedevel -DDEBUGGING -Doptimize='-O3 -g' (since I
get -O2 by default) and I get no such messages from any test.

I tried both with and without -DDEBUGGING in an environment similar
to
alh's (Linux x86_64). I could not reproduce this warning, either.

That puts the onus on me to debug it, then. :-( I can reproduce this
with multiple compilers and multiple configurations. Maybe it is Mac-
specific.

I just tried to reproduce this on my Macs and didn't see the problem.

My macs are running OS X 10.10.2 (uname -r 14.1.0) and 10.9.5 (13.4.0), from
your -V output it looks like you're still running 10.8.

Tony

@p5pRT
Copy link
Author

p5pRT commented Mar 30, 2015

From @bulk88

On Wed Mar 25 22​:11​:21 2015, sprout wrote​:

That puts the onus on me to debug it, then. :-( I can reproduce this
with multiple compilers and multiple configurations. Maybe it is Mac-
specific.

DEBUG_LEAKING_SCALARS+conditional breakpoints in SvREFCNT_dec and newSV (or equivelent), set on the absoolute ptr addr of the SV being in var sv. Atleast for me on Win32 Perl, SV*s are identical from run to run assuming all inputs/PP/XS code is the same.

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Apr 7, 2015

From @jkeenan

On Wed Mar 25 09​:22​:14 2015, sprout wrote​:

I just noticed this​:

ext/XS-Typemap/t/Typemap ......................................
Attempt to free unreferenced scalar​: SV 0x7fd68ba42750 during global
destruction.
Attempt to free unreferenced scalar​: SV 0x7fd68b885f88 during global
destruction.
Attempt to free unreferenced scalar​: SV 0x7fd68ba3fb98 during global
destruction.
Attempt to free unreferenced scalar​: SV 0x7fd68ba42600 during global
destruction.
ok

We need to fix it (or at least look into it) before 5.22.

FWIW, I was unable to reproduce these warnings on an older Darwin.

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented Apr 7, 2015

From @jkeenan

Summary of my perl5 (revision 5 version 21 subversion 11) configuration​:
  Commit id​: 497a6d9
  Platform​:
  osname=darwin, osvers=8.11.0, archname=darwin-2level
  uname='darwin macintosh-9.local 8.11.0 darwin kernel version 8.11.0​: wed oct 10 18​:26​:00 pdt 2007; root​:xnu-792.24.17~1release_ppc power macintosh powerpc '
  config_args='-des -Dusedevel -DDEBUGGING'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  use64bitint=undef, use64bitall=undef, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-fno-common -DPERL_DARWIN -DDEBUGGING -fno-strict-aliasing -pipe -I/usr/local/include -I/opt/local/include -D_FORTIFY_SOURCE=2',
  optimize='-O3 -g',
  cppflags='-fno-common -DPERL_DARWIN -DDEBUGGING -fno-strict-aliasing -pipe -I/usr/local/include -I/opt/local/include'
  ccversion='', gccversion='4.0.1 (Apple Computer, Inc. build 5250)', gccosandvers=''
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=4321, doublekind=4
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=6
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='env MACOSX_DEPLOYMENT_TARGET=10.3 cc', ldflags =' -L/usr/local/lib -L/opt/local/lib'
  libpth=/usr/local/lib /usr/lib /opt/local/lib
  libs=-lpthread -ldbm -ldl -lm -lc
  perllibs=-lpthread -ldl -lm -lc
  libc=, so=dylib, useshrplib=false, libperl=libperl.a
  gnulibc_version=''
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
  cccdlflags=' ', lddlflags=' -bundle -undefined dynamic_lookup -L/usr/local/lib -L/opt/local/lib'

Characteristics of this binary (from libperl)​:
  Compile-time options​: DEBUGGING HAS_TIMES PERLIO_LAYERS
  PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP
  PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV
  PERL_USE_DEVEL USE_LARGE_FILES USE_LOCALE
  USE_LOCALE_COLLATE USE_LOCALE_CTYPE
  USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO
  USE_PERL_ATOF
  Built under darwin
  Compiled at Apr 6 2015 19​:18​:11
  %ENV​:
  PERLBREW_BASHRC_VERSION="0.59"
  PERLBREW_HOME="/Users/jimk/.perlbrew"
  PERLBREW_ROOT="/Users/jimk/perl5/perlbrew"
  PERL_WORKDIR="gitwork/perl"
  @​INC​:
  lib
  /usr/local/lib/perl5/site_perl/5.21.11/darwin-2level
  /usr/local/lib/perl5/site_perl/5.21.11
  /usr/local/lib/perl5/5.21.11/darwin-2level
  /usr/local/lib/perl5/5.21.11
  /usr/local/lib/perl5/site_perl
  .

@p5pRT
Copy link
Author

p5pRT commented May 9, 2015

From @hvds

I'm able to reproduce this. Here are cut-down versions of the 4 cases​:

% PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package XS​::Typemap; require XSLoader; XSLoader​::load() } XS​::Typemap​::T_STDIO_open("stdio.tmp")'
Attempt to free unreferenced scalar​: SV 0x1d1b6a8 during global destruction.
% PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package XS​::Typemap; require XSLoader; XSLoader​::load() } $buf = ""; open $fh, "+<", \$buf; XS​::Typemap​::T_INOUT($fh); close $fh'
Attempt to free unreferenced scalar​: SV 0x2844760 during global destruction.
% PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package XS​::Typemap; require XSLoader; XSLoader​::load() } $buf = ""; open $fh, "<", \$buf; XS​::Typemap​::T_IN($fh); close $fh'
Attempt to free unreferenced scalar​: SV 0x2748628 during global destruction.
% PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package XS​::Typemap; require XSLoader; XSLoader​::load() } $buf = ""; open $fh, "+<", \$buf; XS​::Typemap​::T_OUT($fh); close $fh'
Attempt to free unreferenced scalar​: SV 0xa1c7a8 during global destruction.
%

% ./perl -Ilib -V
Summary of my perl5 (revision 5 version 22 subversion 0) configuration​:
  Commit id​: 416c06f
  Platform​:
  osname=linux, osvers=3.13.0-37-generic, archname=x86_64-linux
  uname='linux shad2 3.13.0-37-generic #64-ubuntu smp mon sep 22 21​:28​:38 utc 2014 x86_64 x86_64 x86_64 gnulinux '
  config_args='-des -Dcc=gcc -Dprefix=/opt/blead-d -Doptimize=-g -O6 -DDEBUGGING -Dusedevel -Uversiononly'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  use64bitint=define, use64bitall=define, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='gcc', ccflags ='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
  optimize='-g -O6',
  cppflags='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
  ccversion='', gccversion='4.8.2', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
  ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='gcc', ldflags =' -fstack-protector -L/usr/local/lib'
  libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
  libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  libc=libc-2.19.so, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.19'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -g -O6 -L/usr/local/lib -fstack-protector'

Characteristics of this binary (from libperl)​:
  Compile-time options​: DEBUGGING HAS_TIMES PERLIO_LAYERS
  PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP
  PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV
  PERL_USE_DEVEL USE_64_BIT_ALL USE_64_BIT_INT
  USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE
  USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME
  USE_PERLIO USE_PERL_ATOF
  Locally applied patches​:
  RC0
  Built under linux
  Compiled at May 9 2015 09​:35​:23
  %ENV​:
  PERL_TEST_MEMORY="8"
  @​INC​:
  lib
  /opt/blead-d/lib/perl5/site_perl/5.22.0/x86_64-linux
  /opt/blead-d/lib/perl5/site_perl/5.22.0
  /opt/blead-d/lib/perl5/5.22.0/x86_64-linux
  /opt/blead-d/lib/perl5/5.22.0
  /opt/blead-d/lib/perl5/site_perl/5.21.9
  /opt/blead-d/lib/perl5/site_perl
  .
%

Not sure I know how to diagnose this, but here's a stack trace for the complaint on the first example​:

PERL_DESTRUCT_LEVEL=2 gdb ./perl
GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+​: GNU GPL version 3 or later <http​://gnu.org/licenses/gpl.html>
This is free software​: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see​:
<http​://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at​:
<http​://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./perl...done.
(gdb) break sv.c​:7069
Breakpoint 1 at 0x528811​: file sv.c, line 7069.
(gdb) run -e '@​INC = "lib"; { package XS​::Typemap; require XSLoader; XSLoader​::load() } XS​::Typemap​::T_STDIO_open("stdio.tmp")'
Starting program​: /src/package/lang/perl/gitperl/perl -e '@​INC = "lib"; { package XS​::Typemap; require XSLoader; XSLoader​::load() } XS​::Typemap​::T_STDIO_open("stdio.tmp")'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, Perl_sv_free2 (sv=sv@​entry=0x9815b0, rc=rc@​entry=0) at sv.c​:7069
7069 Perl_warner(aTHX_ packWARN(WARN_INTERNAL),
(gdb) where
#0 Perl_sv_free2 (sv=sv@​entry=0x9815b0, rc=rc@​entry=0) at sv.c​:7069
#1 0x0000000000527d9a in S_SvREFCNT_dec (sv=0x9815b0) at inline.h​:166
#2 Perl_sv_free (sv=0x9815b0) at sv.c​:7002
#3 Perl_sv_clear (orig_sv=orig_sv@​entry=0x9536b0) at sv.c​:6846
#4 0x00000000005286e1 in Perl_sv_free2 (sv=sv@​entry=0x9536b0,
  rc=<optimized out>) at sv.c​:7033
#5 0x000000000045c20e in S_SvREFCNT_dec (sv=0x9536b0) at inline.h​:166
#6 Perl_gp_free (gv=gv@​entry=0x953698) at gv.c​:2547
#7 0x00000000005280ba in Perl_sv_clear (orig_sv=orig_sv@​entry=0x953680)
  at sv.c​:6684
#8 0x00000000005286e1 in Perl_sv_free2 (sv=sv@​entry=0x953680,
  rc=<optimized out>) at sv.c​:7033
#9 0x000000000045c20e in S_SvREFCNT_dec (sv=0x953680) at inline.h​:166
#10 Perl_gp_free (gv=gv@​entry=0x953668) at gv.c​:2547
#11 0x00000000005280ba in Perl_sv_clear (orig_sv=orig_sv@​entry=0x935fe0)
  at sv.c​:6684
#12 0x00000000005286e1 in Perl_sv_free2 (sv=sv@​entry=0x935fe0,
  rc=<optimized out>) at sv.c​:7033
#13 0x0000000000456715 in S_SvREFCNT_dec (sv=0x935fe0) at inline.h​:166
#14 perl_destruct (my_perl=<optimized out>) at perl.c​:1089
#15 0x0000000000420ab4 in main (argc=3, argv=0x7fffffffe5c8,
  env=0x7fffffffe5e8) at perlmain.c​:127
(gdb) p *sv
$1 = {sv_any = 0x9362b0, sv_refcnt = 0, sv_flags = 255, sv_u = {svu_pv = 0x0,
  svu_iv = 0, svu_uv = 0, svu_nv = 0, svu_rv = 0x0, svu_rx = 0x0,
  svu_array = 0x0, svu_hash = 0x0, svu_gp = 0x0, svu_fp = 0x0}}
(gdb) p /x *(SV*)0x9536b0
$2 = {sv_any = 0x93bf00, sv_refcnt = 0x0, sv_flags = 0x3200000c, sv_u = {
  svu_pv = 0x96dc90, svu_iv = 0x96dc90, svu_uv = 0x96dc90, svu_nv = 0x0,
  svu_rv = 0x96dc90, svu_rx = 0x96dc90, svu_array = 0x96dc90,
  svu_hash = 0x96dc90, svu_gp = 0x96dc90, svu_fp = 0x96dc90}}
(gdb) p *(XPVHV*)0x93bf00
$3 = {xmg_stash = 0x0, xmg_u = {xmg_magic = 0x0, xmg_hash_index = 0},
  xhv_keys = 29, xhv_max = 63}

I'm not sure how to determine what's actually being freed here, suggestions welcome.

@p5pRT
Copy link
Author

p5pRT commented May 9, 2015

From @jkeenan

On Sat May 09 04​:53​:53 2015, hv wrote​:

I'm able to reproduce this. Here are cut-down versions of the 4 cases​:

% PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package
XS​::Typemap; require XSLoader; XSLoader​::load() }
XS​::Typemap​::T_STDIO_open("stdio.tmp")'
Attempt to free unreferenced scalar​: SV 0x1d1b6a8 during global
destruction.
% PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package
XS​::Typemap; require XSLoader; XSLoader​::load() } $buf = ""; open $fh,
"+<", \$buf; XS​::Typemap​::T_INOUT($fh); close $fh'
Attempt to free unreferenced scalar​: SV 0x2844760 during global
destruction.
% PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package
XS​::Typemap; require XSLoader; XSLoader​::load() } $buf = ""; open $fh,
"<", \$buf; XS​::Typemap​::T_IN($fh); close $fh'
Attempt to free unreferenced scalar​: SV 0x2748628 during global
destruction.
% PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package
XS​::Typemap; require XSLoader; XSLoader​::load() } $buf = ""; open $fh,
"+<", \$buf; XS​::Typemap​::T_OUT($fh); close $fh'
Attempt to free unreferenced scalar​: SV 0xa1c7a8 during global
destruction.
%

% ./perl -Ilib -V
Summary of my perl5 (revision 5 version 22 subversion 0)
configuration​:
Commit id​: 416c06f
Platform​:
osname=linux, osvers=3.13.0-37-generic, archname=x86_64-linux
uname='linux shad2 3.13.0-37-generic #64-ubuntu smp mon sep 22
21​:28​:38 utc 2014 x86_64 x86_64 x86_64 gnulinux '
config_args='-des -Dcc=gcc -Dprefix=/opt/blead-d -Doptimize=-g -O6
-DDEBUGGING -Dusedevel -Uversiononly'
hint=recommended, useposix=true, d_sigaction=define
useithreads=undef, usemultiplicity=undef
use64bitint=define, use64bitall=define, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler​:
cc='gcc', ccflags ='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe
-fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64',
optimize='-g -O6',
cppflags='-fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-
protector -I/usr/local/include'
ccversion='', gccversion='4.8.2', gccosandvers=''
intsize=4, longsize=8, ptrsize=8, doublesize=8,
byteorder=12345678, doublekind=3
d_longlong=define, longlongsize=8, d_longdbl=define,
longdblsize=16, longdblkind=3
ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
lseeksize=8
alignbytes=8, prototype=define
Linker and Libraries​:
ld='gcc', ldflags =' -fstack-protector -L/usr/local/lib'
libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.8/include-
fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu
/lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
libs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
libc=libc-2.19.so, so=so, useshrplib=false, libperl=libperl.a
gnulibc_version='2.19'
Dynamic Linking​:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -g -O6 -L/usr/local/lib
-fstack-protector'

Characteristics of this binary (from libperl)​:
Compile-time options​: DEBUGGING HAS_TIMES PERLIO_LAYERS
PERL_DONT_CREATE_GVSV
PERL_HASH_FUNC_ONE_AT_A_TIME_HARD
PERL_MALLOC_WRAP
PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV
PERL_USE_DEVEL USE_64_BIT_ALL USE_64_BIT_INT
USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE
USE_LOCALE_CTYPE USE_LOCALE_NUMERIC
USE_LOCALE_TIME
USE_PERLIO USE_PERL_ATOF
Locally applied patches​:
RC0
Built under linux
Compiled at May 9 2015 09​:35​:23
%ENV​:
PERL_TEST_MEMORY="8"
@​INC​:
lib
/opt/blead-d/lib/perl5/site_perl/5.22.0/x86_64-linux
/opt/blead-d/lib/perl5/site_perl/5.22.0
/opt/blead-d/lib/perl5/5.22.0/x86_64-linux
/opt/blead-d/lib/perl5/5.22.0
/opt/blead-d/lib/perl5/site_perl/5.21.9
/opt/blead-d/lib/perl5/site_perl
.
%

Not sure I know how to diagnose this, but here's a stack trace for the
complaint on the first example​:

PERL_DESTRUCT_LEVEL=2 gdb ./perl
GNU gdb (Ubuntu 7.7-0ubuntu3.1) 7.7
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+​: GNU GPL version 3 or later
<http​://gnu.org/licenses/gpl.html>
This is free software​: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show
copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see​:
<http​://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at​:
<http​://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./perl...done.
(gdb) break sv.c​:7069
Breakpoint 1 at 0x528811​: file sv.c, line 7069.
(gdb) run -e '@​INC = "lib"; { package XS​::Typemap; require XSLoader;
XSLoader​::load() } XS​::Typemap​::T_STDIO_open("stdio.tmp")'
Starting program​: /src/package/lang/perl/gitperl/perl -e '@​INC =
"lib"; { package XS​::Typemap; require XSLoader; XSLoader​::load() }
XS​::Typemap​::T_STDIO_open("stdio.tmp")'
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-
gnu/libthread_db.so.1".

Breakpoint 1, Perl_sv_free2 (sv=sv@​entry=0x9815b0, rc=rc@​entry=0) at
sv.c​:7069
7069 Perl_warner(aTHX_ packWARN(WARN_INTERNAL),
(gdb) where
#0 Perl_sv_free2 (sv=sv@​entry=0x9815b0, rc=rc@​entry=0) at sv.c​:7069
#1 0x0000000000527d9a in S_SvREFCNT_dec (sv=0x9815b0) at inline.h​:166
#2 Perl_sv_free (sv=0x9815b0) at sv.c​:7002
#3 Perl_sv_clear (orig_sv=orig_sv@​entry=0x9536b0) at sv.c​:6846
#4 0x00000000005286e1 in Perl_sv_free2 (sv=sv@​entry=0x9536b0,
rc=<optimized out>) at sv.c​:7033
#5 0x000000000045c20e in S_SvREFCNT_dec (sv=0x9536b0) at inline.h​:166
#6 Perl_gp_free (gv=gv@​entry=0x953698) at gv.c​:2547
#7 0x00000000005280ba in Perl_sv_clear
(orig_sv=orig_sv@​entry=0x953680)
at sv.c​:6684
#8 0x00000000005286e1 in Perl_sv_free2 (sv=sv@​entry=0x953680,
rc=<optimized out>) at sv.c​:7033
#9 0x000000000045c20e in S_SvREFCNT_dec (sv=0x953680) at inline.h​:166
#10 Perl_gp_free (gv=gv@​entry=0x953668) at gv.c​:2547
#11 0x00000000005280ba in Perl_sv_clear
(orig_sv=orig_sv@​entry=0x935fe0)
at sv.c​:6684
#12 0x00000000005286e1 in Perl_sv_free2 (sv=sv@​entry=0x935fe0,
rc=<optimized out>) at sv.c​:7033
#13 0x0000000000456715 in S_SvREFCNT_dec (sv=0x935fe0) at inline.h​:166
#14 perl_destruct (my_perl=<optimized out>) at perl.c​:1089
#15 0x0000000000420ab4 in main (argc=3, argv=0x7fffffffe5c8,
env=0x7fffffffe5e8) at perlmain.c​:127
(gdb) p *sv
$1 = {sv_any = 0x9362b0, sv_refcnt = 0, sv_flags = 255, sv_u =
{svu_pv = 0x0,
svu_iv = 0, svu_uv = 0, svu_nv = 0, svu_rv = 0x0, svu_rx = 0x0,
svu_array = 0x0, svu_hash = 0x0, svu_gp = 0x0, svu_fp = 0x0}}
(gdb) p /x *(SV*)0x9536b0
$2 = {sv_any = 0x93bf00, sv_refcnt = 0x0, sv_flags = 0x3200000c, sv_u
= {
svu_pv = 0x96dc90, svu_iv = 0x96dc90, svu_uv = 0x96dc90, svu_nv =
0x0,
svu_rv = 0x96dc90, svu_rx = 0x96dc90, svu_array = 0x96dc90,
svu_hash = 0x96dc90, svu_gp = 0x96dc90, svu_fp = 0x96dc90}}
(gdb) p *(XPVHV*)0x93bf00
$3 = {xmg_stash = 0x0, xmg_u = {xmg_magic = 0x0, xmg_hash_index = 0},
xhv_keys = 29, xhv_max = 63}

I'm not sure how to determine what's actually being freed here,
suggestions welcome.

With the configuration attached, I could not reproduce your findings​:

#####
[perl] 33 $ PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package XS​::Typemap; require XSLoader; XSLoader​::load() } XS​::Typemap​::T_STDIO_open("stdio.tmp")'
[perl] 34 $ PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package XS​::Typemap; require XSLoader; XSLoader​::load() } $buf = ""; open $fh, "+<", \$buf; XS​::Typemap​::T_INOUT($fh); close $fh'
[perl] 35 $ PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package XS​::Typemap; require XSLoader; XSLoader​::load() } $buf = ""; open $fh, "<", \$buf; XS​::Typemap​::T_IN($fh); close $fh'
[perl] 36 $ PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package XS​::Typemap; require XSLoader; XSLoader​::load() } $buf = ""; open $fh, "+<", \$buf; XS​::Typemap​::T_OUT($fh); close $fh'
#####

No error output in any of the 4 cases. Am I doing something wrong? Is my configuration significantly different from yours?

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented May 9, 2015

From @jkeenan

Summary of my perl5 (revision 5 version 22 subversion 0) configuration​:
  Commit id​: 416c06f
  Platform​:
  osname=linux, osvers=3.13.0-52-generic, archname=x86_64-linux
  uname='linux zareason 3.13.0-52-generic #86-ubuntu smp mon may 4 04​:32​:59 utc 2015 x86_64 x86_64 x86_64 gnulinux '
  config_args='-des -Dusedevel -Uversiononly -Dprefix=/home/jkeenan/testing/blead -Dman1dir=none -Dman3dir=none'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  use64bitint=define, use64bitall=define, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
  optimize='-O2',
  cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
  ccversion='', gccversion='4.8.2', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
  ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -fstack-protector -L/usr/local/lib'
  libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.8/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib /lib64 /usr/lib64
  libs=-lpthread -lnsl -ldb -ldl -lm -lcrypt -lutil -lc
  perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  libc=libc-2.19.so, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.19'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector'

Characteristics of this binary (from libperl)​:
  Compile-time options​: HAS_TIMES PERLIO_LAYERS PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP
  PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV
  PERL_USE_DEVEL USE_64_BIT_ALL USE_64_BIT_INT
  USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE
  USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME
  USE_PERLIO USE_PERL_ATOF
  Locally applied patches​:
  RC0
  Built under linux
  Compiled at May 9 2015 07​:27​:49
  %ENV​:
  PERL5LIB="/home/jkeenan/perl5/lib/perl5"
  PERLBREW_BASHRC_VERSION="0.67"
  PERLBREW_HOME="/home/jkeenan/.perlbrew"
  PERLBREW_MANPATH="/home/jkeenan/perl5/perlbrew/perls/perl-5.20.1/man"
  PERLBREW_PATH="/home/jkeenan/perl5/perlbrew/bin​:/home/jkeenan/perl5/perlbrew/perls/perl-5.20.1/bin"
  PERLBREW_PERL="perl-5.20.1"
  PERLBREW_ROOT="/home/jkeenan/perl5/perlbrew"
  PERLBREW_VERSION="0.67"
  PERL_LOCAL_LIB_ROOT="/home/jkeenan/perl5"
  PERL_MB_OPT="--install_base "/home/jkeenan/perl5""
  PERL_MM_OPT="INSTALL_BASE=/home/jkeenan/perl5"
  PERL_WORKDIR="gitwork/perl"
  @​INC​:
  lib
  /home/jkeenan/perl5/lib/perl5/x86_64-linux
  /home/jkeenan/perl5/lib/perl5
  /home/jkeenan/testing/blead/lib/perl5/site_perl/5.22.0/x86_64-linux
  /home/jkeenan/testing/blead/lib/perl5/site_perl/5.22.0
  /home/jkeenan/testing/blead/lib/perl5/5.22.0/x86_64-linux
  /home/jkeenan/testing/blead/lib/perl5/5.22.0
  .

@p5pRT
Copy link
Author

p5pRT commented May 9, 2015

From perl@profvince.com

With the configuration attached, I could not reproduce your findings​:

#####
[perl] 33 $ PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package XS​::Typemap; require XSLoader; XSLoader​::load() } XS​::Typemap​::T_STDIO_open("stdio.tmp")'
[perl] 34 $ PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package XS​::Typemap; require XSLoader; XSLoader​::load() } $buf = ""; open $fh, "+<", \$buf; XS​::Typemap​::T_INOUT($fh); close $fh'
[perl] 35 $ PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package XS​::Typemap; require XSLoader; XSLoader​::load() } $buf = ""; open $fh, "<", \$buf; XS​::Typemap​::T_IN($fh); close $fh'
[perl] 36 $ PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package XS​::Typemap; require XSLoader; XSLoader​::load() } $buf = ""; open $fh, "+<", \$buf; XS​::Typemap​::T_OUT($fh); close $fh'
#####

No error output in any of the 4 cases. Am I doing something wrong? Is my configuration significantly different from yours?

Thank you very much.

You don't have DEBUGGING enabled, for starters.

Vincent

@p5pRT
Copy link
Author

p5pRT commented May 9, 2015

From @jkeenan

On Sat May 09 07​:01​:15 2015, perl@​profvince.com wrote​:

With the configuration attached, I could not reproduce your findings​:

No error output in any of the 4 cases. Am I doing something wrong?
Is my configuration significantly different from yours?

Thank you very much.

You don't have DEBUGGING enabled, for starters.

Yes, confirmed.

#####
$ !33;!34;!35;!36
PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package XS​::Typemap; require XSLoader; XSLoader​::load() } XS​::Typemap​::T_STDIO_open("stdio.tmp")';PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package XS​::Typemap; require XSLoader; XSLoader​::load() } $buf = ""; open $fh, "+<", \$buf; XS​::Typemap​::T_INOUT($fh); close $fh';PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package XS​::Typemap; require XSLoader; XSLoader​::load() } $buf = ""; open $fh, "<", \$buf; XS​::Typemap​::T_IN($fh); close $fh';PERL_DESTRUCT_LEVEL=2 ./perl -e '@​INC = "lib"; { package XS​::Typemap; require XSLoader; XSLoader​::load() } $buf = ""; open $fh, "+<", \$buf; XS​::Typemap​::T_OUT($fh); close $fh'
Attempt to free unreferenced scalar​: SV 0x1702fa8 during global destruction.
Attempt to free unreferenced scalar​: SV 0x1249a30 during global destruction.
Attempt to free unreferenced scalar​: SV 0x22e5270 during global destruction.
Attempt to free unreferenced scalar​: SV 0xea4148 during global destruction.
#####
--
James E Keenan (jkeenan@​cpan.org)

@p5pRT
Copy link
Author

p5pRT commented May 9, 2015

From @bulk88

Confirmed on Win32 with DEBUGGING.

C​:\p521\srcpara>perl -Ilib -e " { package XS​::Typemap; require XSLoader; XSLoade
r​::load() } XS​::Typemap​::T_STDIO_open(\"stdio.tmp\")"
Attempt to free unreferenced scalar​: SV 0x1ca24dc, Perl interpreter​: 0x1c44dec d
uring global destruction.

C​:\p521\srcpara>perl -V
Summary of my perl5 (revision 5 version 21 subversion 12) configuration​:
  Derived from​: 5d370191488b4e919956a033fa450813df23ee36
  Platform​:
  osname=MSWin32, osvers=5.2, archname=MSWin32-x86-multi-thread
  uname=''
  config_args='undef'
  hint=recommended, useposix=true, d_sigaction=undef
  useithreads=define, usemultiplicity=define
  use64bitint=undef, use64bitall=undef, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cl', ccflags ='-nologo -GF -W3 -O1 -MD -Zi -DDEBUGGING -GL -DWIN32 -D_CO
NSOLE -DNO_STRICT -D_CRT_SECURE_NO_DEPRECATE -D_CRT_NONSTDC_NO_DEPRECATE -DPERL
_TEXTMODE_SCRIPTS -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS',
  optimize='-O1 -MD -Zi -DDEBUGGING -GL',
  cppflags='-DWIN32'
  ccversion='15.00.30729.01', gccversion='', gccosandvers=''
  intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234, doublekind=3

  d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=8, longdblki
nd=0
  ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', lseeksi
ze=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='link', ldflags ='-nologo -nodefaultlib -debug -libpath​:"c​:\per
l\lib\CORE" -machine​:x86 "/manifestdependency​:type='Win32' name='Mic
rosoft.Windows.Common-Controls' version='6.0.0.0' processorArchitecture='*' publ
icKeyToken='6595b64144ccf1df' language='*'"'
  libpth=\lib
  libs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg32.li
b advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uuid.lib ws2_32.l
ib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt.lib

  perllibs=oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib comdlg3
2.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uuid.lib ws2_
32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib comctl32.lib msvcrt
.lib
  libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl521.lib
  gnulibc_version=''
  Dynamic Linking​:
  dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
  cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug
-libpath​:"c​:\perl\lib\CORE" -machine​:x86 "/manifestdependency​:type='
Win32' name='Microsoft.Windows.Common-Controls' version='6.0.0.0' processorArchi
tecture='*' publicKeyToken='6595b64144ccf1df' language='*'"'

Characteristics of this binary (from libperl)​:
  Compile-time options​: DEBUGGING HAS_TIMES HAVE_INTERP_INTERN MULTIPLICITY
  PERLIO_LAYERS PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD
  PERL_IMPLICIT_CONTEXT PERL_IMPLICIT_SYS
  PERL_MALLOC_WRAP PERL_NEW_COPY_ON_WRITE
  PERL_PRESERVE_IVUV PERL_TRACK_MEMPOOL USE_ITHREADS
  USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE
  USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME
  USE_PERLIO USE_PERL_ATOF
  Locally applied patches​:
  uncommitted-changes
  Built under MSWin32
  Compiled at May 9 2015 12​:47​:14
  %ENV​:
  PERL_DESTRUCT_LEVEL="2"
  @​INC​:
  C​:/p521/srcpara/lib
  .

C​:\p521\srcpara>

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented May 9, 2015

From @bulk88

I am preparing a patch for this ticket.

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented May 10, 2015

From @bulk88

On Sat May 09 15​:55​:43 2015, bulk88 wrote​:

I am preparing a patch for this ticket.

stack traces of the SV * that caused "Attempt to free unreferenced scalar"

created at

+ cop_file 0x008f6f34 "-e" char *
  cop_line 1 unsigned long

  perl521.dll!S_new_SV(interpreter * my_perl=0x003853cc, const char * file=0x282bd4ac, int line=5640, const char * func=0x282beb44) Line 348 C
  perl521.dll!Perl_newSV(interpreter * my_perl=0x003853cc, const unsigned int len=0) Line 5640 + 0x18 C
  perl521.dll!Perl_hv_common(interpreter * my_perl=0x003853cc, hv * hv=0x0090d9e4, sv * keysv=0x00000000, const char * key=0x0098b4b9, unsigned int klen=6, int flags=0, int action=48, sv * val=0x00000000, unsigned long hash=70165563) Line 775 + 0x22 C
  perl521.dll!Perl_hv_common_key_len(interpreter * my_perl=0x003853cc, hv * hv=0x0090d9e4, const char * key=0x0098b4b9, long klen_i32=6, const int action=48, sv * val=0x00000000, const unsigned long hash=0) Line 333 + 0x27 C
  perl521.dll!Perl_gv_fetchpvn_flags(interpreter * my_perl=0x003853cc, const char * nambeg=0x0098b4ac, unsigned int full_len=19, long flags=1, svtype sv_type=SVt_PVGV) Line 2265 + 0x3d C
  perl521.dll!Perl_gv_fetchpv(interpreter * my_perl=0x003853cc, const char * nambeg=0x0098b4ac, long add=1, svtype sv_type=SVt_PVGV) Line 1513 + 0x22 C
  perl521.dll!Perl_newGVgen_flags(interpreter * my_perl=0x003853cc, const char * pack=0x1000ca30, unsigned long flags=0) Line 2466 + 0x5f pack is 0x1000ca30 "XS​::Typemap"
  Typemap.dll!XS_XS__Typemap_T_STDIO_open(interpreter * my_perl=0x003853cc, cv * cv=0x00922bcc) Line 1698 + 0x11 C
  perl521.dll!Perl_pp_entersub(interpreter * my_perl=0x003853cc) Line 3270 + 0x10 C
  perl521.dll!Perl_runops_debug(interpreter * my_perl=0x003853cc) Line 2234 + 0xd C
  perl521.dll!S_run_body(interpreter * my_perl=0x003853cc, long oldscope=1) Line 2448 + 0xd C
  perl521.dll!perl_run(interpreter * my_perl=0x003853cc) Line 2374 C
  perl521.dll!RunPerl(int argc=4, char * * argv=0x00384eb8, char * * env=0x00382830) Line 258 + 0x9 C++
  perl.exe!main(int argc=4, char * * argv=0x00384eb8, char * * env=0x00383298) Line 39 + 0x12 C
  perl.exe!mainCRTStartup() Line 398 + 0xe C
  kernel32.dll!_BaseProcessStart@​4() + 0x23

first dec event, but the SV on temp stack isn;t the one that gets double freed, the doublefreed one "is inside" the tmps stack one, I can't dump the SVs since p5p rejected this patch https://rt-archive.perl.org/perl5/Ticket/Display.html?id=121932 and I am on a virgin blead without my normal 30 patches applied, curcop is "-e" line 0

  perl521.dll!Perl_sv_clear(interpreter * my_perl=0x003853cc, sv * const orig_sv=0x0093edfc) Line 6849 C
  perl521.dll!Perl_sv_free2(interpreter * my_perl=0x003853cc, sv * const sv=0x0093edfc, const unsigned long rc=0x00000001) Line 7033 + 0xd C
  perl521.dll!S_SvREFCNT_dec_NN(interpreter * my_perl=0x003853cc, sv * sv=0x0093edfc) Line 177 + 0x11 C
  perl521.dll!Perl_free_tmps(interpreter * my_perl=0x003853cc) Line 178 + 0xd C
  perl521.dll!perl_run(interpreter * my_perl=0x003853cc) Line 2376 + 0x17 C
  perl521.dll!RunPerl(int argc=0x00000004, char * * argv=0x00384eb8, char * * env=0x00382830) Line 258 + 0x9 C++
  perl.exe!main(int argc=0x00000004, char * * argv=0x00384eb8, char * * env=0x00383298) Line 39 + 0x12 C
  perl.exe!mainCRTStartup() Line 398 + 0xe C
  kernel32.dll!_BaseProcessStart@​4() + 0x23

  /* unrolled SvREFCNT_dec and sv_free2 follows​: */

  if (!sv)
  continue;
  if (!SvREFCNT(sv)) {
  sv_free(sv);
  continue;
  }
  if (--(SvREFCNT(sv)))<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<line 6849
  continue;
#ifdef DEBUGGING
  if (SvTEMP(sv)) {
  Perl_ck_warner_d(aTHX_ packWARN(WARN_DEBUGGING),
  "Attempt to free temp prematurely​: SV 0x%"UVxf
  pTHX__FORMAT, PTR2UV(sv) pTHX__VALUE);
  continue;
  }
#endif

2nd dec event, now with "Attempt to free unreferenced scalar"

  perl521.dll!Perl_sv_free2(interpreter * my_perl=0x003853cc, sv * const sv=0x0094ad24, const unsigned long rc=0x00000000) Line 7071 C
  perl521.dll!S_SvREFCNT_dec(interpreter * my_perl=0x003853cc, sv * sv=0x0094ad24) Line 166 + 0x11 C
  perl521.dll!Perl_sv_free(interpreter * my_perl=0x003853cc, sv * const sv=0x0094ad24) Line 7002 + 0xd C
  perl521.dll!Perl_sv_clear(interpreter * my_perl=0x003853cc, sv * const orig_sv=0x0090d9e4) Line 6846 + 0xd C
  perl521.dll!Perl_sv_free2(interpreter * my_perl=0x003853cc, sv * const sv=0x0090d9e4, const unsigned long rc=0x00000001) Line 7033 + 0xd C
  perl521.dll!S_SvREFCNT_dec(interpreter * my_perl=0x003853cc, sv * sv=0x0090d9e4) Line 166 + 0x11 C
  perl521.dll!Perl_gp_free(interpreter * my_perl=0x003853cc, gv * gv=0x0090d9c4) Line 2547 + 0xd C
  perl521.dll!Perl_sv_clear(interpreter * my_perl=0x003853cc, sv * const orig_sv=0x0090d9a4) Line 6684 + 0xd C
  perl521.dll!Perl_sv_free2(interpreter * my_perl=0x003853cc, sv * const sv=0x0090d9a4, const unsigned long rc=0x00000001) Line 7033 + 0xd C
  perl521.dll!S_SvREFCNT_dec(interpreter * my_perl=0x003853cc, sv * sv=0x0090d9a4) Line 166 + 0x11 C
  perl521.dll!Perl_gp_free(interpreter * my_perl=0x003853cc, gv * gv=0x0090d984) Line 2547 + 0xd C
  perl521.dll!Perl_sv_clear(interpreter * my_perl=0x003853cc, sv * const orig_sv=0x003890ec) Line 6684 + 0xd C
  perl521.dll!Perl_sv_free2(interpreter * my_perl=0x003853cc, sv * const sv=0x003890ec, const unsigned long rc=0x00000001) Line 7033 + 0xd C
  perl521.dll!S_SvREFCNT_dec(interpreter * my_perl=0x003853cc, sv * sv=0x003890ec) Line 166 + 0x11 C
  perl521.dll!perl_destruct(interpreter * my_perl=0x003853cc) Line 1089 + 0xd C
  perl521.dll!RunPerl(int argc=0x00000004, char * * argv=0x00384eb8, char * * env=0x00382830) Line 262 + 0x9 C++
  perl.exe!main(int argc=0x00000004, char * * argv=0x00384eb8, char * * env=0x00383298) Line 39 + 0x12 C
  perl.exe!mainCRTStartup() Line 398 + 0xe C
  kernel32.dll!_BaseProcessStart@​4() + 0x23

PERL_DESTRUCT_LEVEL = 2 didn't cause a crash for me on non-DEBUGGING for me, just "Attempt to free unreferenced" warnings, but there seems to be no way to capture or make fatal that to harness, those warnings that come from perl_destruct, but I did figure out another way to trigger the crash. I split it into 2 patches to be sure that Typemap.t fails reliably, perhaps some other people should smoke it with just the first patch to make sure it fails (and it randomly SEGVed for me on non-DEBUGGING) on platforms other than Win32. Maybe the 2 patches should be squashed into one for bisectablity or something.

harness doesn't listen to STDERR, and a warning doesn't normally trigger a non-zero exit, and use warnings fatal doesn't seem to work inside perl_destruct (is that a bug?). Maybe those "panic" style internal warnings should be croaks/fatal, not warnings. Emitting a warning about guaranteed "memory corruption" and continuing execution even though the Perl VM is corrupt is a bit silly to me. A year or 2 ago khw did something that was causing unreferenced warnings on the George Win32 smoker but still passing for months until I spotted it one day while happening to watch a "make test" on my machine. The issue is nobody reviews the 1 MB smoke logs by eye and harness doesn't care about STDERR.

Not only were the typemaps causing memory corruption, there were some other bugs and shoddy code with the typemap entries I fixed such as the typemap entries only working for RETVAL and not an outgoing @​_ arg, using raw decimal numbers as flags instead of named constants flags, and duplicate hash lookups.

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented May 10, 2015

From @bulk88

0001-add-test-that-fails-for-124181-to-Typemap.t.patch
From 594d4089be5c8783f6fa1f53d315e4a701145d0a Mon Sep 17 00:00:00 2001
From: Daniel Dragan <bulk88@hotmail.com>
Date: Sun, 10 May 2015 11:36:05 -0400
Subject: [PATCH 1/2] add test that fails for #124181 to Typemap.t

These tests will either fail with harness, and randomly SEGV for
me, which is intentional since they are testing memory
corruption.
---
 ext/XS-Typemap/Typemap.pm  |    4 ++--
 ext/XS-Typemap/Typemap.xs  |    9 +++++++++
 ext/XS-Typemap/t/Typemap.t |   20 ++++++++++++++++++--
 3 files changed, 29 insertions(+), 4 deletions(-)

diff --git a/ext/XS-Typemap/Typemap.pm b/ext/XS-Typemap/Typemap.pm
index de3319b..a1ae021 100644
--- a/ext/XS-Typemap/Typemap.pm
+++ b/ext/XS-Typemap/Typemap.pm
@@ -36,7 +36,7 @@ require XSLoader;
 
 use vars qw/ $VERSION @EXPORT /;
 
-$VERSION = '0.13';
+$VERSION = '0.14';
 
 @EXPORT = (qw/
 	   T_SV
@@ -76,7 +76,7 @@ $VERSION = '0.13';
 	   T_OPAQUEPTR_IN T_OPAQUEPTR_OUT T_OPAQUEPTR_OUT_short
            T_OPAQUEPTR_IN_struct T_OPAQUEPTR_OUT_struct
 	   T_ARRAY
-	   T_STDIO_open T_STDIO_close T_STDIO_print
+	   T_STDIO_open T_STDIO_open_ret_in_arg T_STDIO_close T_STDIO_print
            T_PACKED_in T_PACKED_out
            T_PACKEDARRAY_in T_PACKEDARRAY_out
            T_INOUT T_IN T_OUT
diff --git a/ext/XS-Typemap/Typemap.xs b/ext/XS-Typemap/Typemap.xs
index 3fa0e74ab..8314cc2 100644
--- a/ext/XS-Typemap/Typemap.xs
+++ b/ext/XS-Typemap/Typemap.xs
@@ -906,6 +906,15 @@ T_STDIO_open( file )
  OUTPUT:
   RETVAL
 
+void
+T_STDIO_open_ret_in_arg( file, io)
+  const char * file
+  FILE * io = NO_INIT
+ CODE:
+  io = xsfopen( file );
+ OUTPUT:
+  io
+
 SysRet
 T_STDIO_close( f )
   PerlIO * f
diff --git a/ext/XS-Typemap/t/Typemap.t b/ext/XS-Typemap/t/Typemap.t
index 27b4086..49ac479 100644
--- a/ext/XS-Typemap/t/Typemap.t
+++ b/ext/XS-Typemap/t/Typemap.t
@@ -6,10 +6,11 @@ BEGIN {
     }
 }
 
-use Test::More tests => 152;
+use Test::More tests => 156;
 
 use strict;
-use warnings;
+#catch WARN_INTERNAL type errors, and anything else unexpected
+use warnings FATAL => 'all';
 use XS::Typemap;
 
 pass();
@@ -213,6 +214,7 @@ is( T_PV("a string"), "a string");
 is( T_PV(52), 52);
 ok !defined T_PV_null, 'RETVAL = NULL returns undef for char*';
 {
+    use warnings NONFATAL => 'all';
     my $uninit;
     local $SIG{__WARN__} = sub { ++$uninit if shift =~ /uninit/ };
     () = ''.T_PV_null;
@@ -393,6 +395,16 @@ if (defined $fh) {
   }
 }
 
+$fh = "FOO";
+T_STDIO_open_ret_in_arg( $testfile, $fh);
+ok( $fh ne "FOO", 'return io in arg open succeeds');
+ok( print($fh "first line\n"), 'can print to return io in arg');
+ok( close($fh), 'can close return io in arg');
+$fh = "FOO";
+#now with a bad file name to make sure $fh is written to on failure
+T_STDIO_open_ret_in_arg( "", $fh);
+ok( !defined$fh, 'return io in arg open failed successfully');
+
 # T_INOUT
 note("T_INOUT");
 SCOPE: {
@@ -439,6 +451,10 @@ SCOPE: {
   ok(!close $fh2);
 }
 
+# Perl RT #124181 SEGV due to double free in typemap
+# "Attempt to free unreferenced scalar"
+%{*{main::XS::}{HASH}} = ();
+
 sub is_approx {
   my ($l, $r, $n) = @_;
   if (not defined $l or not defined $r) {
-- 
1.7.9.msysgit.0

@p5pRT
Copy link
Author

p5pRT commented May 10, 2015

From @bulk88

0002-fix-124181-double-free-refcnt-problems-in-IO-types-i.patch
From a5ccb571fc79c106e20af7e9bf690ab98748c7de Mon Sep 17 00:00:00 2001
From: Daniel Dragan <bulk88@hotmail.com>
Date: Sat, 9 May 2015 23:46:11 -0400
Subject: [PATCH 2/2] fix #124181 double free/refcnt problems in IO types in
 typemap

commit 50e5165b96 "stop T_IN/OUT/INOUT/STDIO typemaps leaking" changed
newRV to newRV_noinc, but the GV * returned by newGVgen() is owned by the
package tree, like the SV * returned by get_sv(). Now when the RV to GV is
freed on mortal stack, the GV * in the package tree is freed, and now there
is a freed GV * in the package tree, if you turn on "PERL_DESTRUCT_LEVEL=2"
(and perhaps DEBUGGING is needed too), the package tree is destroyed SV *
by SV *, and perl will eventually warn with
"Attempt to free unreferenced scalar" which a very bad panic type warning.

Also fix the problem, that if this OUTPUT: type is being used for an
incoming arg, not the outgoing RETVAL arg, you can't assign a new SV*
ontop of the old one, that only works for perl stack return args, so
replace "$arg = &PL_sv_undef;" with "sv_setsv($arg, &PL_sv_undef);" if its
not RETVAL, this way OUTPUT on incoming args also works if it goes down the
error path. For efficiency, in a RETVAL siutation, let the undef original
SV* in $arg which is typically obtained from sv_newmortal() by xsubpp pass
through if we error out.

Also for efficiency, if it is RETVAL (which is more common) dont do the
sv_setsv/SvREFCNT_dec_NN stuff (2 function calls), just mortalize
(1 function call) the ex-temp RV and arrange for the RV to wind up on
perl stack.

Also, the GV * already knows what HV * stash it belongs to, so avoid the
stash lookup done by gv_stashpv() and just use GvSTASH which are simple
pointer derefs.
---
 lib/ExtUtils/typemap |   48 ++++++++++++++++++++++++------------------------
 1 files changed, 24 insertions(+), 24 deletions(-)

diff --git a/lib/ExtUtils/typemap b/lib/ExtUtils/typemap
index 831baad..5f61527 100644
--- a/lib/ExtUtils/typemap
+++ b/lib/ExtUtils/typemap
@@ -389,47 +389,47 @@ T_STDIO
 	    GV *gv = newGVgen("$Package");
 	    PerlIO *fp = PerlIO_importFILE($var,0);
 	    if ( fp && do_open(gv, "+<&", 3, FALSE, 0, 0, fp) ) {
-                SV *rv = newRV_noinc((SV*)gv);
-                rv = sv_bless(rv, gv_stashpv("$Package",1));
-		sv_setsv($arg, rv);
-                SvREFCNT_dec_NN(rv);
-            }
+		SV *rv = newRV_inc((SV*)gv);
+		rv = sv_bless(rv, GvSTASH(gv));
+		${"$var" eq "RETVAL" ? \"$arg = sv_2mortal(rv);"
+		    : \"sv_setsv($arg, rv);\n\t\tSvREFCNT_dec_NN(rv);"}
+	    }${"$var" ne "RETVAL" ? \"
 	    else
-		$arg = &PL_sv_undef;
+		sv_setsv($arg, &PL_sv_undef);\n" : \""}
 	}
 T_IN
 	{
 	    GV *gv = newGVgen("$Package");
 	    if ( do_open(gv, "<&", 2, FALSE, 0, 0, $var) ) {
-                SV *rv = newRV_noinc((SV*)gv);
-                rv = sv_bless(rv, gv_stashpv("$Package",1));
-		sv_setsv($arg, rv);
-                SvREFCNT_dec_NN(rv);
-            }
+		SV *rv = newRV_inc((SV*)gv);
+		rv = sv_bless(rv, GvSTASH(gv));
+		${"$var" eq "RETVAL" ? \"$arg = sv_2mortal(rv);"
+		    : \"sv_setsv($arg, rv);\n\t\tSvREFCNT_dec_NN(rv);"}
+	    }${"$var" ne "RETVAL" ? \"
 	    else
-		$arg = &PL_sv_undef;
+		sv_setsv($arg, &PL_sv_undef);\n" : \""}
 	}
 T_INOUT
 	{
 	    GV *gv = newGVgen("$Package");
 	    if ( do_open(gv, "+<&", 3, FALSE, 0, 0, $var) ) {
-                SV *rv = newRV_noinc((SV*)gv);
-                rv = sv_bless(rv, gv_stashpv("$Package",1));
-		sv_setsv($arg, rv);
-                SvREFCNT_dec_NN(rv);
-            }
+		SV *rv = newRV_inc((SV*)gv);
+		rv = sv_bless(rv, GvSTASH(gv));
+		${"$var" eq "RETVAL" ? \"$arg = sv_2mortal(rv);"
+		    : \"sv_setsv($arg, rv);\n\t\tSvREFCNT_dec_NN(rv);"}
+	    }${"$var" ne "RETVAL" ? \"
 	    else
-		$arg = &PL_sv_undef;
+		sv_setsv($arg, &PL_sv_undef);\n" : \""}
 	}
 T_OUT
 	{
 	    GV *gv = newGVgen("$Package");
 	    if ( do_open(gv, "+>&", 3, FALSE, 0, 0, $var) ) {
-                SV *rv = newRV_noinc((SV*)gv);
-                rv = sv_bless(rv, gv_stashpv("$Package",1));
-		sv_setsv($arg, rv);
-                SvREFCNT_dec_NN(rv);
-            }
+		SV *rv = newRV_inc((SV*)gv);
+		rv = sv_bless(rv, GvSTASH(gv));
+		${"$var" eq "RETVAL" ? \"$arg = sv_2mortal(rv);"
+		    : \"sv_setsv($arg, rv);\n\t\tSvREFCNT_dec_NN(rv);"}
+	    }${"$var" ne "RETVAL" ? \"
 	    else
-		$arg = &PL_sv_undef;
+		sv_setsv($arg, &PL_sv_undef);\n" : \""}
 	}
-- 
1.7.9.msysgit.0

@p5pRT
Copy link
Author

p5pRT commented May 15, 2015

From @rjbs

On Sat May 09 15​:55​:43 2015, bulk88 wrote​:

I am preparing a patch for this ticket.

Could we get a review here? I'd like to know how we feel about getting this into RC1, which I'd like to get out the door in the next few days.

--
rjbs

@p5pRT
Copy link
Author

p5pRT commented May 17, 2015

From @iabyn

On Fri, May 15, 2015 at 02​:49​:33PM -0700, Ricardo SIGNES via RT wrote​:

On Sat May 09 15​:55​:43 2015, bulk88 wrote​:

I am preparing a patch for this ticket.

Could we get a review here? I'd like to know how we feel about getting
this into RC1, which I'd like to get out the door in the next few days.

I've just looked at it. I don't like it at all, from the perspective that,
this close to RC1, the one commit is doing several things​: fixing the
premature free that is the issue of this ticket, but also
fixing/changing/optimising a whole bunch of other stuff too, which I
certainly don't understand without looking at it a whole lot more closely.

In more detail​:

Since perl 5.000 for T_IN, T_OUT, T_INOUT, and since 5.8.0 for T_STDIO,
these typemaps have leaked an RV and GV per invocation.

My commit 50e5165 of Dec 2014 fixed those leaks, but was
over-enthusiastic, in that it "fixed" the GV leak twice, so now the GV may
be prematurely freed.

Daniel's suggested change fixes that (changing the newRV_noinc back to
newRV_inc), but does a bunch of extra stuff related to those typemaps.

I recommend that at this very late stage, we revert my 50e5165 commit
and release 5.22 with known leaks in those typemaps (leaks that nobody
noticed for around 20 years), then readdress this issue post 5.22.

I've submitted the revert for smoking​: smoke-me/davem/rt124181.

Failing that, we should do just the minimal fix of s/newRV_noinc/newRV_inc/,
and readdress the remaining issues post-5.22.

--
Spock (or Data) is fired from his high-ranking position for not being able
to understand the most basic nuances of about one in three sentences that
anyone says to him.
  -- Things That Never Happen in "Star Trek" #19

@p5pRT
Copy link
Author

p5pRT commented May 18, 2015

From @khwilliamson

On 05/17/2015 01​:00 PM, Dave Mitchell wrote​:

recommend that at this very late stage, we revert my 50e5165 commit
and release 5.22 with known leaks in those typemaps (leaks that nobody
noticed for around 20 years), then readdress this issue post 5.22.

Karl Williamson pushed to smoke-me/khw-revert50e5165b9
(v5.21.11-144-g8c93413)​: created

@p5pRT
Copy link
Author

p5pRT commented May 18, 2015

From @iabyn

On Mon, May 18, 2015 at 08​:42​:57AM -0600, Karl Williamson wrote​:

On 05/17/2015 01​:00 PM, Dave Mitchell wrote​:

recommend that at this very late stage, we revert my 50e5165 commit
and release 5.22 with known leaks in those typemaps (leaks that nobody
noticed for around 20 years), then readdress this issue post 5.22.

Karl Williamson pushed to smoke-me/khw-revert50e5165b9
(v5.21.11-144-g8c93413)​: created

Later on in my email I said​:

  I've submitted the revert for smoking​: smoke-me/davem/rt124181.

I'm assuming your smoke is an accidental duplicate of that???

--
Wesley Crusher gets beaten up by his classmates for being a smarmy git,
and consequently has a go at making some friends of his own age for a
change.
  -- Things That Never Happen in "Star Trek" #18

@p5pRT
Copy link
Author

p5pRT commented May 18, 2015

From @khwilliamson

On 05/18/2015 08​:49 AM, Dave Mitchell wrote​:

On Mon, May 18, 2015 at 08​:42​:57AM -0600, Karl Williamson wrote​:

On 05/17/2015 01​:00 PM, Dave Mitchell wrote​:

recommend that at this very late stage, we revert my 50e5165 commit
and release 5.22 with known leaks in those typemaps (leaks that nobody
noticed for around 20 years), then readdress this issue post 5.22.

Karl Williamson pushed to smoke-me/khw-revert50e5165b9
(v5.21.11-144-g8c93413)​: created

Later on in my email I said​:

 I've submitted the revert for smoking&#8203;: smoke\-me/davem/rt124181\.

I'm assuming your smoke is an accidental duplicate of that???

Yes. Sorry.

@p5pRT
Copy link
Author

p5pRT commented May 18, 2015

From @iabyn

On Mon, May 18, 2015 at 09​:34​:41AM -0600, Karl Williamson wrote​:

Yes. Sorry.

No worries :-)

My branch has smoked ok. I'll merge it sometime later this evening (GMT+1)
unless there's any further feedback.

--
Never do today what you can put off till tomorrow.

@p5pRT
Copy link
Author

p5pRT commented May 18, 2015

From @iabyn

On Mon, May 18, 2015 at 06​:20​:51PM +0100, Dave Mitchell wrote​:

On Mon, May 18, 2015 at 09​:34​:41AM -0600, Karl Williamson wrote​:

Yes. Sorry.

No worries :-)

My branch has smoked ok. I'll merge it sometime later this evening (GMT+1)
unless there's any further feedback.

Now reverted with bae466e.

--
That he said that that that that is is is debatable, is debatable.

@p5pRT
Copy link
Author

p5pRT commented Jun 24, 2015

From @bulk88

Updated patches for blead/5.23.

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Jun 24, 2015

From @bulk88

0001-add-test-that-fails-for-124181-to-Typemap.t.patch
From 1c8b0e23ba3b68cc69354e585bdfb7ec7518b6fe Mon Sep 17 00:00:00 2001
From: Daniel Dragan <bulk88@hotmail.com>
Date: Sun, 10 May 2015 11:36:05 -0400
Subject: [PATCH 1/2] add test that fails for #124181 to Typemap.t

These tests will either fail with harness, and randomly SEGV for
me, which is intentional since they are testing memory
corruption.
---
 ext/XS-Typemap/Typemap.pm  |    4 ++--
 ext/XS-Typemap/Typemap.xs  |    9 +++++++++
 ext/XS-Typemap/t/Typemap.t |   20 ++++++++++++++++++--
 3 files changed, 29 insertions(+), 4 deletions(-)

diff --git a/ext/XS-Typemap/Typemap.pm b/ext/XS-Typemap/Typemap.pm
index de3319b..a1ae021 100644
--- a/ext/XS-Typemap/Typemap.pm
+++ b/ext/XS-Typemap/Typemap.pm
@@ -36,7 +36,7 @@ require XSLoader;
 
 use vars qw/ $VERSION @EXPORT /;
 
-$VERSION = '0.13';
+$VERSION = '0.14';
 
 @EXPORT = (qw/
 	   T_SV
@@ -76,7 +76,7 @@ $VERSION = '0.13';
 	   T_OPAQUEPTR_IN T_OPAQUEPTR_OUT T_OPAQUEPTR_OUT_short
            T_OPAQUEPTR_IN_struct T_OPAQUEPTR_OUT_struct
 	   T_ARRAY
-	   T_STDIO_open T_STDIO_close T_STDIO_print
+	   T_STDIO_open T_STDIO_open_ret_in_arg T_STDIO_close T_STDIO_print
            T_PACKED_in T_PACKED_out
            T_PACKEDARRAY_in T_PACKEDARRAY_out
            T_INOUT T_IN T_OUT
diff --git a/ext/XS-Typemap/Typemap.xs b/ext/XS-Typemap/Typemap.xs
index 3fa0e74ab..8314cc2 100644
--- a/ext/XS-Typemap/Typemap.xs
+++ b/ext/XS-Typemap/Typemap.xs
@@ -906,6 +906,15 @@ T_STDIO_open( file )
  OUTPUT:
   RETVAL
 
+void
+T_STDIO_open_ret_in_arg( file, io)
+  const char * file
+  FILE * io = NO_INIT
+ CODE:
+  io = xsfopen( file );
+ OUTPUT:
+  io
+
 SysRet
 T_STDIO_close( f )
   PerlIO * f
diff --git a/ext/XS-Typemap/t/Typemap.t b/ext/XS-Typemap/t/Typemap.t
index 27b4086..49ac479 100644
--- a/ext/XS-Typemap/t/Typemap.t
+++ b/ext/XS-Typemap/t/Typemap.t
@@ -6,10 +6,11 @@ BEGIN {
     }
 }
 
-use Test::More tests => 152;
+use Test::More tests => 156;
 
 use strict;
-use warnings;
+#catch WARN_INTERNAL type errors, and anything else unexpected
+use warnings FATAL => 'all';
 use XS::Typemap;
 
 pass();
@@ -213,6 +214,7 @@ is( T_PV("a string"), "a string");
 is( T_PV(52), 52);
 ok !defined T_PV_null, 'RETVAL = NULL returns undef for char*';
 {
+    use warnings NONFATAL => 'all';
     my $uninit;
     local $SIG{__WARN__} = sub { ++$uninit if shift =~ /uninit/ };
     () = ''.T_PV_null;
@@ -393,6 +395,16 @@ if (defined $fh) {
   }
 }
 
+$fh = "FOO";
+T_STDIO_open_ret_in_arg( $testfile, $fh);
+ok( $fh ne "FOO", 'return io in arg open succeeds');
+ok( print($fh "first line\n"), 'can print to return io in arg');
+ok( close($fh), 'can close return io in arg');
+$fh = "FOO";
+#now with a bad file name to make sure $fh is written to on failure
+T_STDIO_open_ret_in_arg( "", $fh);
+ok( !defined$fh, 'return io in arg open failed successfully');
+
 # T_INOUT
 note("T_INOUT");
 SCOPE: {
@@ -439,6 +451,10 @@ SCOPE: {
   ok(!close $fh2);
 }
 
+# Perl RT #124181 SEGV due to double free in typemap
+# "Attempt to free unreferenced scalar"
+%{*{main::XS::}{HASH}} = ();
+
 sub is_approx {
   my ($l, $r, $n) = @_;
   if (not defined $l or not defined $r) {
-- 
1.7.9.msysgit.0

@p5pRT
Copy link
Author

p5pRT commented Jun 24, 2015

From @bulk88

0002-fix-124181-double-free-refcnt-problems-in-IO-types-i.patch
From 7afd92de70add9ddcab27b094be848c432e3ec32 Mon Sep 17 00:00:00 2001
From: Daniel Dragan <bulk88@hotmail.com>
Date: Wed, 24 Jun 2015 15:48:12 -0400
Subject: [PATCH 2/2] fix #124181 double free/refcnt problems in IO types in
 typemap

commit 50e5165b96 "stop T_IN/OUT/INOUT/STDIO typemaps leaking" changed
newRV to newRV_noinc, but the GV * returned by newGVgen() is owned by the
package tree, like the SV * returned by get_sv(). Now when the RV to GV is
freed on mortal stack, the GV * in the package tree is freed, and now there
is a freed GV * in the package tree, if you turn on "PERL_DESTRUCT_LEVEL=2"
(and perhaps DEBUGGING is needed too), the package tree is destroyed SV *
by SV *, and perl will eventually warn with
"Attempt to free unreferenced scalar" which a very bad panic type warning.

commit 50e5165b96 was reverted in commit bae466e878
"Revert "stop T_IN/OUT/INOUT/STDIO typemaps leaking" for 5.22's release
to stop the panic, but reintroduced the SV/RV leak. So fix the RV leak (the val
passed as source arg of sv_setsv) by freeing it after the copying. In a very
unlikely scenario, the RV could still leak if sv_setsv dies.

Also fix the problem, that if this OUTPUT: type is being used for an
incoming arg, not the outgoing RETVAL arg, you can't assign a new SV*
ontop of the old one, that only works for perl stack return args, so
replace "$arg = &PL_sv_undef;" with "sv_setsv($arg, &PL_sv_undef);" if its
not RETVAL, this way OUTPUT on incoming args also works if it goes down the
error path. For efficiency, in a RETVAL siutation, let the undef original
SV* in $arg which is typically obtained from sv_newmortal() by xsubpp pass
through if we error out.

Also for efficiency, if it is RETVAL (which is more common) dont do the
sv_setsv/SvREFCNT_dec_NN stuff (2 function calls), just mortalize
(1 function call) the ex-temp RV and arrange for the RV to wind up on
perl stack.

Also, the GV * already knows what HV * stash it belongs to, so avoid the
stash lookup done by gv_stashpv() and just use GvSTASH which are simple
pointer derefs.
---
 lib/ExtUtils/typemap |   40 ++++++++++++++++++++++++++++------------
 pod/perldelta.pod    |    7 +++++++
 2 files changed, 35 insertions(+), 12 deletions(-)

diff --git a/lib/ExtUtils/typemap b/lib/ExtUtils/typemap
index 0b09641..5f61527 100644
--- a/lib/ExtUtils/typemap
+++ b/lib/ExtUtils/typemap
@@ -388,32 +388,48 @@ T_STDIO
 	{
 	    GV *gv = newGVgen("$Package");
 	    PerlIO *fp = PerlIO_importFILE($var,0);
-	    if ( fp && do_open(gv, "+<&", 3, FALSE, 0, 0, fp) )
-		sv_setsv($arg, sv_bless(newRV((SV*)gv), gv_stashpv("$Package",1)));
+	    if ( fp && do_open(gv, "+<&", 3, FALSE, 0, 0, fp) ) {
+		SV *rv = newRV_inc((SV*)gv);
+		rv = sv_bless(rv, GvSTASH(gv));
+		${"$var" eq "RETVAL" ? \"$arg = sv_2mortal(rv);"
+		    : \"sv_setsv($arg, rv);\n\t\tSvREFCNT_dec_NN(rv);"}
+	    }${"$var" ne "RETVAL" ? \"
 	    else
-		$arg = &PL_sv_undef;
+		sv_setsv($arg, &PL_sv_undef);\n" : \""}
 	}
 T_IN
 	{
 	    GV *gv = newGVgen("$Package");
-	    if ( do_open(gv, "<&", 2, FALSE, 0, 0, $var) )
-		sv_setsv($arg, sv_bless(newRV((SV*)gv), gv_stashpv("$Package",1)));
+	    if ( do_open(gv, "<&", 2, FALSE, 0, 0, $var) ) {
+		SV *rv = newRV_inc((SV*)gv);
+		rv = sv_bless(rv, GvSTASH(gv));
+		${"$var" eq "RETVAL" ? \"$arg = sv_2mortal(rv);"
+		    : \"sv_setsv($arg, rv);\n\t\tSvREFCNT_dec_NN(rv);"}
+	    }${"$var" ne "RETVAL" ? \"
 	    else
-		$arg = &PL_sv_undef;
+		sv_setsv($arg, &PL_sv_undef);\n" : \""}
 	}
 T_INOUT
 	{
 	    GV *gv = newGVgen("$Package");
-	    if ( do_open(gv, "+<&", 3, FALSE, 0, 0, $var) )
-		sv_setsv($arg, sv_bless(newRV((SV*)gv), gv_stashpv("$Package",1)));
+	    if ( do_open(gv, "+<&", 3, FALSE, 0, 0, $var) ) {
+		SV *rv = newRV_inc((SV*)gv);
+		rv = sv_bless(rv, GvSTASH(gv));
+		${"$var" eq "RETVAL" ? \"$arg = sv_2mortal(rv);"
+		    : \"sv_setsv($arg, rv);\n\t\tSvREFCNT_dec_NN(rv);"}
+	    }${"$var" ne "RETVAL" ? \"
 	    else
-		$arg = &PL_sv_undef;
+		sv_setsv($arg, &PL_sv_undef);\n" : \""}
 	}
 T_OUT
 	{
 	    GV *gv = newGVgen("$Package");
-	    if ( do_open(gv, "+>&", 3, FALSE, 0, 0, $var) )
-		sv_setsv($arg, sv_bless(newRV((SV*)gv), gv_stashpv("$Package",1)));
+	    if ( do_open(gv, "+>&", 3, FALSE, 0, 0, $var) ) {
+		SV *rv = newRV_inc((SV*)gv);
+		rv = sv_bless(rv, GvSTASH(gv));
+		${"$var" eq "RETVAL" ? \"$arg = sv_2mortal(rv);"
+		    : \"sv_setsv($arg, rv);\n\t\tSvREFCNT_dec_NN(rv);"}
+	    }${"$var" ne "RETVAL" ? \"
 	    else
-		$arg = &PL_sv_undef;
+		sv_setsv($arg, &PL_sv_undef);\n" : \""}
 	}
diff --git a/pod/perldelta.pod b/pod/perldelta.pod
index 58ece4a..2a94b60 100644
--- a/pod/perldelta.pod
+++ b/pod/perldelta.pod
@@ -236,6 +236,13 @@ caused sub-threads in threaded -DPERL_TRACE_OPS builds to spew exceedingly
 large op-counts at destruct.  These counts would print %x as "ABABABAB",
 clearly a mem-poison value.
 
+=item *
+
+A leak in the XS typemap caused one scalar to be leaked each time a C<FILE *>
+or a C<PerlIO *> was C<OUTPUT:>ed or imported to Perl, since perl 5.000. These
+particular typemap entries are thought to be extremely rarely used by XS
+modules. [perl #124181]
+
 =back
 
 =head1 Acknowledgements
-- 
1.7.9.msysgit.0

@p5pRT
Copy link
Author

p5pRT commented Jul 7, 2015

From @bulk88

On Wed Jun 24 12​:50​:22 2015, bulk88 wrote​:

Updated patches for blead/5.23.

Bump.

--
bulk88 ~ bulk88 at hotmail.com

@p5pRT
Copy link
Author

p5pRT commented Jul 8, 2015

From @tonycoz

On Wed Jun 24 12​:50​:22 2015, bulk88 wrote​:

Updated patches for blead/5.23.

Thanks, appled as c1b8440 and 7ed1d85.

If you can put the perldelta changes in a separate patch (or just as prose in the ticket) it will simplify applying these changes, since perldelta sees a lot of churn.

Thanks,
Tony

@p5pRT
Copy link
Author

p5pRT commented Jul 8, 2015

@tonycoz - Status changed from 'open' to 'resolved'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant