Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GC corruption when testing DataFrames #25176

Closed
ararslan opened this issue Dec 19, 2017 · 5 comments
Closed

GC corruption when testing DataFrames #25176

ararslan opened this issue Dec 19, 2017 · 5 comments
Labels
bug Indicates an unexpected problem or unintended behavior GC Garbage collector

Comments

@ararslan
Copy link
Member 

GC error (probable corruption) :
Allocations: 11258018 (Pool: 11252032; Big: 5986); GC: 25

!!! ERROR in jl_ -- ABORTING !!!
0x3aca4a0: Queued root: 0x7f537f39c310 :: 0x7f537d1017b0 (bits: 3)
        of type Core.MethodInstance
0x3aca4b8: Queued root: 0x7f538539aee0 :: 0x7f537d100f60 (bits: 3)
of type Array{Any, 1}
...
0x3acd440: Queued root: 0x7f538561ec00 :: 0x7f537d100160 (bits: 3)
        of type SimpleVector
0x3acd458:  r-- Stack frame 0x7fff4f158200 -- 1 of 2 (indirect)
signal (6): Aborted
in expression starting at no file:0
gsignal at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
abort at /lib/x86_64-linux-gnu/libc.so.6 (unknown line)
gc_assert_datatype_fail at /buildworker/worker/package_linux64/build/src/gc.c:1378
gc_mark_loop at /buildworker/worker/package_linux64/build/src/gc.c:2183
...

Full log here: https://gist.github.com/ararslan/8a287510a3111681641cde87d6f8cd32

cc @Keno
@ararslan ararslan added GC Garbage collector bug Indicates an unexpected problem or unintended behavior labels Dec 19, 2017
@ararslan ararslan mentioned this issue Dec 19, 2017
Merged
@StefanKarpinski StefanKarpinski added this to the 1.0 milestone Dec 19, 2017
@maleadt
Copy link
Member

maleadt commented Dec 19, 2017

Repro? Or a set of Julia+pkg hashes? Tried to reproduce with an ASAN build, but the tests are probably bailing out early due to recent changes (printf, bindir, or whatnot).

@ararslan
Copy link
Member Author

ararslan commented Dec 19, 2017 •
edited
Loading

I don't have a minimal reproducible example at the moment, but the DataFrames commit was JuliaData/DataFrames.jl@b2075f4, and according to Travis, these are the packages it was using:

Package Version
BinDeps v0.8.1
CategoricalArrays v0.3.3
CodecZlib v0.4.1
Compat v0.40.0
DataStreams v0.3.1
DataStructures v0.7.3
Missings v0.2.4
NamedTuples v4.0.0
Reexport v0.0.3
SHA v0.5.2
SortingAlgorithms v0.2.0
SpecialFunctions v0.3.6
StatsBase v0.19.2
TranscodingStreams v0.4.1
URIParser v0.3.0
WeakRefStrings v0.4.0

And this was its Julia version:

$ julia -e 'versioninfo()'
Julia Version 0.7.0-DEV.3018
Commit ff0e1f7fcf* (2017-12-15 23:34 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU @ 2.50GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, ivybridge)
Environment:
  TRAVIS_JULIA_VERSION = nightly

@maleadt
Copy link
Member

maleadt commented Dec 20, 2017 •
edited
Loading

ASAN detects the corruption a little earlier, but still in precompilation:

=================================================================
==3727==ERROR: AddressSanitizer: global-buffer-overflow on address 0x7f6cf5fc6858 at pc 0x7f6cf5af9eb9 bp 0x7ffeba142c40 sp 0x7ffeba142c38
READ of size 8 at 0x7f6cf5fc6858 thread T0
    #0 0x7f6cf5af9eb8 in gc_try_setmark julia/src/gc.c:1432:24
    #1 0x7f6cf5af15d5 in gc_mark_loop julia/src/gc.c:1871:22
    #2 0x7f6cf5afc27b in _jl_gc_collect julia/src/gc.c:2480:5
    #3 0x7f6cf5afba19 in jl_gc_collect julia/src/gc.c:2645:13
    #4 0x7f6cf5aef451 in maybe_collect julia/src/gc.c:689:9
    #5 0x7f6cf5aef1e8 in jl_gc_big_alloc julia/src/gc.c:744:5
    #6 0x7f6cf5aefbe4 in jl_gc_pool_alloc julia/src/gc.c:956:12
    #7 0x7f6cf5aef166 in jl_gc_alloc_ julia/src/julia_internal.h:272:13
    #8 0x7f6cf5afdb3d in jl_gc_alloc julia/src/gc.c:2680:12
    #9 0x7f6cf5a9f1b9 in jl_new_structv julia/src/datatype.c:741:22
    #10 0x7f6cf59f7ac4 in jl_f_tuple julia/src/builtins.c:710:12
    #11 0x7f6cdfaad823 in julia_next_997 generator.jl:47
    #12 0x7f6cdfaad9d9 in japi1_copy!_995 abstractarray.jl:592
    #13 0x7f6cdfaadbd3 in japi1__collect_994 array.jl:436
    #14 0x7f6cdfaadc74 in japi1_collect_993 array.jl:433
    #15 0x7f6cdfaae13a in japi1_abstract_eval_call_990 inference.jl:2641
    #16 0x7f6cdfaaf3f0 in japi1_abstract_eval_985 inference.jl:2690
    #17 0x7f6cdfab0cb9 in japi1_abstract_interpret_984 inference.jl:2843
    #18 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #19 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #20 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #21 0x7f6cdfa535b5 in julia_typeinf_work_42 inference.jl:3488
    #22 0x7f6cdfaa7875 in japi1_typeinf_39 inference.jl:3619
    #23 0x7f6cdfacd734 in japi1_typeinf_edge_1280 inference.jl:3350
    #24 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #25 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #26 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #27 0x7f6cdfacb496 in japi1_abstract_call_method_1268 inference.jl:2186
    #28 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #29 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #30 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #31 0x7f6cdfb0badb in japi1_abstract_call_gf_by_type_1079 inference.jl:1955
    #32 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #33 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #34 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #35 0x7f6cdfabb14b in japi1_abstract_call_1015 inference.jl:2637
    #36 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #37 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #38 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #39 0x7f6cdfaae7bb in japi1_abstract_eval_call_990 inference.jl:2667
    #40 0x7f6cdfaaf3f0 in japi1_abstract_eval_985 inference.jl:2690
    #41 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #42 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #43 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #44 0x7f6cdfab0a43 in japi1_abstract_interpret_984 inference.jl:2835
    #45 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #46 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #47 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #48 0x7f6cdfa535b5 in julia_typeinf_work_42 inference.jl:3488
    #49 0x7f6cdfaa7875 in japi1_typeinf_39 inference.jl:3619
    #50 0x7f6cdfacd734 in japi1_typeinf_edge_1280 inference.jl:3350
    #51 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #52 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #53 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #54 0x7f6cdfacb496 in japi1_abstract_call_method_1268 inference.jl:2186
    #55 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #56 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #57 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #58 0x7f6cdfb0badb in japi1_abstract_call_gf_by_type_1079 inference.jl:1955
    #59 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #60 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #61 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #62 0x7f6cdfabb14b in japi1_abstract_call_1015 inference.jl:2637
    #63 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #64 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #65 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #66 0x7f6cdfaae7bb in japi1_abstract_eval_call_990 inference.jl:2667
    #67 0x7f6cdfaaf3f0 in japi1_abstract_eval_985 inference.jl:2690
    #68 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #69 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #70 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #71 0x7f6cdfab0a43 in japi1_abstract_interpret_984 inference.jl:2835
    #72 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #73 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #74 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #75 0x7f6cdfa535b5 in julia_typeinf_work_42 inference.jl:3488
    #76 0x7f6cdfaa7875 in japi1_typeinf_39 inference.jl:3619
    #77 0x7f6cdfacd734 in japi1_typeinf_edge_1280 inference.jl:3350
    #78 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #79 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #80 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #81 0x7f6cdfacb496 in japi1_abstract_call_method_1268 inference.jl:2186
    #82 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #83 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #84 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #85 0x7f6cdfb0badb in japi1_abstract_call_gf_by_type_1079 inference.jl:1955
    #86 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #87 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #88 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #89 0x7f6cdfabb14b in japi1_abstract_call_1015 inference.jl:2637
    #90 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #91 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #92 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #93 0x7f6cdfaae7bb in japi1_abstract_eval_call_990 inference.jl:2667
    #94 0x7f6cdfaaf3f0 in japi1_abstract_eval_985 inference.jl:2690
    #95 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #96 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #97 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #98 0x7f6cdfab0a43 in japi1_abstract_interpret_984 inference.jl:2835
    #99 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #100 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #101 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #102 0x7f6cdfa535b5 in julia_typeinf_work_42 inference.jl:3488
    #103 0x7f6cdfaa7875 in japi1_typeinf_39 inference.jl:3619
    #104 0x7f6cdfabd18d in typeinf_frame inference.jl:3309
    #105 0x7f6cdfabd18d in julia_typeinf_code_1092 inference.jl:3402
    #106 0x7f6cdfabdc30 in jlcall_typeinf_code_1091 (julia/build/sanitize/usr/lib/julia/sys-debug.so+0x74c30)
    #107 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #108 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #109 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #110 0x7f6cdfaa85f9 in julia_typeinf_ext_1 inference.jl:3444
    #111 0x7f6cdfaa8c85 in jlcall_typeinf_ext_0 (julia/build/sanitize/usr/lib/julia/sys-debug.so+0x5fc85)
    #112 0x7f6cf59c571a in jl_call_fptr_internal julia/src/julia_internal.h:380:16
    #113 0x7f6cf59a878a in jl_call_method_internal julia/src/julia_internal.h:399:12
    #114 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #115 0x7f6cf5b659f4 in jl_apply julia/src/julia.h:1474:12
    #116 0x7f6cf5b65473 in jl_apply_with_saved_exception_state julia/src/rtutils.c:257:13
    #117 0x7f6cf59ac583 in jl_type_infer julia/src/gf.c:263:48
    #118 0x7f6cf59bbc00 in jl_compile_for_dispatch julia/src/gf.c:1705:15
    #119 0x7f6cf59c0727 in jl_compile_method_internal julia/src/julia_internal.h:348:17
    #120 0x7f6cf59a86ef in jl_call_method_internal julia/src/julia_internal.h:395:21
    #121 0x7f6cf59c2d85 in jl_apply_generic julia/src/gf.c:2011:23
    #122 0x7f6cf5b27864 in jl_apply julia/src/julia.h:1474:12
    #123 0x7f6cf5b27b5c in jl_call0 julia/src/jlapi.c:182:13
    #124 0x7f6cf5a7e754 in jl_get_loaded_modules julia/src/toplevel.c:141:29
    #125 0x7f6cf5a4081b in jl_save_incremental julia/src/dump.c:2307:17
    #126 0x7f6cf5adc059 in jl_write_compiler_output julia/src/precompile.c:58:17
    #127 0x7f6cf5a1aa2b in jl_atexit_hook julia/src/init.c:237:9
    #128 0x4f4a41 in main julia/ui/repl.c:238:5
    #129 0x7f6cf46482b0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x202b0)
    #130 0x41cc29 in _start (julia/build/sanitize/usr/bin/julia-debug+0x41cc29)

0x7f6cf5fc6858 is located 8 bytes to the left of global variable '<string literal>' defined in 'julia/src/precompile.c:51:54' (0x7f6cf5fc6860) of size 9
  '<string literal>' is ascii string '__init__'
0x7f6cf5fc6858 is located 40 bytes to the right of global variable '<string literal>' defined in 'julia/src/precompile.c:42:30' (0x7f6cf5fc67e0) of size 80
  '<string literal>' is ascii string 'WARNING: --output-jit-bc is meaningless with options for dumping sysimage data
'
SUMMARY: AddressSanitizer: global-buffer-overflow julia/src/gc.c:1432:24 in gc_try_setmark
Shadow bytes around the buggy address:
  0x1fed9ebf8cb0: f9 f9 f9 f9 00 00 00 00 00 00 01 f9 f9 f9 f9 f9
  0x1fed9ebf8cc0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1fed9ebf8cd0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1fed9ebf8ce0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1fed9ebf8cf0: 00 00 00 00 00 00 00 00 f9 f9 f9 f9 00 00 00 00
=>0x1fed9ebf8d00: 00 00 00 00 00 00 f9 f9 f9 f9 f9[f9]00 01 f9 f9
  0x1fed9ebf8d10: f9 f9 f9 f9 00 00 00 00 00 00 00 06 f9 f9 f9 f9
  0x1fed9ebf8d20: 00 00 00 00 00 00 00 05 f9 f9 f9 f9 00 00 00 00
  0x1fed9ebf8d30: 00 07 f9 f9 f9 f9 f9 f9 00 00 00 00 00 05 f9 f9
  0x1fed9ebf8d40: f9 f9 f9 f9 00 02 f9 f9 f9 f9 f9 f9 00 00 00 00
  0x1fed9ebf8d50: 00 00 03 f9 f9 f9 f9 f9 00 00 00 00 00 00 00 01
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==3727==ABORTING
W- The call to compilecache failed to create a usable precompiled cache file for module DataFrames -Warn:Base:loading.jl:423
|  exception = ErrorException("Precompile file header verification checks failed.")
ERROR: LoadError: LoadError: Declaring __precompile__(true) is only allowed in module files being imported.
Stacktrace:
 [1] __precompile__(::Bool) at ./loading.jl:250
 [2] top-level scope
 [3] include at ./boot.jl:283 [inlined]
 [4] include_relative(::Module, ::String) at ./loading.jl:503
 [5] _require(::Symbol) at ./loading.jl:434
 [6] require(::Symbol) at ./loading.jl:298
 [7] include at ./boot.jl:283 [inlined]
 [8] include_relative(::Module, ::String) at ./loading.jl:503
 [9] include(::Module, ::String) at ./sysimg.jl:15
 [10] process_options(::Base.JLOptions) at ./client.jl:322
 [11] _start() at ./client.jl:373
in expression starting at DataFrames/src/DataFrames.jl:1
in expression starting at DataFrames/test/runtests.jl:10

EDIT: switched build trees (in order to bisect) and now I can't reproduce it anymore...

@ararslan
Copy link
Member Author

If you can no longer reproduce this, should we call it a fluke and close this?

@StefanKarpinski StefanKarpinski modified the milestones: 1.0, 1.0.x Jan 4, 2018
@Keno
Copy link
Member

Keno commented May 20, 2019

There've been lots of missing gc roots fixed since 1.0. Absent a reproducer, I'll close this. Of course please open a new issue for any reoccurence.

@Keno Keno closed this as completed May 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Indicates an unexpected problem or unintended behavior GC Garbage collector
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@StefanKarpinski @maleadt @Keno @ararslan