Is the C++ target Multithread safe?

[alainmarcel]

I started a large project with the C++ target a few months back with great success so far, Kudos!
A few days ago though, I tried for the 1rst time to run some of the lexers and parsers in parallel in different threads, no data shared (no statics, all allocated objects) but my code base is quite large at this point so coding errors are possible. I get mostly core dumps at runtime and occasionally good results (Typical of bad multithread coding).
I want to rule out options, the primary one being, are the lexers, parsers and listeners Multithread safe?

Most of the time I crash here:

Thread 2 "myproject" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff6f4f700 (LWP 25394)]
0x000000000080f9a9 in std::vector<antlr4::misc::Interval, std::allocatorantlr4::misc::Interval >::operator=(std::vector<antlr4::misc::Interval, std::allocatorantlr4::misc::Interval > const&) ()
(gdb) where
#0 0x000000000080f9a9 in std::vector<antlr4::misc::Interval, std::allocatorantlr4::misc::Interval >::operator=(std::vector<antlr4::misc::Interval, std::allocatorantlr4::misc::Interval > const&) ()
#1 0x0000000000817d79 in antlr4::atn::ATN::nextTokens(antlr4::atn::ATNState*) const ()
#2 0x00000000008136a2 in antlr4::DefaultErrorStrategy::sync(antlr4::Parser*) ()
...

Thanks

[alainmarcel]

Any chance of getting it done in the foreseeable future? That would save me from the multi-process, shared memory tricky approach.
Thanks

[ghost]

@alainmarcel Unlikely. This is non-trivial and the C++ maintainer is currently not very active.

[KvanTTT]

Just interesting: does this issue related to cache clearing? C# runtime also not thread safe but it can be resolved with using of ReadWriterLockSlim synchronization primitive

[mike-lischke]

I fixed a bug wrt. multithreading. Try the next release, which is due soon and see if that helps you.

[alainmarcel]

Will certainly do. Thanks, Alain

…

On Mar 30, 2017, at 1:01 PM, Mike Lischke ***@***.***> wrote: I fixed a bug wrt. multithreading. Try the next release, which is due soon and see if that helps you. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ksedgwic]

I think I'm seeing a similar problem. I'm using antlr-4.7-complete.jar and antlr4-cpp-runtime-4.7-source.zip.

Here are two different backtraces:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffcb7ae700 (LWP 19046)]
0x00007ffff7273493 in antlr4::misc::IntervalSet::add (this=0x7fffcb7ac8c0, addition=...) at /usr/local/src/antlr4-cpp-runtime-4.7-source/runtime/src/misc/IntervalSet.cpp:91
91	    if (addition.adjacent(r) || !addition.disjoint(r)) {
(gdb) bt
#0  0x00007ffff7273493 in antlr4::misc::IntervalSet::add (this=0x7fffcb7ac8c0, addition=...) at /usr/local/src/antlr4-cpp-runtime-4.7-source/runtime/src/misc/IntervalSet.cpp:91
#1  0x00007ffff7273949 in antlr4::misc::IntervalSet::addAll (this=0x7fffcb7ac8c0, set=...) at /usr/local/src/antlr4-cpp-runtime-4.7-source/runtime/src/misc/IntervalSet.cpp:139
#2  0x00007ffff7272d46 in antlr4::misc::IntervalSet::IntervalSet (this=0x7fffcb7ac8c0, set=...) at /usr/local/src/antlr4-cpp-runtime-4.7-source/runtime/src/misc/IntervalSet.cpp:37
#3  0x00007ffff71eaa04 in antlr4::DefaultErrorStrategy::sync (this=0x7fffc0000c50, recognizer=0x7fffcb7acd20) at /usr/local/src/antlr4-cpp-runtime-4.7-source/runtime/src/DefaultErrorStrategy.cpp:103
#4  0x00007ffff71ba8f4 in PredicateExpressionParser::expr (this=0x7fffcb7acd20, precedence=0) at GEN/PredicateExpressionParser.cpp:318
#5  0x00007ffff71b9758 in PredicateExpressionParser::start (this=0x7fffcb7acd20) at GEN/PredicateExpressionParser.cpp:84
#6  0x00007ffff71d871c in as_query_predicate_add (query=0x7fffcb7ad250, stmt=0x411662 "amenity == \"restaurant\"") at as_predicate.cpp:378
#7  0x000000000040782e in (anonymous namespace)::do_query_worker (argp=0x659df0) at osm_pir_bench.cpp:334
#8  0x0000000000407b3c in (anonymous namespace)::query_worker (argp=0x659df0) at osm_pir_bench.cpp:378
#9  0x00007ffff7bc561a in start_thread (arg=0x7fffcb7ae700) at pthread_create.c:334
#10 0x00007ffff58a65fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb)

Also:

Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffcb7ae700 (LWP 19369)]
0x00007ffff57d8a28 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55
55	  return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);

bt
#0  0x00007ffff57d8a28 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007ffff57da62a in __GI_abort () at abort.c:89
#2  0x00007ffff581bdea in __libc_message (do_abort=do_abort@entry=2, fmt=fmt@entry=0x7ffff592e360 "*** Error in `%s': %s: 0x%s ***\n") at ../sysdeps/posix/libc_fatal.c:175
#3  0x00007ffff582422a in malloc_printerr (ar_ptr=<optimized out>, ptr=<optimized out>, str=0x7ffff592e428 "double free or corruption (fasttop)", action=3) at malloc.c:5004
#4  _int_free (av=<optimized out>, p=<optimized out>, have_lock=<optimized out>) at malloc.c:3865
#5  0x00007ffff582778c in __GI___libc_free (mem=<optimized out>) at malloc.c:2966
#6  0x00007ffff71e65a0 in __gnu_cxx::new_allocator<antlr4::misc::Interval>::deallocate (this=0x649ed8, __p=0x7fffc4003680) at /usr/include/c++/5.3.1/ext/new_allocator.h:110
#7  0x00007ffff71e5562 in std::allocator_traits<std::allocator<antlr4::misc::Interval> >::deallocate (__a=..., __p=0x7fffc4003680, __n=2) at /usr/include/c++/5.3.1/bits/alloc_traits.h:517
#8  0x00007ffff71e3790 in std::_Vector_base<antlr4::misc::Interval, std::allocator<antlr4::misc::Interval> >::_M_deallocate (this=0x649ed8, __p=0x7fffc4003680, __n=2) at /usr/include/c++/5.3.1/bits/stl_vector.h:178
#9  0x00007ffff71e20ea in std::vector<antlr4::misc::Interval, std::allocator<antlr4::misc::Interval> >::operator= (this=0x649ed8, __x=std::vector of length 2, capacity 2 = {...}) at /usr/include/c++/5.3.1/bits/vector.tcc:198
#10 0x00007ffff71e1361 in antlr4::misc::IntervalSet::operator= (this=0x649ed0) at /usr/local/src/antlr4-cpp-runtime-4.7-source/runtime/src/misc/IntervalSet.h:25
#11 0x00007ffff7260671 in antlr4::atn::ATN::nextTokens (this=0x7ffff754de40 <PredicateExpressionParser::_atn>, s=0x649eb0) at /usr/local/src/antlr4-cpp-runtime-4.7-source/runtime/src/atn/ATN.cpp:92
#12 0x00007ffff71ea9f2 in antlr4::DefaultErrorStrategy::sync (this=0x7fffc0000c50, recognizer=0x7fffcb7acd20) at /usr/local/src/antlr4-cpp-runtime-4.7-source/runtime/src/DefaultErrorStrategy.cpp:103
#13 0x00007ffff71bacd6 in PredicateExpressionParser::expr (this=0x7fffcb7acd20, precedence=0) at GEN/PredicateExpressionParser.cpp:388
#14 0x00007ffff71b9758 in PredicateExpressionParser::start (this=0x7fffcb7acd20) at GEN/PredicateExpressionParser.cpp:84
#15 0x00007ffff71d871c in as_query_predicate_add (query=0x7fffcb7ad250, stmt=0x411662 "amenity == \"restaurant\"") at as_predicate.cpp:378
#16 0x000000000040782e in (anonymous namespace)::do_query_worker (argp=0x659df0) at osm_pir_bench.cpp:334
#17 0x0000000000407b3c in (anonymous namespace)::query_worker (argp=0x659df0) at osm_pir_bench.cpp:378
#18 0x00007ffff7bc561a in start_thread (arg=0x7fffcb7ae700) at pthread_create.c:334
#19 0x00007ffff58a65fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

[ksedgwic]

If I run the program with a single thread it doesn't seem to crash.

[mike-lischke]

@ksedgwic How do you use the parser in threads? I hope you are not sharing them between threads. From your stacktrace I see it's crashing in the same code in different threads (when manipulating an IntervalSet. That doesn't make much sense, since this is not a shared structure. Maybe it's failing on certain values, not because of concurrent access to something?

[ksedgwic]

The two backtraces are from two separate runs.

Each thread creates a parser and uses it by itself, no inter-thread sharing at all.

[alainmarcel]

4.7 works like a charm in multi thread and is also faster!
When switching to it from 4.6, the bad stack trace I saw earlier moved to my own code and I could debug my own issue (I was deleting in a thread A a parser created in another thread B....).

Just be extra careful with caching any parser (like caching pre-compiled headers in some thread) and accessing (walking a listener) the resulting created parser in another thread.
Thanks!