Segmentation fault when using libxml-ruby & nokogiri in the same project.

[Coren]

Hi,

I'm having this kind of crash for a job in a ruby on rails project.

I have never seen it when I launch it from a "normal" ruby process (irb or webrick)
It always happens when I launch it from Sidekiq.

I have upgraded to latest versions with rvm, and I have seen same behaviour.

Basically, what my source code is doing:

1. Collect & parse data in xml (with ruby-oai gem, so libxml-ruby)
2. Take a node and its children, get a string representation, and give it to xml2json gem (which parses it with nokogiri)

Here is the trace

/home/mloiseleur/.rvm/gems/ruby-2.2.1/gems/libxml-ruby-2.7.0/lib/libxml/node.rb:75:in `find'
/home/mloiseleur/.rvm/gems/ruby-2.2.1/gems/libxml-ruby-2.7.0/lib/libxml/node.rb:58:in `context'
/home/mloiseleur/.rvm/gems/ruby-2.2.1/gems/libxml-ruby-2.7.0/lib/libxml/node.rb:58:in `new'

-- Machine register context ------------------------------------------------
 RIP: 0x00000000105003fd RBP: 0x00000000193d39d0 RSP: 0x00000000193d39b0
 RAX: 0x636e6174736e692d RBX: 0x00000000063bdbb0 RCX: 0x00000000055282e0
 RDX: 0x000000001c565a30 RDI: 0x00000000144438b0 RSI: 0x0000000000000001
  R8: 0x0000000000000000  R9: 0x0000000010646c22 R10: 0x0000000000000000
 R11: 0x0000000011ba4fa0 R12: 0x00000000063bc000 R13: 0x00000000063bcf30
 R14: 0x000000000639bce0 R15: 0x00000000193d3be8 EFL: 0x0000000000000004

-- C level backtrace information -------------------------------------------
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(rb_vm_bugreport+0x4ea) [0x501d30a] vm_dump.c:693
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(rb_bug_context+0xcb) [0x4eb30ab] error.c:425
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(sigsegv+0x3e) [0x4f9145e] signal.c:879
/lib/x86_64-linux-gnu/libpthread.so.0 [0x531f8d0]
/home/mloiseleur/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/lib/nokogiri/nokogiri.so(xmlFreeNodeList+0x87) [0x105003fd]
/home/mloiseleur/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/lib/nokogiri/nokogiri.so(xmlFreeProp+0xb8) [0x104fe019]
/home/mloiseleur/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/lib/nokogiri/nokogiri.so(xmlFreePropList+0x2f) [0x104fdf50]
/home/mloiseleur/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/lib/nokogiri/nokogiri.so(xmlFreeNodeList+0x148) [0x105004be]
/home/mloiseleur/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/lib/nokogiri/nokogiri.so(xmlFreeNodeList+0x107) [0x1050047d]
/home/mloiseleur/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/lib/nokogiri/nokogiri.so(xmlFreeNodeList+0x107) [0x1050047d]
/home/mloiseleur/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/lib/nokogiri/nokogiri.so(xmlFreeNodeList+0x107) [0x1050047d]
/home/mloiseleur/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/lib/nokogiri/nokogiri.so(xmlFreeNodeList+0x107) [0x1050047d]
/home/mloiseleur/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/lib/nokogiri/nokogiri.so(xmlFreeNodeList+0x107) [0x1050047d]
/home/mloiseleur/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/lib/nokogiri/nokogiri.so(xmlFreeNodeList+0x107) [0x1050047d]
/home/mloiseleur/.rvm/gems/ruby-2.2.1/gems/nokogiri-1.6.6.2/lib/nokogiri/nokogiri.so(xmlFreeDoc+0x161) [0x104fc75f]
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(finalize_list+0x51) [0x4ed1021] gc.c:2463
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(gc_finalize_deferred+0x50) [0x4ed2550] gc.c:2500
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(rb_postponed_job_flush+0x133) [0x5024563] vm_trace.c:1572
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(rb_threadptr_execute_interrupts.part.41+0x139) [0x502a7f9] thread.c:1971
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_call0_body.constprop.78+0x52e) [0x501046e] vm_eval.c:252
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(rb_call0+0x192) [0x5011562] vm_eval.c:59
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(rb_class_new_instance+0x21) [0x4f1f281] object.c:1856
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_call_cfunc+0x127) [0x5003827] vm_insnhelper.c:1382
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_exec_core+0x124d) [0x5009c4d] insns.def:1054
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_exec+0x78) [0x500e3d8] vm.c:1400
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_call0_body.constprop.78+0x1ce) [0x501010e] vm_eval.c:180
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(rb_call0+0x192) [0x5011562] vm_eval.c:59
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(rb_class_new_instance+0x21) [0x4f1f281] object.c:1856
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_call_cfunc+0x127) [0x5003827] vm_insnhelper.c:1382
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_exec_core+0x124d) [0x5009c4d] insns.def:1054
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_exec+0x78) [0x500e3d8] vm.c:1400
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(rb_yield+0x492) [0x50190f2] vm.c:813
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(rb_ary_each+0x52) [0x4e63e22] array.c:1803
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_call_cfunc+0x127) [0x5003827] vm_insnhelper.c:1382
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_exec_core+0x1197) [0x5009b97] insns.def:1024
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_exec+0x78) [0x500e3d8] vm.c:1400
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_call0_body.constprop.78+0x1ce) [0x501010e] vm_eval.c:180
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(rb_call0+0x192) [0x5011562] vm_eval.c:59
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(rb_iterate+0xea) [0x5005c0a] vm_eval.c:1129
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(rb_block_call+0x2b) [0x5005dcb] vm_eval.c:1198
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(enum_to_a+0x38) [0x4ea7968] enum.c:503
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_call_cfunc+0x127) [0x5003827] vm_insnhelper.c:1382
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_exec_core+0x124d) [0x5009c4d] insns.def:1054
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_exec+0x78) [0x500e3d8] vm.c:1400
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(invoke_block_from_c+0x6be) [0x501462e] vm.c:813
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_invoke_proc+0xe0) [0x50147f0] vm.c:878
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(rb_vm_invoke_proc+0x18) [0x50148d8] vm.c:897
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(proc_call+0x52) [0x4ec2452] proc.c:731
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_call_cfunc+0x127) [0x5003827] vm_insnhelper.c:1382
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_call_method+0x11e) [0x501652e] vm_insnhelper.c:1691
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_exec_core+0x124d) [0x5009c4d] insns.def:1054
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_exec+0x78) [0x500e3d8] vm.c:1400
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_call0_body.constprop.78+0x1ce) [0x501010e] vm_eval.c:180
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(rb_call0+0x192) [0x5011562] vm_eval.c:59
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(send_internal+0xd2) [0x5016fc2] vm_eval.c:928
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_call_cfunc+0x127) [0x5003827] vm_insnhelper.c:1382
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_call_method+0x11e) [0x501652e] vm_insnhelper.c:1691
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_exec_core+0x1197) [0x5009b97] insns.def:1024
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_exec+0x78) [0x500e3d8] vm.c:1400
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(invoke_block_from_c+0x6be) [0x501462e] vm.c:813
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(vm_invoke_proc+0xe0) [0x50147f0] vm.c:878
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(rb_vm_invoke_proc+0x18) [0x50148d8] vm.c:897
/home/mloiseleur/.rvm/rubies/ruby-2.2.1/lib/libruby.so.2.2(rb_fiber_start+0x110) [0x5031c70] cont.c:1263

[flavorjones]

@Coren Thanks for asking this question.

Nokogiri tries extremely hard to work well with libxml-ruby, but unfortunately that gem never tried to meet us halfway, and there are edge cases we simply can't work around. Without going into crazy details, libxml-ruby hooks into the libxml memory lifecycle in inconvenient ways.

We try to work around this, but GC is complex, in particular regarding the order in which objects may be freed (which may not agree with the order that libxml2 would like (which is 90% of the difficulty of wrapping Ruby around libxml2)). An example of this is viewable here:

https://github.com/sparklemotion/nokogiri/blob/master/ext/nokogiri/xml_document.c#L38-L62

You'll note that the stack walkback indicates this is happening at document free-time.

Unfortunately, the mixed-use case of libxml-ruby and nokogiri is simply something we can't guarantee is going to work, and is therefore not really supported.

I'd love to know why both libraries are being used at the same time. What dependency is using libxml-ruby (and is it HappyMapper)?

I'm going to close this, but feel free to follow up with more information that might be actionable.

And thanks for using Nokogiri.

[flavorjones]

Ah, I missed that you said you were using ruby-oai. I'm not familiar with it, unfortunately.

[flavorjones]

Forgot to be explicit: if you can provide a reproducible test case, I'm happy to look into what we can do to work better with libxml-ruby. It's just that this comes up once a year or so, and I generally don't get lucky enough to get such a repro script. :(

[Coren]

Ok. I'll see if I can do something about a repro script.

[flavorjones]

I'm going to cut v1.6.8.rc3 tonight, which addresses another edge case with libxml-ruby, and so it may be worth trying that version out to see if it addresses your case.

I've also added the start of some tests around libxml-ruby conflicts, and so any test case you can provide would be easy to merge into our test suite.