segfault from wasm-bindgen CLI on OSX with recent rust nightlies

[TimHambourger]

The wasm-bindgen command is reliably giving me a segfault on OSX 10.10.5 when I install with recent rust nightlies. I'm seeing an older rust version fix the issue, so I'm guessing this is a recent nightly regression. I figured I'd give my report in case it helps ID the issue.

The symptom:

Macintosh:~ roberthambourger$ wasm-bindgen --help
Segmentation fault: 11
Macintosh:~ roberthambourger$ wasm-bindgen --version
Segmentation fault: 11
Macintosh:~ roberthambourger$ wasm-bindgen path/to/my/wasm.wasm --out-dir .
Segmentation fault: 11

Sample rustup show output:

Macintosh:~ roberthambourger$ rustup show
Default host: x86_64-apple-darwin

installed toolchains
--------------------

stable-x86_64-apple-darwin
nightly-2018-01-01-x86_64-apple-darwin
nightly-2018-04-30-x86_64-apple-darwin
nightly-x86_64-apple-darwin (default)

installed targets for active toolchain
--------------------------------------

wasm32-unknown-unknown
x86_64-apple-darwin

active toolchain
----------------

nightly-x86_64-apple-darwin (default)
rustc 1.27.0-nightly (91db9dcf3 2018-05-04)

I've tried different combos of wasm-bindgen-cli versions installed with various rust nightly versions. My observations are that

• wasm-bindgen-cli 0.2.8 is bad when installed with nightly 2018-05-04, 2018-05-03, and 2018-04-30, but good with 2018-01-01.

• wasm-bindgen-cli 0.2.4 through 0.2.8 are all bad with nightly 2018-05-03

Also note that wasm2es6js works for me as expected with all combos tested:

Macintosh:~ roberthambourger$ wasm2es6js --help
Converts a wasm file to an ES6 JS module

Usage:
    wasm2es6js [options] <input>
    wasm2es6js -h | --help

etc.

Thanks as always for the great work and let me know if I can provide more info.

[TimHambourger]

Update: I did a bisection search. Apparently

cargo +nightly-2018-03-16 install --force wasm-bindgen-cli

is good and

cargo +nightly-2018-03-17 install --force wasm-bindgen-cli

is bad.

[alexcrichton]

Oh dear this sounds bad! Are you able to attach a debugger and see where the fault is happening?

This doesn't reproduce locally for me unfortunately :(

[TimHambourger]

Hmm, ok. The callstack apparently passes through thread_local via regex via docopt. It makes sense that docopt is involved since this is affecting every wasm-bindgen option.

Here's my rust-lldb output (this is after re-installing wasm-bindgen-cli with the --debug option with nightly 2018-05-04):

Process 2273 stopped
* thread #1: tid = 0x407f, 0x000000010051db92 wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec [inlined] core::ptr::swap_nonoverlapping_bytes::h6df82f8d5ea7e526 + 28 at ptr.rs:237, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x000000010051db92 wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec [inlined] core::ptr::swap_nonoverlapping_bytes::h6df82f8d5ea7e526 + 28 at ptr.rs:237
(lldb) bt
* thread #1: tid = 0x407f, 0x000000010051db92 wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec [inlined] core::ptr::swap_nonoverlapping_bytes::h6df82f8d5ea7e526 + 28 at ptr.rs:237, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
  * frame #0: 0x000000010051db92 wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec [inlined] core::ptr::swap_nonoverlapping_bytes::h6df82f8d5ea7e526 + 28 at ptr.rs:237
    frame #1: 0x000000010051db76 wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec [inlined] core::ptr::swap_nonoverlapping::hdae0ad8ddd3a4807 at ptr.rs:187
    frame #2: 0x000000010051db76 wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec [inlined] core::mem::swap::h079abbed39293f6c at mem.rs:634
    frame #3: 0x000000010051db76 wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec [inlined] core::mem::replace::h33caec918b20cf74 at mem.rs:691
    frame #4: 0x000000010051db76 wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::init::hf29d791cc8dc2c2f at local.rs:270
    frame #5: 0x000000010051db76 wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::hf4098ad3d27e0fb7 + 34 at local.rs:296
    frame #6: 0x000000010051db54 wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h07ea88e18fd4e1fd at local.rs:248
    frame #7: 0x000000010051db54 wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec [inlined] std::panicking::update_panic_count::h96721aee7d30548f at panicking.rs:240
    frame #8: 0x000000010051db54 wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec + 4 at panicking.rs:317
    frame #9: 0x000000010020824d wasm-bindgen`std::thread::panicking::hf2650f0279c3587b + 13
    frame #10: 0x0000000100209381 wasm-bindgen`std::sys_common::poison::Flag::borrow::h6a204c3cf5f3e9a0 + 17
    frame #11: 0x00000001002062d7 wasm-bindgen`_$LT$std..sync..mutex..MutexGuard$LT$$u27$mutex$C$$u20$T$GT$$GT$::new::h61ce00daf227bcd8 + 39
    frame #12: 0x00000001002062a3 wasm-bindgen`_$LT$std..sync..mutex..Mutex$LT$T$GT$$GT$::lock::h7159199f04e5e3d7 + 51
    frame #13: 0x00000001002095f8 wasm-bindgen`thread_local::thread_id::ThreadId::new::h6921b80d28b8a599 + 40
    frame #14: 0x00000001002097bd wasm-bindgen`thread_local::thread_id::THREAD_ID::__init::h5478d05251a82a10 + 13
    frame #15: 0x0000000100206b85 wasm-bindgen`_$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::init::hc377c97075f0478c + 37
    frame #16: 0x0000000100206dee wasm-bindgen`_$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::he41f193bcfa8bc26 + 334
    frame #17: 0x0000000100206c65 wasm-bindgen`_$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h3dea01a5e8940192 + 21
    frame #18: 0x0000000100209727 wasm-bindgen`thread_local::thread_id::get::hfa631e905ee35447 + 23
    frame #19: 0x00000001000cc90d wasm-bindgen`_$LT$thread_local..CachedThreadLocal$LT$T$GT$$GT$::get_or_try::hc8f09a6d92b53fc9 + 29
    frame #20: 0x0000000100155e48 wasm-bindgen`regex::re_unicode::Regex::locations::he5308b7fe79a3196 + 104
    frame #21: 0x000000010014ccef wasm-bindgen`regex::re_unicode::Regex::captures::h3f9d5d85220d7ec1 + 79
    frame #22: 0x000000010000c678 wasm-bindgen`docopt::parse::Parser::parse::hd4f1e73726c69527 + 120
    frame #23: 0x000000010000be5b wasm-bindgen`docopt::parse::Parser::new::h08a1db9f339f3acf + 523
    frame #24: 0x00000001000050ce wasm-bindgen`docopt::dopt::Docopt::new::h999d56b70313ae9b + 110
    frame #25: 0x0000000100000d28 wasm-bindgen`wasm_bindgen::main::h2ac10bbd516a199d + 40
    frame #26: 0x0000000100005472 wasm-bindgen`std::rt::lang_start::_$u7b$$u7b$closure$u7d$$u7d$::h4eac75b514bea9ce + 18
    frame #27: 0x000000010051db38 wasm-bindgen`std::panicking::try::do_call::h0f8a789012cd34c6 [inlined] std::rt::lang_start_internal::_$u7b$$u7b$closure$u7d$$u7d$::h3e3d4117cb482f1c + 24 at rt.rs:59
    frame #28: 0x000000010051db2c wasm-bindgen`std::panicking::try::do_call::h0f8a789012cd34c6 + 12 at panicking.rs:310
    frame #29: 0x000000010052981f wasm-bindgen`__rust_maybe_catch_panic + 31 at lib.rs:105
    frame #30: 0x00000001005140e2 wasm-bindgen`std::rt::lang_start_internal::h775766efcbaa9c6c [inlined] std::panicking::try::h5a637b8342249be1 + 51 at panicking.rs:289
    frame #31: 0x00000001005140af wasm-bindgen`std::rt::lang_start_internal::h775766efcbaa9c6c [inlined] std::panic::catch_unwind::h095eff007ddeac63 at panic.rs:374
    frame #32: 0x00000001005140af wasm-bindgen`std::rt::lang_start_internal::h775766efcbaa9c6c + 191 at rt.rs:58
    frame #33: 0x0000000100005452 wasm-bindgen`std::rt::lang_start::heb39064736550b2c + 66
    frame #34: 0x0000000100003805 wasm-bindgen`main + 37
    frame #35: 0x00007fff889415c9 libdyld.dylib`start + 1

If I'm reading that right, there's a panic from trying to lock a poisoned mutex, but then the panic code hits an EXC_BAD_ACCESS which is what actually crashes the app. Is that what you're seeing too? If so, it seems like there are two separate issues, the poisoned mutex and the EXC_BAD_ACCESS, no?

And wasm-bindgen and wasm2es6js both call Docopt::new, so the reason this is only affecting wasm-bindgen is... some non-obvious difference in their usage strings??

There's more I'd like to debug on my end. But in the meantime, this all looks pretty far removed from wasm-bindgen. So I'd be fine closing this issue. Do you have any advice on where to file this instead? It's not clear to me if the fault lies in libstd, thread_local, regex, or some combo.

Thanks for your help!

[alexcrichton]

cc @BurntSushi have you ever seen a segfault like this? I can't really explain what's going on :(

@TimHambourger can you get a diassembly of precisely what the faulting instruction is?

[TimHambourger]

@alexcrichton Sure thing. Does this give you what you need?

Process 2080 stopped
* thread #1: tid = 0x34c2, 0x000000010051db92 wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec [inlined] core::ptr::swap_nonoverlapping_bytes::h6df82f8d5ea7e526 + 28 at ptr.rs:237, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x000000010051db92 wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec [inlined] core::ptr::swap_nonoverlapping_bytes::h6df82f8d5ea7e526 + 28 at ptr.rs:237
(lldb) disassemble --frame
wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec at panicking.rs:316:
   0x10051db50:  pushq  %rbp
   0x10051db51:  movq   %rsp, %rbp
   0x10051db54:  subq   $0x10, %rsp
   0x10051db58:  leaq   0x22e701(%rip), %rdi      ; std::panicking::update_panic_count::PANIC_COUNT::__getit::__KEY::h0d46fda1cd45c9ae
   0x10051db5f:  callq  *(%rdi)
   0x10051db61:  cmpq   $0x1, (%rax)
   0x10051db65:  jne    0x10051db76               ; std::panicking::panicking::hba52d83258fa66ec + 38 [inlined] core::ptr::swap_nonoverlapping_bytes::h6df82f8d5ea7e526 at ptr.rs:187
std::panicking::panicking::hba52d83258fa66ec + 38 [inlined] core::ptr::swap_nonoverlapping::hdae0ad8ddd3a4807 at mem.rs:634
std::panicking::panicking::hba52d83258fa66ec + 38 [inlined] core::mem::swap::h079abbed39293f6c at mem.rs:691
std::panicking::panicking::hba52d83258fa66ec + 38 [inlined] core::mem::replace::h33caec918b20cf74 at local.rs:270
std::panicking::panicking::hba52d83258fa66ec + 38 [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::init::hf29d791cc8dc2c2f at local.rs:296
std::panicking::panicking::hba52d83258fa66ec + 38 [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::hf4098ad3d27e0fb7 + 34 at local.rs:248
std::panicking::panicking::hba52d83258fa66ec + 4 [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h07ea88e18fd4e1fd at panicking.rs:240
std::panicking::panicking::hba52d83258fa66ec + 4 [inlined] std::panicking::update_panic_count::h96721aee7d30548f at panicking.rs:317
std::panicking::panicking::hba52d83258fa66ec + 4 at panicking.rs:317
   0x10051db67:  leaq   0x22e6f2(%rip), %rdi      ; std::panicking::update_panic_count::PANIC_COUNT::__getit::__KEY::h0d46fda1cd45c9ae
   0x10051db6e:  callq  *(%rdi)
   0x10051db70:  movq   0x8(%rax), %rcx
   0x10051db74:  jmp    0x10051db97               ; std::panicking::panicking::hba52d83258fa66ec + 71 [inlined] core::ptr::swap_nonoverlapping_bytes::h6df82f8d5ea7e526 at ptr.rs:187
std::panicking::panicking::hba52d83258fa66ec + 71 [inlined] core::ptr::swap_nonoverlapping::h68a2b8a25e0adf7d at mem.rs:634
std::panicking::panicking::hba52d83258fa66ec + 71 [inlined] core::mem::swap::h5da607e8df09307d at mem.rs:691
std::panicking::panicking::hba52d83258fa66ec + 71 [inlined] core::mem::replace::ha2d91b19f1abcf93 at cell.rs:480
std::panicking::panicking::hba52d83258fa66ec + 71 [inlined] _$LT$core..cell..Cell$LT$T$GT$$GT$::replace::h7d512da2f3effa47 at cell.rs:437
std::panicking::panicking::hba52d83258fa66ec + 71 [inlined] _$LT$core..cell..Cell$LT$T$GT$$GT$::set::h44a09895ea8d28b8 at panicking.rs:242
std::panicking::panicking::hba52d83258fa66ec + 71 [inlined] std::panicking::update_panic_count::_$u7b$$u7b$closure$u7d$$u7d$::h8e6b5150de66179a at local.rs:294
std::panicking::panicking::hba52d83258fa66ec + 71 [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::hf4098ad3d27e0fb7 + 67 at local.rs:248
std::panicking::panicking::hba52d83258fa66ec + 4 [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h07ea88e18fd4e1fd at panicking.rs:240
std::panicking::panicking::hba52d83258fa66ec + 4 [inlined] std::panicking::update_panic_count::h96721aee7d30548f at panicking.rs:317
std::panicking::panicking::hba52d83258fa66ec + 4 at panicking.rs:317
   0x10051db76:  movl   $0x1, %eax
   0x10051db7b:  movd   %rax, %xmm0
   0x10051db80:  movdqa %xmm0, -0x10(%rbp)
   0x10051db85:  leaq   0x22e6d4(%rip), %rdi      ; std::panicking::update_panic_count::PANIC_COUNT::__getit::__KEY::h0d46fda1cd45c9ae
   0x10051db8c:  callq  *(%rdi)
   0x10051db8e:  movaps -0x10(%rbp), %xmm0
-> 0x10051db92:  movaps %xmm0, (%rax)
   0x10051db95:  xorl   %ecx, %ecx
   0x10051db97:  leaq   0x22e6c2(%rip), %rdi      ; std::panicking::update_panic_count::PANIC_COUNT::__getit::__KEY::h0d46fda1cd45c9ae
   0x10051db9e:  callq  *(%rdi)
   0x10051dba0:  movq   %rcx, 0x8(%rax)
   0x10051dba4:  testq  %rcx, %rcx
   0x10051dba7:  setne  %al
   0x10051dbaa:  addq   $0x10, %rsp
   0x10051dbae:  popq   %rbp
   0x10051dbaf:  retq   
(lldb) disassemble --line
wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec + 38 [inlined] core::ptr::swap_nonoverlapping_bytes::h6df82f8d5ea7e526 at ptr.rs:187
   0x10051db76:  movl   $0x1, %eax
   0x10051db7b:  movd   %rax, %xmm0
   0x10051db80:  movdqa %xmm0, -0x10(%rbp)
   0x10051db85:  leaq   0x22e6d4(%rip), %rdi      ; std::panicking::update_panic_count::PANIC_COUNT::__getit::__KEY::h0d46fda1cd45c9ae
   0x10051db8c:  callq  *(%rdi)
   0x10051db8e:  movaps -0x10(%rbp), %xmm0
-> 0x10051db92:  movaps %xmm0, (%rax)
   0x10051db95:  xorl   %ecx, %ecx

FYI, I'll be traveling for the next few days and won't have the computer where I've been repro'ing this.

[alexcrichton]

Ok thanks! That's a little suspicious... If you get a chance can you see what the value of %rax is? I'm curious if it's a valid pointer and if it's aligned properly

[BurntSushi]

@alexcrichton Sadly, no, I haven't seen this particular blend before. I have seen segfaults with thread_local! on darwin, but not with the thread_local crate. cc @Amanieu

[alexcrichton]

I've run this through a gauntlet of OSX releases on Travis and unfortunately none of them turned up segfaults :(

@TimHambourger in all likelihood this is either a linker bug, a compiler bug, or a libc runtime bug. In that sense the "fix" here would be figuring out how to just flat out avoid whatever is triggering the bug. The wasm-bindgen crate doesn't otherwise use the regex crate so we could switch to something like clap perhaps, but I'm otherwise not entirely sure what to do about this :(

[TimHambourger]

@alexcrichton Thanks for the gauntlet of tests! I'm more than willing to believe that this is particular to this box. I tested on my newer mac w/ OSX 10.12 and saw no problems there.

So I don't see any reason to refactor wasm-bindgen for this. I have a few workarounds available (including sticking to the newer box or installing with the version of rust that I know works). I'll go ahead and close this issue.

But, if you're curious, I did check %rax. I get 0x0000000100f04e58:

Process 1517 stopped
* thread #1: tid = 0x4ee6, 0x000000010051ef32 wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec [inlined] core::ptr::swap_nonoverlapping_bytes::h6df82f8d5ea7e526 + 28 at ptr.rs:237, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
    frame #0: 0x000000010051ef32 wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec [inlined] core::ptr::swap_nonoverlapping_bytes::h6df82f8d5ea7e526 + 28 at ptr.rs:237
(lldb) disassemble --line
wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec + 38 [inlined] core::ptr::swap_nonoverlapping_bytes::h6df82f8d5ea7e526 at ptr.rs:187
   0x10051ef16:  movl   $0x1, %eax
   0x10051ef1b:  movd   %rax, %xmm0
   0x10051ef20:  movdqa %xmm0, -0x10(%rbp)
   0x10051ef25:  leaq   0x22f404(%rip), %rdi      ; std::panicking::update_panic_count::PANIC_COUNT::__getit::__KEY::h0d46fda1cd45c9ae
   0x10051ef2c:  callq  *(%rdi)
   0x10051ef2e:  movaps -0x10(%rbp), %xmm0
-> 0x10051ef32:  movaps %xmm0, (%rax)
   0x10051ef35:  xorl   %ecx, %ecx
(lldb) register read
General Purpose Registers:
       rax = 0x0000000100f04e58
       rbx = 0x00007fff5fbff7b0
       rcx = 0x0000000000000000
       rdx = 0x0000000000000000
       rdi = 0x000000010074e330  wasm-bindgen`std::panicking::update_panic_count::PANIC_COUNT::__getit::__KEY::h0d46fda1cd45c9ae
       rsi = 0x0000000101a18068
       rbp = 0x00007fff5fbfd9b0
       rsp = 0x00007fff5fbfd9a0
        r8 = 0x0000000101a18060
        r9 = 0x00007fff5fbfd8f0
       r10 = 0x0000000101a0d008
       r11 = 0xffff8001a1e1a770
       r12 = 0x0000000000000000
       r13 = 0x0000000000000000
       r14 = 0x00007fff5fbff7e8
       r15 = 0x00007fff5fbff7a8
       rip = 0x000000010051ef32  wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec + 66 [inlined] core::ptr::swap_nonoverlapping_bytes::h6df82f8d5ea7e526 + 28 at ptr.rs:187
  wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec + 38 [inlined] core::ptr::swap_nonoverlapping::hdae0ad8ddd3a4807 at mem.rs:634
  wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec + 38 [inlined] core::mem::swap::h079abbed39293f6c at mem.rs:691
  wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec + 38 [inlined] core::mem::replace::h33caec918b20cf74 at local.rs:270
  wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec + 38 [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::init::hf29d791cc8dc2c2f at local.rs:296
  wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec + 38 [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::try_with::hf4098ad3d27e0fb7 + 34 at local.rs:248
  wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec + 4 [inlined] _$LT$std..thread..local..LocalKey$LT$T$GT$$GT$::with::h07ea88e18fd4e1fd at panicking.rs:240
  wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec + 4 [inlined] std::panicking::update_panic_count::h96721aee7d30548f at panicking.rs:317
  wasm-bindgen`std::panicking::panicking::hba52d83258fa66ec + 4 at panicking.rs:317
    rflags = 0x0000000000010202
        cs = 0x000000000000002b
        fs = 0x0000000000000000
        gs = 0x0000000000000000

[alexcrichton]

Ok so the specific cause of this segfault is a misaligned value. The faulting instruction is movaps %xmm0, (%rax) which seems to require that the memory location has a 16-byte alignment, but the destination %rax does not (0x0100f04e58).

The return value of the previous indirect call to the __getit function this is returning the address of a #[thread_local] in memory which means I think this is basically a bog-standard misaligned TLS value on OSX. We've seen this in a number of locations unfortunately, and it looks like it was a bug in Xcode's linker from long ago (since been fixed).

In that sense it looks like upgrading fixed the issue for you which is indeed expected!

[TimHambourger]

Ah, makes sense. That was very informative, thanks! Yes, I saw that one of your Travis CI tests hit the same OSX version as me but had a slightly newer XCode version (6.4 vs 6.2). So could be consistent with XCode being the culprit.

[stuhood]

This seems potentially related to a thread_local issue that we are seeing on older OSX versions with rust 1.26. We reproduce on travis's images for xcode6.4 and xcode7.3 (OSX 10.10 and 10.11, respectively): see the stack from pantsbuild/pants#5803 (comment) (although ignore the forking diagnosis, as there is no forking happening in this case).

EDIT: Although it would seem that if there is nothing we can do about the older linker, possibly it would be better to chase a fix in the regex/ignore crates?

[inferiorhumanorgans]

FWIW I'm seeing similar problems on OSX 10.9 w/ rust 1.27.