Neovim hangs after LSP aborts mid-completion

[mqudsi]

• Did you upgrade to latest plugin version? yes
• Did you upgrade to/compile latest binary? languageclient 0.1.160
• (Neovim users only) Did you check output of :checkhealth LanguageClient? yes
• Did you check [troubleshooting]? yes

Describe the bug

Due to an upstream rust-analyzer bug, the language server was crashing in the middle of a call. Here is the LC log reporting the LSP output and confirming the abort:

thread '<unnamed>' panicked at 'index out of bounds: the len is 1 but the index is 1', crates/hir_ty/src/lib.rs:1008:31
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:78
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:59
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1076
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1537
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:62
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:198
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:217
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:526
  11: rust_begin_unwind
             at src/libstd/panicking.rs:437
  12: core::panicking::panic_fmt
             at src/libcore/panicking.rs:85
  13: core::panicking::panic_bounds_check
             at src/libcore/panicking.rs:62
  14: <hir_ty::Ty as hir_ty::TypeWalk>::walk_mut_binders
  15: <hir_ty::Ty as hir_ty::TypeWalk>::walk_mut_binders
  16: hir_ty::method_resolution::is_valid_candidate
  17: hir_ty::method_resolution::iterate_inherent_methods
  18: hir_ty::method_resolution::iterate_method_candidates_by_receiver
  19: hir_ty::method_resolution::iterate_method_candidates_impl
  20: hir::code_model::Type::iterate_method_candidates
  21: ide::completion::complete_dot::complete_dot
  22: ide::completion::completions
  23: std::panicking::try
  24: ide::Analysis::completions
  25: rust_analyzer::handlers::handle_completion
  26: <F as threadpool::FnBox>::call_box
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
client exited without proper shutdown sequence

Of course, that isn't LanguageClient's problem at all. The problem is that this ultimately causes Neovim itself to hang, completely unresponsive (killall -9 languageclient does not help).

I have LC installed and integrated into neovim via ncm2, and the completion was triggered by <c-x><c-o>. Attaching a debugger to neovim reveals that is hanging waiting on libuv to return from an epoll call, after a call to LanguageClient_runSync(...):

#0  0x00007ff8ec65aa47 in epoll_wait (epfd=3, events=events@entry=0x7ffec7bf0120, maxevents=maxevents@entry=1024, timeout=timeout@entry=99)
    at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#1  0x000056345da94369 in uv__io_poll (loop=loop@entry=0x56345de785c0 <main_loop>, timeout=99)
    at /home/mqudsi/neovim/.deps/build/src/libuv/src/unix/linux-core.c:309
#2  0x000056345da841c4 in uv_run (loop=loop@entry=0x56345de785c0 <main_loop>, mode=UV_RUN_ONCE) at /home/mqudsi/neovim/.deps/build/src/libuv/src/unix/core.c:375
#3  0x000056345d8e39dc in loop_poll_events (loop=0x56345de785c0 <main_loop>, ms=100) at ../src/nvim/event/loop.c:62
#4  0x000056345d9069c1 in do_sleep (msec=<optimized out>) at ../src/nvim/ex_docmd.c:7619
#5  0x000056345d906b18 in ex_sleep (eap=<optimized out>) at ../src/nvim/ex_docmd.c:7608
#6  0x000056345d8fe405 in do_one_cmd (cmdlinep=cmdlinep@entry=0x7ffec7bf3488, flags=flags@entry=7, cstack=cstack@entry=0x7ffec7bf3510, 
    fgetline=fgetline@entry=0x56345d8fa1f7 <get_loop_line>, cookie=cookie@entry=0x7ffec7bf34b0) at ../src/nvim/ex_docmd.c:1955
#7  0x000056345d8fef84 in do_cmdline (cmdline=<optimized out>, fgetline=0x56345d8e6629 <get_func_line>, cookie=0x5634605e1650, flags=7)
    at ../src/nvim/ex_docmd.c:601
#8  0x000056345d8e143b in call_user_func (fp=fp@entry=0x5634604ad5d0, argcount=argcount@entry=2, argvars=argvars@entry=0x7ffec7bf4230, 
    rettv=rettv@entry=0x7ffec7bf48a0, firstline=firstline@entry=42, lastline=lastline@entry=42, selfdict=0x0) at ../src/nvim/eval/userfunc.c:1059
#9  0x000056345d8e1b71 in call_func (funcname=funcname@entry=0x563460510100 "LanguageClient_runSync", len=<optimized out>, len@entry=22, 
    rettv=rettv@entry=0x7ffec7bf48a0, argcount_in=argcount_in@entry=2, argvars_in=argvars_in@entry=0x7ffec7bf4230, argv_func=argv_func@entry=0x0, firstline=42, 
    lastline=42, doesrange=0x7ffec7bf4404, evaluate=true, partial=0x0, selfdict_in=0x0) at ../src/nvim/eval/userfunc.c:1501
#10 0x000056345d8e1e24 in get_func_tv (name=0x563460510100 "LanguageClient_runSync", len=22, rettv=0x7ffec7bf48a0, arg=0x7ffec7bf4850, firstline=42, lastline=42, 
    doesrange=0x7ffec7bf4404, evaluate=1, partial=0x0, selfdict=0x0) at ../src/nvim/eval/userfunc.c:430
#11 0x000056345d8ba78b in eval7 (arg=0x7ffec7bf4850, rettv=0x7ffec7bf48a0, evaluate=1, want_string=<optimized out>) at ../src/nvim/eval.c:4128
#12 0x000056345d8ab8c7 in eval6 (arg=arg@entry=0x7ffec7bf4850, rettv=rettv@entry=0x7ffec7bf48a0, evaluate=evaluate@entry=1, want_string=want_string@entry=0)
    at ../src/nvim/eval.c:3826
#13 0x000056345d8adccd in eval5 (arg=arg@entry=0x7ffec7bf4850, rettv=rettv@entry=0x7ffec7bf48a0, evaluate=evaluate@entry=1) at ../src/nvim/eval.c:3674
#14 0x000056345d8ae04a in eval4 (arg=arg@entry=0x7ffec7bf4850, rettv=rettv@entry=0x7ffec7bf48a0, evaluate=evaluate@entry=1) at ../src/nvim/eval.c:3415
#15 0x000056345d8ae99e in eval3 (arg=arg@entry=0x7ffec7bf4850, rettv=rettv@entry=0x7ffec7bf48a0, evaluate=evaluate@entry=1) at ../src/nvim/eval.c:3333
#16 0x000056345d8aeae3 in eval2 (arg=arg@entry=0x7ffec7bf4850, rettv=rettv@entry=0x7ffec7bf48a0, evaluate=evaluate@entry=1) at ../src/nvim/eval.c:3264
#17 0x000056345d8aec29 in eval1 (arg=arg@entry=0x7ffec7bf4850, rettv=rettv@entry=0x7ffec7bf48a0, evaluate=evaluate@entry=1) at ../src/nvim/eval.c:3190
#18 0x000056345d8b0d8e in eval0 (
    arg=0x5634605df647 "LanguageClient_runSync( 'LanguageClient#omniComplete', { 'character': LSP#character() + len(a:base), 'complete_position': LSP#character(), 'text': s:completeText, })", rettv=rettv@entry=0x7ffec7bf48a0, nextcmd=nextcmd@entry=0x7ffec7bf4938, evaluate=1) at ../src/nvim/eval.c:3150
--Type <RET> for more, q to quit, c to continue without paging--
#19 0x000056345d8b181f in ex_let_const (eap=0x7ffec7bf4930, is_const=is_const@entry=false) at ../src/nvim/eval.c:1420
#20 0x000056345d8b1986 in ex_let (eap=<optimized out>) at ../src/nvim/eval.c:1348
#21 0x000056345d8fe405 in do_one_cmd (cmdlinep=cmdlinep@entry=0x7ffec7bf4b68, flags=flags@entry=7, cstack=cstack@entry=0x7ffec7bf4bf0, 
    fgetline=fgetline@entry=0x56345d8e6629 <get_func_line>, cookie=cookie@entry=0x5634605e37e0) at ../src/nvim/ex_docmd.c:1955
#22 0x000056345d8fef84 in do_cmdline (cmdline=<optimized out>, fgetline=0x56345d8e6629 <get_func_line>, cookie=0x5634605e37e0, flags=7)
    at ../src/nvim/ex_docmd.c:601
#23 0x000056345d8e143b in call_user_func (fp=fp@entry=0x5634604b0ee0, argcount=argcount@entry=2, argvars=argvars@entry=0x7ffec7bf5920, 
    rettv=rettv@entry=0x7ffec7bf5910, firstline=firstline@entry=42, lastline=lastline@entry=42, selfdict=0x0) at ../src/nvim/eval/userfunc.c:1059
#24 0x000056345d8e1b71 in call_func (funcname=0x563460683990 "LanguageClient#complete", len=<optimized out>, rettv=0x7ffec7bf5910, argcount_in=<optimized out>, 
    argvars_in=0x7ffec7bf5920, argv_func=0x0, firstline=42, lastline=42, doesrange=0x7ffec7bf58c4, evaluate=true, partial=0x0, selfdict_in=0x0)
    at ../src/nvim/eval/userfunc.c:1501
#25 0x000056345d8abcd9 in call_vim_function (func=0x563460683990 "LanguageClient#complete", argc=2, argv=0x7ffec7bf5920, rettv=0x7ffec7bf5910)
    at ../src/nvim/eval.c:1052
#26 0x000056345d8a1a47 in expand_by_function (type=type@entry=13, base=base@entry=0x5634605e3260 "") at ../src/nvim/edit.c:3874
#27 0x000056345d8a315c in ins_compl_get_exp (ini=ini@entry=0x56345de7d320 <compl_startpos>) at ../src/nvim/edit.c:4246
#28 0x000056345d8a1e2f in ins_compl_next (allow_get_expansion=allow_get_expansion@entry=1, count=<optimized out>, insert_match=insert_match@entry=1, 
    in_compl_func=in_compl_func@entry=0) at ../src/nvim/edit.c:4632
#29 0x000056345d8a386b in ins_complete (c=15, enable_pum=enable_pum@entry=true) at ../src/nvim/edit.c:5250
#30 0x000056345d8a49f6 in insert_do_complete (s=s@entry=0x7ffec7bf5c00) at ../src/nvim/edit.c:1325
#31 0x000056345d8a6ce6 in insert_handle_key (s=s@entry=0x7ffec7bf5c00) at ../src/nvim/edit.c:859
#32 0x000056345d8a78b2 in insert_execute (state=0x7ffec7bf5c00, key=<optimized out>) at ../src/nvim/edit.c:802
#33 0x000056345da1fef4 in state_enter (s=0x7ffec7bf5c00) at ../src/nvim/state.c:69
#34 0x000056345d89e1f6 in insert_enter (s=s@entry=0x7ffec7bf5c00) at ../src/nvim/edit.c:484
#35 0x000056345d89e560 in edit (cmdchar=65, startln=<optimized out>, count=1) at ../src/nvim/edit.c:1404
#36 0x000056345d98c8aa in invoke_edit (cap=cap@entry=0x7ffec7bf5da8, repl=repl@entry=0, cmd=65, startln=startln@entry=0) at ../src/nvim/normal.c:7693
#37 0x000056345d98d85d in nv_edit (cap=0x7ffec7bf5da8) at ../src/nvim/normal.c:7665
#38 0x000056345d985a19 in normal_execute (state=0x7ffec7bf5d20, key=<optimized out>) at ../src/nvim/normal.c:1142
#39 0x000056345da1fef4 in state_enter (s=0x7ffec7bf5d20) at ../src/nvim/state.c:69
#40 0x000056345d980f22 in normal_enter (cmdwin=<optimized out>, noexmode=<optimized out>) at ../src/nvim/normal.c:463
#41 0x000056345d95591a in main (argc=-943759744, argv=<optimized out>) at ../src/nvim/main.c:554

Environment

• neovim/vim version (nvim --version or vim --version): NVIM v0.5.0-dev+918-g4383c0f95

• This plugin version (git rev-parse --short HEAD): 4037b1d

• This plugin's binary version (bin/languageclient --version): 0.1.160

• Minimal vimrc content (A minimal vimrc is the smallest vimrc that could
  reproduce the issue. Refer to an example [here][min-vimrc.vim]): n/a, dependent on upstream bug

• Language server link and version: rust-analyzer 277488b

To Reproduce

Try completing rust code that triggers a panic in rust-analyzer:

struct Slice<T> {}
struct Box<T, A> {}
struct Vec<T, A> {}

impl<T> Slice<T> {
    pub fn into_vec<A>(self: Box<Self, A>) -> Vec<T, A> { }
}

fn main() {
    let foo: Slice<u32>;
    foo. // try completing here
}

Current behavior

rust-analyzer crashes and neovim hangs.

Expected behavior

Neovim should not hang.

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

[martskins]

Yep, I could reproduce. Just a minor correction though, the server is not crashing, it just isn't replying. If you wait long enough (60 seconds) the request will time out and it will recover. But indeed, the client is looping here

LanguageClient-neovim/autoload/LanguageClient.vim

Line 1106 in 4037b1d

while len(l:LanguageClient_runSync_outputs) == 0

.

I'm looking into this to see if we can get rid of that runSync call and invoke completion in a more async way

[martskins]

@mqudsi I think #1170 should do the trick, I'll give it a try in the next few days and if everything works as expected we can merge that into dev.

[mqudsi]

Thanks, removing that blocking call would indeed be a welcome change. I'll test that branch myself locally as well today, if I get the chance.

[mqudsi]

Just a note that I have never witnessed neovim return to a usable state after an LS hang, even after waiting well past the specified sixty seconds (which imho is already too generous).