Multiple IntelliSense stale processes on Linux

[poetinha]

VS Code leaves multiple IntelliSense stale processes running after quit. A process can use even 99% of CPU sometimes and just hangs.

will 1772 0.3 2.4 1232188 75816 tty2 Sl+ 10:29 0:23 /home/will/.vscode/extensions/ms-vscode.cpptools-0.14.2/bin/Microsoft.VSCode.CPP.Extension.linux will 2646 0.0 0.2 318928 7912 tty2 Sl+ 10:59 0:00 /home/will/.vscode/extensions/ms-vscode.cpptools-0.14.2/bin/Microsoft.VSCode.CPP.IntelliSense.Msvc.linux 1772 8 will 2657 0.0 0.2 319060 7172 tty2 Sl+ 10:59 0:00 /home/will/.vscode/extensions/ms-vscode.cpptools-0.14.2/bin/Microsoft.VSCode.CPP.IntelliSense.Msvc.linux 1772 9 will 2916 0.0 0.2 319060 8572 tty2 Sl+ 11:14 0:00 /home/will/.vscode/extensions/ms-vscode.cpptools-0.14.2/bin/Microsoft.VSCode.CPP.IntelliSense.Msvc.linux 1772 24 will 3157 0.0 1.2 319060 37252 tty2 Sl+ 11:34 0:00 /home/will/.vscode/extensions/ms-vscode.cpptools-0.14.2/bin/Microsoft.VSCode.CPP.IntelliSense.Msvc.linux 1772 29 will 3179 0.0 0.2 319060 6640 tty2 Sl+ 11:36 0:00 /home/will/.vscode/extensions/ms-vscode.cpptools-0.14.2/bin/Microsoft.VSCode.CPP.IntelliSense.Msvc.linux 1772 30

OS: Debian 9.2 (x64)
VS Code: 1.18.0 (x64)
C/C++ extension: 0.14.2

[bobbrow]

If you ever see one of the "IntelliSense" processes running at 99% for an extended period of time with no significant change in memory usage, then you are likely hitting an infinite loop bug in the engine. If you are able to detect when this happens and can send us the code snippet or code file that causes it, we'd be happy to investigate and get the issue resolved.

Also, if you are able to share the steps you took that resulted in orphaned extension processes, we can investigate that too.

[ehadmra]

I'm facing the same issue. I noticed it gets worse as the project becomes bigger and when opening multiple files. I have to keep terminating it from time to time. This is so annoying.

[sbrw]

I've also experience this. Specially with a workspace with many files, target specific directories and pre-processor directives to select target. I've tried to enable the most verbose debug level but there is nothing in the output window. Would it be possible to temporarily add extra debug prints?

When I switch configuration (target) the intellisense flame (lower right toolbar) is continually red, and intellisense does not work. When hovering over a symbol the tool-tip says "Loading...". Only way out is to quit vscode, kill the stale processe(s) and restart.

[sean-mcmanus]

@sbrw Our pending 0.14.3 update fixed a process deadlock when switching configurations, which sounds like what you're describing. You can set your C_Cpp.loggingLevel to 7, 8, or 9 to get more debug output, but you want to lower it during normal usage, because outputting too much slows things down a lot.

[bobbrow]

I don't recommend ever using 8 or 9. 6 should be enough to see that the extension is alive and processing editor commands.

[sean-mcmanus]

We've fixed a bunch of issues that could lead to stale processes and stuck "Loading..." deadlocks. Does 0.16.1 still have problems for anyone?

[flixr]

I still have the problem that IntelliSense seems to be stuck (red flame never goes away) and there are stale processes around after closing VS Code.
OS: Ubuntu 14.04 x64
VS Code: 1.21.1
cpptools: 0.16.1

[gbonacini]

I have the same problem on OS X: multiple VSCode.CPP.IntelliSense instance running, in stale , apparently for a deadlock (I did a trace with dtruss). It's very annoying and force me to kill continuously the processes to gain 30 second of code navigation.

[sean-mcmanus]

@gbonacini @flixr Any advice on how to repro this? Is the status of the process "Zombie"? That would indicate it's crashing. It's not reproing on my machine. @grdowns got a repro on his machine of the stuck red flame, but only with a debugger attached and when switching configs, so maybe that's what you guys are hitting?...when we find/fix the root cause we might know. Any call stacks of threads with work that seemed deadlocked might help.

[gbonacini]

Thank you for your help. Today I'll collect all the data I can to reproduce the problem. For now, I can provide these information:

• The problem only happen when I'm working on a project very large, with hundreds of source files. Working on smaller project all work just fine.
• I installed vscode and this plugin three months ago: the problem began after few upgrades. At the beginning, absolutely no problem.

[gbonacini]

This could help to reproduce the problem:

• Starting a new session of vscode (the second one, other one started on other branch);
• Opening source files and navigating the code, nothing happen until I open the 18th tab in vscode with a c++ header;
• Trying to navigate the code with the option "Go to Definition" of the contextual menu, the pop-up the editor stucks on the "Loading ..." pop-up;
• I have 4 VSCode.CPP.IntelliSense processes in execution:

502 6998 6410 0 11:54AM ?? 0:09.61 /Users/bg/.vscode/extensions/ms-vscode.cpptools-0.16.1/bin/Microsoft.VSCode.CPP.IntelliSense.Msvc.darwin 6410 10
502 7006 6410 0 11:56AM ?? 0:05.36 /Users/bg/.vscode/extensions/ms-vscode.cpptools-0.16.1/bin/Microsoft.VSCode.CPP.IntelliSense.Msvc.darwin 6410 11
502 7016 6410 0 11:57AM ?? 0:09.65 /Users/bg/.vscode/extensions/ms-vscode.cpptools-0.16.1/bin/Microsoft.VSCode.CPP.IntelliSense.Msvc.darwin 6410 12
502 7044 6410 0 11:59AM ?? 0:06.24 /Users/bg/.vscode/extensions/ms-vscode.cpptools-0.16.1/bin/Microsoft.VSCode.CPP.IntelliSense.Msvc.darwin 6410 13

• All the VSCode.CPP.IntelliSense processes are blocked:

• First:
  $ sudo dtruss -p 7044

SYSCALL(args) = return

• Second
  $ sudo dtruss -p 7016

SYSCALL(args) = return

• Third:
  $ sudo dtruss -p 7006

SYSCALL(args) = return

• Fourth:
  --------- Stating here the trace of the last one ----------

madvise(0x7FC0BC878000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BC885000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BCBB9000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BCAC8000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BD5A2000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BD5AA000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BCF7A000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BC9E3000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BCF17000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BCB6B000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BD14C000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BA57E000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BA57D000, 0x1000, 0x7) = 0 0
madvise(0x7FC0C047C000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BC9A4000, 0x1000, 0x7) = 0 0
madvise(0x7FC0C0352000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BCFD8000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BCFE4000, 0x1000, 0x7) = 0 0
madvise(0x7FC0C02F5000, 0x1000, 0x7) = 0 0
madvise(0x7FC0C02F6000, 0x1000, 0x7) = 0 0
madvise(0x7FC0C02F2000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BD21B000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BD21A000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BD217000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BD2A9000, 0x1000, 0x7) = 0 0
madvise(0x7FC0C021C000, 0x1000, 0x7) = 0 0
madvise(0x7FC0C021F000, 0x1000, 0x7) = 0 0
madvise(0x7FC0C021E000, 0x1000, 0x7) = 0 0
madvise(0x7FC0C021B000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BD00E000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BD00D000, 0x1000, 0x7) = 0 0
madvise(0x7FC0C0072000, 0x1000, 0x7) = 0 0
madvise(0x7FC0C00B0000, 0x1000, 0x7) = 0 0
madvise(0x7FC0C00B2000, 0x1000, 0x7) = 0 0
madvise(0x7FC0C00AF000, 0x1000, 0x7) = 0 0
madvise(0x7FC0C00AE000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BD16B000, 0x1000, 0x7) = 0 0
madvise(0x7FC0C00DE000, 0x1000, 0x7) = 0 0
madvise(0x7FC0BD16C000, 0x0, 0x0) = 0 0
read(0x3, "\0", 0x4) = -1 Err#4
psynch_cvwait(0x7FC0BA50B0B0, 0x1C0100001D00, 0x1C00) = -1 Err#260
dtrace: error on enabled probe ID 2199 (ID 886: syscall::thread_selfid:return): invalid user access in action #5 at DIF offset 0
dtrace: error on enabled probe ID 2199 (ID 886: syscall::thread_selfid:return): invalid user access in action #5 at DIF offset 0
dtrace: error on enabled probe ID 2199 (ID 886: syscall::thread_selfid:return): invalid user access in action #5 at DIF offset 0
dtrace: error on enabled probe ID 2199 (ID 886: syscall::thread_selfid:return): invalid user access in action #5 at DIF offset 0

------------------- End trace of the fourth -------------------------

• After waiting 15 minutes, I killed all the 4 processes, and the code navigation is in working condition again.

I hope this information will be useful, thank you again for the help.

[flixr]

@sean-mcmanus maybe the backtraces and log I posted in #1777 helps?

[sean-mcmanus]

@flixr Yeah, thanks for the ping on the backtraces. I think we have enough info to make progress on this (hopefully there is only this one deadlock).

[gbonacini]

Hi I think I found the problem: the source code is on a remote file system locally mounted. If for some reason there are network problem, or a remount is needed, the Intellisense processes stuck in a perpetual wait state, blocking the correct scanning (the "Loading..." popup remains untill I kill all the Intellisense processes):

ps -ef | grep -i intell
502 41382 21568 0 3:02PM ?? 1:25.99 /Users/bg/.vscode/extensions/ms-vscode.cpptools-0.16.1/bin/Microsoft.VSCode.CPP.IntelliSense.Msvc.darwin 21568 37
502 41534 21568 0 3:04PM ?? 1:15.96 /Users/bg/.vscode/extensions/ms-vscode.cpptools-0.16.1/bin/Microsoft.VSCode.CPP.IntelliSense.Msvc.darwin 21568 38
502 42199 21568 0 3:29PM ?? 0:15.62 /Users/bg/.vscode/extensions/ms-vscode.cpptools-0.16.1/bin/Microsoft.VSCode.CPP.IntelliSense.Msvc.darwin 21568 39
502 42292 21568 0 3:32PM ?? 0:11.89 /Users/bg/.vscode/extensions/ms-vscode.cpptools-0.16.1/bin/Microsoft.VSCode.CPP.IntelliSense.Msvc.darwin 21568 40
502 50085 50073 0 5:57PM ?? 0:00.06 /Users/bg/.vscode/extensions/ms-vscode.cpptools-0.16.1/bin/Microsoft.VSCode.CPP.IntelliSense.Msvc.darwin 50073 1
502 50088 50073 0 5:57PM ?? 0:00.06 /Users/bg/.vscode/extensions/ms-vscode.cpptools-0.16.1/bin/Microsoft.VSCode.CPP.IntelliSense.Msvc.darwin 50073 2
502 50092 50073 0 5:57PM ?? 0:00.06 /Users/bg/.vscode/extensions/ms-vscode.cpptools-0.16.1/bin/Microsoft.VSCode.CPP.IntelliSense.Msvc.darwin 50073 3
502 50513 11214 0 8:27PM ?? 0:58.52 /Users/bg/.vscode/extensions/ms-vscode.cpptools-0.16.1/bin/Microsoft.VSCode.CPP.IntelliSense.Msvc.darwin 11214 44
502 51715 11214 0 10:17AM ?? 0:07.97 /Users/bg/.vscode/extensions/ms-vscode.cpptools-0.16.1/bin/Microsoft.VSCode.CPP.IntelliSense.Msvc.darwin 11214 46

If I use dtruss, all the processes are bloched like this :

dtruss -p 52435
Password:

SYSCALL(args) = return

Thank you for your assistance,

G.

[gbonacini]

P.S. Trying to restart vscode, the processes won't die, the kill command seems the only way to solve the problem temporarily.

[NoSuchProcess]

I attached gdb to the one stale process, it shows following backtrack:

$ sudo gdb -p 11402 $HOME/.vscode/extensions/ms-vscode.cpptools-0.16.1/bin/Microsoft.VSCode.CPP.IntelliSense.Msvc.linux
...
(gdb) bt
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f80c3034664 in _L_lock_952 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f80c30344c6 in __GI___pthread_mutex_lock (mutex=0xfcb6a0 <_cs>) at ../nptl/pthread_mutex_lock.c:114
#3  0x0000000000b3915a in std::__1::recursive_mutex::lock() ()
#4  0x00000000005a51f5 in intellisense_client_factory::clear() ()
#5  0x00000000005b2e7a in vscode::message_handler::shutdown() ()
#6  0x00000000005b307f in vscode::message_handler::shutdown(vscode::vscode_client_message const&) ()
#7  0x00000000005b2992 in vscode::message_handler::main_loop() ()
#8  0x000000000061d21c in main ()

Hope this would help to solve the problem

[sean-mcmanus]

Is this still a problem with 0.17.1? We fixed some deadlocks and crashes.

[NoSuchProcess]

It looks like that this problem in 0.17.1 has been fixed. Thanks a lot!

[samikama]

Unfortunately there still seems to be a deadlock that is leading to stale processes. vscode seems to be deadlocking when reading the output of clang-format. The version I have is 0.17.1

gdb) bt
#0  0x00007fb08055606d in __GI___libc_read (fd=0, buf=0x24291e0, nbytes=4096) at ../sysdeps/unix/sysv/linux/read.c:26
#1  0x00007fb0804d69a0 in _IO_new_file_underflow (fp=0x7fb08082c9e0 <_IO_2_1_stdin_>) at fileops.c:584
#2  0x00007fb0804d7c82 in __GI__IO_default_uflow (fp=0x7fb08082c9e0 <_IO_2_1_stdin_>) at genops.c:413
#3  0x00007fb0804d19c8 in _IO_getc (fp=0x7fb08082c9e0 <_IO_2_1_stdin_>) at getc.c:40
#4  0x0000000000b23cb5 in std::__1::__stdinbuf<char>::__getchar(bool) ()
#5  0x0000000000609eb9 in std::__1::basic_istream<char, std::__1::char_traits<char> >& std::__1::getline<char, std::__1::char_traits<char>, std::__1::allocator<char> >(std::__1::basic_istream<char, std::__1::char_traits<char> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, char) ()
#6  0x00000000005bc9b3 in vscode::message_handler::main_loop() ()
#7  0x00000000006293dc in main ()
(gdb) inf thr
  Id   Target Id         Frame 
* 1    Thread 0x7fb080f98740 (LWP 10684) "Microsoft.VSCod" 0x00007fb08055606d in __GI___libc_read (fd=0, buf=0x24291e0, nbytes=4096)
    at ../sysdeps/unix/sysv/linux/read.c:26
......
 14   Thread 0x7fb06d7fa700 (LWP 10698) "Microsoft.VSCod" 0x00007fb080840786 in futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fb06d7f9e70, 
    expected=0, futex_word=0x2424358) at ../sysdeps/unix/sysv/linux/futex-internal.h:205
  15   Thread 0x7fb06cff9700 (LWP 10699) "Microsoft.VSCod" 0x00007fb080844a4a in __waitpid (pid=13067, stat_loc=0x7fb06cff7ce0, options=0)
    at ../sysdeps/unix/sysv/linux/waitpid.c:29
  16   Thread 0x7fb043fff700 (LWP 10700) "Microsoft.VSCod" 0x00007fb080840072 in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffdf406d990)
(gdb) bt
#0  0x00007fb08055606d in __GI___libc_read (fd=0, buf=0x24291e0, nbytes=4096) at ../sysdeps/unix/sysv/linux/read.c:26
#1  0x00007fb0804d69a0 in _IO_new_file_underflow (fp=0x7fb08082c9e0 <_IO_2_1_stdin_>) at fileops.c:584
#2  0x00007fb0804d7c82 in __GI__IO_default_uflow (fp=0x7fb08082c9e0 <_IO_2_1_stdin_>) at genops.c:413
#3  0x00007fb0804d19c8 in _IO_getc (fp=0x7fb08082c9e0 <_IO_2_1_stdin_>) at getc.c:40
#4  0x0000000000b23cb5 in std::__1::__stdinbuf<char>::__getchar(bool) ()
#5  0x0000000000609eb9 in std::__1::basic_istream<char, std::__1::char_traits<char> >& std::__1::getline<char, std::__1::char_traits<char>, std::__1::allocator<char> >(std::__1::basic_istream<char, std::__1::char_traits<char> >&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, char) ()
#6  0x00000000005bc9b3 in vscode::message_handler::main_loop() ()
#7  0x00000000006293dc in main ()
(gdb) thr 15
[Switching to thread 15 (Thread 0x7fb06cff9700 (LWP 10699))]
#0  0x00007fb080844a4a in __waitpid (pid=13067, stat_loc=0x7fb06cff7ce0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
29	../sysdeps/unix/sysv/linux/waitpid.c: No such file or directory.
(gdb) bt
#0  0x00007fb080844a4a in __waitpid (pid=13067, stat_loc=0x7fb06cff7ce0, options=0) at ../sysdeps/unix/sysv/linux/waitpid.c:29
#1  0x0000000000662822 in exec_command_and_get_output(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, unsigned int) ()
#2  0x00000000005e1d38 in vscode::message_handler::textDocument_formattingHelper(vscode::DocumentRangeFormattingParams) ()
#3  0x00000000005d0bf2 in vscode::message_handler::textDocument_formatting(vscode::DocumentFormattingParams) ()
#4  0x00000000005c0426 in vscode::message_handler::dispatch(vscode::vscode_client_message&&, vscode::vscode_server_message&) ()
#5  0x00000000005bd941 in vscode::message_handler::handle_message(vscode::vscode_client_message&&) ()
#6  0x00000000005e63d9 in void* std::__1::__thread_proxy<std::__1::tuple<vscode::message_handler::main_loop()::$_28> >(void*) ()
#7  0x00007fb0808397fc in start_thread (arg=0x7fb06cff9700) at pthread_create.c:465
#8  0x00007fb080566b5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

I hope this helps.

Thanks,
Sami

[sean-mcmanus]

@samikama Okay -- yeah, other users have reported some sort of freeze when formatting. I've been able to obtain a repro with the chromium workspace -- it doesn't repro in all scenarios for some reason. We'll investigate. Thanks for reporting this. So far it looks like clang-format is just getting stuck. Maybe we could time out or something.

[samikama]

@sean-mcmanus,

Thanks, considering that I wasn't observing this issue since I started using vscode and symptoms started after I upgraded to most recent vscode and cpp plugin, I would have looked at possible pipe related issues since bt seems to point that clang is waiting on write buffer while cpptools is waiting on read. See below for clang-format backtrace. I hope these help.

(gdb) bt
#0  0x00007ff6da5060c4 in __GI___libc_write (fd=1, buf=0x55f0a0c468a2, nbytes=32767) at ../sysdeps/unix/sysv/linux/write.c:26
#1  0x00007ff6db2f48b8 in llvm::raw_fd_ostream::write_impl(char const*, unsigned long) () from /usr/lib/x86_64-linux-gnu/libLLVM-5.0.so.1
#2  0x00007ff6db2f540d in llvm::raw_ostream::write(char const*, unsigned long) () from /usr/lib/x86_64-linux-gnu/libLLVM-5.0.so.1
#3  0x000055f09f198d0b in ?? ()
#4  0x000055f09f12ffa5 in ?? ()
#5  0x000055f09f12b90e in ?? ()
#6  0x00007ff6da4231c1 in __libc_start_main (main=0x55f09f12b650, argc=5, argv=0x7ffe656f3808, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, 
    stack_end=0x7ffe656f37f8) at ../csu/libc-start.c:308
#7  0x000055f09f12b9fa in ?? ()
(gdb) 
#0  0x00007ff6da5060c4 in __GI___libc_write (fd=1, buf=0x55f0a0c468a2, nbytes=32767) at ../sysdeps/unix/sysv/linux/write.c:26
#1  0x00007ff6db2f48b8 in llvm::raw_fd_ostream::write_impl(char const*, unsigned long) () from /usr/lib/x86_64-linux-gnu/libLLVM-5.0.so.1
#2  0x00007ff6db2f540d in llvm::raw_ostream::write(char const*, unsigned long) () from /usr/lib/x86_64-linux-gnu/libLLVM-5.0.so.1
#3  0x000055f09f198d0b in ?? ()
#4  0x000055f09f12ffa5 in ?? ()
#5  0x000055f09f12b90e in ?? ()
#6  0x00007ff6da4231c1 in __libc_start_main (main=0x55f09f12b650, argc=5, argv=0x7ffe656f3808, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, 
    stack_end=0x7ffe656f37f8) at ../csu/libc-start.c:308
#7  0x000055f09f12b9fa in ?? ()

[sean-mcmanus]

@samikama Yeah, we have a fix pending, we're working on releasing an update.

[samikama]

@sean-mcmanus,

Great! Thanks. I found it very useful and looking forward to the fix.

Cheers,
Sami

[sean-mcmanus]

We believe this is fixed with 0.17.7 -- but you may need to restart your machine or kill any old process from prevoius versions that are still dangling.

@samikama The formatting deadlocked was fixed in 0.17.3: #2007 .