Reduce delay of completion

[lantyn]

Is your feature request related to a problem? Please describe it.
When a completion invokes kpsewhich there is usually a noticeable delay for the first time. Setting g:vimtex_include_search_enabled = 0 is only a compromise:

• you lose access to, for example, manually installed packages and master bib database under your local texmf.
• when used together with autocompletion plugins (I tried coc-vimtex), kpsewhich is still triggered and causes delay.

As a workaround, I turned off vimtex completion completely and used texlab LSP to provide the completion feature.

Describe the solution you'd like
Maybe exploiting the async feature of vim8 and neovim to run kpsewhich asynchronously could reduce the delay.

[lervag]

Can you be more specific about which type of completion where you experience the delay? Cite completion? If I am to work on this, it would be quite useful to have a good example where the delay is present. I.e.,

For me, the delay is really quite small (but I agree it is noticable). Perhaps 0.5 seconds. And just the first time. Note that this is for cite completion with a relatively large bib file.

I'm not quite sure how to use the async features, to be honest. I would be happy to do this, but it would be easier if someone who are already familiar with the features could guide me.

[lantyn]

I am on windows, and vim calls cmd.exe to run kpsewhich. You know terminal on windows is considerably slower than that on linux or mac. For me, cite completion takes about 2 seconds. The worse thing is when I edit an old project and try to insert an environment. There are usually many packages included in the preamble. After \begin{ triggering autocompletion, it usually takes more than 5 seconds for kpsewhich to finish, meanwhile a bunch of cmd windows appear and closes.

[lervag]

I am on windows, and vim calls cmd.exe to run kpsewhich. You know terminal on windows is considerably slower than that on linux or mac.

I really didn't know, and I have to admit I don't really know how to improve that. Except, perhaps, adding the async idea you propose.

For me, cite completion takes about 2 seconds. The worse thing is when I edit an old project and try to insert an environment. There are usually many packages included in the preamble. After \begin{ triggering autocompletion, it usually takes more than 5 seconds for kpsewhich to finish, meanwhile a bunch of cmd windows appear and closes.

I assume this only occurs the first time, right? There is caching now, so subsequent completion should be fast.

Please note that I don't use Windows. This makes it difficult to work on Windows specific issues, because I don't have a good way of testing. I don't mind having and improving support for Windows, but I am also quite limited with regard to both my own ability (especially with Windows related issues) and my time.

I'm curious if you perhaps would be interested in slightly changing your workflow and use either WSL for your LaTeX workflow (i.e. run a small Linux instance) or use something like MSYS. Both of these should give you a way to work with LaTeX with Vim (or neovim, if you prefer) on Windows, where things should as on Linux.

[lantyn]

I assume this only occurs the first time, right? There is caching now, so subsequent completion should be fast.

Yes, only the first time. But the first time is important for new users transferring to vimtex, isn't it?

I'm curious if you perhaps would be interested in slightly changing your workflow and use either WSL for your LaTeX workflow (i.e. run a small Linux instance) or use something like MSYS. Both of these should give you a way to work with LaTeX with Vim (or neovim, if you prefer) on Windows, where things should as on Linux.

Haha, good suggestion but there are other issues causing more pain than tolerating a small delay.

I believe the compiler is already done in the right way; it runs in the background and the writer can continue editing, no need to wait for the compiler to complete. Instead of having to wait for a delay the first time, I think it a better UX, if the user could continue inserting the command/environment name while caching completion is done in the background. I'm not sure what is appropriate for the first-time completion menu, though, maybe the items already available together with the info "completion caching in progress" "completion caching completed" similar to that of the compiler, to remind the user a second time completion could provide more items.

[lervag]

I assume this only occurs the first time, right? There is caching now, so subsequent completion should be fast.

Yes, only the first time. But the first time is important for new users transferring to vimtex, isn't it?

Of course, but it is also a significant improvement from every time.

I believe the compiler is already done in the right way; it runs in the background and the writer can continue editing, no need to wait for the compiler to complete. Instead of having to wait for a delay the first time, I think it a better UX, if the user could continue inserting the command/environment name while caching completion is done in the background. I'm not sure what is appropriate for the first-time completion menu, though, maybe the items already available together with the info "completion caching in progress" "completion caching completed" similar to that of the compiler, to remind the user a second time completion could provide more items.

The problem is that I fear this will take quite abit of work. But let's keep the issue open. I'll look into if it is possible to work on the caching in a different thread that runs in the background. But: I don't think this is possible, because Vimtex is implemented in vimscript. I don't think Vim or neovim supports running vimscript "in parallell". What they do support is running external jobs in the background with the jobstart family of functions.

Another option is to promote the completion caches to make them persistent. This does mean that vimtex will create a lot more cache files, but that might not really be a big problem.

[lervag]

By the way, and again: it would be very nice with a simple example combined with steps to reproduce the particular delays you are thinking of.

[lantyn]

I did a test with a quite minimal example. Of course, clear the cache at first. Include the packages I commonly used.

\documentclass[t,mathserif,aspectratio=43]{beamer}
\usepackage{amsmath,amssymb,amsfonts,bm,graphicx,ulem,mathtools}
%\usepackage{multirow,makecell,mathdots}
\usepackage[all,2cell]{xy}
\UseTwocells
\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage{times}
\usepackage[T1]{fontenc}

\begin{document}

\end{document}

After \begin{, C-X C-O. Now check the delay.

To my surprise, it in fact takes more than 40 seconds. Notice the icon on the taskbar that flashes. It is vim continuously calling kpsewhich in a cmd.exe window. Each flash is a single call, and the window closes after the call is finished, so it is hard to capture the content of the window. But it is basically something like blablabla kpsewhich blablabla which I suppose is redirecting the output to the cache file.

More about my platform details:
Microsoft Windows 10 Home 10.0.19042 Build 19042, gVim 8.2.1955, MikTex 20.11, all are 64bit versions. Latest vimtex.

[lervag]

Ok; I've tried to reproduce this on Linux:

1. I deleted my vimtex cache.
2. I disabled my autocomplete plugin (coc.nvim). I want manual completion for testing.
3. I created the minimal document.
4. I type \begin{ then <c-x><c-o>.

The initial delay when the cache was cleared is perhaps 1 second. But then all subsequent completions are instant. Can you confirm this on your end?

[lervag]

Please note: The delay on the first invocation before the kpsewhich cache is created is something I think we can live with. However, if you get similar delays everytime you open a document for the first completion, then I will consider to look into fixing it.

[lantyn]

The initial delay when the cache was cleared is perhaps 1 second. But then all subsequent completions are instant. Can you confirm this on your end?

The subsequent completions are instant, provided that no uncached new packages are included.
Each time a new package or bib file is included, there will be a delay of several seconds, as I previously mentioned, which is very typical for another document.
If I turn to a different journal/book template of certain complexity, 10 more different packages = another 40 seconds delay.

But yes, in the end things get settled down and smooth, so this issue may not enjoy very high priority.

[lervag]

But yes, in the end things get settled down and smooth, so this issue may not enjoy very high priority.

Ok, thanks for the feedback. I'll take a look if I can find a simple way to improve things.

However, I'm still curious which types of completion are problematic. The one we are discussing now is what I've called environment completion. It uses kpsewhich to find packages and load available environment candidates from the discovered packages. Are there other types of completion where you find there is a problematic delay, and if so, could you provide a list with simple examples or at least specification that makes it easier for me to look into each of them?

[lantyn]

My discovery is that the delay scales with the number of files to locate.

I checked my cache files. There is bib cache file as large as the package cache file (200kb). In the corresponding tex file, there are 4 bib files included in \bibliography. I did the delay test of \cite{ command for that tex file, and it takes about 6 seconds. Other tex files including only one bib file usually have a delay of 1.5~2 seconds.

I found 3 kinds of cache files: bib, package and ref. Latex commands and environments are the same type of completion, are they? Indeed trying to complete commands after a single backslash \ has a similar delay as that for \begin{. The completion for \ref{ has no delay (I suppose they depend on the aux file and no kpsewhich).

I have a reasonable conjecture: the performance bottleneck on windows is the multiple call of cmd.exe. Each call starts a new cmd window, runs the external command from vimscript, and closes the window. So a simple fix may be to avoid the start and close of cmd of so many times, by first organizing the packages and bib files into a list or a batch file, and make a single call of cmd.exe (for vimscript on windows, that means a single external command) to locate the files.

I am happy to help with test on windows if you implemented any changes.

[lervag]

I did the delay test of \cite{ command for that tex file, and it takes about 6 seconds. Other tex files including only one bib file usually have a delay of 1.5~2 seconds.

Ok, thanks. I think this may also be because of a call to kpsewhich.

I have a reasonable conjecture: the performance bottleneck on windows is the multiple call of cmd.exe. Each call starts a new cmd window, runs the external command from vimscript, and closes the window. So a simple fix may be to avoid the start and close of cmd of so many times, by first organizing the packages and bib files into a list or a batch file, and make a single call of cmd.exe (for vimscript on windows, that means a single external command) to locate the files.

I think your conjecture is correct, but your solution is not viable. I don't want batch processing here, because the calls to kpsewhich are cached and I want to check one file at a time.

I am happy to help with test on windows if you implemented any changes.

Ok, I'll let you know if I get some idea on how to improve this. I am thinking of rewriting the process.vim module to rely on the more modern job control features in Vim and neovim, which I think might also solve your issue as a side effect. But it is not so straightforward, unfortunately.

[lervag]

Btw, I'll repeat my earlier statement, which I still think holds true:

I'm curious if you perhaps would be interested in slightly changing your workflow and use either WSL for your LaTeX workflow (i.e. run a small Linux instance) or use something like MSYS. Both of these should give you a way to work with LaTeX with Vim (or neovim, if you prefer) on Windows, where things should work well.

[lervag]

I've started to work on a modernised jobs backend now, see #2195. I'll close this issue, as I don't see any other relevant activities.