Performance issues when loading org files (parsed multiple times)

[andreadev-it]

Describe the bug

It seems that, both on load and on write, the org files get processed multiple times. When processing a large file, this causes visible performance issues (it takes about 5-6 seconds to open my neovim config file of ~1000 lines).
I've put some debugging print around the File:_parse() function call, here it is how the code for the File:new() function looks right now (this file):

  setmetatable(data, self)
  self.__index = self
  print("Before parsing " .. filename .. ": " .. vim.loop.now())
  data:_parse()
  print("After parsing: " .. vim.loop.now())
  return data

This is what gets printed when I load my init.org file:

As you can see, the file parsing method gets called 6 times (usually it is 4 times). I don't mind the parsing taking more than a second, that's understandable, but it is a bit weird that it is being done so many times, and the slowdown is noticable.
Other files only get parsed 2 or 4 times, this seem to be related to the file containing a source code block or not (with source code blocks it parses the file 4 times, otherwise 2).

Also, when writing the file, here it is what the messages show:

Again, that code gets executed 3 times.

I will be exploring this a little bit more, but I thought to share this findings, so that maybe someone can point me in the right direction. Maybe there is an explanation for this, so if there's a reason, it's fine for me, but it seems like there's room for improvement on our part :)

Steps to reproduce

Not much to say here, it happens every time I open or save a file

Expected behavior

The parsing should be done once when opening the file, and once when the file gets saved. At least, that's what I would expect.

Emacs functionality

No response

Minimal init.lua

I've tested with my own config, but testing it with the minimal init found in tests/minimal_init.vim gives the same results.

Screenshots and recordings

No response

OS / Distro

Windows 10

Neovim version/commit

0.8.0

Additional context

No response

[andreadev-it]

I've done a little more digging and found out that the following function runs the command filetype detect always when opening a file, and this triggers the FileType event, which in turn doubles the loading time.

Files._check_source_blocks

This function can be found in parser/files.lua at line 296.

I do not understand what this function should do, to be honest, why is there the need to run the filetype detect command? It seems like it might be to prevent some issues, but I don't seem to get why...

[andreadev-it]

Ok. I understood that it seems to be needed for treesitter injection into the source code blocks, because without it all code blocks just shows as simple text, without any highlighting or any other feature.

[andreadev-it]

I found out that using vim.loop.now() is not the right thing to do. I'm sorry for making a fuss, but in the end it seems that the issue might be with the filetype detection, and not with this plugin code. However, would it be possible to avoid checking for the filetype twice? Currently it seems to be necessary to get the filetype of source code blocks

[joaomsa]

Nice catch, been running into this slowdown as well with my telescope plugin, but I haven't investigated as well as you've done.

I'm wondering possible optimization routes, like paralleling each file parsing with the job control api, or perhaps some lazy behavior for hidden buffers.

[kristijanhusak]

filetype detect should be triggered only when you add a new code block via #+begin_src, otherwise it should not trigger.

We do some tracking of the changedtick on the file itself to avoid parsing when not necessary, but the problem is that file gets requested by 3 sources at the same time, and at that point, none of the sources have a "fresh" file.

We could probably stick with a stale file state for some of the things, but I didn't investigate it too deeply because my files are not that big (I archive a lot). I'll check if we can maybe cut the "refresh" on some places where it's not necessary, that should speed things up a bit.

[kristijanhusak]

I just pushed some changes that should reduce the number of parsing. On first open, it will parse a bit more, but editing a single file should trigger parsing only when the file is actually changed. Please give it a test and let me know how it works.

[andreadev-it]

Hy @kristijanhusak , in the meantime I had split my init file into different files. I merged them all into a test file to check for the performance, and indeed it feels like there's a big difference. Now it's about 1-2 seconds, when before it was about 6 seconds of latency when opening the file. Also, writing the file to disk seems a little bit faster (but that wasn't too much of an issue for me before either)
Nice job!

[kristijanhusak]

I'll close this for now since we made some progress. In future, we might try to address this further, but I guess this is ok for now. Thanks for reporting.