Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix slow editor load on large projects #95672

Closed

Conversation

Hilderin
Copy link
Contributor

I tested a project with all the image from https://github.com/PMDCollab/SpriteCollab/tree/master/sprite which contains 61332 png.

Problem found:

  • In the new EditorFileSystem::_first_scan_process_scripts, the method ResourceLoader::get_resource_type was called on each file and this method loops on all the loader and calls get_resource_type which for some loader read the resource file. So all files were read multiple times.

Before this PR:

  • First scan (before Filesystem Dock progress bar appears): 1 min

After this PR:

  • First scan (before Filesystem Dock progress bar appears): 4-5 sec.

I found another performance problem with EditorFileSystem::_test_for_reimport but that will take more modifications to fix, I'll create another PR later.

@Jordyfel
Copy link
Contributor

It is possible to get all valid script extensions from ScriptServer. Something like

for (int i = 0; i < ScriptServer::get_language_count(); i++) {
	ScriptServer::get_language(i)->get_extension();
}

@smix8
Copy link
Contributor

smix8 commented Aug 17, 2024

Great! Can confirm this makes the scanning files dialog appear near instant on a 100.000+ files project.

It still does the kinda ridiculous reloading from disk for all the files in a project on a file change / rename but I guess that is the other issue mentioned with the EditorFileSystem::_test_for_reimport?

@Hilderin
Copy link
Contributor Author

It is possible to get all valid script extensions from ScriptServer. Something like

for (int i = 0; i < ScriptServer::get_language_count(); i++) {
	ScriptServer::get_language(i)->get_extension();
}

Thanks @Jordyfel, I did not thought about it. It even faster, I'm down to 1-3 seconds for the first scan on the same project instead of 4-5 seconds. I pushed the modifications to use your suggestion.

@Hilderin Hilderin force-pushed the fix-slow-load-on-large-project branch from a2920b5 to e92e7a3 Compare August 17, 2024 01:36
@Hilderin
Copy link
Contributor Author

Great! Can confirm this makes the scanning files dialog appear near instant on a 100.000+ files project.

Nice!!! Thanks to test it.

It still does the kinda ridiculous reloading from disk for all the files in a project on a file change / rename but I guess that is the other issue mentioned with the EditorFileSystem::_test_for_reimport?

Exactly, each .import file are read like 3 times on editor load. I'm looking into it. Pretty sure I will not be able to do miracles but I'll do my best.

@Hilderin
Copy link
Contributor Author

I'm closing this PR since the V2 has been approved.

@Hilderin Hilderin closed this Sep 26, 2024
@AThousandShips AThousandShips removed this from the 4.4 milestone Sep 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants