GDScript parsing causes large editor stalls

[gokiburikin]

Godot version

v4.1.stable.official [9704596]

System information

Windows 10

Issue description

Some part of the GDScript parsing routine is run on EditorSettings change, after idle parse, and when offering code completion suggestions. In Godot 3.5 this either isn't run or is significantly faster as to not severely impact editor performance.

In Godot 4.1 editor performance is heavily impacted by "code complexity." Below is a video of I/O and CPU usage from changing EditorSettings and typing with a code_completion_delay of 0.01.

NOTE: My CPU is an i7-10700K so the 6.2% CPU usage is Godot maxing out 1 core.

C is an autoload file with just a few properties that point to "complex" Class instances

The complex classes are just a bunch of properties

EditorSettings changes cause this stalling

Godot_v4.1-stable_win64_2023-08-14_15-44-34.mp4

Code completion suggestions cause this stalling

Godot_v4.1-stable_win64_2023-08-14_15-45-54.mp4

While these examples are contrived, the performance on my actual project is quite bad.

Godot_v4.1-stable_win64_2023-08-14_15-56-16.mp4

The same codebase in Godot 3.5 has no performance issues (it might be stalling only after pressing close)

Godot_v3.5.2-stable_win64_2023-08-14_16-08-48.mp4

Godot_v3.5.2-stable_win64_2023-08-14_16-10-05.mp4

This stalling also causes input events to be dropped or late, so if I do things that aren't typing about idle_parse_delay seconds after writing code they can get messed up during this stall, like using my mouse to select text or hitting ctrl + f. I can't reliably reproduce these for a video but they're common.

Semi-relevant discussion thread godotengine/godot-proposals#7487

Profiling

I downloaded a version of 4.0.3 with debugging symbols and confirmed moving the code completion delay slider around (changing EditorSettings) causes a validate call if any scripts are open in the script editor, but code completion seems to just be particularly slow.

VerySleepy call stacks

When changing EditorSettings

When triggering code completion with 0.01 delay

Steps to reproduce

Have a relatively "complex" codebase and attempt to change code complete delay or idle parse delay EditorSettings, or set code completion delay to 0.01 and start typing to trigger it.

MRPs attached.

Minimal reproduction project

Here's an MRP for Godot 4.1: gdscript-parse-mrp.zip
Here's an MRP for Godot 3.5: gdscript-parse-mrp3.zip

I get basically no CPU usage or disk thrashing from the Godot 3.5 MRP.

[HolonProduction]

I can reproduce the stalling in 4.x (current master).

About the stalling during completion I created a flamegraph for that:

Interactive svg

We spend way to much time parsing the dependencies. (Everything above raise_status) In theory those should be cached, but since the parser for completion is bound to the GDScriptLanguage::complete_code and is ref counted, it gets destroyed after every completion request, and with it all dependent parsers. (Although it seems this was introduced in #90601 and #92616 so I will probably take another look at the situation before that.)

[HolonProduction]

Apparently when calling GDScriptParser.parse the parser recreates itself, which will destruct it as well and therefor remove cache entries. So apparently there is no way to cache parse results across multiple calls to parse at the moment, if I'm not missing something essential.

[rune-scape]

possibly fixed by #95115
i can't actually reproduce this, but my PR does avoid invalidating parsers in this specific situation where a GDScriptParserRef is created, used, then destroyed

id appreciate if someone else could test it

[gokiburikin]

Doesn't appear to have any impact on the issue. Reproduction in this MRP is a lot more obvious if Idle Parse Delay and Code Completion are set to the lowest possible values since it will retrigger a parse with each character instead of once, but here's a more complete visual example of the issue and how to recreate it.

godot-parse-mrp.mp4

As a bonus, setting this up also informed me that there's definitely a problem when hovering over something that doesn't resolve while holding control (shown at 1:04 in the video) but I don't know if that's related or if I should open a new PR for it.

ETA: Here's the codeblock for printing stalls since it's not in the MRP linked above. Also requires @tool at the top.

var last := 0
var expected_value := 16
func _process(_delta: float) -> void:
	var d := Time.get_ticks_msec() - last
	last = Time.get_ticks_msec()
	if d - expected_value > 1:
		print("%d ms stall" % (d - expected_value))

[rune-scape]

i see, im reproducing it now, after profiling i think that #95115 either doesn't change this issue or only helps a small amount (as in it fixes a performance regression from #92616 that might be contributing to the stalling)