Different behavior when running individual vs. all tests when using custom resources and type hints

[doctor-g]

I am getting two different behaviors depending on whether I run a single test or all of my tests. I have created and attached a project to reproduce the problem.

You can open either test_custom_object.gd or test_custom_resource.gd and run the tests, and they are fine. They use the warning fiddling approach discussed in #482 in order to run properly when using custom resource types.

If you run all the tests, instead of running successfully as expected, one gets "Invalid Call. Nonexistent function 'foo' in base 'Nil.'" at line 26 of test_double_with_type_hints.

There is a way to make the tests run, though: remove the type hint on line 4 of custom_object.gd. That is, change:

var resource : CustomResource

to

var resource

Speaking as someone who loves type hints, I don't think this is a good workaround.

Godot 4.0.3 stable
Gut 9.0.1
Linux

gut_issue_reproduction-master.zip

[bitwes]

TL;DR

TL;DR for some reason, if the first test GUT runs creates a double of CustomObject then everything falls apart. If not, then it seems to run fine. You can also move the test code to a function and call that from tests, and that seems to fix it. I have no idea why.

Simplifying the steps to reproduce

I'm not sure what is causing the problem. I was able to replicate the issue with the following test script. test_problem runs fine, but test_problem_again will blow up, but only when GUT runs it as the second test, not when called from test_problem. This is the first weird thing. There are more.

const _REDUNDANT_AWAIT := "debug/gdscript/warnings/redundant_await"

func before_all():
	set_double_strategy(DOUBLE_STRATEGY.SCRIPT_ONLY)
	ProjectSettings.set_setting(_REDUNDANT_AWAIT, 0)

func test_problem():
	var dbl_custom_obj : CustomObject =  double(CustomObject).new() as CustomObject
	var dbl_custom_resource : CustomResource = double(CustomResource).new() as CustomResource
	var partial_dbl_custom_resource : CustomResource = partial_double(CustomResource).new() as CustomResource
	assert_not_null(dbl_custom_obj, 'dbl_custom_obj')
	assert_not_null(dbl_custom_resource, 'dbl_custom_resource')
	assert_not_null(partial_dbl_custom_resource, 'partail_dbl_custom_resource')

	# call the next test from within this test to see what happens.
	gut.p('-- Calling test_problem_again --')
	test_problem_again()

func test_problem_again():
	var dbl_custom_obj : CustomObject =  double(CustomObject).new() as CustomObject
	var dbl_custom_resource : CustomResource = partial_double(CustomResource).new() as CustomResource
	var partial_dbl_custom_resource : CustomResource = partial_double(CustomResource).new() as CustomResource
	assert_not_null(dbl_custom_obj, 'dbl_custom_obj')
	assert_not_null(dbl_custom_resource, 'dbl_custom_resource')
	assert_not_null(partial_dbl_custom_resource, 'partail_dbl_custom_resource')

Some results of testing:

• If a double of CustomObject is never made, then everything runs fine (comment out lines in both tests).
• I tried loading the scripts into vars and using that instead of referencing their class_name and it didn't make a difference.
• I tried removing all the type hints and as clauses from the tests, and it made no difference.

Weird Stuff

What is weird is that test_problem_again does not cause an error when called from test_problem, but when GUT runs it as the second test it blows up the same way you were seeing when running all the tests.

REALLY Weird Stuff

If the first test run does not create a double of CustomObject then everything works as expected. In the following, I only added the test_nothing test, everything passed, no errors or warnings.

extends GutTest

const _REDUNDANT_AWAIT := "debug/gdscript/warnings/redundant_await"

func before_all():
	set_double_strategy(DOUBLE_STRATEGY.SCRIPT_ONLY)
	ProjectSettings.set_setting(_REDUNDANT_AWAIT, 0)

func test_nothing():
	pass_test('this passes')

func test_problem():
	var dbl_custom_obj : CustomObject =  double(CustomObject).new() as CustomObject
	var dbl_custom_resource : CustomResource = double(CustomResource).new() as CustomResource
	var partial_dbl_custom_resource : CustomResource = partial_double(CustomResource).new() as CustomResource
	assert_not_null(dbl_custom_obj, 'dbl_custom_obj')
	assert_not_null(dbl_custom_resource, 'dbl_custom_resource')
	assert_not_null(partial_dbl_custom_resource, 'partail_dbl_custom_resource')

	# call the next test from within this test to see what happens.
	gut.p('-- Calling test_problem_again --')
	test_problem_again()


func test_problem_again():
	var dbl_custom_obj : CustomObject =  double(CustomObject).new() as CustomObject
	var dbl_custom_resource : CustomResource = partial_double(CustomResource).new() as CustomResource
	var partial_dbl_custom_resource : CustomResource = partial_double(CustomResource).new() as CustomResource
	assert_not_null(dbl_custom_obj, 'dbl_custom_obj')
	assert_not_null(dbl_custom_resource, 'dbl_custom_resource')
	assert_not_null(partial_dbl_custom_resource, 'partail_dbl_custom_resource')

Want More Weird? I Got You.

Here's something REALLY REALLY REALLY weird (yea, we just went from one "REALLY" to three). If you put all code from the tests into a function, and call that from the tests, then the error doesn't happen. The following is basically the same thing as the first script, except all the doubling and asserts are in a function.

extends GutTest

const _REDUNDANT_AWAIT := "debug/gdscript/warnings/redundant_await"

func before_all():
	set_double_strategy(DOUBLE_STRATEGY.SCRIPT_ONLY)
	ProjectSettings.set_setting(_REDUNDANT_AWAIT, 0)

func _do_the_asserts():
	var dbl_custom_obj : CustomObject =  double(CustomObject).new() as CustomObject
	var dbl_custom_resource : CustomResource = partial_double(CustomResource).new() as CustomResource
	var partial_dbl_custom_resource : CustomResource = partial_double(CustomResource).new() as CustomResource
	assert_not_null(dbl_custom_obj, 'dbl_custom_obj')
	assert_not_null(dbl_custom_resource, 'dbl_custom_resource')
	assert_not_null(partial_dbl_custom_resource, 'partail_dbl_custom_resource')


func test_problem():
	_do_the_asserts()
	_do_the_asserts()


func test_problem_again():
	_do_the_asserts()

?

I have no idea what's happening. There does appear to be some ugly workarounds for your tests, which is better than having to have an ugly workaround in your game code. I can't wait to fully understand what is going on here.

[bitwes]

Good news, I think I got a lead on this issue. It started happening in one of my own tests and I couldn't make it stop. See the Godot issue referenced for more details. I think I have a fix, but need to test it out a lot.