Support typed arrays in the extension API dump

[raulsntos]

Summary

Currently we're using {"type: "typedarray::Node"} to represent a typed array.
This PR changes this to: {"type": "Array", "array_type": "Node"}.

Changing the value of type breaks compatiblity.

Details

Adds optional array_type field to specify the type of the array element when type is Array (for example in Node::get_children).

The JSON for Node::get_children with this PR is now:

{
	"name": "get_children",
	"is_const": true,
	"is_vararg": false,
	"is_static": false,
	"is_virtual": false,
	"hash": 873284517,
	"return_value": {
		"type": "Array", // <-- This is modified with this PR
		"array_type": "Node" // <-- This is added with this PR
	},
	"arguments": [
		{
			"name": "include_internal",
			"type": "bool",
			"default_value": "false"
		}
	]
},

Alternative design

• Support generic types in the extension API dump #64469

[CedNaru]

Your issue with the hint seems to be similar to what I have already encountered at some point too.

godotengine/godot-headers#67.

I think it was never really fixed in the end but you can have a starting point with the posts there.

[raulsntos]

@CedNaru Thanks! After reading that it's still not clear to me if the JSON should leave that as is. I think other places use the type name directly without the need to specify the hint so I think I'll trim the leading 24/17: then.

[CedNaru]

edit: removing that comment. I made a mistake

[Bromeon]

Regarding

"return_value": {
  "type": "Array",
  "array_type": "Node" # <-- This is added with this PR
},

Are we sure Array is the only class that will have generics? What about Dictionary, or possibly even user-defined classes in the future?

Also, are nested annotations allowed, e.g. Array<Array<int>>?

---

Maybe the notation could be written more general:

# Array<Node>
"return_value": {
  "type": "Array",
  "generic_types": ["Node"]
},

# Dictionary<String, *> (if that's ever a thing)
"return_value": {
  "type": "Dictionary",
  "generic_types": ["String", "Variant"]
},

# Array<Array<int>>
"return_value": {
  "type": "Dictionary",
  "generic_types": [{
    "type": "Array",
    "generic_types": ["int"]
  ]},
},

In essence, a type could then either be a string "Ty" or a JSON document { "type": "Ty", generic_types: [...] }.
It's also possible to rename generic_types to generic_args.

[raulsntos]

Are we sure Array is the only class that will have generics? What about Dictionary, or possibly even user-defined classes in the future?

My idea was other classes would use a different JSON field, so for dictionaries it could be:

"return_value": {
	"type": "Dictionary",
	"dictionary_key_type": "String",
	"dictionary_value_type": "int"
},

The type field value would let you know which optional fields you can check for, and if the generic type is Variant (unspecified) then the field would not appear in the JSON. Something like:

auto return_type = read_json_return_type();

if (return_type["type"] == "Array") {
	String array_type = "Variant";
	if (return_type.has("array_type")) {
		array_type = return_type["array_type"];
	}
	// Do something with the Array type
} else if (return_type["type"] == "Dictionary") {
	String key_type = "Variant";
	String value_type = "Variant";
	if (return_type.has("dictionary_key_type")) {
		key_type = return_type["dictionary_key_type"];
	}
	if (return_type.has("dictionary_value_type")) {
		value_type = return_type["dictionary_value_type"];
	}
	// Do something with the Dictionary type
}

Also, are nested annotations allowed, e.g. Array<Array<int>>?

I don't think they are supported in ClassDB at all because I'm using the PROPERTY_HINT_ARRAY_TYPE which I don't think supports nested generics and I don't think there's any API in Godot right now that uses nested generics.

---

I can see how this won't scale if we have more generic classes than Array and Dictionary but I don't really think any other generic classes would be added to the engine anytime soon and user-defined generic classes is also unlikely to happen in my opinion (not that I don't want it to happen because I love generics but it just doesn't seem likely, we don't even support method overloading in ClassDB).

I do like your idea better and kinda wanted to implement it like that at the beginning since it's more similar to how generic parameters work in C# reflection APIs but I thought it'd be better to keep it simple.

[Bromeon]

The question is: do we anticipate that at some point, more classes may have generic parameters than just currently Array?

A generic_types or generic_args field holding a list would support both the current case and any future cases, and not need hardcoded field names such as dictionary_key_type. On the other hand, with hardcoded fields, the Variant means field absent works well as you mentioned.

Your deserialization example is procedural because C++ doesn't have JSON mapping frameworks. Languages that support some sort of reflection however, would likely use a declarative version, e.g. Rust:

// Note: Option is like std::optional, meaning value can be absent
// Vec is like std::vector

#[derive(Deserialize)]
struct ReturnType {
    type: String,
    generic_args: Option<Vec<String>>,
}

and with your proposal:

#[derive(Deserialize)]
struct ReturnType {
    type: String,
    array_type: Option<String>,
    dictionary_key_type: Option<String>,
    dictionary_value_type: Option<String>,
    ... // more for others?
}

I'm pretty sure C# offers something like that as well. I think both approaches are possible, let's maybe see what engine developers think about it.

[raulsntos]

@Bromeon I opened #64469 with your proposed design.

The C# bindings generator is implemented as a module so it can access ClassDB directly and won't be using extension_api.json so I don't really care which one gets merged, but if the alternative is more convenient to GDExtension users then we should prefer that one. I'll keep both open to see which one the @godotengine/gdextension team prefers.

[touilleMan]

regarding #64469 vs #54249, I'd rather have dedicated array_type/dictionary_key_type/dictionary_value_type than a generic_args.
This is because the first approach is more explicit which is really important considering there is basically no documentation for extension_api.json (appart from reading Godot source code ^^)

Generics support with variant is only informative (i.e. there is no guarantee the declared type will be respected at runtime). On top of that generic typing support for custom classes is a complex topic (see for instance how much energy Python community put into this). So I don't think support for deep nested generic types (Array[Array[Dictionary[String, int]]]) is coming any time soon, and when/if it come I don't see a real benefit for the extension_api.json (especially since there is the native_structures that already fill the need for typed parameters/return value in Godot standard api)

So I'd much rather take the YAGNI pragmatic approach there 😃

[akien-mga]

See also #65817.

[akien-mga]

I agree with following YAGNI here - there are no concrete plans for further generics support so we should keep it simple/explicit for what TypedArray provides (which is really still just an array of Variants with a type hint).

[akien-mga]

Needs rebase.

[raulsntos]

I'm pretty sure this is superseded by #65817.

This PR's goal was to add the type of the array's elements as a separate property so consumers of extension_api.json that don't support typed arrays can ignore the new property and avoids breaking compatibility with the old type property.

"return_value": {
	"type": "Array",
	"array_type": "Node"
}

With #65817 the type property for typed arrays was changed to include the type of the array's elements so now consumers must parse the type string.

"return_value": {
	"type": "typedarray::Node"
}

With this PR, parsing the type could be implemented with this pseudo-code:

auto return_type = read_json_return_type();

if (return_type["type"] == "Array") {
	String array_type = "Variant";
	if (return_type.has("array_type")) {
		array_type = return_type["array_type"];
	}
	// Do something with the Array type
}

With #65817, consumers would need something like this instead:

auto return_type = read_json_return_type();

if (return_type["type"] == "Array") {
	String array_type = "Variant";
	// Do something with the Array type
} else if (return_type["type"].starts_with("typedarray::")) {
	String array_type = return_type["type"].substring("typedarray::".size());
	// Do something with the Array type
}

And existing code that only handles type == "Array" no longer works.

Since #65817 has already been merged, we already broke compatibility, so that ship has already sailed but it's fine since we're still in beta.

---

The only difference between this PR and #65817 is that I slice the hint string:

https://github.com/raulsntos/godot/blob/df2698dceee29486eb127e61b871c5e268433fef/core/extension/extension_api_dump.cpp#L73

As opposed to:

godot/core/extension/extension_api_dump.cpp

Line 59 in a8a8819

return String("typedarray::") + p_info.hint_string;

I do this because in some cases it uses the syntax subType/subTypeHint:nextSubtype (see #64249 (comment)) and it seems we only care about nextSubtype. I've only seen this syntax documented in EditorPropertyArray:

godot/editor/editor_properties_array_dict.cpp

Lines 593 to 594 in df2698d

	// The format of p_hint_string is:
	// subType/subTypeHint:nextSubtype ... etc.

An example of a property that uses this syntax:

godot/scene/resources/font.cpp

Line 1019 in a8a8819

ADD_PROPERTY(PropertyInfo(Variant::ARRAY, "fallbacks", PROPERTY_HINT_ARRAY_TYPE, vformat("%s/%s:%s", Variant::OBJECT, PROPERTY_HINT_RESOURCE_TYPE, "Font"), PROPERTY_USAGE_STORAGE), "set_fallbacks", "get_fallbacks");

Which currently results in this JSON:

{
	"type": "typedarray::24/17:Font"
}

[akien-mga]

We could rediscuss this in next week's GDExtension meeting and see if we're all happy with the merged option or if we want to change it further. I think it's still OK to break compatibility if we need to to get a solid base for 4.x.

[Bromeon]

Glad to hear it's implemented 🙂

I'm not too convinced the :: approach is the best one (in general, not just here), because it's essentially a poor-man's reinvention of JSON fields -- another undocumented protocol inside JSON that requires consumers to manually parse strings. It probably works as long as all parts are alphanumeric, but if ever double-colons enter the value space, it will cause problems.

If there were a small spec of this internal protocol, things would already be quite a bit easier.

[raulsntos]

Closing for now:

• The changes in this PR break compat and we'd rather not do that unnecessarily.
• Generic typed arrays are already supported, although it uses a different format (see [GDExtension] Implement support for typed arrays. #65817). So this PR doesn't allow anything that isn't already possible.
• If/when we need to add additional information to the API dump, we can re-asses the format.
• The current format (typedarray::Node) needs to be documented.