Enable scripts to generate XML API documentation for the editor's GUI

[willnationsdev]

Describe the project you are working on:
Creating plugins for Godot.

Describe the problem or limitation you are having in your project:

Let's say I define an addon with a collection of script classes. If someone else installs my addon, they have no way to easily determine what the intended use/API is for those scripts. I can't state what the purpose of my script is for, which methods are public versus protected/private, which methods are static, which methods are virtual (and intended to be overridden). I can't specify in which cases a method or class should be used, how constants and signals relate to the API, etc.

The only way for someone to glean any of this information is to go to the script itself and read the source code which is antithetical to the purpose of building an addon for others to use. They shouldn't need to know the details of how the addon works; they only need to know what they can use the addon for and how to use it.

We should provide a mechanism by which people can generate in-editor documentation for script classes and be able to rebuild this documentation on command / via a recurring process / something-or-other.

Describe how this feature / enhancement will help you overcome this problem or limitation:

And God said, "Let there be script documentation," and there was script documentation.

Integrating user-defined script documentation would be the most direct, clean, and efficient means of solving the problem. The editor already has built-in support for this with engine types. There is just no code at the moment that actively searches for and integrates user-defined documentation data and rebuilds the in-editor docs cache as scripts change, nor is there anything that generates that information.

Show a mock up screenshots/video or a flow diagram explaining how your proposal will work:

Add a button to rebuild documentation manually in the ScriptEditor.

The information above would all be filled in with our own custom script class's name, properties, methods, etc.

Since private/public data concepts are not built into GDScript and Godot's scripting except through naming conventions, we could set things so that only items without leading underscores appear in the documentation.

Once, later on, the scripting API supports recognizing annotations for scripts in general, and GDScript gets annotation support, then we can start to directly mark things as private/public, virtual, etc.

Would one day like to be able to do something like this:

extends Node
class_name MyNode

# Despite method having a leading underscore, this specially marks that
# the method should be exposed to API documentation.
@public
@desc Summary about how this does a thing.
    More information on it doing that thing.
    This is injected straight into the XML, i.e. you can use Godot's XML doc features:
    Now I'm referencing a Godot class: [Node].
    Default implementation can be found in [method MyNode._override_me].
    Here's an example of how you should override it...
    [codeblock]
    extends MyNode
    @override # indented @-signs don't impact outer annotation
    func _override_me() -> int:
        return 1
    [/codeblock]
@virtual
func _override_me() -> int:
    return 0

But none of the docs stuff is 100% dependent on the annotations. It just helps fill in information. A lot of the boilerplate functionality can be achieved without annotations thanks to the inclusion of scripting API reflection information (godotengine/godot#31021)

Describe implementation detail for your proposal (in code), if possible:

1. Cache an original copy of the engine's DocData information.
2. Write a method in the Editor's DocData or other docs-related, centralized API, to fetch a collection of XML files from all existing addons directories (res://addons/*/docs/*.xml for example) or sub-projects (whenever godotengine/godot#19178 is implemented). It would then build the content of those files into a DocData instance and merge it into the editor's cached DocData instance in order to update and replace the main DocData instance. The main one will then automatically cause Godot's ScriptEditor GUI to update with the correct script information.
3. Write a method to generate boilerplate XML files from one or more scripts.
   • If annotation support is in the engine (Annotations in GDScript godot#20318), then use that information to help fill out the content inside the XML data.
4. Whenever script classes are regenerated, we would also want to regenerate XML docs in a background thread.

If this enhancement will not be used often, can it be worked around with a few lines of script?:

It would be used extremely often. The entire addon script ecosystem is hamstrung by the lack of this feature. We need an integrated solution. It would be a decently large amount of script work to build the tools necessary to do all of this.

Is there a reason why this should be core and not an add-on in the asset library?:

Honestly, with the exception of the annotation stuff (cherry-on-top material) and the DocData merging/searching code (which isn't exposed to scripts yet), a lot of this can be done from the scripting API with an EditorPlugin now that script reflection info is exposed to scripts. But even if we fixed the basic problems of exposing the right stuff, a user shouldn't need to install a plugin that then enables them to generate documentation on how to use a plugin. That would be ridiculous.

Other Related Issues:

• Allow class_name to also add a description godot#23047
• Add to In-Engine Scripting API Documentation with a Plugin godot#10635

[golddotasksquestions]

Integrated API documentation for Plugins and Addons is very much needed. Without it, pushing features out of the engine to the Asset Library is completely pointless imho.

[Calinou]

@luizcarlos1405 Please don't bump issues without contributing significant new information; use the 👍 reaction button on the first post instead.

[akien-mga]

For the reference, there's a new tool by @NathanLovato et al. which solves part of this proposal: https://github.com/GDQuest/gdscript-docs-maker

I haven't tried it nor reviewed this proposal in depth to compare, so I don't know if it's sufficient to consider this solved, or if a built-in solution would still be needed.

[NathanLovato]

The feature is already partly there, inside Godot: the language server builds the docs as JSON, gathering your information about GDScript code and your docstrings. It's just not used by Godot, but e.g. the VSCode plugin can display docs and signature from your GDScript code through the language server.

The tool we wrote writes your code reference as markdown docs, like this: https://www.gdquest.com/docs/godot-steering-ai-framework/reference/classes/gsaikinematicbody2dagent/. It could output xml or any other format without too much trouble, as it's in Python, and built to be extensible. But that wouldn't be a built-in solution.

So right now at least, you can use a python program (or one of our shell scripts that does the job for you) to get a reference as markdown docs. This means you can put the docs on the web, make them available locally, and use e.g. a release hook to re-generate the docs automatically.

Then, if people are fine with an external solution to generate the addon docs, we would just need a way to load extra XML files or something into the engine to register more classes.

[NathanLovato]

Note that I get the point of having it built-in, especially as we build tools ourselves. But in a context where all devs are busy on the engine, and where we're few add-on developers, I would suggest going for a solution that doesn't require too much work and maintenance on the engine's side.

Python is especially efficient when it comes to writing and maintaining those kinds of tools, compared to having the feature in the engine. If someone finds a bug, we can make a fix and a release a new version in no time. Python is also a popular language for games' asset pipelines and developer tooling so it'd be contributor-friendly.

And for the few of us who make and distribute tools that need a code reference (rather than a manual), it shouldn't be too bothersome to use python or a shell command (we can make a detailed guide if needed).

[NathanLovato]

Another note, before I forget: it might be worth asking @Geequlim for his thoughts on the matter, as in the VSCode extension, he added support for showing docs from the built-in classes and your godot project, and I believe it's all markdown.

For now, iirc every editor plugin or client is responsible for parsing the json and converting it, generally into markdown. But the language server's features aren't really documented, so there might be more to it than I'm aware of.

[aaronfranke]

Just for reference, here's how C# does inline XML documentation:

    /// <summary>
    /// Called once per frame.
    /// </summary>
    /// <param name="delta">Time passed since last frame.</param>
    public override void _Process(float delta)
    {

The documentation shows up in most IDEs when you hover over the method name, and also when the method shows up in autocomplete, it looks like this:

When you type ///, most IDEs automatically create the XML structure for you, it looks like this:

    /// <summary>
    /// 
    /// </summary>
    /// <param name="delta"></param>

Then, you just have to fill it in with text. GDScript could have a similar format and a macro for auto-creating a documentation template, perhaps if users type ### or similar?

[NathanLovato]

I'm not too fond of xml in docstrings as without snippets or vim it's longer to write. That's in GDScript. If xml is standard in C#, then C# devs should probably go with xml docstrings.

For comparison, using xml vs jsdoc style:

# <summary>
# Called once per frame.
# </summary>
# <param name="delta">Time passed since last frame.</param>
func _process(delta):

jsdoc style (javascript):

# Called once per frame.
# @param delta - Time passed since last frame.
func _process(delta):

Python and lisp have it even simpler: just a docstring, and it works pretty well.

In any case, there'll be some parsing and conversion, be it in text editor plugins, or to add in the language server, as it outputs everything as json and plain strings for docstrings.

Note that the language server does extract type information, function signature, etc. I have to double-check but iirc it can extract docstrings for individual enum members as well. We saw something like that when François was fixing a bug in the parser/server.

At the end of the day, I'm fine with any style as long as we can have some standard for GDScript. I'd personally lean towards having the least markup possible, so that the docstrings read well even without conversion.

[willnationsdev]

@NathanLovato I agree with keeping the syntax lightweight for generating stuff. As for how the generation is done though, I think it's fine to have an external Python solution while still maintaining plans in the future to introduce a built-in solution. Otherwise, the integration with the editor's existing search functionalities and lookups are missing. That can really drag down a user's experience with using addons.

And while addons may not be especially prolific right now, I expect that 4.0's release and the revision of the Asset Library will largely change that. The more we reduce barriers for people to publish, maintain, and discover addons, the more quickly people's use of addon content will improve. Having addon documentation integrated with editor documentation will become increasingly important.

Edit: And as far as I'm concerned, the content of this proposal is targeted at the latter built-in solution, not an external Python one.

[NathanLovato]

it's fine to have an external Python solution while still maintaining plans in the future to introduce a built-in solution.

Sure, that's the idea. And we also made it for online docs generation or distributing documents otherwise. E.g. right now we have a company that's interested in having an online reference. Also, imagine if someday you have some official Godot addons (like you have for Krita, blender), while ideally you want the docs to be built-in, you also have people who'll browse them online, use search engines...

The more we reduce barriers for people to publish, maintain, and discover addons, the more quickly people's use of addon content will improve.

Definitely.

Could we settle on some markup, until someone gets to implement this proposal? We'll use it in gdscript docs maker.

Right now, we have it as simple as possible, I followed the docstrings in the XML reference in Godot's repo:

• You can link to classes and methods writing [ClassName], [method_or_property] (searches in current class), and [ClassName.method_or_property]
• You can add metadata to anything writing key: value[, value2, value3] in your doc comments. This is YAML-style. For example, we support keywords like tags: virtual, or tags: abstract (for abstract classes), but also keywords like deprecated. I don't mind changing for anything else, we just went with something simple.

[akien-mga]

@ThakeeNathees is working on a documentation system for GDScript as part of GSoC 2020, mentored by @vnen, which will rely on the annotations support in GDScript 2.0.

See also godotengine/godot#4370 which was the earlier proposal about this on the main engine repo.

@ThakeeNathees please do not hesitate to share details about what you plan to implement here, or if better suited, in a new proposal (I'm not sure your documentation system would rely on XML like the core engine docs as suggested here).

[ThakeeNathees]

@akien-mga Ok, I'll open a proposal soon on documentation generation about what I've planed to implement and ideas, for suggestion and discussion.

[willnationsdev]

@akien-mga @vnen Will the annotation support in GDScript be exposed to the editor via methods in the core Script interface? That is, could one conceivably define and extract metadata of this sort from any script, such that all scripting languages could generate documentation for their code?

[Mantissa-23]

Currently, classes, functions and properties get docstrings when you write a comment above them:

# Return traverse angle of a vector in local space
static func calculate_traverse(pos : Vector3) -> float:
	return wrap_angle(atan2(pos.x, pos.z))

Will show up as

Both in LSP popups and in the built-in script editor popups

Even if we're seeking a more complicated markup language so well-formatted documentation can be generated, I see no reason these docstrings shouldn't be forwarded to editor tooltips in the meantime.

[Calinou]

This is now implemented for GDScript in the master branch by godotengine/godot#41095.

As for other languages, it may be better to wait for the new GDNative extension system to be finalized and see what can be done with it.

[YuriSizov]

This should be closed, as this feature has been implemented in master (and also improved recently).