C++ Game module

[AndreaCatania]

Describe the project you are working on:
I'm working on a game that needs some specific functionalities coded in C++.

Describe the problem or limitation you are having in your project:
In Godot we already have different ways to introduce extra features coded in C++ (or other languages).
However, all of these come with the disadvantages:

• Complicate build mechanism.
• Difficult debugging during segfaults of these modules.

Due to these complications, usually the best approach is to make a Godot fork, and add the features on top of it.

While this is a good practice, it has the following disadvantages:

• Jump between Godot versions is not immediate and always require work.
• The game logic code is directly inside the engine and you have to maintain the entire engine code per each project.
• It's not easy and however not clean, to share the same code between two game projects.

When the fork solution is the best choice, these disadvantage make it not ergonomic.

Describe the feature / enhancement and how it helps to overcome the problem or limitation:
To improve these, we may introduce the game_module build argument. It can be used to specify an external path that Godot will build all together with the engine.

In this way, you have all the advantages that the Godot fork has plus is possible to decouple the engine code with the game code, allowing:

• Easier building mechanism.
• Easier debugging.
• Easier feature sharing between projects.
• Easier Godot version change.
• It's possible to use 1 Godot engine source for all the projects (and since the engine core building is cached, when you change project you don't need a clean build, saving a lot of time).

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:
To use it you will need:

• To specify the path in the build arguments, like: target=debug platform=x11 -j8 game_module=/home/andrea/Workspace/MyGameModule.
• The game module is similar to any Godot module, you can check it here: https://github.com/AndreaCatania/MyGameModule

Note: Here the integration of this feature: godotengine/godot#36883
Node 2: Here a much better integration of this proposal: godotengine/godot#36922

If this enhancement will not be used often, can it be worked around with a few lines of script?:
It can't be work around using scripts.

Is there a reason why this should be core and not an add-on in the asset library?:
It's a change done on the Godot build configuration.

[AndreaCatania]

Here the integration of this feature: godotengine/godot#36883

[Xrayez]

Related proposals:

• Ways to link C++ modules to Godot: Allow putting modules that compile into the engine in a project's folder godot#31457.
• Godot as part of a larger project: Proposal: Building Godot as part of a larger project, changes to Build System godot#27842.
• Built-in per-project C++ experience: Built-in GDNative compilation godot#11080.
• My first attempts at decoupling engine and modules: Ignore Godot Engine source from Godot projects by default godot#28343.

[Xrayez]

When the fork solution is the best choice, these disadvantage make it not ergonomic.

I agree, I try hard not to maintain my own fork, so far C++ modules has provided most needed functionality. But I have to say that I went on the darkside a bit and come up with an auto-patching system to alleviate some small engine limitations (like hardcoded constants). When you update engine source, the patches may fail to apply, so they have to be updated too, but usually it's trivial to maintain a pack of small bug/enhancement patches. The patches are simple git diffs which are versioned along the project, and it may look like this:

diff --git a/core/image.h b/core/image.h
index f29a30cda0..59ff19f28a 100644
--- a/core/image.h
+++ b/core/image.h
@@ -59,8 +59,8 @@ public:
 	static SaveEXRFunc save_exr_func;
 
 	enum {
-		MAX_WIDTH = 16384, // force a limit somehow
-		MAX_HEIGHT = 16384 // force a limit somehow
+		MAX_WIDTH = 32768, // force a limit somehow
+		MAX_HEIGHT = 32768 // force a limit somehow
 	};
 
 	enum Format {

I use some Python script wrappers over scons/git to make this even easier:

def apply_patch(self, patch_filename):
    patch_filepath = os.path.join(self.patches_abs_path, patch_filename)
    try:
        subprocess.run(['git', 'apply', '-3', patch_filepath], cwd=self.engine_abs_path).check_returncode()
    except:
        raise self.ApplyPatchError("failed to apply patch: " + patch_filename)
        
        
def generate_patch(self, patch_filename):
    patch_filepath = os.path.join(self.patches_abs_path, patch_filename)
    # Diff between the latest commit and the working tree
    subprocess.run(['git', '-C', self.engine_abs_path, 'diff', 'HEAD', '>', patch_filepath], shell=True)

[Xrayez]

To improve these, we may introduce the game_module build argument. It can be used to specify an external path that Godot will build all together with the engine.

Regarding game_module build argument, I'm not entirely sure of the added benefit, the only thing this seems to achieve is being able to link a single C++ module using scons alone, and the rest is just plain C++ modules.

I usually have more than one module in a project which I'm currently symlinking with the engine source. Both modules and the engine are git submodules in my project. Saying that it would be better to provide a way to point to several modules to make this more flexible. But I get the idea of a "dedicated game module" which acts as a major project's extension, so to speak.

It can't be work around using scripts.

It can but I have to say that it's really difficult to setup, certainly not a few lines.

[AndreaCatania]

The idea of the game_module argument is to specify a directory that is build all together with the engine, without enforcing any specific workflow to the developer.

By doing so, you can compose your game directory as you wish:

• A collection of extra nodes.
• A collection of other modules.
• A collection of third-party software + modules.
• Binding to other language code.

My idea is to have something similar to what UE4 does.
There, you can download the engine from their launcher (any version), then you specify your game directory (where the c++ code is) and the launcher build the engine with the game code.

To change engine version you just need download a new engine version from the editor and build again with the game code.

In ue4 fork the engine or use the above work-flow is the same thing. Thanks to this feature, fork the engine is discouraged and a better game and engine separation is achieved.

The change that I'm proposing allow the same concept.

[Xrayez]

I haven't used UE4 so can't contribute on that topic unfortunately.

A collection of other modules.

So to clarify, does that mean I must define my own SCsub/Sconstruct and list them all there? Sounds good. But I imagine the process of "collecting modules" would have to be all replicated the same way as it's done in Godot's root Sconstruct. What about custom module documentation/icons? The engine would have to know about those things somehow as well, else a game_module would be a blackbox to Godot, unless that's the point of this proposal.

In that case, game_module seems to be somewhat misleading here. To me it seems like what you propose goes towards an enhanced and simplified way of developing project-specific features using C++ which just happens to use the existing modules mechanism (modules/SCsub).

What if we could instead provide a dedicated way for that? The argument would look like this:

scons tools=yes target=release_debug bits=64 project_path=/home/Xrayez/my_godot_project

(assuming my_godot_project contains project.godot, but you can point it to any path, too)

Given the project_path, the engine can then optionally run godot/project/SCsub which will use the supplied project_path to build self-contained, project-specific C++ stuff, again all externally:

This just avoids treating this as a module rather than project-specific thing, as what you propose is not necessarily "external module linking" but actually "external project/package linking" for building C++ code which can contain more than a single module, that's where my confusion came from.

[AndreaCatania]

Yes you are right, what I did until now is not enough - icons are not supported and support modules will simplify things.

However, my idea is to see what the community response is before doing anything more sophisticated; and from what I see I'm not the only who need this feature.

Regarding path to project: This may make things a lot more complicate to integrate and I don't see a real benefits over integrating it using the scons config. Consider that scons config are good enough to allow make it a lot more easy to use.

[reduz]

IMO this is just workarounds to gdnative not offering enough, I prefer we fix GDNative.

[AndreaCatania]

The problem that this feature is trying to solve are caused by some fundamental GDNative concept, that I don't think are solvable without altering the nature of GDNative.

For example, GDNative allows hot swap thanks to the use of shared objects. Using SO adds complexity in:

• Usability (see the GDN layer)
• Compiling
• Debugging

When you don't need such versatility, and you just need to use the godot core classes, GDN is never the best solution.

So we can't really talk about GDNative fixes, because these are trade-off to achieve such versatility.

[jknightdoeswork]

Can you elaborate on each of the 3 points: Usability, Compiling and Debugging?

Compiling - Are you referring to initial setup? Development iteration time?

Debugging - Can you fill me in on what the problem is?

I see 2 core issues in this thread: "complicated build system" and "debugging during segfaults of modules". Both of these are very hard problems. The build system is very good, in my opinion. scons pretty much just works. Maybe we can fix the "debugging during segfaults" problem?

[reduz]

• Usability (see the GDN layer): Technically we should be able to make it perfectly possible to do anything you do accessing core classes by accessing gdnative. It may not be possible now, but it should definitely be doable.
• Compiling: This is not really a problem, the main issue with GDNative is that we don't have nice presets that you can use to download and compile them, but again, nothing should avoid us from fixing this.
• Debugging Normally Godot will not crash within Godot, and if this crashes within your DLL/so, you whould be able to see and debug this fine.

All these problems are solvable in GDNative, we should work on fixing this so we can start asking people to not do modules unless they really need to.

[AndreaCatania]

Usability (see the GDN layer): Technically we should be able to make it perfectly possible to do anything you do accessing core classes by accessing gdnative. It may not be possible now, but it should definitely be doable.

You can't access core classes using a shared object, you need a shared interface to do it. So this doesn't seems doable.

Compiling: This is not really a problem, the main issue with GDNative is that we don't have nice presets that you can use to download and compile them, but again, nothing should avoid us from fixing this.

The preset, solves the problem when you just need to setup the project. But what about the build complexity? Sometimes the entire pipeline is more complex than the bare godot, and you may prefer to not add complexity without a specific reason.

Debugging Normally Godot will not crash within Godot, and if this crashes within your DLL/so, you whould be able to see and debug this fine.

The reality is that it's not the pleasant work with shared objects, and I would prefer use it when I really need it.

All these problems are solvable in GDNative, we should work on fixing this so we can start asking people to not do modules unless they really need to.

But what about good godot nodes workflow then?

Be able to add new nodes is the key concept here... Let's say we want to support a new physics engine NVidia Flex or https://fmod.com/ to integrate it I prefer add new nodes rather doing it using scripts...

[AndreaCatania]

By the way, this was not meant to be a GDNative replacement, rather it is meant to be an extra available tool in the developer hand.

[Xrayez]

Yeah, there are other core stuff like implementing your own custom Godot servers which have to be currently done via modules AFAIK.

I want to believe that GDNative can be improved/rewritten to the point to be on par with C++ modules development. But to reiterate what @AndreaCatania said, I'm not sure whether any kind of implementation could provide as much freedom, tight integration and stability as dealing with the Godot source code, including circumventing possible limitations which can't be solved by GDNative alone (because, well, you need to recompile the actual engine to make such changes).

So this proposal has more practical aspect rather than theoretical basis. See other keyword in this proposal: "fork". So there has to be a good reason for doing so, and lets be honest that Godot cannot solve all possible use cases and needs, so it would be good to at least make the existing workflow easier and more flexible without reinventing anything.

There's another aspect in that, everything is implemented in core first. I think some people prefer to use the latest features. I'm not sure how GDNative can keep up with such rapid development changes.

But yeah the proposal is not about GDNative replacement, it has its own uses (language bindings etc).

[reduz]

@Xrayez, @AndreaCatania: again, the "accessing core stuff" is not an argument here. If there is an use case, access can be added in GDNative. You can' t really add most server types as modules anyway because they need to be instantiated from the main loop. Adding new types can also be done with GDNative, and it can be improved usability wise.

For the rest, I really think it becomes a much more corner use cases, which does not make sense for having an entire feature for, given the idea is to discourage the use of modules as much as possible towards the future.

[Xrayez]

Maybe the "game module" name does not actually reflect the original proposal which lead the discussion the wrong way (which I suppose was inspired by UE4 terminology, and btw they mention both pros and cons similarly). So lets talk about in terms of "C++ packages/extensions" rather than "C++ modules" perhaps.

Again, I'll quote @AndreaCatania:

By doing so, you can compose your game directory as you wish:

A collection of extra nodes.
A collection of other modules. [emphasis mine]
A collection of third-party software + modules.
Binding to other language code.

The most intriguing selling point of this proposal which I'm able recognize is that you can compile an entire hierarchy of independent SCsub units with this. A regular module cannot do this currently AFAIK, or the actual usages will be limited in comparison.

More importantly, with implementation similar to godotengine/godot#36883 it's possible to feed the absolute/relative paths to such C++ package (it's important to have versioned and associated engine+modules/package configuration to have reproducible builds).

In theory you could develop your own C++ engine/framework with this, building upon other core classes. You could then publish your C++ package and name it "Godot Extended". In a way, such a package can be seen as a general purpose C++ extension which can be tailored towards specific type of genres written in C++ for performance reasons. Another way of thinking about this is "Godot C++ frameworks" which can be written by independent developers.

So that's why I kinda insisted on separating the two workflows: C++ module development and C++ package development. Each will have its own set of callbacks for preregistering, registering, and unregistering types (lets call them presetup, setup, and teardown), to avoid further confusion for developers.

[AndreaCatania]

@reduz This feature is not about give the possibility to interact at such deepness in Godot, that due to its limitations is not even possible to do from a module. But this is not the argument here, so let's put the focus on this proposal:

This proposal and GDNative are using two technologies, that are available on all most any system programming language, these provides the same final result in different approaches.
Each approach has its own benefits and features depending the use case.

• Static Library -> This proposal
• Shared Object / DLL -> GDNative

This to say that one is not mutually exclusive for the other.

---

To integrate this proposal I've added 103 line of code:

and with @Xrayez proposted features at max I will add another 100 line of rows that will never need to be touched again.

This proposal will result in an addition that is by really far less complex than GDNative, and will not need an heavy development support in future. This is a fundamental parameter to consider in the decision process.

[groud]

I think what this PR implements is a good solution until we find a way to make Gdnative easier to use. The fact it does not need a lot of code make it worth IMHO.

[Xrayez]

@reduz:

For the rest, I really think it becomes a much more corner use cases, which does not make sense for having an entire feature for

See my interpretation and implementation for this which doesn't touch the core, nor introduces new concepts, just makes the modules development process more flexible: godotengine/godot#36922.

[Xrayez]

First of all, thanks @AndreaCatania for preferring godotengine/godot#36922 to implement this proposal. As of now, I'd consider this to be feature-complete and ready for review.

But I'd like to build upon the ideas introduced here and share my insights while working on the feature, at least on the conceptual level.

We've previously talked about C++ packages as something which a single module can provide (initial proposal). But now I see a C++ package as a collection of C++ modules instead. In a way, all the built-in modules can be considered as a single core C++ package. Imagine if godot/modules could be renamed to godot/package. I'm not saying that we should do this, but that's just to convey the idea.

Now talking in terms of packages (modules == package), godotengine/godot#36922 provides a way to point to a single external C++ package via custom_modules build option (project game package). What I'm thinking would be even more powerful is having ability to compile several packages.

So, instead of custom_modules, you could define a list of packages as:

scons platform=x11 tools=yes target=release_debug packages="../physics/modules;../ai/modules/"

In a way, this is very similar to PATH environment variable, but for Godot modules. That's why initially the option had the name modules_search_path, so that the logic could be extended to support several search paths to compile modules at different locations, giving pretty much the ultimate freedom for C++ modules development (not GDNative development, it has its own uses). 🙂

EDIT: this was trivial to add, it doesn't necessarily has the notion of "packages", but it does provide a way to search modules at multiple paths: godotengine/godot#36922 (comment)

---

As I've shown how it's possible to (re)move all built-in modules to be compiled as part of Godot externally as in godotengine/godot#36922 (comment), this gives quite a bunch of opportunities to consider:

1. Move all non-essential modules out of core and ask people to maintain their own C++ packages.

2. Create default godot-modules repository as a git submodule (likely won't happen due to project's structure philosophy, and requires some more modifications to the build system).

Just saying about possible positive outcomes which can help to prevent engine bloat, which seems to be one of the most important aspects of Godot's development philosophy.

[starry-abyss]

Useful to me! Currently as a workaround I added make install to my Qt Creator project which "deploys" sources of the module to Godot directory, and made it run after each build.

[ghost]

don't forget to add some documentation for it, otherwise it's unusable by 99% of the userbase