Implement a transpiler for GDScript to C++

[nonunknown]

Describe the project you are working on

None atm

Describe the problem or limitation you are having in your project

PS: This is more a tracker than a proposal (its already being implemented). The reason for this proposal is because:

• Clarify for people out there how GDPP will work
• Find people willing to help on the project

GDScript is a very pratical language, but for some things like use of external libraries or intesive operations it can be a problem.

IMPORTANT

My intention is not to ignore/hate/kill the GDScript from Godot, but instead empower the language so we can have the best of both worlds.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

What if we can use gdscript with the performance and built-in types from c++ this way we can mix toguether the good and the best!

The idea came from the cumbersome process of using GDNative, where the end-user needs a lot of knowledge of building projects and also attach them to the project! Yes I know, Godot 4.0 will not have it anymore, but instead will have GDExtension which I dont know nothing about yet! (if someone could explain in the comments would be nice) But keep in mind that:

• GDPP will be Godot4+ it will not be available for 3.x and older ones!
• GDPP will be a standalone tool (a full featured IDE specialized in GDExtension)

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

• GDPP or "GDScript++" is a programming language heavily inspired by GDscript2.0
• This language compiles/output to C++ in conjunction with GDExtension library
• It is statically/strong typed (types always must be declared)
• Contains built-in types from C++ (shown below)

Some examples of GDPP:

ifndef MY_CLASS_C
define MY_CLASS_C #defines are very useful for preprocessing like a debug only feature

#class_name and extends is required for every file

class_name Myclass
extends RefCounted 

# some basic types declaration, must be always strong typed
var value:int = 10
var text:String = "Test"

# Some types that came from c++

var my_map:map[String,int] = [ [ "hello", 1 ], [ "world", 2 ] ]

var my_vector:vector[ float ] = [10.2, 20.44] # you cant use "20" as it was a integer, it must be a float like "20."

var my_char:char = 'a'

var my_tuple:(String,char) = ("Hi",'a')

#Private and Public concept exists in cpp, and to follow gdscript pattern you use a '_' before the name
var _my_private:String = "Hey I'm a private string"

#function declaration

func say(msg:String):
    print("Hello World")

#The new `unsafe` keyword:

unsafe: #Everthing you type here will be c++ pure code so you can have control over low level stuf
    #include <vector>
    std::vector<std::String> v = {"Hey sup!"};
    void a_method(int a)
    {
        std::cout << v[0] << std::endl;
    }


endif #MY_CLASS_C

The code above will be converted to the following

Myclas.hpp

#ifndef MY_CLASS_C
#define MY_CLASS_C

#include <godot.h>
#include <RefCounted.h>
#include <vector>
#include <map>
#include <tuple>
#include <string>
class Myclass : RefCounted
{
    //Keep in mind that comments are ignored so they will be mantained here just for readablity, but in gdpp itself it will not be transpiled
    public:
      // some basic types declaration, must be always strong typed
      int value = 10;
      godot::String text = "Test";

      // Some types that came from c++
      std::map<godot::String,int> my_map = { { "hello",1 }, {"world, 2"} };

      std::vector<float> my_vector = { 10.2, 20.44 };

      char my_char = 'a';
      
      std::tuple<godot::String,char> = { "Hi", 'a' };
    
    //Private and Public concept exists in cpp, and to follow gdscript pattern you use a '_' before the name
  private:
      godot::String _my_private = "Hey I'm a private string";
    
    #function declaration
    
    void say(godot::String msg)
      {
           godot::print("Hello World");
      }

        std::vector<std::string> v = {"Hey sup!"};
        void a_method(int a)
        {
            std::cout << v[0] << std::endl;
        }
}
#endif //MY_CLASS_C

Evey other thing that I didnt mentioned in the example will be exacly as GDScript itself

Project Page

The proof of concept can be found here: https://github.com/nonunknown/gdscript-pp

The link above is the old implementation for 3.x, it shows that yes, this is possible! Even its not on a usable state it can be tested to see the output, the thing is that this one read the source line by line and its not a proper language

The actual project can be found here: https://github.com/nonunknown/gdpp-standalone

It is C++17 only, and it is being implemented as actually a new programming language, at babysteps.
It already contains a tokenizer, and the parser is being done.

FAQ

Q: Why remake everything instead of just modifying godot's source?
A: I'm learning a lot doing things from ZERO (like the GDScript Tokenizer) I learned a lot and want that for the project and eveyone involved

Q: What is a Transpiler
A: A Transpiler is program that converts source to source, instead like GDScript which converts source to bytecode

Q: This will make GDScript obsolete?
A: Nope, In any way, you will be able to use Both if you want, and use GDPP to fast prototype C++ stuff altough you can use only it!

Thanks to

This project is only possible today, because a lot of people gave me support/materials for that

@Calinou
@vnen
@willnationsdev

If this enhancement will not be used often, can it be worked around with a few lines of script?

it will be in every possible project (if the user needs or simply wants to) Where the user needs very performant code.

Is there a reason why this should be core and not an add-on in the asset library?

This atm will not be core! (maybe on the future when the project is mature)

[Ansraer]

Hmmm, you definetly have my attention with this project.
Have you considered using pragma once instead of the define include guard? I am aware that it is not standard (yet) but by now it is universaly supported and recommended by most organisations. Plus, given that most users of this will be more familiar with gdscript than the intricacies of c++ one single line at the top is probably easier to remember than needing multiple lines at different locations.
I also noticed that you use several containers from stl. Not sure how many of them are actually exposed to gdscript and gdnative, but godots builtin containers are probably preferable, especially given how reduz is currently reworking and modernizing them.

[Xrayez]

In my opinion, the best results can be achieved by directly dealing with GDScript's parser tree/AST, especially when it comes to maintenance. I've also done some experimental stuff using this kind of approach at Goost, see https://github.com/goostengine/goost/tree/gd3/modules/gdscript_transpiler (not in official builds).

The implementation is probably quite naive and it's just proof of concept, but that's how the whole process of parsing could be simplified. This way, you don't have to re-implement GDScript tokenizer and parser yourself, but of course, this kind of approach requires compiling the engine from source. If this could be implemented directly in Godot, this would be the natural route to take.

One of the difficulties I see is being able to transpile coroutines (yield or await in 4.0) in C++, there should be likely a way to achieve this by looking how GDScript does this in C++ itself. But a code produced this way may not be necessarily readable anymore, but that's acceptable given that one of the major reasons why you'd want to transpile GDScript code to C++ is basically performance. In any case, having a tool which allows to automate the laborious process of porting GDScript to C++ would be very nice to have, even with suboptimal quality.

Another challenge/limitation is the requirement that all variables/functions must always have types explicitly defined. This might be easier to do from within C++ because GDScript parser may infer types implicitly from constants.

[nonunknown]

@Ansraer can you send me some material bout pragma once in modern programs?

@Xrayez This is very interesting, since your transpiler is already attached to the engine itself is something I really want to do when the project reaches the mature stage! I'll keep this in mind!

[Ansraer]

Best article on pragma once I could find on short notice: https://luckyresistor.me/2019/07/13/why-its-time-to-use-pragma-once/

[erayzesen]

What you're describing sounds great.

I'm asking out of curiosity; After converting the script you call gdpp to cpp with transpiler, will godot be able to compile it? If it can do that, can we already use cpp files (various libraries)? For example, I want to do image processing, there is a nice library written in cpp for this, it would be great to be able to add it.

[YuriSizov]

can we already use cpp files (various libraries)? For example, I want to do image processing, there is a nice library written in cpp for this, it would be great to be able to add it.

You can already achieve that by either making custom engine modules (requires a full engine recompilation with modules included, both for the editor and for the export templates), or by creating a GDNative plugin (using C/C++ binding, or any other available bindings).

[Galomortal47]

this is quite an interesting project, would that save the work from having to port GDnative or compiling the entire engine in case of modules? i really wanted an cross platform way of having C++ code in godot so i can better optimise for lower end hardware.

[nonunknown]

yeah in that case the transpiler will work only 4.0 onwards, which is not GDNative anymore but GDExtension, also you will basically type gdscript and it will output a .cpp and .h file and will be compiled automatically by the plugin! so just type/compile/done process!

[Malcolmnixon]

I've been thinking on similar lines - a transpiler from gdscript to GDExtension. I was thinking on the lines of an additional "Compiled native" option to the project exporter which would:

• Generate an application.gdextension file
• Generate C++ source code from all scripts in the application (with optional warnings for untyped/variant)
• Generate an SConstruct file
• Run scons to generate the GDExtension binary

This would allow all gdscript code to get the benefit of native execution speed - with the trade-off of a significant increase in build time, and the developer having to provide scons and compilers for all supported targets - a perfect scenario to leverage github actions.

Improving the set of available container types (Vector, Queue, Fifo, Set, PriorityQueue, FixedSizedFifo, etc.) would be additional improvements which gdscript (both interpreted and compiled) could benefit from.

[Xrayez]

Improving the set of available container types (Vector, Queue, Fifo, Set, PriorityQueue, FixedSizedFifo, etc.) would be additional improvements which gdscript (both interpreted and compiled) could benefit from.

I'd like to clarify that, from what you mentioned, only Vector, Set, List (stack/queue could be emulated with list, but those would suffer performance issues) are available in Godot natively. I've recently implemented PriorityQueue/Queue/Stack in Goost though: https://github.com/goostengine/goost/tree/gd3/core/types/templates 🙂

[kisg]

Instead of generating C++ source code, an alternative could be to generate LLVM IR directly (of course together with DWARF metadata to allow source level debugging), that can be then compiled either to native code or to WASM. The GDScript implementation in Godot 4 already has a (bytecode) generation interface, which could probably be used as a starting point.

Another option would be to compile GDScript code to WASM bytecode (See proposal #3370). WASM runtimes already support interpreted, JITed and AOT compiled execution modes, with the later case reaching near-native performance. The advantage of this option over the LLVM IR output option is that it does not add LLVM as a dependency to Godot. The WASM micro runtime is less than ~100KB: https://github.com/bytecodealliance/wasm-micro-runtime

In either case, this would work best with the implementation of #3369, in which case classes implemented with GDScript would always be standard GDExtension classes regardless whether they run in interpreted mode or in AOT compiled mode.

Full disclosure: I submitted the mentioned proposals.

[Zireael07]

@nonunknown Links in OP don't work (404)

Also, a vote for WASM target

[be-thomas]

@kisg llvm implementation would be pretty bloated.
The major reason people use Godot is because it lightweight & small.
Downloads quickly & doesn't get in your way of doing things.

If you just try to go ahead & download the LLVM source code it will be 20GBs+
Also C++ can be compiled to WASM anyways. So I expect people to get a far better experience with a C++ transpiler.

I had also suggested a very quick speed boost that can be gained if we transpile gdscript to lua behind the scene.
There is LuaJIT runtime which will hardly take up an additional half MB in native binaries.
However, LuaJIT performance is almost equal to .NET.
Plus, even with JIT mode off (i.e., interpreted mode), it is still the fastest scripting language there is to date approaching the likes of C#
The major advantage of this is that there won't be any issue with dynamic typing, since lua supports it already. No one needs to refactor their code. It will work with or without type information.
Moreover, if you provide some type information, the JIT compiler optimizes on the fly & code becomes even faster(through built-in ffi in LuaJIT).
Lastly, it does all this while consuming merely a fraction of the RAM that .NET takes. Because it was originally supposed to be an embedded language.

[Zireael07]

LuaJIT's big disadvantage is that it's a JIT, which means it won't be accepted on iOS, as Calinou pointed out in your original proposal.

[be-thomas]

@Zireael07 , as I have mentioned in the comment & my original proposal, LuaJIT with JIT off(interpreter mode) still blows every other scripting language by huge margins. Because the interpreter is hand optimized & the language is simple.
source: https://news.ycombinator.com/item?id=16949222

JIT can be turned off with a simple directive on those platforms which don't support it.
Also, some people seem to be living under a rock. iOS 14.2+ supports JIT officially.
source: https://9to5mac.com/2020/11/06/ios-14-2-brings-jit-compilation-support-which-enables-emulation-apps-at-full-performance/

[Calinou]

Also, some people seem to be living under a rock. iOS 14.2+ supports JIT officially.

The linked article says:

However, that doesn’t mean you’ll see emulation apps in the App Store, or even any other app using JIT. Testut told us that the current implementation works only for sideloaded apps, which are usually installed through Xcode and other developer tools instead of the App Store. In other words, this is a feature intended for developers with debugging purposes.

I've seen the sideloading procedure required for JIT-enabled apps and it's something… 😶
Not exactly something you'd want to use as a daily driver.

[be-thomas]

Sorry for being a little flashy with my words.

However, I request documenting the bytecode of GDscript.
If that is done, a lot of contributors can come forward, including myself.
I have searched a lot & the only thing I have as a reference is the source code itself, which are all a few thousands of lines at the very least.

Btw, LISP is a dynamic language but it can be compiled to machine code.
Also, haskell uses type inference & compiles to machine code too.

There are a lot of possibilities but people cannot do much because its hard to even get started because of lack of documentation.

[kisg]

@kisg llvm implementation would be pretty bloated. The major reason people use Godot is because it lightweight & small. Downloads quickly & doesn't get in your way of doing things.

If you just try to go ahead & download the LLVM source code it will be 20GBs+ Also C++ can be compiled to WASM anyways. So I expect people to get a far better experience with a C++ transpiler.

If you read my previous comment to the end, I suggested to emit WASM bytecode directly from the GDScript runtime, exactly to avoid the dependency on LLVM.
WASM bytecode has the advantage that it is supported on all platforms targeted by Godot, including the Web.

[Zireael07]

However, I request documenting the bytecode of GDscript.

More than reasonable ask.

[be-thomas]

However, I request documenting the bytecode of GDscript.

More than reasonable ask.

I take it that, it means 'reasonable++'.
Sounds good.

[Calinou]

However, I request documenting the bytecode of GDscript.

https://github.com/bruvzg/gdsdecomp has documentation on GDScript's bytecode format.

[be-thomas]

https://github.com/bruvzg/gdsdecomp has documentation on GDScript's bytecode format.

I could not find a documentation there. The only thing I found that could be referred to, is the code itself.
It would be much better to just have a list of all bytecodes, with description of what it does & how it mutates the VM/interpreter.
That's all I'm asking for.
I can host the documentation too, if you just help me out with the descriptions.
I believe, this small effort can go a long way.

[kisg]

@be-thomas If you want to create a GDScript to Lua transpiler (or any other alternative backend), then probably the best way is to use the code generator abstraction already present: https://github.com/godotengine/godot/blob/8adf04804550957f43c810b13fcce66dd76e5fd8/modules/gdscript/gdscript_codegen.h

The GDScript bytecode generator is just an implementation of the above interface. You don't need to use the GDScript bytecode directly, because it is an implementation detail of the GDScript VM that you are trying to replace with the LuaJIT VM.

[minim271]

LuaJIT performance is almost equal to .NET.

reference: https://programming-language-benchmarks.vercel.app/lua-vs-csharp

And based on my experience in the past, luajit is unlikely to have the "almost equal" performance of C#.

To clarify, I'm not saying that luajit is not good. It's excellent. But I totally don't think it's a good idea to use it as GDScript's transpiler target. Wasm is a more reasonable solution, especially considering that GDScript has no garbage collector.

[be-thomas]

LuaJIT performance is almost equal to .NET.

reference: https://programming-language-benchmarks.vercel.app/lua-vs-csharp

And based on my experience in the past, luajit is unlikely to have the "almost equal" performance of C#.

To clarify, I'm not saying that luajit is not good. It's excellent. But I totally don't think it's a good idea to use it as GDScript's transpiler target. Wasm is a more reasonable solution, especially considering that GDScript has no garbage collector.

Those benchmarks are using the vanilla Lua Interpreter.
Luajit interpreter will be faster, Plus the Luajit with JIT on will be even faster.
If you expand the Lua section, it is using the vanilla Lua 5.4.4 for benchmarks, evident here -
https://github.com/hanabi1224/Programming-Language-Benchmarks/runs/8051842043?check_suite_focus=true

It easily beats v8 & is faster than unoptimized C & C++.
For actual LuaJIT comparison, check here (and try benchmarking yourself too) -
https://gist.github.com/spion/3049314

On what grounds, am I saying it will have "almost equal" performance of C#?

• LuaJIT initially starts with a very fast interpreter(written in assembly) & only the HOT parts are JIT compiled when required.
  It uses a fraction of C#'s RAM & won't be too far away from C# in terms of performance.

• C# is supposed to suffer from speed drops because it keeps JITing all the code (JITing is expensive), whereas LuaJIT uses a mix of both JIT & a hand assembled interpreter. This means running code is instantaneous in LuaJIT (unlike Java & C#).

• Also, Lua to C calls are very cheap, its faster than C++, Rust & C too (source: https://nullprogram.com/blog/2018/05/27/ ) because it does direct calls instead of going through PLT. You can see the assembly generated by the JIT to get an idea of how it achieves it. Well, if it can be faster than C, then just forget about C#, its defeated in this regard, hands down.

• if you define a variable with type (using ffi), operations on those variables will be just as fast as C itself. Because it is JITed without any overhead. Godot has optional typing which can be integrated with this to get amazing speed ups.

\

My Questions -

1. Shouldn't we embrace the open source community by using each other's tools?
2. While Godot is open source, does the Godot community has enough resources to optimize GDscript to this extent?

[Calinou]

1. Shouldn't we embrace the open source community by using each other's tools?

One of the Best practice for engine contributors guidelines is to only reuse libraries if it makes sense. We are intentionally self-conscious about this and want to avoid relying on third-party decisions (see Bullet shifting its focus from games to robotics and simulation).

2. While Godot is open source, does the Godot community has enough resources to optimize GDscript to this extent?

GDScript has already received a rewrite and several optimizations for 4.0, such as typed instructions. I'm confident contributors will be able to continue this work for future 4.x releases 🙂

[be-thomas]

1. Shouldn't we embrace the open source community by using each other's tools?

One of the Best practice for engine contributors guidelines is to only reuse libraries if it makes sense. We are intentionally self-conscious about this and want to avoid relying on third-party decisions (see Bullet shifting its focus from games to robotics and simulation).

I'm curious how a general purpose programming language could shift its focus, such as to be unusable for programming.
Moreover, if you didn't know Cloudflare is powered by LuaJIT (they picked it just because of the raw speed).
Also the popular indie game marketplace - https://itch.io is written in Moonscript which transpiles to LuaJIT. A very clever way to get it up & running quickly.

Now, really I can just keep on saying & supporting LuaJIT but it seems pretty useless now. Since, I'm the only one talking about it. People are disregarding my opinions because LuaJIT supposedly uses JIT which they don't want.
And its laughable, that others disregard all my opinions because a JIT is coming in 4.X (could you guys decide already if you want a JIT or not? is this dependencyphobia?)

It really doesn't make sense & I give up convincing you all.
I honesty, don't care much about what happens to GDscript as long as it is fast enough.

[minim271]

@be-thomas

I used to use LuaJIT(not interperter) heavily in unity in my full-time work. It is not so fast in the real scene, but I respect you to keep your own point of view, the result of this problem may be greatly affected by specific conditions and is not so important.

If you want to use LuaJIT as the script engine, I think you should use the GDNative/Extension to achieve this goal, and then write external tools to transpile GDScript(or just use lua). This will be an exciting project. And if your goal is like this, it is not necessary to discuss it in this repo's issues, just make a plugin.

If you want Godot officially change GDScript's engine to LuaJIT, I think that is definitely a bad idea. Really need performance? You should choose the more mainstream C# or GDNative. GDScript provides excellent flexibility and does not have performance bottlenec in most scenarios.

Lua is a language with GC, and it has very different design from GDScript. The great advantage of GDScript is that it is customized for Godot. Changing the script engine will definitely have an impact on this point, and at least it will increase a lot of maintenance costs. And "No GC" is very meaningful in many games (Many unity developers need to deal with GC related optimization problems all year round)

[minim271]

2. While Godot is open source, does the Godot community has enough resources to optimize GDscript to this extent?

The answer to this question is simple: it doesn't have to.

GDScript do not intend and do not need to be the fastest scripts language.

If the main problem is performance, Python should not exist as the most widely used language all the year round.

I certainly don't mean that we should not optimize GDScript, but considering that people are working hard to do this, and that the current performance is sufficient in most scenarios, and other scenarios have other solutions, it seems unnecessary to be strict on this issue.

[erayzesen]

I think this idea(or another "GDSciprt to binary" transpiler solutions) should be rethink. In particular, most of game engines alternative to Godot have this convenience.

Game Maker Studio:
GML->C++->binary (Yoyo Compiler)

Unity
C#-> C++-> binary (Unity ilcpp)

Godot is a engine promising the fast development process and the ease of use. Of course, GD Extension (Or Currently Gdnative) is an important solution. But do you think this is enough? Is it realistic to seek C ++ experience for users who use GDScript and want performance? When we look at Godot's user base, we do not see this depiction.

I think this would be a game-changer alternative both for making GDScript more valuable and in terms of performance solutions.

And I would like to thank the developer team again here. Please think about this once again.

[Lcbx]

for those interested, I made a Gdscript to C# and c++ transpiler.
it's written in python though so the parser has to be kept up to date with current Gdscript syntax.
The output code contains bindings for exported properties. It uses godot classes/containers but it should still save you time.
Cheers 🫡

[Calinou]

@Foxchandaisuki Please don't bump issues without contributing significant new information. Use the 👍 reaction button on the first post instead.