A whole new builder.

[Faless]

Rewritten in python (tested 3.8).
Only replace build.sh (but can easily replace scons runners and build-release.sh).
See ./cli.py -h for help.

Highlights

• Use release action for the old build.sh behaviour.
• Most options can be overridden via a json config file, or cli arguments before the action.
• Use run to only build containers from current godot.tar.gz, can specific one or more (e.g. -k JavaScript), default all, can also start the container in interactive mode.
• Can now select using local or remote images, override repository, and their paths (only priv/public list is hardcoded, maybe we can do something about that).
• Containers are configured in builder/images.py. Add a new config for a new container, or edit their parameters
• Config options are in builder/config.py. Adding a variable to the Config class will automatically add it as a cli option.
• PodmanRunner can run in non-block mode (not exposed), so it could be also used as part of a rocket chat bot.

Note: Some config options are unused while I port build-release.sh (which will probably be action dist).
Note2: I also want to add the action build which would run scons (and replace build-plat/build.sh)
Note3: There's quite a few things I couldn't test. So this is draft for now.

@akien-mga I hope this makes up for all my ranting about the build scripts ❤️

Usage:

usage: cli.py [-h] [--apple_id APPLE_ID] [--apple_id_password APPLE_ID_PASSWORD] [--build_name BUILD_NAME] [--num_core NUM_CORE] [--osx_bundle_id OSX_BUNDLE_ID] [--osx_host OSX_HOST]
              [--osx_key_id OSX_KEY_ID] [--password PASSWORD] [--private_path PRIVATE_PATH] [--public_path PUBLIC_PATH] [--registry REGISTRY] [--sign_keystore SIGN_KEYSTORE]
              [--sign_name SIGN_NAME] [--sign_password SIGN_PASSWORD] [--sign_url SIGN_URL] [--username USERNAME] [-c CONFIG]
              {config,checkout,fetch,run,release} ...

positional arguments:
  {config,checkout,fetch,run,release}
                        The requested action
    config              Print or save config file
    checkout            git checkout, version check, tar
    fetch               Fetch remote build containers
    run                 Run the desired containers
    release             Make a full release cycle, git checkout, reset, version check, tar, build all

optional arguments:
  -h, --help            show this help message and exit
  --apple_id APPLE_ID
  --apple_id_password APPLE_ID_PASSWORD
  --build_name BUILD_NAME
  --num_core NUM_CORE
  --osx_bundle_id OSX_BUNDLE_ID
  --osx_host OSX_HOST
  --osx_key_id OSX_KEY_ID
  --password PASSWORD
  --private_path PRIVATE_PATH
  --public_path PUBLIC_PATH
  --registry REGISTRY
  --sign_keystore SIGN_KEYSTORE
  --sign_name SIGN_NAME
  --sign_password SIGN_PASSWORD
  --sign_url SIGN_URL
  --username USERNAME
  -c CONFIG, --config CONFIG
                        Configuration override

usage: cli.py config [-h] [-n] [-s SAVE]

optional arguments:
  -h, --help            show this help message and exit
  -n, --dry-run
  -s SAVE, --save SAVE

usage: cli.py checkout [-h] [-n] [-c] [-t] [--skip-check] treeish godot_version

positional arguments:
  treeish              git treeish, possibly a git ref, or commit hash.
  godot_version        godot version (e.g. 3.1-alpha5)

optional arguments:
  -h, --help           show this help message and exit
  -n, --dry-run
  -c, --skip-checkout
  -t, --skip-tar
  --skip-check

usage: cli.py fetch [-h] [-n] [-f] [-i IMAGE]

optional arguments:
  -h, --help            show this help message and exit
  -n, --dry-run
  -f, --force-download
  -i IMAGE, --image IMAGE
                        The image to fetch, all by default. Possible values: mono-glue, windows, ubuntu-64, ubuntu-32, javascript, macosx, android, ios, uwp

usage: cli.py run [-h] [-n] [-b {all,classical,mono}] [-k CONTAINER] [-r] [-i]

optional arguments:
  -h, --help            show this help message and exit
  -n, --dry-run
  -b {all,classical,mono}, --build {all,classical,mono}
  -k CONTAINER, --container CONTAINER
                        The containers to build, one of ['MonoGlue', 'Windows', 'Linux64', 'Linux32', 'JavaScript', 'MacOSX', 'Android', 'IOS', 'Server', 'UWP']
  -r, --remote          Run with remote containers
  -i, --interactive     Enter an interactive shell inside the container instead of running the default command

usage: cli.py release [-h] [-n] [-b {all,classical,mono}] [-s] [-c] [-g GIT] [-f] [-l] godot_version

positional arguments:
  godot_version         godot version (e.g. 3.1-alpha5)

optional arguments:
  -h, --help            show this help message and exit
  -n, --dry-run
  -b {all,classical,mono}, --build {all,classical,mono}
  -s, --skip-download
  -c, --skip-git
  -g GIT, --git GIT     git treeish, possibly a git ref, or commit hash.
  -f, --force-download
  -l, --localhost

[Faless]

Draft, as it needs some testing, and I realize some help strings are missing

[akien-mga]

CC @Xrayez who might be interested too :)

[Faless]

Will likely need a more graceful KeyboardInterrupt too (now the subprocess keep going a bit, we should add the "double ctrl+c" style for exiting program immediately, vs sending term to subprocess).

[Xrayez]

Looks promising, I may give this a try once 3.3-stable is released. 🙂

I've had a problem of git synchronization from remote when using tags vs commits, so oftentimes I just had to skip this in current build.sh, perhaps this PR resolves those usability issues.

Another thing which can be added is custom modules support. I believe that's the reason why people like me would like to use official build scripts in the first place. Probably there may be no need to use custom_modules build option specifically because it seems like Godot source is just packed in godot.tar.gz for all podman builds, which forces the user to copy modules directly under Godot source anyways. But doing so makes the process of synchronizing custom modules themselves more cumbersome, and it's not guaranteed that those modules will remain there once all build steps are completed... In any case, I think using Python will make this process easier for third-parties, at least.

But that's for another PR, I could possibly work on improving the usability behind using custom modules myself if the above makes sense.

[Faless]

I've had a problem of git synchronization from remote when using tags vs commits, so oftentimes I just had to skip this in current build.sh, perhaps this PR resolves those usability issues.

So right now the full git step will be

git fetch --all --tags
checktout --detach treeish
git archive HEAD

So you can fetch any ref with proper addressing, even tags. E.g.:

./cli.py release 3.2.3-stable -g refs/tags/3.2.3-stable

Or even different remotes (assuming you added it manually in the git folder beforehand):

./cli.py release 3.3-rc -g my_remote/3.x

Another thing which can be added is custom modules support.

I think an easy addition as soon as I port build-plat/build.shs too, is to add the custom_modules option to scons builds, and mount the selected folder accordingly. Shouldn't be hard once that's done.

It could be even done without the porting, by just editing all the platforms build.sh to receive some custom_opts env variable, but I honestly would rather just get the whole build step done properly (and selectively).

[Xrayez]

Missing godot-uwp

DEBUG:root:Dry run: ['/usr/bin/podman', 'run', '--rm', '-w', '/root/', '--env', 'BUILD_NAME=custom_build', '--env', 'NUM_CORES=4', '--env', 'CLASSICAL=1', '--env', 'MONO=1', '-v', '/home/Xrayez/src/godot/godot-build-scripts/mono-glue:/root/mono-glue', '-v', '/home/Xrayez/src/godot/godot-build-scripts/godot.tar.gz:/root/godot.tar.gz', '-v', '/home/Xrayez/src/godot/godot-build-scripts/build-uwp:/root/build', '-v', '/home/Xrayez/src/godot/godot-build-scripts/out/uwp:/root/out', '--ulimit', 'nofile=32768:32768', 'localhost/uwp:3.3-mono-6.12.0.114', 'bash', '/root/build/build.sh']

Non-configurable image version

Had to modify images.py directly to match container images with mono tag (current stable version used):

image_version = "3.x-mono-6.12.0.122"

Git checkout issue

[Xrayez@localhost godot-build-scripts]$ ./cli.py release -b all -g 3.x --localhost 3.3-rc
DEBUG:root:Running command: ['which', 'podman']
DEBUG:root:/usr/bin/podman
DEBUG:root:
DEBUG:root:RUNNING: ['git', 'clone', '/home/Xrayez/src/godot/godot-build-scripts/git'], PID: 1227088
fatal: destination path 'git' already exists and is not an empty directory.
DEBUG:root:Program failed with return code: 128
DEBUG:root:fatal: destination path 'git' already exists and is not an empty directory.

DEBUG:root:Try running: git clone /home/Xrayez/src/godot/godot-build-scripts/git
DEBUG:root:RUNNING: ['git', '-C', '/home/Xrayez/src/godot/godot-build-scripts/git', 'fetch', '--all'], PID: 1227091
Fetching origin
From https://github.com/godotengine/godot
   9952a5039a..94a0fc47f7  3.2        -> origin/3.2
 * [new branch]            3.x        -> origin/3.x
   01851defb5..d540875bc0  master     -> origin/master
DEBUG:root:RUNNING: ['git', '-C', '/home/Xrayez/src/godot/godot-build-scripts/git', 'checkout', '--detach', '3.x'], PID: 1227113
fatal: '--detach' cannot be used with '-b/-B/--orphan'
DEBUG:root:Program failed with return code: 128
DEBUG:root:fatal: '--detach' cannot be used with '-b/-B/--orphan'

DEBUG:root:Try running: git -C /home/Xrayez/src/godot/godot-build-scripts/git checkout --detach 3.x
Command failed ['git', '-C', '/home/Xrayez/src/godot/godot-build-scripts/git', 'checkout', '--detach', '3.x']

Missing godot.tar.gz

Only with --skip-git I think. I wanted to skip git checkout, but not archive.

[Xrayez@localhost godot-build-scripts]$ ./cli.py release -b all -g 3.x --localhost --skip-git 3.3-rc
DEBUG:root:Running command: ['which', 'podman']
DEBUG:root:/usr/bin/podman
DEBUG:root:
DEBUG:root:RUNNING: ['/usr/bin/podman', 'run', '--rm', '-w', '/root/', '--env', 'BUILD_NAME=goost', '--env', 'NUM_CORES=4', '--env', 'CLASSICAL=1', '--env', 'MONO=1', '-v', '/home/Xrayez/src/godot/godot-build-scripts/mono-glue:/root/mono-glue', '-v', '/home/Xrayez/src/godot/godot-build-scripts/godot.tar.gz:/root/godot.tar.gz', '-v', '/home/Xrayez/src/godot/godot-build-scripts/build-mono-glue:/root/build', 'localhost/godot-mono-glue:3.x-mono-6.12.0.122', 'bash', '/root/build/build.sh'], PID: 1228242
Error: statfs /home/Xrayez/src/godot/godot-build-scripts/godot.tar.gz: no such file or directory
DEBUG:root:Program failed with return code: 125
DEBUG:root:Error: statfs /home/Xrayez/src/godot/godot-build-scripts/godot.tar.gz: no such file or directory

DEBUG:root:Try running: /usr/bin/podman run --rm -w /root/ --env BUILD_NAME=goost --env NUM_CORES=4 --env CLASSICAL=1 --env MONO=1 -v /home/Xrayez/src/godot/godot-build-scripts/mono-glue:/root/mono-glue -v /home/Xrayez/src/godot/godot-build-scripts/godot.tar.gz:/root/godot.tar.gz -v /home/Xrayez/src/godot/godot-build-scripts/build-mono-glue:/root/build localhost/godot-mono-glue:3.x-mono-6.12.0.122 bash /root/build/build.sh
Command failed ['/usr/bin/podman', 'run', '--rm', '-w', '/root/', '--env', 'BUILD_NAME=goost', '--env', 'NUM_CORES=4', '--env', 'CLASSICAL=1', '--env', 'MONO=1', '-v', '/home/Xrayez/src/godot/godot-build-scripts/mono-glue:/root/mono-glue', '-v', '/home/Xrayez/src/godot/godot-build-scripts/godot.tar.gz:/root/godot.tar.gz', '-v', '/home/Xrayez/src/godot/godot-build-scripts/build-mono-glue:/root/build', 'localhost/godot-mono-glue:3.x-mono-6.12.0.122', 'bash', '/root/build/build.sh']

Patch

Here are some of the changes (including fixing above issues) which allowed me to start running the builds (haven't completed all builds as of writing yet, but seems to work fine).

diff --git a/.gitignore b/.gitignore
index 355d9d3..79d04c7 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,5 +1,6 @@
 # User-specific configuration and signing key
 config.sh
+config.json
 *.pkcs12
 
 # Generated by build scripts
diff --git a/builder/builder.py b/builder/builder.py
old mode 100644
new mode 100755
index fc3c792..4d43627
--- a/builder/builder.py
+++ b/builder/builder.py
@@ -86,11 +86,11 @@ class PodmanRunner(Runner):
         for d in run_config.dirs:
             ensure_dir(os.path.join(self.base_dir, d))
 
-        cores = os.environ.get('NUM_CORES', os.cpu_count())
+        cores = os.environ.get('NUM_CORES', Config.num_core)
         cmd = [self._podman, "run", "--rm", "-w", "/root/"]
         cmd += env({
-            "BUILD_NAME": os.environ.get("BUILD_NAME", "custom_build"),
-            "NUM_CORES": os.environ.get("NUM_CORES", os.cpu_count()),
+            "BUILD_NAME": os.environ.get("BUILD_NAME", Config.build_name),
+            "NUM_CORES": os.environ.get("NUM_CORES", Config.num_core),
             "CLASSICAL": 1 if classical else 0,
             "MONO": 1 if mono else 0,
         })
@@ -153,7 +153,7 @@ class GitRunner(Runner):
     def checkout(self, ref):
         repo = "https://github.com/godotengine/godot"
         dest = os.path.join(self.base_dir, "git")
-        self.git("clone", dest, can_fail=True)
+        self.git("clone", repo, "git", can_fail=True)
         self.git("-C", dest, "fetch", "--all")
         self.git("-C", dest, "checkout", "--detach", ref)
 
diff --git a/builder/images.py b/builder/images.py
old mode 100644
new mode 100755
index d37700f..6733b2b
--- a/builder/images.py
+++ b/builder/images.py
@@ -12,7 +12,7 @@ class ImageConfig:
     extra_opts = []
     cmd = ["bash", "/root/build/build.sh"]
     mounts = {}
-    image_version = "3.3-mono-6.12.0.114"
+    image_version = "3.x-mono-6.12.0.122"
     log = None
 
 
@@ -89,7 +89,7 @@ class UWPConfig(ImageConfig):
     extra_opts = ["--ulimit", "nofile=32768:32768"]
     cmd = ["bash", "/root/build/build.sh"]
     mounts = {"build-uwp": "build"}
-    image = "uwp"
+    image = "godot-uwp"
     log = "uwp"

[Xrayez]

Is there a way to build a list of containers at once? Somehow my laptop rebooted tonight (crashed?) and I've only managed to compile for Windows, so I'd like to "resume" the build except for Windows. With the old script, I could just comment out those build steps to prevent them from building again.

I know that it would be safe to re-run all builds for production, though.

[Faless]

You can use the `run` command, and chain multiple `-k container` to specify which container to build

…

On Sun, Apr 11, 2021, 13:45 Andrii Doroshenko ***@***.***> wrote: Is there a way to build a list of containers at once? Somehow my laptop rebooted tonight (crashed?) and I've only managed to compile for Windows, so I'd like to "resume" the build except for Windows. With the old script, I could just comment out those build steps to prevent them from building again. I know that it would be safe to re-run all builds for production, though. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#31 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAM4C3TU3GJ66DO4SQOCHO3TIGDW5ANCNFSM42BTEW5A> .

[Xrayez]

Ah, I didn't realize that the same argument can be specified multiple times.

---

I've re-run the script and stumbled upon this error:

i686-w64-mingw32-g++ -o bin/godot.windows.opt.tools.32.mono.exe -static -static-libgcc -static-libstdc++ -flto=4 -Wl,--stack,8388608 -Wl,--nxcompat -Wl,-whole-archive /root/dependencies/mono-32/lib/libmonosgen-2.0.a -Wl,-no-whole-archive platform/windows/godot_windows.windows.opt.tools.32.o platform/windows/context_gl_windows.windows.opt.tools.32.o platform/windows/crash_handler_windows.windows.opt.tools.32.o platform/windows/os_windows.windows.opt.tools.32.o platform/windows/key_mapping_windows.windows.opt.tools.32.o platform/windows/joypad_windows.windows.opt.tools.32.o platform/windows/power_windows.windows.opt.tools.32.o platform/windows/windows_terminal_logger.windows.opt.tools.32.o platform/windows/godot_res.windows.opt.tools.32.o -L/root/dependencies/mono-32/lib main/libmain.windows.opt.tools.32.a main/tests/libtests.windows.opt.tools.32.a modules/libmodules.windows.opt.tools.32.a platform/libplatform.windows.opt.tools.32.a drivers/libdrivers.windows.opt.tools.32.a editor/libeditor.windows.opt.tools.32.a scene/libscene.windows.opt.tools.32.a servers/libservers.windows.opt.tools.32.a core/libcore.windows.opt.tools.32.a modules/freetype/libfreetype_builtin.windows.opt.tools.32.a -lmingw32 -lopengl32 -ldsound -lole32 -ld3d9 -lwinmm -lgdi32 -liphlpapi -lshlwapi -lwsock32 -lws2_32 -lkernel32 -loleaut32 -ldinput8 -ldxguid -lksuser -limm32 -lbcrypt -lavrt -luuid -ldwmapi -lpsapi -lversion
/usr/lib/gcc/i686-w64-mingw32/10.2.1/../../../../i686-w64-mingw32/bin/ld: hq2x.windows.opt.tools.32.o (symbol from plugin): warning: no symbol for section '__ZZN4Math9fast_ftoiEfE1b' found
Terminated

I have no idea if it's caused by using the new build script or it's simply LTO error or so, I literally haven't touched anything to cause Ctrl + C...

[Faless]

I have no idea if it's caused by using the new build script or it's simply LTO error or so, I literally haven't touched anything to cause Ctrl + C...

@Xrayez did the python script terminate or just the container (if you where doing multiple run)? Is it possible it went out of memory? LTO can require quite a few GiB of RAM.

[Xrayez]

Is it possible it went out of memory? LTO can require quite a few GiB of RAM.

This is likely an issue. I have 8 GB of RAM, but it wasn't enough even for compiling mono (see my newbie endeavors in godotengine/build-containers#74). I then configured/increased the swap file to 8 GB, but perhaps even that is not enough?

But then it would also fail on classical Windows builds, but those completed just fine. It's mono which get crashed.

So likely not an issue with the Python build script.

[Xrayez]

So I investigated/tested the problem further, and I get my statements back.

I think it's likely due to Python at the end of the day (literally). I think it would take me more than 24 hours to build everything with LTO on my Intel i3 laptop, and I got either "Terminated" or "Killed" messages occuring sporadically (either at LTO stage, or even compilation stages). Increasing the swap size to 16 GB seems to help and I've managed to build Linux, but then got "Killed" signal at Javascript build. Incidentally, this only happened at night time. I also disable sleep/hibernation so the builds don't get interrupted on idle time.

I've tried building everything with the old build.sh, and I didn't experience the errors above (so far). So, perhaps somehow Python is taking more memory than needed?

• https://serverfault.com/questions/522349/how-to-prevent-termination-of-long-running-python-script
• https://superuser.com/questions/1137934/python-script-suddenly-getting-terminated-with-no-other-messages
• https://unix.stackexchange.com/questions/614950/python-programs-suddenly-get-killed

That said, you may not even experience the error yourself if you have a powerful machine.