feat: switch to xmake #377
Conversation
|
I get this error after following your steps, after checking out this branch + the xmake branch in the Unreal submodule: The command I ran was: |
|
When running The |
|
After the latest changes, It also appears that Running |
|
Which environment are you running this in? Is it the x64 development tools prompt, some other? |
I ran it in PowerShell, without running any other script like the vcvarsall script first. |
|
Testing again after the latest commit gives me the following warnings when running The errors regarding dwmapi.asm and "bad VCC option" when running I also figured out my OutputDevice errors. This is a potential problem because we might have some unused files (and I do remember at least one file but unfortunately the name escapes me) that really should not be compiled. |
|
It looks to me like ASMHelper and DynamicOutput are being dynamically linked. |
|
I think unused files are better just deleted from the repo rather than not being selected in the build system, will check regarding the asmhelper/dynamicoutput linking incorrectly |
|
The latest commit appears to be properly linking statically. |
|
The |
|
How will this change work with IDEs that depend on CMake, like CLion ? |
|
You could be using generated cmake files, but usually the way I see most used for clion/clion nova is using the compiler commands generator ( |
|
With fc251ba, I could not figure out a way to make cmake fallback to http if ssh fails like the others. Tried a bunch of things. So I hope it's not much of an issue if I just change this one clone from ssh to http |
Tbh I gave up using CLion/CLion Nova for UE4SS, CLion is way too slow analysing and debugging (and thinks that UE4SSProgram.cpp has 10k errors) and Nova is brand new so is still fairly buggy (but doesn't have the same issue as CLion). Not to mention the pain in the arse that is trying to switch between configurations. In the end I switched to using Rider which has VS Solution support so you can just run the vsxmake2022 command and then open the .sln with Rider with all the configs there. I don't know if you rely on specific features of CLion/CLion Nova but just for general IDE stuff n things, Rider is surprisingly poggers. |
|
Are more instructions required for C++ mod building? |
I've been testing CLion Nova for a while now and it's faster but yeah it's awfully glitchy right now, hopefully it will get better. I have CLion set up very specifically so I don't really want to change to another IDE even if it's from JetBrains because some plugins aren't compatible or settings are named differently (so I can't find them) or missing or whatever. I'll try figure something out after this is merged, I don't want to be holding it up just because of CLion. |
|
Merge checklist:
|
|
I updated the merge checklist to clarify some things regarding updating the readme file. |
This problem appears to be fixed. |
|
Could the default VS startup project be changed from |
|
My approval is pending a final test of the build instructions, which I will do soon.
|
|
The regular build instructions appear to work fine, but the mod instructions are not working for me. When following the mod build instructions, I get this error when executing When building through VS instead, I get these errors: |
For the record, I could not reproduce this, I could build MyAwesomeMod no problemo. My guess is that since xmake UEPsuedo branch isn't merged yet, you are still using the old UEPsuedo version. I basically just copied my RE-UE4SS clone that's on the xmake branch in both repos, to MyMods directory. |
I deleted everything and recloned again on latest commit on this and on uepsuedo, I'm still having no luck. What xmake.lua file are you using for your mod? I uploaded my test mod files. See if you can build it? |
Turns out that I didn't update my xmake file completely. local projectName = "MyAwesomeMod"
target(projectName)
set_kind("shared")
set_languages("cxx20")
set_exceptions("cxx")
add_includedirs(".")
add_files("main.cpp")
add_deps("UE4SS")
on_load(function (target)
import("build_configs", { rootdir = get_config("scriptsRoot") })
build_configs:set_output_dir(target)
end)
on_config(function (target)
import("build_configs", { rootdir = get_config("scriptsRoot") })
build_configs:config(target)
end)
after_clean(function (target)
import("build_configs", { rootdir = get_config("scriptsRoot") })
build_configs:clean_output_dir(target)
end)Everything below |
|
I'm starting to have doubts about switching to xmake. At least everyone knows roughly how to use cmake but if we switch to xmake we're kind of screwed until someone decides to learn it if something needs to be changed for whatever reason, and with cmake there's tons of information available whenever someone gets stuck and needs help while with xmake I haven't had much luck searching for help. These things combined makes me doubt that changing to xmake is going to be easier in the long run than staying with cmake. |
At least if you were still in the discord you'd know that every man and his dog has had some form of trouble or another with cmake build as it currently is. The rust integration with cmake (corrosion) is fragile at best. No one wants to maintain it. cmake is uncapable of falling back from ssh to http if ssh clone fails (which it does on the regular). I'm aware this PR has been rocky but I'm fairly confident that once xmake starts working it will just work. I mean, it was working fine, until I suggested localcc move the buildscripts into the existing tools folder just to keep the root repo clean and consistent. And the progress is only slow because people who want to work on it don't because they're busy or whatever, while everyone else is too hesitant to really look at xmake because it's new and are skeptical of its abilities. That being said, narknon has always expressed the same concerns as you about xmake and the fact no one really knows about it. I do think you've got a fair point there. On the other hand, cmake's documentation is terrible and it has a serious problem with new versions releasing new features and then documentation either just not being updated properly, or is updated but its not obvious which versions support those. When I was first learning cmake it took me multiple painful months to realise about fetch content. It's just not documented obviously anywhere (unless you know to look for it), despite arguably being one of the most useful features cmake has to offer (imo). Meanwhile xmake is fresh and its docs are really easy to follow. That's my two cents on it. I think it just needs to be given a chance (once everything works again). |
| after_load(function (target) | ||
| import("build_configs", { rootdir = get_config("scriptsRoot") }) | ||
| import("target_helpers", { rootdir = get_config("scriptsRoot") }) | ||
| build_configs:set_output_dir(target) |
There was a problem hiding this comment.
Does our set_output_dir(target) helper accomplish something that the built-in target:set_targetdir/target:set_objectdir/target:set_dependir cannot?
Build engineer perspective to try and contextualize some of the kinks in the mod compilation context. In my opinion, the templated
I also think a custom Proposing something like the following (syntax/options can be whatever we want):
This would enable the modder to bootstrap a buildable mod without having to create/touch an
I have plenty of other ideas (and I'm sure we could get feedback on the onboard process from maintainers/modders) if we want to roll a UE4SS xmake task that can act as the entry point for modders in the development environment. It would certainly reduce the barrier to entry at the cost of up-front transparency to onboarding modders. |
We have the mod template repo already so we have to ask ourselves, is it easier to tell people to execute a command or is it easier to tell people to use the template repo ? |
Seems a bit overkill tbh when I doubt the mod xmake.lua will be changing at all once this is done. It will be the repo's main xmake.lua and buildscripts that will be changing.
Template already does all this. Though currently it is using bats which probably isn't the best way to do it. The best way would probably be an xmake lua script. So feel free to take a look at what is already in those bats and port them to lua script. Note what the Currently in the docs we are describing the setup process and then at the end saying "oh by the way, there is a template you can use to do this for the rest of your mods". My reasoning for this order (rather than automated process first then manual setup) is so that modders understand exactly what the automated setup process is doing, which I'd say is pretty important when setting up your dev environment. Obviously if the process was super complex I wouldn't suggest users to do it manually. But for the most part it's just cloning the repo, copy and paste, change some names, and run a couple of commands. |
I'm pretty sure I just fucked up on the mod's xmake.lua file. It builds fine for me with yours (and doesn't fail to find the build_scripts either, which is what I thought was wrong with yours). I'll update the c++ docs with the correct file... I guess? I.e. reversing the xmake.lua in the docs commit since apparently the old one worked fine. |
|
Could everyone re-check the compilation following the docs once again so the PR can get unblocked? |
|
Builds successfully following the updated Has anyone actually tested the CI scripts ? |
|
I built UE4SS standalone (without mods) and it still works. Worth noting is that the output directory for UE4SS.dll has changed.
I don't think this will matter for our CI scripts, because build.py scans every directory until it finds the files its looking for. |
|
I ran the experimental release action on this branch and it doesn't recognize the I know our pre-existing CI won't be able to compile as is even if it finds xmake because it doesn't checkout the xmake branch for the Unreal submodule, so either someone has to fork it and set all the CI up in their fork or we can make sure xmake is installed in our current CI and fix any potential problems after the merge. |
Output StructureProject compilation will involve the following categories of outputs:
We should probably decide if we want to
|
|
We should follow either GNU standards or mimic UE for output structure. |
|
I much rather prefer the ue structure so we should try mimicking that. |
|
Anything else that needs to be addressed? Would be best to have this merged asap so me, and maybe some other maintainers can get unblocked and get going. |
|
@Buckminsterfullerene02 @bitonality I would like to request a build from either of you also, to verify it runs on different machines. |
|
On second thought, this needs to be addressed: #377 (comment) |
|
Builds fine for me. |
Xmake is a nicer build system, which doesn't require as much effort to add new build configurations in that work with every generator. Xmake also allows to interface with CMake projects which allows us to easily use existing CMake external dependencies. All internal dependencies were converted to use xmake. Xmake also has a nice package manager which doesn't require any configuration on the user's side, allowing us to use packages from that if needed.
In case anyone stumbles across this and wonders if I got CLion to work. Another problem was that I'm on Linux and we don't yet support Linux so I've had to figure out some way to use xmake to generate Windows files. @echo off
net use Z: \\192.168.0.182\stuff_shared /u:<username> *
Z:
cd Z:\repos\oss_ue4ss\cxx_mods
:: Note profilerFlavor being None.
:: It's set to Tracy by default if left out for debug builds and my VM doesn't like that.
xmake f --profilerFlavor=None -m "Game__Debug__Win64" -y
xmake project -k compile_commands
:: This file is stored in the cxx_mods directory.
:: Because reasons, I need to execute it multiple times.
:: If I don't, the search & replace only partially completes.
py fix_compile_commands.py
py fix_compile_commands.py
py fix_compile_commands.py
py fix_compile_commands.py
py fix_compile_commands.py
:: I store this bat file at C:\Users\<username>
:: As a result, re-entering the C drive will automatically put me right where I have my bat script.
:: That means I can re-execute this bat without needing to cd to it.
C:This is the file I use to convert my # This python script fixes all the paths in compile_commands.json generated by the 'xmake project -k compile_commands' command.
# The purpose of this is to allow compile_commands.json to be generated on a Windows machine (i.e. VM) via SSH, but a build server should also be possible.
# This script is necessary because:
# 1. I want to use the CLion IDE on my Linux machine.
# 2. Our project (UE4SS) doesn't support Linux builds yet so any attempt to use xmake will fail.
# 3. We no longer use cmake and, I've failed to figure out how to use xmake successfully via wine.
# Absolute path to compile_commands.json.
# This can be either on the client or the server, it depends on where you execute this script.
# This file needs to exist on the machine that you execute this script on.
# I prefer to execute this on the server alongside all my other toolchain scripts.
compile_commands_file = r"Z:\repos\oss_ue4ss\cxx_mods\compile_commands.json"
# Absolute path that compile_commands.json uses for its 'directory' field on the server.
server_project_root = r"Z:/repos/oss_ue4ss/cxx_mods"
# Absolute path to the actual project on the client.
client_project_root = r"/stuff/repos/oss_ue4ss/cxx_mods"
# Absolute path to cl.exe and ml64.exe on the server.
server_cl_exe = r"C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.37.32822/bin/HostX64/x64/cl.exe"
server_ml64_exe = r"C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.37.32822/bin/HostX64/x64/ml64.exe"
# Absolute path to clang-cl on the client.
client_clang_cl = r"/stuff/repos/llvm/llvm-project/build_clang_cxx_cxxabi_unwind_18rc3/bin/clang-cl"
with open(compile_commands_file, mode="r") as file:
data = file.read()
data = data.replace(server_project_root, client_project_root)
data = data.replace(r"\\", "/")
data = data.replace(server_cl_exe, client_clang_cl)
data = data.replace(server_ml64_exe, client_clang_cl)
with open(compile_commands_file, mode="w") as file:
file.write(data)My workflow is this:
My Z:
cd Z:\repos\oss_ue4ss\cxx_mods
xmake
C: |
Xmake is a nicer build system, which doesn't require as much effort to
add new build configurations in that work with every generator. Xmake
also allows to interface with CMake projects which allows us to easily
use existing CMake external dependencies.
All internal dependencies were converted to use xmake.
Xmake also has a nice package manager which doesn't require any
configuration on the user's side, allowing us to use packages from that
if needed.
Setting up
Checkout the
xmakebranch for the unreal submoduleThe project should now be compilable with xmake.
Example
To compile the full project in release mode using xmake you can use this to configure first:
After configuring run:
Vs project files
If you want to generate visual studio project files, run: