Issue with AppVeyor

[warden307]

So not sure if this is just me or not. Never used AppVeyor before, but when I go to download the build it gives me this message...

{"message":"Artifacts download limit (1024 MB/day) exceeded."}

Is there something I'm doing wrong orrrrrrrrr?

[Jules-A]

AppVeyor is meant to be used for automated build testing, not releases and looks like they are starting to enforce it by placing limits. Ryujinx recently got taken down off AppVeyor without any message as to why.

The AppVeyor builds only used to be for testing as there used to be nightly builds every now and then but they've stopped and the old 0.9.11 build is so old it's not recommended so the AppVeyor builds became the main build that people who visit the Desmume site see.

[warden307]

So there is no proper way to get DeSmuMe outside of getting lucky on the reset or the really old build then?

[rofl0r]

i guess it wouldn't be a bad idea if someone added a github action that builds a win32 release. github downloads are unlimited. even better would be if a tag was pushed now and then (maybe like 0.9.12-pre-1), so source users also have something to rely on.

[stinyc]

Any updates on this? Been trying to download for a couple of days now and haven't been lucky.

[zeromus]

Someone with time for this bullshit needs to copy this and make it work on a desmume fork for evaluation

https://github.com/Arisotura/melonDS

It's the best reference we've found. The builds go to:

https://github.com/Arisotura/melonDS/actions?query=workflow%3A%22CMake+Build+%28Windows+x86-64%29%22+event%3Apush

This project is actually very easy. Just build the sln and grab this artifact

desmume/src/frontend/windows/__bins/DeSmuME-VS2019-x64-Release.exe

[Launders]

Someone with time for this bullshit needs to copy this and make it work on a desmume fork for evaluation

https://github.com/Arisotura/melonDS

It's the best reference we've found. The builds go to:

https://github.com/Arisotura/melonDS/actions?query=workflow%3A%22CMake+Build+%28Windows+x86-64%29%22+event%3Apush

This project is actually very easy. Just build the sln and grab this artifact

desmume/src/frontend/windows/__bins/DeSmuME-VS2019-x64-Release.exe

what exactly does this mean in layman terms?

[SimonAfek]

what exactly does this mean in layman terms?

MelonDS has a build system based on GitHub Actions, rather than the AppVeyor buildbot that desmume currently uses. Zeromus is suggesting someone fork the repo and set up a similar GHA build flow for desmume.

[rofl0r]

melonds uses CMake though, so the workflow isn't applicable. i've been experimenting yesterday with a GHA that uses MSVC solutions directly: https://github.com/rofl0r/desmume-gtk2/runs/5083221475?check_suite_focus=true and it seems to work except that the solution has a prebuild hook executing a script to get a git build number, and it seems that the cscript executable it tries to use for that isn't present (at least in the hardcoded location). so we'd need to figure out whether and where the windows container has cscript.exe or install it ourselves (and eventually a git executable too).

[SimonAfek]

In the meantime, here's a VS2022 build of the most recent master code. Haven't tested it beyond "it starts".

DeSmuME-VS2022-x64-Release.zip

[SimonAfek]

melonds uses CMake though, so the workflow isn't applicable. i've been experimenting yesterday with a GHA that uses MSVC solutions directly: https://github.com/rofl0r/desmume-gtk2/runs/5083221475?check_suite_focus=true and it seems to work except that the solution has a prebuild hook executing a script to get a git build number, and it seems that the cscript executable it tries to use for that isn't present (at least in the hardcoded location). so we'd need to figure out whether and where the windows container has cscript.exe or install it ourselves (and eventually a git executable too).

Shot in the dark, but have you tried changing the prebuild-event to looking for cscript via %windir%\system32 instead of %windir%\Sysnative?

[rofl0r]

indeed, that fixes the issue: https://github.com/rofl0r/desmume-gtk2/runs/5097084147?check_suite_focus=true

however now the build fails running out of memory when compiling GPU.cpp

[SimonAfek]

@rofl0r I took your GHA .yml and fiddled around with it a bit on a fork from the latest master, and I get the same thing.

• The "Release" configuration runs out of memory: https://github.com/SimonAfek/desmume/runs/5106299542?check_suite_focus=true#step:4:1142
• However, the "Release Fastbuild" configuration works fine: https://github.com/SimonAfek/desmume/actions/runs/1811639765

[rofl0r]

@SimonAfek cool! did you test the resulting artifact ? there's some warning spam about int vs void* in arm_jit.cpp which looks quite suspicious, as if the proper x64 configuration wasn't properly detected.

[SimonAfek]

I played the first few minutes of an Ace Attorney game with no issues. The AppVeyor builds have the exact same thing going on: https://ci.appveyor.com/project/zeromus/desmume#L64

I haven't checked, but I assume the same is probably true if you manually build in Visual Studio.

I don't know what the performance ramifications are of the fastbuild config vs. the normal one, though! That's way out of my wheelhouse.

[rofl0r]

I played the first few minutes of an Ace Attorney game with no issues.

with jit enabled ?

[SimonAfek]

"Dynamic Recompiler" checkbox in the emulation settings, right? In that case, yes.

[zeromus]

You will have to figure out how to get it to use the 64bit build tools in order for the release build to work without running out of memory, probably. See here, "To workaround this issue, please use the x64 hosted toolset or use /cgthreads1 to disable multithreaded compilation."
https://developercommunity.visualstudio.com/t/fatal-error-c1002-compiler-is-out-of-heap-space-in/653301
Typically one uses "x64 Native Tools Command Prompt for VS 2019" or similar.
You can "call" the appropriate batch file, which is located at e.g. "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvars64.bat" from another batch file so that the 64bit tools will be selected. there may be other ways, but.. maybe not.

[rofl0r]

i think msbuild GHA recipe does just that. the usually recommended workaround for running out of mem is to use your own VMs to build. also the output of the job logs that "in order to use multiple build jobs, use option /m" or similar. so i'm assuming it already runs at a single thread. is the fastbuild release config much worse than the release one ?

[zeromus]

it's worse enough that we need to fix it. I doubt it's running on a single thread. I don't understand your rationale for believing that. Your sentence about option /m was actually garbled and doesn't make sense.

[rofl0r]

https://github.com/TASEmulators/desmume/runs/5112100739?check_suite_focus=true

if you expand the "build solution" dropdown, one of the first messages is:

Building the projects in this solution one at a time. To enable parallel build, please add the "-m" switch.

hmm, that might actually refer to projects, not translation units.

trying with cgthreads option now -> still fails, but much later: https://github.com/TASEmulators/desmume/runs/5115562582?check_suite_focus=true

as you can see from time elapsed, it did indeed only use a single thread now.

[zeromus]

This problem is caused by single files with massive templates. Even if building with one thread fixed it, it might not be reliable. We need to figure out how to use the 64bit build tools.

[SimonAfek]

If I'm reading this right, rofl0r is correct that the setup-msbuild recipe sets you up to use the 64-bit build tools when you specify x64 in the msbuild-architecture variable: https://github.com/microsoft/setup-msbuild/blob/HEAD/dist/index.js#L1094

The job log also supports that - it calls HostX64\x64\CL.exe. Here: https://github.com/SimonAfek/desmume/runs/5115985727?check_suite_focus=true#step:4:72

And setting /cgthreads1 still gives the same error, unfortunately. I made a separate "ReleaseST"-config based on "Release" and gave it that parameter - no dice: https://github.com/SimonAfek/desmume/runs/5115985727?check_suite_focus=true#step:4:537

[zeromus]

try using vs2019 instead, there was no need to switch to 2022

[SimonAfek]

Same thing: https://github.com/SimonAfek/desmume/runs/5116408441?check_suite_focus=true#step:4:534

[rofl0r]

i've tried various things in the pr, no dice. vs 2019, wind 2019, single thread. the windows VM has 7GB of ram which should be plenty. reading this apache/mxnet#13958 it would appear that the "Release" configuration uses some insane optimization levels such as LTO or PGO, which probably should be turned off. or maybe the failing files can be refactored to use less template tricks.

[rofl0r]

another option might be to add a 32 bit x86 target, which would lead to a much slimmer binary and compilation process. this is actually what i tried first, before i realized there isn't even a target in the .sln. x86 build has the additional advantage that it works on all windows systems, and even wine.

[SimonAfek]

the windows VM has 7GB of ram which should be plenty

Counterpoint: This is what task manager's Visual Studio entry looks like during a build on my machine

[zeromus]

x86 was nuked because of msvc code generation bugs related to the wifi module. I don't plan on readding it, although if you dug up the issue on github here where I decided to do it, and determined that the bugs were fixed on a more recent vs2019 compiler or maybe even a vs2022, then I might bless restoring the x86 configuration

I've seen vs2019 using up to 17GB of memory linking the x64 version (but I never had it set to run on a single thread). I wouldn't be surprised if 7GB is too little

What are our options for getting more ram?

[rofl0r]

What are our options for getting more ram?

3 options afaict:

• host own github-compatible-and-connected cloud server
• paid github account for builds
• using a macos X vm instead as those are equipped with 14 GB. with sufficient trickery, we might be able to run a windws VM inside the OSX vm. // edit or msbuild and msvc via wine

apart from that, it seems the main culprit here is the incredibly resource-hungry microsoft compiler. might make sense into looking into the use of mingw-w64 instead. or just decrease the optimization level a little for the automated build. a super-highly-optimized build could now and then be uploaded manually, but i think it shouldnt matter a lot if nightlies run 3% slower.

[zeromus]

OK, those options all suck.
I dont think there's good odds x86 will work any better. The templates are still there.
Can you try setting the project to use clang (visual studio offers that as an option now)? And use thin LTO if possible. This will be better than setting it up to build with mingw-w64, which.. I would say, it will be a godawful mess due to the conflation of WIN32 and MSC_VER, although I feel like someone cleaned that up in recent years.

[rofl0r]

Can you try setting the project to use clang (visual studio offers that as an option now)?

that's a good idea. i managed to set up the GHA to use it in #503 - however clang seems to be a little more stringent:

..\..\utils\AsmJit\x86\x86assembler.cpp(836,43): error : constant expression evaluates to -1 which cannot be narrowed to type 'uint32_t' (aka 'unsigned int') [-Wc++11-narrowing] [D:\a\desmume-gtk2\desmume-gtk2\desmume\src\frontend\windows\DeSmuME.vcxproj]

we're getting a couple of those in https://github.com/rofl0r/desmume-gtk2/runs/5119513501?check_suite_focus=true

[zeromus]

1. try manually casting them to uint32_t
2. if this gets too out of control, you may try updating asmjit. but if desmume melts down because it needs an old asmjit... then revert to technique 1, which WILL eventually work

[rofl0r]

got this far: https://github.com/TASEmulators/desmume/runs/5120381221?check_suite_focus=true#step:4:11666

some string guid concatenation stuff in directx headers, i dont even know where they come from.. at the beginning of the build some zips are unpacked, like "Processing archive: directx\directx.7z", it maybe that those aren't compatible with clang in this version.

[zeromus]

I'll try to figure out what to do about that but youll have to give me a couple of days.

[boundingbox]

Thanks to everyone for the hard work, y'all probably don't hear this enough but you're appreciated. Maybe you'll see a commit from me someday once I've got enough cpp under my belt. I'm already learning a lot because of this project's code so I'm really appreciative of the work here.

[rofl0r]

in case it helps, i once came across this set of fixed microsoft headers and libs for use with pellesc ANSI C 99/11 compiler, they should be compatible with clang too. at least i've seen commits fixing GUID-specific parts in some d3d headers.

[zeromus]

just pushed a commit which might work. give it a shot

[Maarcis12]

Still nothing, the same error message.

[rofl0r]

fails here now: https://github.com/TASEmulators/desmume/runs/5134349422?check_suite_focus=true

[zeromus]

tried to fix it another way

[SimonAfek]

Dies here now: https://github.com/SimonAfek/desmume/runs/5137296808?check_suite_focus=true#step:4:12515

oamView.cpp(222,30): error : non-constant-expression cannot be narrowed from type 'unsigned int' to 'LONG' (aka 'long') in initializer list [-Wc++11-narrowing] [D:\a\desmume\desmume\desmume\src\frontend\windows\DeSmuME.vcxproj]
oamView.cpp(222,47): error : non-constant-expression cannot be narrowed from type 'unsigned int' to 'LONG' (aka 'long') in initializer list [-Wc++11-narrowing] [D:\a\desmume\desmume\desmume\src\frontend\windows\DeSmuME.vcxproj]

[zeromus]

more errors fixed

[rofl0r]

succeeded finally: https://github.com/TASEmulators/desmume/actions/runs/1826091610

anyone can try the attached build artifact whether it has expected performance of full release build ?

[zeromus]

as far as I'm concerned it's on par (tested file menu in golden sun dark dawn with all defaults, got ~122 fps in both cases) and made sure the dynarec still worked. I don't care if it's slower in any single case, as long as it's not an unambiguous regression (and it's not).

One more thing to try before we call this case closed. Can you fix it to run a batch file after the msbuild is done? can be nothing but "echo hello world". I want to make sure we can do that so we can move the msbuild command to there if needed and so we can do some packaging of additional stuff later if needed (lua dlls, etc). Since nobody's left around who cares enough to do PGO runs, we might just start releasing builds made by appveyor. Would improve the release cycle time, after the next release is done, that is. I am under heavy pressure from people I care about to make a new release (pressure from people I don't care about doesn't exist)

[SimonAfek]

Purely out of curiosity, does anyone have any idea why the clang builds are so much smaller? They're like 6 MB instead of 40.

E: Also, @rofl0r, you might consider going back to setup-msbuild v1.1 instead of v1.0.3. I assume you downgraded just to see if it would solve the memory issues, but 1.0.3 doesn't seem to support the msbuild-architecture variable that allows for use of the 64-bit build tools:

[rofl0r]

@zeromus done, the requested postbuild script executes successfully now https://github.com/TASEmulators/desmume/runs/5157234725?check_suite_focus=true#step:4:13508

does anyone have any idea why the clang builds are so much smaller? They're like 6 MB instead of 40.

i don't. maybe clang-cl is just a lot more efficient than cl ? it is at least when it comes to memory use. i guess you just extract a previous 40 mb and the new artifact and copare them side by side. are some files missing? are all binaries smaller ? or is it just a single one ?

E: Also, @rofl0r, you might consider going back to setup-msbuild v1.1 instead of v1.0.3. I assume you downgraded just to see if it would solve the memory issues, but 1.0.3 doesn't seem to support the msbuild-architecture variable that allows for use of the 64-bit build tools:

this was needed to be able to choose an MSVC version < 17. the newer action doesn't provide 16.x gen MSVCs from what i can tell.

[SimonAfek]

this was needed to be able to choose an MSVC version < 17. the newer action doesn't provide 16.x gen MSVCs from what i can tell.

I think which version of MSVC/VS you get depends on which windows image you use. windows-2019 / windows-latest has VS2019 (16.x) installed, while windows-2022 has VS2022 (17.x).

[zeromus]

clang builds are smaller because they instantiate fewer specialized templates. this may reduce performance by 0.001%. It's a matter.

OK, I guess we're done. What URL should I put on the download pages and stuff?

[rofl0r]

first pr needs to be merged (i'll squash it intop a single commit and push a new PR, done - it's #505)

[rofl0r]

@SimonAfek do you have an idea how to get a link that always points to latest build artifact of master branch ?

[SimonAfek]

Not sure about linking to latest, but this search will filter so that only windows builds from master are shown: https://github.com/TASEmulators/desmume/actions?query=workflow%3A%22DeSmuME+Windows+build%22+branch%3Amaster

Then it's just a matter of clicking on the top result

[rofl0r]

@theCapypara - since you added the github action for and the interface code to use for your py-desmume project - (which surprisingly seems to use a fork of desmume) - did you find a good way to link to the latest interface build artifacts ?

[zeromus]

looks like this is gonna work
https://nightly.link/TASEmulators/desmume/workflows/build_win/master/desmume-win-x64.zip
https://nightly.link/TASEmulators/desmume/workflows/build/master/macos.zip

is everyone satisfied now?

[theCapypara]

Hi, yeah, nightly.link is the best way.