Updated ScummVM core based on latest sources available if you want it

[diablodiab]

I've made a new updated version of the core based on the existing libretro wrapper and the newest ScummVM sources. There are also some adjustments in the approach that completely separates the wrapper from the original source, hopefully making it easier to keep the core in sync going forward.

If this is of interest you can find it here:
https://github.com/diablodiab/scummvm

The libretro-wrapper is added in one commit on top of the base source.

You are more than welcome to incorporate or use it for inspiration towards the official core if you want to.
Just wanted to share.

[hizzlekizzle]

oh, that's very exciting, actually! Thanks for doing this work.

[dhwz]

@diablodiab
Can you tell me how to build it properly for aarch64?
I always get an compile error.

make: *** [Makefile:559: engines/ags/engine/ac/characterinfo_engine.o] Error 1

[diablodiab]

You can try commenting out "ENABLE_AGS" line in config.hd to see if it might be an issue with the AGS engine for the platform you are compiling for. Out of curiosity, what device are you trying to compile it for?

[dhwz]

Oh yes commenting out helped great thx!
I tried to disable it by setting it to 0 but that didn't help. :-)

I'm building it for 351ELEC (Anbernic RG351).

[dhwz]

@diablodiab
Another question maybe you have an idea, saving core overrides doesn't work (with your version and also the old version).
"Error saving overrides"

[diablodiab]

I'm not sure this is related to the core or to RetroArch itself. I get the same error with many other cores at the moment.

[dhwz]

Yes that was also my guess that it is RetroArch related but some user doesn't believe me.

[DrUm78]

Amazing work, thanks! I compiled it successfully for the FunKey S (after disabling AGS that failed too). :)

[DrUm78]

@diablodiab Here is the error message I get when trying to compile the Libretro core with your wrapper with last sources from ScummVM repo (DrUm78@4aa78f2):

Compiling libretro_os.cpp...
../../../../backends/platform/libretro/libretro_os.cpp: In function ‘OSystem* retroBuildOS(bool)’:
../../../../backends/platform/libretro/libretro_os.cpp:1271:45: error: invalid new-expression of abstract class type ‘OSystem_RETRO’
 1271 |    return new OSystem_RETRO(aEnableSpeedHack);
      |                                             ^
../../../../backends/platform/libretro/libretro_os.cpp:288:7: note:   because the following virtual functions are pure within ‘OSystem_RETRO’:
  288 | class OSystem_RETRO : public EventsBaseBackend, public PaletteManager {
      |       ^~~~~~~~~~~~~
In file included from ../../../../backends/base-backend.h:26,
                 from ../../../../backends/platform/libretro/libretro_os.cpp:33:
../../../../common/system.h:1369:15: note:     ‘virtual void OSystem::getTimeAndDate(TimeDate&, bool) const’
 1369 |  virtual void getTimeAndDate(TimeDate &td, bool skipRecord = false) const = 0;
      |               ^~~~~~~~~~~~~~
make: *** [Makefile:564: ../../../../backends/platform/libretro/libretro_os.o] Error 1

The last working commit for me is DrUm78@10310cc. Seems like the ScummVM team changed something in the code that does not work anymore with your wrapper.
My repo: https://github.com/DrUm78/scummvm

Any way to fix it on your side? Thanks!

[diablodiab]

I've merged with base a couple of times since the first post here. I think I fixed this specific issue in this commit on my repo:
diablodiab@b4064be

[diablodiab]

Nvm, I see changes have been made yesterday that causes your the issue you are having. You need to apply this update to the getTimeAndDate function libretro_os.cpp:
scummvm@a0c237f#diff-736a7d91012aa31c738fe94bc53f4443daf2a2f6bee4aa623c8ca2c73360e50f

[DrUm78]

Amazing, that works, thank you for your very quick help and your great work. 🥇

[mrmatteastwood]

Nothing to contribute here, but just wanted to say: freaking brilliant. I really hope your work makes it into the official RA repos, it would be amazing to have an up-to-date ScummVM core!

[hizzlekizzle]

I compiled this and did some admittedly brief testing, and everything that I tried seemed great. I did have to disable AGS (linux x86_64) to get it to compile, but that was the only hitch.

The current scummvm core is newer than I thought (it was apparently updated circa 2018 vs the 2011 I was thinking), but the git history is completely buggered, so I think switching over to this would be a good idea anyway.

[mrmatteastwood]

Would one of you fine folks be willing to share a build of this for Linux? I'm no good at building stuff myself, but I'll be more than happy to do testing.

[DrUm78]

@hunterk Your file was deleted it seems.

[hunterk]

weird. oh well. deleted the post, too /shrug

[DrUm78]

I deleted mine too as it has a few perks for my targetted device (240x240 resolution and some other changes) so that won't apply for a generic Linux distro.

[DrUm78]

@mrmatteastwood Here is the 2.3.0 core for Linux from last diablodiab's sources (sources are 3 days old only) as my repo has too many changes for another platform:
https://www.mediafire.com/file/gov4s3a1ld0deqo/scummvm_libretro.zip/file

I did not test it but should work as it does for my ARM device. Binary information is:
scummvm_libretro.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=bfaea00c56b3f3c4817413e5f972a7ee0225aeb8, stripped

It has all the engines, except AGS (because of the compilation error). Enjoy.

[diablodiab]

I don't use Linux that much myself, but I've also compiled a Linux version of the core now:
http://lindqvist.synology.me/linux/x86_64/scummvm_libretro.so.zip

This one includes the AGS engine also.

[mrmatteastwood]

I don't use Linux that much myself, but I've also compiled a Linux version of the core now:
http://lindqvist.synology.me/linux/x86_64/scummvm_libretro.so.zip

This one includes the AGS engine also.

Thanks so much for your replies, guys! I am going to use this one and will test it with the games I have over the coming days, running RA 1.6.0 on Linux Mint 20.2.

[mrmatteastwood]

Hey @diablodiab, is it possible to report bugs directly in your GitHub repo? I don't see an "Issues" tab. If not, I can report the issues here.

[diablodiab]

I've enabled issues on my repo now, just to keep things a little separated for now.

[mrmatteastwood]

I found 2 bugs so far, though I have to say one seems quite specific, so not sure how much you can do about them. Either way, perhaps this helps a little to refine your new version of this core.

That said, I tried a number of other games, including the Gobliiins series, some Lucas Arts adventures, Broken Sword. Didn't play them very far, but so far, no issues, all seems stable and reliable. The animations in Broken Sword felt a little slow and the mouse cursor felt a little laggy, but I seem to recall that's how this game always was, so I don't think it's a bug.

[i30817]

It's not surprising, new engines don't often change the rest of the code nowadays.

I think that upstream is not up to date in having all the checksums to all AGS < 3.5 games yet though (and may never will since they're thousands even not including demos, patches and incomplete 'games').

Oh and i wanted to say 'thank you' for the linux ags enabled core posted here which i locked in my retroarch to keep it until this is merged.

AGS 3.5 which scummvm ported is a moderately good place with only some major games with minor problems in games of the Kathy Rain studio currently.
Only major thing left besides that is getting rid of Allegro and keeping MIDI emulation with samples working and one or two minor games that use the lua plugin (i'd call them demos even), that i think might not happen now since AGS 4.0 is breaking away with compatibility anyway, so if it happens it'll happen on the scummvm repo i think.

[mrmatteastwood]

It's not surprising, new engines don't often change the rest of the code nowadays.

I was thinking the same thing. Glad you went into some more depth and details, this is more than I can do at the moment. You're experiencing the same issue with distorted music when using sound fonts?

[i30817]

I haven't tested ags in the core that much.

The problem it had with sound fonts upstream until recently i was writing about was that Allegro controlled the soundfont location with a config file or 'fell back' to the executable directory or the 'home' directory.

This worked fine in windows, because the 'executable directory' is the same as the game data. It didn't work in linux because people won't ever write config files for libraries, much less libraries they didn't install and are statically linked, and the executable was not in the game dir (libretro core isn't either), and so people were forced to put in the soundfonts in the home dir to have functioning MIDI if they hadn't setup timidity and configured things just right. If a MIDI game came with a soundfont - which is rare because they add 15 mb to the download - they'd never have music and would never know better if they hadn't figured this out (which took me a while).

I'm talking about the 'emulation' allegro midi driver using the soundfont; not sending the MIDI data directly to the soundcard, which frankly speaking never worked right to me. I heavily suspect nearly all soundcards in windows use midi emulation now because that shit hasn't been working in linux without a weird timidity or fluidsynth system configuration almost no one does for literal decades, so having the option to configure AGS to use the allegro MIDI emulator directly (probably a old static version of timidity) was a configuration blessing to make the games move across computers with a minimum of breakage.

This location bug was fixed upstream though, now the engine will pickup soundfont files in the same directory as the game data files.

I did notice a bug where the fonts are broken (transparency artifacts) in several games, which i suppose is due to scummvm ripping out whatever library the original engine uses for font handling for theirs. This is going to be a 'problem' for more stylized games even after that is fixed i think.

[i30817]

Yo, I don't know if you're interested in it, but early macromedia games just got added to the main build upstream. This is not that interesting yet, but cosmology of kyoto and frankenstein through the eyes of the monster may be supported soon.

[hizzlekizzle]

this one builds differently. we need to work out how to integrate it with our existing CI infrastructure. That's part of why I created my github actions recipe.

None of our existing cores have to fetch upstream, fetch the backend, add the backend to the upstream code and then build.

[mrmatteastwood]

Just a quick note, I realized I had reported the sound dropout issue on the wrong repo. Copied it to the right one now (which is also where diablodiab is active :-)).

[diablodiab]

@Boardscanada and Hunterk:
I was told that you wanted to talk with me about getting the libretro-scummvm-backend on the buildbot. At the moment, I have very little time due to work related issues, so it's difficult to prioritize, but...

The work I did with restructuring the ScummVM libretro backend was always intended as a contribution to the official work on the libretro core, for you to adopt if you wanted it. I don't have time to maintain it myself at the moment, so the ideal scenario for me would be if you want to to create a new repo based on the current state of the libretro-backend and allow others to take over and make contributions and improvements here in a more official repo.

If this is what you wanted to talk about, then by all means please proceed :)

[LibretroAdmin]

Sure, make that repo, then transfer it over to me so I can move it to the libretro org. From there, I will give you admin access over it as well. So you won't lose anything in the process. And we'll have an easier time gathering contributors that can contribute to it.

[schmurtzm]

That's an awesome news ! :) Very happy to see that it will happen !
Last month I have worked on libretro ScummVM version for Miyoo Mini (a little linux handheld with low cpu) I've tested and compared the official core and the one from diablodiab. I can tell that ScummVM from Diablodiab is awesome : it make possible to play a lot of games in good condition and it will be a lot better for following the rythm of development of ScummVM (which is quite impressive).

During my exploration I've discussed with the ScummVM team to clarify some points on the way how the ScummVM core is working, they don't support the libretro version at all but they've given me some explanations. So I've discovered some interesting points that I'll expose quickly here for the future maintainer of the core. It's a little anticipated and it will deserve dedicated issues of course but I have to expose it 😄

Two essential points (which are related to the current libretro core and the future diablodiab's core) :

• The most important : the way how the scummvm core runs the games is wrong from the beginning 😅 . The shortcuts with .scummvm files runs a command line that will never use the configuration file (scummvm.ini) from scummvm as it should. For example the consequences are that if you run a game like grim Fandango, the flag which indicate that the game files have been checked will never be set and at each start it will check all the files (long process). Futhermore I emphasize the fact that creating .scummvm files is a complex and long task for the user (they have to check an external website with compatibility list). So it must change to use the powerful game recognition included in ScummVM. There is two easy steps to make it simple and more efficient :
  - use scummvm -p "$gamedir" --add --recursive to detect and add all the present games in the scummvm.ini file
  - use a new shortcut file which contains the name of the game section of the scummvm.ini.
   
  There are numerous advantage to this method :
  - ScummVM works as it should by using its internal game recognition
  - The creation of the shortcuts and the import of games can be automated
  - The ScummVM main GUI (called launcher) can be used with this configuration, it allows to configure game settings
   
  All the information and details about this are in this issue : The command line use in ScummVM Core does not allow to load a game configuration correctly diablodiab/libretro-scummvm-backend#24, we already use the new core in this way with our Onion's community and the users are really happy of the way it works because it transform the experience of adding games for the user in something very easier than the initial way.
   
   
• The second point is about the audio stack which is not as good than the standalone version at all. On low powered devices you'll have quickly sound stuttering on games like Broken Sword 2, Gim, Blade Runnner.... I've made a workaround here which improve a lot the result and allows to play the previous mentioned games on the Miyoo Mini (which is an achievement!) but I think that the core deserve a new sound stack which is a bit beyond my area of expertise 😅 . An issue about this is here.

For now the core from diablodiab can be used as is because it already improve a lot the game compatibility compared to the current libretro core but if in the future these 2 points can be taken into account it will be awesome. I'm fully available to help if you need more details, you can find me here or on Retroarch Discord and ScummVM Discord).
Looking forward to see diablodiab's core available for everyone 😀 !

[i30817]

For the previous post i use this program to do it for me after adding all the stuff in the core manually (from starting the core from the core menu):
https://github.com/i30817/libretro-mkscumm

I don't like to add recursively without checking because the scummvm detector is not perfect and will introduce 'bogus' games even with all the data files correctly named and positioned (which sometimes needs changes from the original game files). Especially if you use the mass add function, which is the only real way not to spend a lot of time.

Then after removing all the buggy entries, and using my program with those games added, the ids inside the .scummvm files that are created are the correct ones instead of the 'generic' ones that the libretro readme mentions, have the real game name as understood by scummvm, as far as possible (forbidden windows characters removed) and also the playlist(s) have correct names for the games (and i can make other playlists for the same core easily too, based on the filepaths), because the label json property is what is used in RA, not the filename.

Since i've also been adding the games i have to the libretro-thumbnails scummvm project, the thumbnails mostly appear, and most of those that don't can be adapted with my other project https://github.com/i30817/libretrofuzz

Although there are some engines where i'm not adding thumbnails yet (AGS, some others). More people contributing would eventually close this gap.

Personally speaking the most annoying thing about these cores (either) is the lack of hardware acceleration. Running any game on these cores compared to running them on upstream will noticeably turn up a laptop fan, if yours has a fan. Much worse if the game has 3d elements. That may be one of the causes of the 'audio stuttering' even.

[schmurtzm]

For the previous post i use this program to do it for me after adding all the stuff in the core manually (from starting the core from the core menu): i30817/libretro-mkscumm

I don't like to add recursively without checking because the scummvm detector is not perfect and will introduce 'bogus' games even with all the data files correctly named and positioned (which sometimes needs changes from the original game files). Especially if you use the mass add function, which is the only real way not to spend a lot of time.

Then after using my program with those games added, the ids inside the .scummvm files are the correct ones instead of the 'generic' ones that the libretro readme mentions, and also the playlist(s) have correct names for the games (and i can make other playlists for the same core easily too, based on the filepaths).

Generally, i'm unsatisfied with how the scummvm core handles adding games for another reason: it requires a additional file in the game directory.

Not only is this a bad idea if your games are zipped or something, it's also completely unnecessary now that the playlist fileformat is json. The 'id' of the section in the scummvm.ini file could be in the playlist as a new filed in the game entry, and the path pointing to a directory instead of the .scummvm file.

I don't know if this has to be implemented by libretro or the core, but it should be to finally get rid of the need for .scummvm, if not for the need to add manually to the scummvm.ini file, which as mentioned, is necessary for correct function.

Doing it this way exclusively would screw up the manual or automatic scanner, so i understand why it wasn't implemented, but i'd like the opinion, for projects like mine that aren't completely generic and actually use the scummvm.ini file as it's supposed to be used.

I agree on this point : the creation of additional files shouldn't exist : to list games in the frontend ideally it should read the scummvm.ini file. But it's not the way how the frontend work (and it is the same for all retrogaming software) : they need a file with a specific extension to launch a game.

From my point of view the scummvm detector must be use by the users : this is efficient and easy way. If you see buggy entry it deserve an issue on the ScummVM repo to be fixed and not a workaround by making it manually. This is the way to improve it.
Obviously it is constraining but we are facing a special case (ScummVM is not an emulator and it is a multi-egnines software embedded in a multi core software 🤪 ).

The most important here is not how to add the games, you can do that as you want from a command line or from the Launcher UI of ScummVM BUT after that the game must be launched in a way to use this entry of the scummvm.ini file. And it's not the case of the .scummvm shortcut files for now.

[i30817]

I removed that part of the comment (and added some others).

The reason i removed it is that the files actually have a 'advantage' and it doesn't actually cost me much to add them and use that method.

The advantage is the the RA scanner and manual scanner understands them. I know 'you can always create them again' and that's what i do, but if you delete/move a game and don't want to go through the annoyance of using libretro-scummvm-playlist again, you can just use the playlist management to 'rescan' a playlist base directory and it will mostly work (the names will be screwed, remember my program eliminates forbidden windows characters from .scummvm filenames).

So although i think the files are a bad design, it doesn't actually affect my workflow that much and has at least 'some' utility for now.

Doing the files manually though, that's horrible and error prone because of the ids need to match scummvm.ini ids to work 'correctly', especially if you have a near complete or complete scummvm collection. That's the main reason i created libretro-scummvm-playlist anyway (you can install it with pip right now).

[schmurtzm]

Stop to edit your messages, I see it always changing when I write an answer 😅 We have to stay focus on the core here, you are mainly talking about the frontend and the playlists, on my side I talk about how the core is launched currently ;)

[mrmatteastwood]

Just wanna chime in real quick to say BOOYAH and KUDOS to you guys for being on top of this!

An up-to-date ScummVM is at the very top of my wishlist for RetroArch and though I have no programming experience, I'd like to contribute in any way I can, e.g., by testing on Windows and Linux (Mint), or by editing the documentation for ScummVM once the new core is ready. I did the whole doc for DOXBox Pure with @schellingb. » https://docs.libretro.com/library/dosbox_pure/

@schmurtzm mentioned audio issues, let me add that Kyrandia 1 is having a weird stuttering issue where it seems like not only the audio stutters, but the whole game micro-freezes. I have a comprehensive bug report about this here:

diablodiab/libretro-scummvm-backend#27

It happens on both my PCs on Linux Mint and on Windows (on the laptop only, no Windows on the desktop). Issue does not occur with the ScummVM core currently present in the RA repo (2.1).

[i30817]

They're basically one and the same. The playlists point to the (invented) scummvm files to have a 'id' to launch the game.

The main problem you're complaining about is that the users creating the scummvm files inevitably use the wrong ids because they're either not adding the games through the scummvm GUI or because they just don't care and don't use the ids that are in the scummvm.ini file. That program solves this issue if you use it, creating those files for you and a playlist pointing to them after you the the GUI mass add.

For RA to do the same without a external program, it would be ... problematic.

For the first issue, 'creating the scummvm.ini file' it would essentially have to proxy the 'scanner' to not use normal methods when you start scanning a directory for that core, but call the core functions. Not sure this will ever fly. But users can do it manually if they're directed to do so.

For creating the 'scummvm id files' (or the alternative of another json property in the playlist) it would have to hook up when a game is added to the scummvm.ini file (not a easy thing to do without adding code to the core or some non-portable file watching). Maybe when the core closes the scummvm.ini could be scanned to build a playlist 'automatically'.

I mean, it's possible, but i kind of doubt it because the information would be going from 'core->frontend' when usually the api is the opposite.

And to clarify i mean 'I doubt it would be accepted in the project', not that it's impossible. Making the playlist creating being dependent on the core changing scummvm.ini would essentially mean 'giving up' on the scanner for the scummvm core and making the core dictate the playlist creation, so even if it's 'the least error prone way', I'll believe it when i see a PR merged.

[schmurtzm]

You continue to edit your messages all the time 😬

They're basically one and the same. The playlists point to the (invented) scummvm files to have a 'id' to launch the game.

No, you misunderstand. Please read my issue carrefully 😉 I 've just explained that the game should not be launched with his ID like this For example :
The right internal command line to run Grim is : scummvm grim-win and with .scummvm file you will obtain scummvm -p SCUMMVMpath/GRIM grim:grim which is wrong due to the points that I've explained.

[i30817]

That's because you're creating your files wrong, aka you're one of those users that is doing something wrong.

Here is my grim fandango file based on the scummvm.ini which is created by the core after i add the game in the scummvm GUI:

i3@sleipnir:/media/i3/Mordred/Games$ cat ScummVM/grim_fandango/Grim\ Fandango\ \(Windows_English\).scummvm 
grim-win

[i30817]

Besides to launch a game from retroarch manually you must do this:

i3@sleipnir:/media/i3/Mordred/Games/ScummVM/grim_fandango$ retroarch -L scummvm Grim\ Fandango\ \(Windows_English\).scummvm 

retroarch -L corename file

If you try it like that, grim fandango will 'verify' the files the first time, but not the second, which is what you were complaining about.

I have no clue from where you think 'grim:grim' comes, or why would you try to start scummvm instead of retroarch to use a retroarch core.

[schmurtzm]

You're not really aware of how ScummVM work currently... and this conversation must be done outside this issue.
Please read the scummvm documentation to understand from where grim:grim comes. Then you can come on discord to discuss with me and to understand each other well before interfering here more 😅 You can also read my conversation with ScummVM team from here.
I'm not complaining of anything but just give some advice to the future maintener from my experience.

[i30817]

I read it now, and i'm telling you that you that that kind of start initializer will not work in this core.

Only names which exist in the scummvm.ini files are (or rather should be) accepted by the core, not the 'engine:game' variant.

What's the point with that anyway? You can't automate that and without the scummvm.ini entries you're not even sure if you're choosing the right 'game' right-side name (entries that already have the engine anyway).

I guess i'm asking 'why would you even want this'. Are you trying to eliminate the need for the scummvm.ini?

[schmurtzm]

Only names which exist in the scummvm.ini files are accepted by the core, not the 'engine:game' variant.

You're wrong one more time ! If we talk about the diablodiab's core you have to specify 'engine:game' or game like "Indiana Jones and the Fate of Atlantis" will not run. Read this issue to understand. But anyway using the game ID it is not the recommended way at all.
Please stop interfering here, come talk to me on discord (I don't find you), I'm sure that we have a lot to share here once we will be agree on these points ;)

[i30817]

That issue is complete nonsense. I can start fate of atlantis fine with a .scummvm file and if you want a video i'll post it in your issue. But even if we ignore your assertion that the scummvm file doesn't work, you're then complaining that 'engine:game' breaks (a unsupported way to use the core anyway).

This is not surprising! Because ':' is a illegal windows character in command lines (it's the path separator in windows). You have to quote it.

[schmurtzm]

That issue is complete nonsense. I can start fate of atlantis fine with a .scummvm file and if you want a video i'll post it in your issue. But even if we ignore your assertion that the scummvm file doesn't work, you're then complaining that 'engine:game' breaks (a unsupported way to use the core anyway).

This is not surprising! Because ':' is a illegal windows character in command lines (it's the path separator in windows).

😞 That's because your core is not up to date. With an old core there wasn't doublon with atlantis. I understand that you are very invested with your projects on playlists but I will not answer here anymore, it breaks the initial discussion. Come talk to me , we will expose our point of view when we will be agree.

[i30817]

Wrong again. I'm using the diablodiab core. Built it last week actually. And i have better uses of my time than playing tech support for someone that doesn't listen.

For the last time, delete your .scummvm files (the ones added in the games directories), delete your scummvm.ini file, start the core manually from 'start core', mass add all games, and either use my project to create the the .scummvm files and playlist or create them yourself only with the ids that appear between [] in the scummvm.ini file and use the manual scanner of retroarch to scan .scummvm files.

And your games will start. Including fate of atlantis.

[schmurtzm]

In fact that's me who is currently making support for you 😅 ...
But it's fun to understand that finally you do the same as me by hijacking the way how the core launcher work. When you do that in fact you don't use the "game ID" but the section name of scummvm.ini (I agree that most of time they are identical but not always). However I think that the internal command line of the core still wrong because it use the path of the game (-p) which should be retrieved by scummvm itself in the scummvm.ini. Are you agree with that ?

By the way, even if it's not a mandatory at all, to give a maximum of flexibility I think that the core should accept any of the existing command line of scummvm standalone.

It's very a shame to not come to discuss to put our efforts in common because we try to solve the same problematic and have the same objectives...

[i30817]

Scummvm files must be created in the same dir as the entries for the game in the scummvm.ini file. And that's what my program does.

I mean, it seems self evident, because if you moved them, the playlist would have no clue where the game was.

And as mentioned, these cores are not 'flexible' because that would require transferring information from 'core to frontend', ie: the frontend would need to parse the scummvm.ini to find the paths.

Removing the requirement for the .scummvm files would mess up the manual scanner and the normal scanner because they have no way to recognize the games just from paths, but only from extensions and CRCs (although the normal scanner already is 'suspect' because it has a pre-set number of 'possible CRCs' for games, which is already wrong because scummvm will happily add multiple versions of the same game for languages by adding '-number' to the end of the id - use the manual scanner for scummvm and my program i guess).

The current use of .scummvm files is just to connect the games paths to the definitions on the scummvm.ini file. You can try to use 'engine:id' inside those files, but it's pointless, it doesn't make the games 'load better', the flags that scanning the game from the scummvm GUI adds are still missing if 'id' doesn't exist in scummvm.ini in the retroarch system directory, afaik. Even if you put in '--autodetect' inside those files and it worked, it wouldn't exactly be ideal because a game directory can add multiple valid games in scummvm ( different languages or 'enhanced' version for example) and you could easily end up with the 'wrong language' for instance if the game is originally spanish and you only read/understand english but the game has a english translation. In short, the step of creating the scummvm.ini file by scanning games from the scummvm GUI is necessary and using the correct ids for entries on that file in the scummvm files is also necessary.

[schmurtzm]

Globally I agree. Some remarks however :

Removing the requirement for the .scummvm files would mess up the manual scanner ...

I don't ask to remove .scummvm files ;)

the frontend would need to parse the scummvm.ini to find the paths.

• It's already working like that : if you just put the target name (= the "scummvm.ini section name" that appear between []) in the command line (and not the -p) it is enough to run the game without modifying anything in the core. You don't need the -p because the scummvm part of the code already reading the corresponding path= in the scummvm.ini. It's interesting because it means that you can put your .scummvm file anywhere and potentially separate it from the games folders.
• be obliged to put .scummvm has one drawback : it can make the scan long due to certain games has numerous subfolders and files

You can try to use 'engine:id' inside those files, but it's pointless

Normally if you indicate a -p (which is always used like this in the current core because it is designed to use 'engine:id' !) you have to use 'engine:id' as documented on ScummVM website :

=========== screenshot of ScummVM documentation : ===============

==========================

What you do currently is that : scummvm -p <path to game files> <target> which is not really supported by ScummVM : I think that the -p is skipped and overrided by the path= value from scummvm.ini. Agree ?

EDIT : no the -p is not skipped but if we don't mention it, the core will look in scummvm.ini

[asciibeats]

I created an AUR package for my fellow Arch Linux users,

[diablodiab]

Sure, make that repo, then transfer it over to me so I can move it to the libretro org. From there, I will give you admin access over it as well. So you won't lose anything in the process. And we'll have an easier time gathering contributors that can contribute to it.

Done :)

[spleen1981]

I think the "libretro backend" structure (used in the legacy core and in @diablodiab's one as well) makes sense only if the intent is to merge with the project upstream, which I don't think is the case here, otherwise it's just a useless complication.

Starting from the legacy core source, I've reorganized from scratch and improved the code here ScummVM mainline libretro core

• ScummVM main repo is included as a submodule. To sync the libretro core it should be sufficient to update the submodule and build, unless modifications to core source/makefiles are needed due to changes upstream.
• No patch is applied to the submodule
• Submodule updates are committed following ScummVM official releases. First three fields of core tags are relevant to upstream ScummVM version, last is libretro core revision (e.g. v2.6.1.1 is ScummVM 2.6.1, core revision 1).

Makefiles have been reworked and refactored as needed:

• Available engines are retrieved and added automatically at build time from ScummVM submodule source code, no need for manual modifications anymore
• Datafiles and themes bundle (scummvm.zip) and core.info files are created automatically at build time based on current submodule source, no need for manual modifications anymore
• All hard-coded dependencies have been turned to submodules from the proper libretro repository (libretro-deps)
• Hard coded libreretro-common is turned to submodule as well.
• Various other fixes and refactorings

Added also a couple of useful options:

• Virtual keyboard toggle ("select" on retropad). Files bundled in scummvm.zip are needed for this to work.
• D-pad acceleration time

Still lot of cleaning/improvements to do, including better input mapping options, games identification without need for .scummvm files, and direct use of upstream configure script. Those will be investigated later.

Couple of commits have been taken from @diablodiab repo and forks (hence all improvements in those are merged here as well), but following modifications were too heavy to be handled through PRs to the said repos, and needed a separated one.

This core is currently merged in Lakka master and RetroArch Kodi add-on for CoreELEC, feel free to submit PRs as needed.

[hizzlekizzle]

@spleen1981 any chance you're on discord? @LibretroAdmin was hoping to talk to you a bit about how we can get it integrated. I agree with your points about maintainability with the submodule strategy.

[spleen1981]

yes I'm there - spleen1981#3705