Mikage fails to access titles in emulated NAND even when they are provided

[Gimzie]

I have dumped all titles from my 3DS system and copied them to ~/.local/share/mikage/data, both in that directory (as generated), as well as a title subdirectory, but Mikage fails to access system titles anyway. Additionally, I have tried placing the files in /usr/share/mikage/nand and /usr/share/mikage/nand/title which also do not work.

I have attached a screenshot below of the error that appears. If it helps, I'm running Fedora 41.

[neobrain]

Hey, thanks for giving it a spin!

Currently Mikage does not implement full title database bookkeeping, so the title files must be renamed to 00000000.cxi. The paths in your filesystem should look like this: ~/.local/share/mikage/nand/title/0004009b/00014002/00000000.cxi.

FWIW note that recent system versions may have some problems, so I recommend sticking to versions 4.0-9.0 for other titles. Then again fixes for newer versions are welcome too :)

Let me know if this helps!

[Gimzie]

Looks like that did the trick! Though, the path should be /mikage/data/.

I haven't renamed every title file yet, and there are some that contain multiple .app's in the same folder, but it does at least get through part of the setup screen. This temporary limitation might be worth mentioning somewhere on the main readme so that others may properly set their environments up...

On a semi-related note regarding titles; we over at Super Mario Bros. Next were recently hoping to implement Mikage support for the game's patcher application (alongside existing support for Citra & Luma3DS) ahead of the emulator's full release, but it's unclear from the initial release whether or not this developer edition of Mikage is currently well suited for this, given the current limitations.

Namely, our requirements are that the patcher application can detect that a supported game title is installed/present, so that appropriate files can be dumped and the patcher itself can chainload the modded game on-the-fly.

It's clear that Mikage is developing nicely, but given that patching on our end relies on robust title support, might it be best to wait for a later milestone? Especially if related mechanisms are prone to changing in the near future. Some clarification on this would be appreciated, given the "developer edition" naming.

[neobrain]

Looks like that did the trick! Though, the path should be /mikage/data/.

Great!

I haven't renamed every title file yet, and there are some that contain multiple .app's in the same folder, but it does at least get through part of the setup screen. This temporary limitation might be worth mentioning somewhere on the main readme so that others may properly set their environments up...

Yeah, the readme needs a minor overhaul given some recent changes, but I didn't want to delay the repo publication just because of that.

Namely, our requirements are that the patcher application can detect that a supported game title is installed/present, so that appropriate files can be dumped and the patcher itself can chainload the modded game on-the-fly.

Very interesting. Could you give me a specific rundown on what exactly the process for this looks like on the real system? Are you using Luma's built-in mod feature and/or any other Luma-specific features / kernel extensions?

On a semi-related note regarding titles; we over at Super Mario Bros. Next were recently hoping to implement Mikage support for the game's patcher application (alongside existing support for Citra & Luma3DS)

As a general rule of thumb, emulator-specific tweaks are discouraged. I know that some of Citra's limitations may leave you little other option, but for Mikage it's preferable to just do whatever you do on the real system (and doing anything else has a chance of just making things worse in the long run).

[RicBent]

Hey there!

Very interesting. Could you give me a specific rundown on what exactly the process for this looks like on the real system? Are you using Luma's built-in mod feature and/or any other Luma-specific features / kernel extensions?

No, we are not using Luma's or Citra's. We wish to make this a truly frictionless experience for our users and found that all existing solutions cause users to do more things than necessary and/or were too limiting for us.

Our mod launcher does this on real hardware to get to the required game files:

1. Launch swkbd with PMLAUNCHFLAG_QUEUE_DEBUG_APPLICATION
2. Override it with a custom applet that takes action later, then run it
3. Main launcher app chainloads New Super Mario Bros. 2 via apt, then exits (*)
4. swkbd dumps launched NSMB2 code and injects a romfs dumper app, runs it, then swkbd exits
5. Dumper app extracts required romfs files to run our mod, then chainloads our launcher again via apt
6. Launcher diff-patches code/romfs files

(*) This part uses Luma's PMDBG extension to force debug the next process so I can intercept the launch of NSMB2 properly. There should be ways to do this part without that part however.

To launch the mod itself:

1. Launcher relaunches itself via apt
2. On entry it replaces its own code with the patched NSMB2 code (with patches to redirect the romfs reads to the sd card)
3. Jump back to NSMB2 entry

This entire process would obviously not work on citra as it does not emulate any processes besides the game. So sadly emulator specific tweaks sadly are required:
On citra one can directly access the ncch archives of installed titles so the insane chainloading to get to the game's files is not required. Also citra doesn't support apt chainloading so I just use NS reboots there.

Considering Mikage emulates all sysmodules, all aspects minus the PMDBG extension we used should work. I will give it a try when I get mikage fully running.

but for Mikage it's preferable to just do whatever you do on the real system (and doing anything else has a chance of just making things worse in the long run)

An extension to svcGetSystemInfo to identify mikage and its build version, similar to citra's/luma's, might be helpful to identify what environment homebrew is dealing with.

[RicBent]

Also related to the main issue, I am getting an out of bounds read error for data/00040138/00000002/content/00000000.cxi. Considering this is one of the supplied replacement titles, I am not quite sure how to work around this issue.

I am running mikage like this with eu Super Mario 3d Land
./source/mikage sm3dl_eur.cci --enable_logging --bootstrap_nand --launch_menu

> Creating view 44 at offset 45056
Creating view 45 at offset 46080
Creating view 46 at offset 47104
Creating view 47 at offset 48128
MemoryManager allocated new block at 0x26c00000 with 0x1000 bytes; 0x13ff000/0x1400000 bytes remaining (owner 0x137ea8d78)
MemoryManager allocated new block at 0x26c01000 with 0x1000 bytes; 0x13fe000/0x1400000 bytes remaining (owner 0x137ea8d78)
MemoryManager allocated new block at 0x26c02000 with 0x1000 bytes; 0x13fd000/0x1400000 bytes remaining (owner 0x137ea8d78)
[17:20:42.006] [OS] [info] FakeBootThread (id 6), N3HLE2OS10BootThreadE: Dispatcher entering
[17:20:42.006] [OS] [info] Attempting to open "/Users/bent/.local/share/mikage/data/00040138/00000002/content/00000000.cxi" (create=false)
[17:20:42.006] [OS] [info] Found section at post-exefs-header offset 0x0 with 0xef000 bytes (ExeFS at 0xc00)
[17:20:42.006] [OS] [info] Opening FileView at offset=0xc00 for 0xef000 bytes of data
[17:20:42.006] [OS] [info] Attempting to get size of file "/Users/bent/.local/share/mikage/data/00040138/00000002/content/00000000.cxi"
[17:20:42.347] [FRONTEND] [error] SDL_WINDOWEVENT_EXPOSED
[17:20:42.358] [FRONTEND] [error] Attempted to open FileView at range 0xc00-0xefc00 exceeding the original file size 0xef000 (result 0x0)
 #0 0x000102cf12b4 in Mikage::Exceptions::Invalid::Invalid<unsigned long long&, unsigned long long, unsigned long long&, unsigned int&>(char const*, unsigned long long&, unsigned long long&&, unsigned long long&, unsigned int&) (sp=0x0001182c7190)
 #1 0x000102cef598 in HLE::PXI::FS::FileView::Open(HLE::PXI::FS::FileContext&, HLE::PXI::FS::OpenFlags) (sp=0x0001182c7270)
 #2 0x000102c4f790 in HLE::OS::BootThread::Run() (sp=0x0001182c72d0)
 #3 0x000102c350f8 in _ZN5boost7context6detail11fiber_entryINS1_12fiber_recordINS0_5fiberENS0_21basic_fixedsize_stackINS0_12stack_traitsEEEZNS_11coroutines26detail14push_coroutineIvE13control_blockC1IS7_ZN3HLE2OS22CoroutineThreadControlC1IZNSF_6ThreadC1ERNSF_7ProcessEjjE3$_0EE (sp=0x0001182c7cb0)
2024-12-29 17:20:42.780 mikage[22488:6248041] +[IMKClient subclass]: chose IMKClient_Legacy
2024-12-29 17:20:42.780 mikage[22488:6248041] +[IMKInputSession subclass]: chose IMKInputSession_Legacy

It is not quite clear to me what titles need to be dumped. The readme suggests everything just comes from game update partitions?

[neobrain]

Also related to the main issue, I am getting an out of bounds read error for data/00040138/00000002/content/00000000.cxi. Considering this is one of the supplied replacement titles, I am not quite sure how to work around this issue.

I'll need to ponder on your other comments, but FTR I think you can safely comment out the error from pxi_fs_host_file.cpp.

It is not quite clear to me what titles need to be dumped. The readme suggests everything just comes from game update partitions?

The readme is currently missing one required step: After bootstrap, you need to copy the folders from <build>/nand_archives to ~/.local/share/mikage/data (where all the other title folders like 00040130 are located). Other than that, you don't need anything else.

[RicBent]

After commenting out that error I'm getting the following:

Not quite sure where that one is coming from as the built-in exception handler doesn't halt lldb.

[RicBent]

After commenting out some try/catches, it seems to come from HLE::OS::CoroutineThreadControl::ResumeFromScheduler(), so probably unrelated to the other stuff here.

(lldb) bt
* thread #6, stop reason = signal SIGABRT
  * frame #0: 0x0000000196aaa600 libsystem_kernel.dylib`__pthread_kill + 8
    frame #1: 0x0000000196ae2f70 libsystem_pthread.dylib`pthread_kill + 288
    frame #2: 0x00000001969ef908 libsystem_c.dylib`abort + 128
    frame #3: 0x0000000196a9944c libc++abi.dylib`abort_message + 132
    frame #4: 0x0000000196a87a24 libc++abi.dylib`demangling_terminate_handler() + 320
    frame #5: 0x000000019672d3f4 libobjc.A.dylib`_objc_terminate() + 172
    frame #6: 0x0000000196a98710 libc++abi.dylib`std::__terminate(void (*)()) + 16
    frame #7: 0x0000000196a986b4 libc++abi.dylib`std::terminate() + 108
    frame #8: 0x0000000196a08910 libc++.1.dylib`std::rethrow_exception(std::exception_ptr) + 24
    frame #9: 0x0000000100095984 mikage`HLE::OS::CoroutineThreadControl::ResumeFromScheduler() + 128
    frame #10: 0x000000010008ae98 mikage`HLE::OS::OS::EnterExecutionLoop() + 1040
    frame #11: 0x000000010008d13c mikage`HLE::OS::OS::Run(std::__1::shared_ptr<Interpreter::Setup>) + 4148
    frame #12: 0x00000001000bebe4 mikage`EmuSession::Run() + 60
    frame #13: 0x00000001000e50b4 mikage`void* std::__1::__thread_proxy[abi:ne180100]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, main::$_2>>(void*) + 56
    frame #14: 0x0000000196ae32e4 libsystem_pthread.dylib`_pthread_start + 136

[neobrain]

An extension to svcGetSystemInfo to identify mikage and its build version, similar to citra's/luma's, might be helpful to identify what environment homebrew is dealing with.

I understand the concern, but this isn't going to happen. In my experience, such emulator-specific hacks help for a short while but then introduce new layers of problems in the long run (when the original problem isn't even there anymore). Sadly this is a consideration that no one even brought up when the feature was added in Citra, but it's a mistake that won't be repeated here.

Also related to the main issue, I am getting an out of bounds read error for data/00040138/00000002/content/00000000.cxi. Considering this is one of the supplied replacement titles, I am not quite sure how to work around this issue.

I'll need to know the exact steps you went through to bootstrap your NAND to reproduce this. SM3DL ran fine on my end last time I checked. EDIT: Just realized you literally just bootstrapped from SM3DL. I'll need to retest this on my end.

Our mod launcher does this on real hardware to get to the required game files:

That's an impressive level of voodoo that I'll assume you indeed have strong enough reasons to prefer. I don't see any major problem with any of these other than the Luma-specific PMDBG extension you mentioned (as long as you invalidate code caches appropriately).

[neobrain]

After commenting out some try/catches, it seems to come from HLE::OS::CoroutineThreadControl::ResumeFromScheduler(), so probably unrelated to the other stuff here.

Actually it's just re-throwing an exception thrown elsewhere. Setting an exception breakpoint in gdb via catch throw should help (though note that it'll also catch FS exceptions that are fine and handled internally).

[neobrain]

Also related to the main issue, I am getting an out of bounds read error for data/00040138/00000002/content/00000000.cxi. Considering this is one of the supplied replacement titles, I am not quite sure how to work around this issue.

I found the issue: It should be fixed in c06c249 and obsoletes the workaround in pxi_fs_host_file.cpp. You'll need to run bootstrap again (ideally removing ~/.local/share/mikage/ first.)

I think this should fix all the issues you had, let me know if there are any other problems!