macOS: Import File fails to locate and load linked system libraries from dyld cache #4162
Assignees
Labels
Feature: Filesystem
Feature: Loader/Mach-O
Status: Internal
This is being tracked internally by the Ghidra team
Milestone
Comments
|
Thanks I'll take a look. |
|
Do you actually have those dylibs on your system at those paths? I do not. |
|
Indeed, they are now part of the dyld cache: /System/Library/dyld. #3345 says that it should be supported, so I expect Ghidra to resolve these references. |
|
I see. This will be a pretty big change to add support for this. |
ryanmkurtz
changed the title
|
@ryanmkurtz In that case could you elaborate what #3345 have fixed then? |
|
Sure. That ticket was mostly about dealing with the "split" dyld cache files..allowing you to load the entire cache (which is now split into .1, .2, etc) into Ghidra, or open the entire cache as a filesystem. |
|
Regarding this ticket, here is a discussion of the changes to macOS that caused the problem: https://developer.apple.com/forums/thread/655588 |
|
In other words it is possible to load linked libraries and then reference them via External Programs, but at this time I have to do it manually. |
|
Yes that is correct. |
|
When I export any of the revealed binaries from the File System dialog, the resulting binary always has size of ~529,4MB. That does not seem right and differs from https://github.com/keith/dyld-shared-cache-extractor |
|
I am not seeing that behavior. What file are you opening in the file system browser, and what are some example dylibs you are exporting that end up with that size? |
ryanmkurtz
added
Status: Internal
I do this and then re-analyze the module I've loaded. But this has no effect -- links to the external programs are still rendered as Thanks in advance, @Kentzo! I find this bug intensely annoying, and I'm sure I'm not alone. It really should have higher priority. |
|
I did not try this in a while. Have you seen the linked issue? |
|
By the "linked issue" do you mean #4175? I just tried it, and didn't see the problem. Your STR no longer work, so mine were a bit different. And once I'd loaded the TimeMachine framework, I measured the entire size of the By the way, I've kept playing with possible workarounds for this bug. I extracted the entire dyld shared cache to a single directory, then made that directory part of Ghidra's search path for "system libraries", also increasing the search depth. That worked for simple dylibs (like those in If possible, I'd still like an answer to my original question :-) |
@steven-michaud Ghidra's external references only really support navigation convenience to the other imported programs. |
|
Is there a build with this fix that I can download? Or do I need to build Ghidra from source, or wait for the 10.4 release? |
|
You'll need to build the |
|
I'm really trying to improve this area of Ghidra though, so it would be great if you could provide feedback. |
|
Over the next few days I'll learn how to build Ghidra from source, and test out the master branch. I'll let you know my results. |
|
Thanks, it's fairly simple. Here are the instructions. |
|
You're right, the build instructions are quite simple. And they worked! I built the master branch on an Apple Silicon machine (an M1 2020 Mac Mini) running macOS 12.6.7. I used I tested the build on Thanks! This makes Ghidra much more usable on macOS 11 and up. |
|
Glad to hear it's working well for you! Feel free to make new GitHub issues for any bugs/annoyances you might find. |
|
Tried that again with Ghidra 10.3.3 on M1 Pro running macOS 13.5.2 and still confused how to get Ghidra to load external libraries.
Please let me know the proper flow of importing macOS binaries alongside external libraries. |
|
@Kentzo In Ghidra 10.3.3, libraries are looked up by just their name and not full path. You can confirm this by selecting the In the master branch and the upcoming 10.4 release, this has been fixed. You'll be able to specify the root of your extracted directory, or the original dyld_shared_cache file with no need to extract. A lot of improvements have been made to our own extractor. I am also working on a feature that lets you add missing components from the dyld_shared_cache into your current program on demand. The idea there is you import your program and start REing it. Eventually you will come across a red missing reference to something in the cache. You right click on that reference, and perform a new Add To Program action. It will find the library/component being referenced in the cache and insert it into your program. This feature will not be ready for 10.4, but if you are interested I can let you know where to test it. |
|
Disregarding the dyld issue, why is Ghidra failing to apply library’s path to roots specified via Edit Paths? Is it going to ve fixed in 10.4? |
|
I attempted to explain this with:
Maybe I am not understanding your question correctly. In 10.3.3 if you add all the subdirectories with libraries in them to Edit Paths (not just the root directory), it should find them. This is undesirable behavior which should be fixed in master/upcoming 10.4. release. |
|
Thank you for the explanation. |
|
@ryanmkurtz Could you explain whether there is a way in 10.4 to resolve "missing" external libraries in existing projects with already imported binaries? |
|
Unfortunately we do not have an automated way to do this. You would have to import the libraries into your project yourself, and then in your already-imported/analyzed programs, go into their External Programs window and specify the linkage. Then, you can rerun the I recommend you make a new feature request ticket asking for the ability to load libraries after a program has already been imported. I can then present it to the team for prioritization. |
Sure, just wanted to make sure I wasn't missing somethinig I just tried to import (File->Import File) /usr/bin/tmutil (x86_64) on macOS 13.5.2 (M1) with Ghidra 10.4 with both "Load Local Libraries From Disk" and "Load System Libraries From Disk" (without specifying additional Paths) and it failed to load system libraries: All of these libraries are system and are part of the dyld cache. What extra do I need to do to let Ghidra automatically load these libraries. Do I need to edit Paths by adding folder that contains the compressed dyld cache, dyld cache itself or a folder with the extracted dyld cache? |
|
@Kentzo, If you enabled the System libraries checkbox, you'll probably see these 2 entries in the Edit Paths window: These are the 2 default dyld_shared_cache files I added. Do you have at least 1 on your system? Maybe i need to add more defaults. You should be able to add your dyld_shared_cache root file in Edit Paths though for it to use it (assuming you are using Ghidra 10.4). |
|
@Kentzo Oh, the issue is likely that your binary is x86 and the dyld_shared_cache files it is looking for are aarch64. |
|
@steven-michaud How did you determine that they were actually loaded? I don't see them in the window project, the imported signatures (e.g. from libswift* and libobjc) remain undefined. All imported libraries do appear in Symbol Tree -> Imports, but they don't seem to point to any external programs. @ryanmkurtz No, nothing was "automatically" added to Edit Paths after I checked "Load System Libraries From Disk". 
|
Both x86_64 and aarch64 dydl caches are present on macOS 13.5.2 on M1 (remember, it can run x86_64 binaries through Rosetta): |
|
And you are using 10.4? Can you give me the full path of the x86 one so i can add it? |
I didn't, or at least not directly. I looked through the assembly code for external references. Every one I found was resolved correctly. |
|
@Kentzo thanks, i'll get the x86 one added. You should be able to add |
|
That's the import output after adding the path you specified: It seems like some libs are still not found, e.g. |
|
I'll investigate that tomorrow. |
|
@Kentzo I believe things like |
|
I see. Is it feasible to put some note in the log explaining why a particular library was not found? |
|
Yea, the log is kind of a confusing mess. I'll try to make it more clear what's happening. |
ryanmkurtz
added a commit
that referenced
this issue
Oct 4, 2023
dyld_shared_cache will be searched for in more default locations (#4162)
|
@ryanmkurtz, you should probably also add the following paths for macOS 11 and 12: |
|
@steven-michaud, thanks, i'll get those in tomorrow. Please let me know if you know of any more. |
|
I found one clear-cut case of a spurious error message about a file not being loaded. I tested with Ghidra 10.4 on macOS 14, on an Apple Silicon Mac. I ran I opened Ghidra's CodeBrowser, imported I opened the "Functions" window and pasted in Because Edit: I did this for both "architectures" of |
ryanmkurtz
added a commit
that referenced
this issue
Oct 5, 2023
dyld_shared_cache will be searched for in even more default locations (#4162)
ryanmkurtz
added a commit
that referenced
this issue
Oct 10, 2023
|
The |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
When importing a file Ghidra fails to locate and load libraries from the dyld cache.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
Ghidra locates and loads external libraries that are part of the dyld cache.
Environment (please complete the following information):
Additional context
#3345
The text was updated successfully, but these errors were encountered: