Track all resources with a unique ID instead of a res:// path

[NathanWarden]

What I think would be a better generalized solution for all resources, would be for each resource whether script/mesh/material/etc to simply have a unique id that would be stored in an import file. So, each resource would have the unique ID instead of a file path. The unique ID could then be associated with each resource. So if a script of any type, material, texture, audio file, etc would no longer be dependent on a specific path.

This would allow a user to rearrange their project even very far into development without having to "Fix Dependencies" since everything would be automatically know where every resource is internally.

This would require Godot to scan the project when you load it and create a Map of each asset ID and it's path, and then when a script/material/audio slot has UniqueID in it, it will internally know that ID X belongs to path X. If an asset is moved, then it's import file is also moved and the Map is updated with the new path.

This would allow all resources to be moved around and literally nothing would ever break.

So assets would be loaded via the following pseudo code:
// This will not break if the user moves the material
mesh.material = LoadResource(resource_paths[material_id]);

instead of:
// This will break if the user moves the material
mesh.material = LoadResource(material_path);

[Zylann]

Ah, someone finally raised the question xD
I've been thinking about this for a long time, but never posted because it's quite debatable, and if it were to happen it would at least be for Godot 4.0, unless you are talking about a system where people define their ID manually, asset by asset... for thousands of them? Renaming will still be problematic as long as people have to read or write these IDs because they would have to name them (naming is fine, the editor using name as ID isn't). If that name needs to change the same problem will arise again.

I had a TXT to note stuff about this, here it is, sorry if it's quite long:

(old https://pastebin.com/tXnLgCaB)

---

This thread is about considering GUID-based asset management for Godot, aka Unity3D-like (you'll see comparison a lot).

I'm pretty sure this was thought about at some point, and I know it may not happen anytime soon,
but I couldn't find any discussion about this here, at least to see why it wasn't considered, or even if people realize what this is.

I gathered my thoughs after all this time working with Godot. Even if a GUID system is rejected, these concerns remain valid.
Paths are not a bad decision, it's simple and makes sense... up to a point.

Here is why I think using paths gets limited in the context of a game engine:

1. Moving or renaming assets in Godot is a dilemma, even more for non-coders. Copying assets from a project to another also doesn't work because the path may vary, even if relative paths are used.
   This is the biggest asset design issue in Godot IMO, and still going to be AFAIK.
   Even if the editor attempts to update references when files are moved from dock, this remains a tremendous amount of work which propagates to IDEs, VCS and other external tools.
   Importance given to file path of everything is too high, as if artists had to think like coders, especially in such a collaborative and dynamic domain. (DISCLAIMER: even if the idea can be extended to nodes, I believe only resources would need it because the scene tree is purely a Godot thing, while the filesystem is a shared resource).

2. The editor tries to update all refs when using the filesystem dock, but it still has a few issues with this at time of writing (they might be fixed now but issues like that keep popping up), and searching all files of a thousands-assets project to replace all occurrences is going to be quite a heavy task. It's also quite some work to fix all bugs rising from this housekeeping, and yet they only work from inside Godot. It also leads to awkward situations where renaming a single file in your project forces Godot to reload open scenes (and loose all changes made to them if not saved!).

3. The editor can't deal with files moved externally (from file explorer or version control)
   How do you git mv? As an artist? Without breaking scenes you don't have open and are referencing the stuff you move/rename? Sure there could be a Git integration somehow but again that's a lot of work just to force using something inside Godot (disregarding any external tool that may exist already).

4. Resolving dependencies on scene opening takes time, especially on large projects

5. Resolving dependencies is not recursive. If a scene contains an instance of a scene which itself has broken dependencies, it will not even allow to open it, and not tell you which instance is broken, so ALL referenced scenes and resources need to be manually opened and fixed.

6. Resolving dependencies manually is prone to embarrassing errors
   For example, swapping a scene to another, which fucks all instances, and you need to re-resolve the scene again

7. Even if some scenes (TSCN) are readable, mostly experienced coders and eventually some artists will modify them directly

8. Resolving dependencies requires to modify too many files, and moving things around is prone to merge nightmare.
   This produces version control diffs due to file paths and ID swaps and conflicts in files that usually have nothing to do with the assets being moved,
   and other people end up having to resolve them (in real life, not everyone is good or even willing to do this).
   This happen rarely with GUIDs. I worked on a big Unity project for several years and used to merge prefabs and scenes just fine, without even having to look inside of them most of the time.

9. External resources have an indirection ID in TRES/TSCN
   This produces even more unwanted version control diffs and conflicts when a reference is added or removed,
   because it shuffles the secondary ID everywhere.

10. Although good organization always helps, things always change. Paths don't always scale well on big projects.
    Making games takes a long time, collaboratively and things can change very often.
    in real life projects with medium to large teams can't afford a full-freeze of the project just to rename and move files around.
    Sure it doesn't prevent some teams to succeed with just file paths, and like all things, with experience you get around it...
    but it should be really nice to not have to workaround the flaws I listed above in the first place, which should improve.

11. Renaming a file causes it to be re-imported, just because Godot sees it as a different file according to its name, which is silly.

bonus) Since Unity became mainstream, many users nowadays take for granted that moving assets just works.
this can make teams on big projects freak out when considering Godot over Unity3D.

Now, here is why I think GUIDs as primary asset referencing would improve Godot:

1. GUIDs make an asset globally identifiable regardless of its path or even its project.
   It makes the asset easy to move around and even copy across projects.
   Even embedded resources (aka "built-ins") can be fetched identically, because a GUID is not tied to a particular storage logic. Alternatively, you can also identify an embedded resource as a combination if its container GUID and its local file ID (like Unity does but for game objects).

2. Moving an asset doesn't produce any version control diff beyond itself, because the GUID doesn't have to change.
   Only people working specifically on the file will have to deal with conflicts

3. Moving or renaming an asset can be done from anywhere (editor, system, VCS...), and the editor has no work to do beyond detecting the same resource at a different location. Also, having a huge project with thousands of files won't affect how long it takes to rename or move an asset.

4. Storing a GUID needs a companion .meta file of some sort.
   It can also contain what in Godot we used to store as .import or .flags.
   This meta file has to be moved/renamed too when the asset is moved/renamed, and that's the only gotcha people need to know.
   In worst case, not doing this makes the engine generate a new GUID and produce the usual dependency fix dialog open.
   Coders might not like meta files because scripts seem to make less sense to have them. But in fact, they still come in handy in case you want to rename a script that was used in zillions of places in your project. Rename the file and the meta file. Done. Also, you won't see meta files often, since they shouldn't be shown in the engine's file browser (or Visual Studio for example if you use C#). The only place you will have to remember them is VCS.
   Many Unity users know this by experience (almost all gamedevs do), and Godot also has already a similar gotcha with .import files anyways. And if you can't stand them, they can be stored in .import folder, at the risk of loosing some of their advantages when moving assets around (just like currently). They could also be placed in a class attribute, or be optional, if you can bear the disadvantages.

5. GUIDs can be read in TSCN once you know how it works.
   They are just a character sequence that "names" the asset uniquely, so they can be read or even written as part of a text resource to reference another. You also don't have to deal with absolute vs relative paths, and searching for that string in the project is easy and guaranteed to be accurate.

6. People don't have to deal with GUIDs directly. A drag-and-drop workflow perfectly works.
   The only time you would have to, is when you merge a conflict in a resource involving a GUID.
   In best case you can solve this with proper diff tool that tells you which GUID is what,
   in worst case you grep the GUID to know which asset is concerned.

7. GUIDs can coexist with fetching assets by path.
   Coders can still preload() and load() with a path if the resource filesystem is important.
   The path is also still accessible obviously, if the resource comes from a file.
   There can be an alternative path format, "id://the_guid", to get an asset by GUID (just a wild proposal),
   which the editor would use by default when resources get assigned from within the UI

8. GUIDs would be the primary resource referencing system used by the engine.
   It has to be only one. However, as point 6) states, it doesn't mean we can't fetch by path ;)

9. Looking up assets by GUID just needs the engine to parse the project files without even loading the assets.
   The editor does this already, for other reasons though. There is also no cost for an exported game because we can list resources in a file (like we ended up doing for named classes in 3.1).

10. A uniquely identified resource doesn't need further indirection (which is the case in TSCN/TRES format)

[NathanWarden]

I've been thinking about this for a long time, but never posted because it's quite debatable, and if it were to happen it would at least be for Godot 4.0

@Zylann I'm glad to know I'm not alone desiring this. I thought about it having to wait until 4.0 too and that was my first impression, but after some thinking I don't think that's actually the case. It's not API breaking and there's a pretty straight forward way of dealing with resources that are already tracked via res://path and that's simply by checking if the resource path/id data starts with res:// or not. If it starts with it then it needs an ID, then it can continue the load process. That could all happen transparently without user interaction as well.

I'm happy to read your detailed thoughts on GUIDs and I share many of the same concerns... especially when it comes to big projects... thanks :)

The long term project we've been working on (for about 5 years now) contains almost 10,000 files in it (not counting meta files), I can't imagine having to deal with broken dependencies in Godot if we were to move it over.

[hsandt]

I'd like to mention a few alternative approaches to feed thought, although I would also be inclined to use GUID everywhere.

(note that imported resources already use a kind of GUID suffix in the .import folder (e.g. icon.png asset -> texture in res://.import/icon.png-487276ed1e3a0c39cad0279d744ee560.stex), so this about other resources like scene files)

Redirectors

Unreal Engine leaves a virtual Redirector at the initial location of a moved asset. This allows referencing assets to continue reference the old path. However, this causes some issues:

1. It adds one level of indirection (discussion above is about avoiding that)
2. The user may think that the reference paths have not been updated and think it's a bug (esp. if the Redirector is invisible)
3. The Redirector must be made visible to show to the user what happens under the hood, but that also pollutes the Assets panel. Depending on the implementation, the Redirector may store as a file at the old location (which will be seen in VCS), or defined inside the project description (also visible in VCS, but many redirectors will be seen as one file change).

The advantages:

1. Referencing assets are not changed
2. It's possible to resolve the indirection at any time (typically when the user has more time to afford a scanning) so old references are updated, and this can then be committed in VCS separately from the change that moved the assets in the first place (if it was a big change, this can help not making it even bigger). This corresponds to a Fixup in Unreal.

Reference: https://docs.unrealengine.com/en-us/Engine/Basics/Redirectors

Recursive/neighborhood scan

This one comes from iTunes (!). If you moved a track in another location, iTunes will add a (!) mark near the track when it detects it. When you try to play the track, iTunes suggests you to Locate the missing file by selecting a new path.
Once you have done that, iTunes will also search for other missing files with similar original paths (e.g. in the same album), searching moved files in the new path. It also seems to be able to find missing files in subfolders and sibling folders.

This method doesn't change the way Move is handled, only makes scanning more performant if you have moved many files.

I couldn't check if Godot is not already doing something similar though, because I don't have an example where you would move many non-imported resources (I only have PNGs to work on...).

---

My first issue was actually with moving the Main scene (set in Project settings > Application > Run) as it is only referred to by path. To create a working example, we could prepare a set of nodes referring to many scenes and move all those scenes elsewhere? What other resources are non-imported so we can demonstrate the issue?

[willnationsdev]

What other resources are non-imported so we can demonstrate the issue?

Scripts are another non-imported resource. If we suddenly decided to use GUIDs, then I assume we'd have to have some kind of .meta file that goes along with it though. Or perhaps we could get away with a .meta folder at the project root so we don't have to move pairs of files around all the time? Idk how most of that works with .import files atm, so I'm not sure.

---

Related to #7402 and my WIP PR #22181. All CustomTypes do is allow us to map a name to a file path and bind an optional icon to it. And it only works in the editor. If we really did have a GUID system, then I believe we could also allow users to optionally bind a name and/or some metadata (via #18591 ?) to it that would allow users to specify names for any resource, even at runtime.

One of the things that Godot is missing is some way of referencing reflection data for user-defined types (basically anything that ClassDB can do for engine types). Having a centralized, runtime-accessible system that knows the names of scripts and scenes would finally allow us to write logic that requires an awareness of a script or scene's name or location. Generating a list of types that meet certain constraints, allowing the user to create their own types, allowing users to query inheritance data, etc. These types of problems are solved much more easily when there is a centralized system that handles everything. You could even have the ScriptServer register script-class names into this same system, only those records would be handled automatically and couldn't be manually edited.

And with the introduction of #20318, we'd be able to add reflection data to scripts using annotations, effectively allowing us to mirror other ClassDB methods for scripts like getting a list of properties affiliated with a class by name alone. These are important features to have if one wishes to make data-driven applications, especially ones that have editors included.

So, I believe this Issue is related to many problems people have found related to gathering data about user-defined types.

[lexum0]

How about doing it like Xcode? The file is internally linked to the file's UUID in the project file and the filename must be unique, if two are found then it uses the first one. The UUID is generated upon import.

So that way it's very simple to use, every filename is it's own ID with an internal UUID for internal referencing and if you need to differentiate files you do it by name. Otherwise it's very confusing to have two files named the same and having a different purpose.

[silkentrance]

How about splitting this one up into an API that will give us instances of UUID or UUID strings, and the other stuff that makes use of it? While the API part could be done rather fast, the rest just needs more work.

Here is a link to a cross platform UUID/GUID library: https://github.com/graeme-hill/crossguid/blob/master/src/guid.cpp

Perhaps this can be included with the OS singleton interface, e.g. OS.uuid() : Uuid/String?

[MuffinManKen]

The bits about decoupling the res: paths would be nice, but I would be thrilled if the sequential resource IDs were replaced with a UUID. My git history is nearly unreadable at times and merging branches with scenes can be nearly impossible.

[Xrayez]

Workaround

Note that currently you can use a ResourcePreloader node which can act as a resource database mapping a resource name of your choosing to an actual res:// path. It still tracks any resource by path but this also has a surprising benefit: when you move those assets, the refactoring system already implemented in Godot would rename those paths in any ResourcePreloader node for you, and you can fetch any resource just by name then:

var my_scene = $preloader.get_resource("my_scene") as PackedScene

You can have multiple resource preloader nodes to group specific assets, and wrap them in a singleton for quick access.

---

Note that any exported resource via script export(Resource) var can also be handled by the refactoring system similarly so that the path doesn't need to be hardcoded into a script.

Talking about IDs, paths and scripts which seems to be related: #33360.

[Zylann]

@Xrayez this only solves part of the issue (and seems to be intented at another one). You have to add such node to scenes or as a singleton (to preload ALL assets of the game?). Moving assets outside of Godot still causes trouble, VCS will still have extra diffs, they would not be portable between projects if paths change, and you still have to give a somewhat readable (and unique) name to every asset which means one day it may need to change too. Also, all the work Godot currently does parsing the entire filesystem to find paths to rename could be made unecessary by using IDs as primary reference.

[Xrayez]

to preload ALL assets of the game?

Given some limitations as in #13954, preloading most commonly used assets may be a decent compromise between slow game startup times and the smooth gameplay experience at run-time.

[Calinou]

Closing in favor of godotengine/godot-proposals#471, as feature proposals are now tracked on the Godot proposals repository.

[ignaloidas]

Closing in favor of godotengine/godot-proposals#471, as feature proposals are now tracked on the Godot proposals repository.

I don't think this issue and that proposal are the same solution. My proposal concerns internal representation in in .tscn files to make them friendlier to VCS's, while this is a wider scope proposal concerning usage inside scripting as well.

[willnationsdev]

@ignaloidas I would recommend that a new proposal is opened to explicitly account for that subtly different case then.

[Xrayez]

Some issues which could be prevented if we had such a system: #31045, #26188, #40991 (mainly talking about resource remapping system which relies on having different file paths).

[Flavelius]

This issue should probably be reopened as the linked proposal is not related and the issue of res based paths is still in effect throughout godot.

[Calinou]

This issue should probably be reopened as the linked proposal is not related and the issue of res based paths is still in effect throughout godot.

A new proposal should be opened for this instead 🙂

[Zireael07]

@Calinou: I was sure we have a proposal for that, and we do afaict: godotengine/godot-proposals#471

[Flavelius]

@Zireael07 #15673 (comment)

[Calinou]

For people stumbling upon this issue from a search engine, note that this is being addressed by #50676.

[Zylann]

I'm not sure the full extent discussed here is being solved by that PR, it seems focused on a particular case (sub-resources).