Add dependency handling and enable users to organize and publish addons and addon-collections as packages

[willnationsdev]

Describe the project you are working on:

The Godot Next plugin: a collection of scripts that are useful in a variety of projects (utilities, editor tools, singletons, nodes, resources, etc.).

Describe the problem or limitation you are having in your project:

Godot Next is large enough that it has a small network of internal dependencies (ranging from 0 and 1 all the way up to 6 in the rarest case). No big deal if you grab the whole thing, but if you just want to solve a problem with one script, then you might have to deal with dependency management, even inside a single addon.

This is a big headache, both for me as the maintainer and for users who get interested and then very quickly frustrated/turned off due to needing to deal with resolving out-of-nowhere dependency chains.

Describe how this feature / enhancement will help you overcome this problem or limitation:

The best solution would be to increase the granularity of addons in Godot Next and introduce dependency handling to Godot. After all, some people want it all; some want just the most necessary bits. The choice of whether to make users manually pick out the parts they want from the whole or piece together the whole from a bunch of parts shouldn't be a mutually exclusive one. Both options should be available.

Right now we have 1 addon with a wide variety of classes. Some are related to others; many aren't. But, they are all a package deal. They are part of Godot Next's brand of, "a generic extension to Godot."

Rather than that, I would rather have N addons, one for each class or minor collection of classes, and have each tiny addon declare the dependencies that it has on other addons. I would then want to be able to declare all of these addons to be part of a thematic/topical "package" that users can use to mass select a collection of other addons. These could be defined as part of an AssetPackage resource type for arbitrary creation and storage. The Godot Asset Library and the EditorSettings would also be able to store these AssetPackage files for remotely distributed and locally shared "packages" of addons.

Furthermore, I would want to be able to browse and configure these "packages" of addons during the project creation and addon browsing processes to provide more concentrated value and discoverability to users.

If a new-ish Godot user opens up the Project Manager to create a new project, they should be able to view a list of popular Packages that they can install alongside their project. Rather than needing to create their project, go to the AssetLib tab, and then browse through hundreds of addons and install interesting ones one-by-one, they could instead see a GUI of addons to select which are batch installed at the end of the process and which can be selected/de-selected in groups by selecting/de-selecting themed Packages.

Users should even be able to create their own AssetPackage resources, either for local use (to auto-install a collection of preferred addons) or to upload and manage for the community (like what Godot Next could do).

A "Package" could then be...

• Godot Next.
• A user-defined Favorites list of editor tools to share across all projects.
  • Perhaps we could add an AssetPackagae field in EditorSettings to accommodate this.
• A "framework" of addons tailored to specific fields:
  • A game genre such as 2D platformer or first-person shooter, etc.
  • A technology category such as Mobile Apps, Data Structures, AI, Particle Effects, etc.
• A team-organized collection of addons for specific tasks:
  • Producers get in-editor tools for task management, allocating team resources, etc.
  • Writers get in-editor tools for narrative dialogue editing, more visual scripting nodes, etc.
  • 3D artists get in-editor tools for voxel modeling or assets for meshes, textures, etc.

Note that one difference between an addon and a package is that addons simply declare a set of dependencies while packages can have both required and optional addons associated with them, as demonstrated in the final mockup below.

Show a mock up screenshots/video or a flow diagram explaining how your proposal will work:

The final UI I imagine for this would be like a blend of the VS Code extensions UI and the Visual Studio Installer.

The former contains a list of elements where selecting any one element displays an editor-rendered Markdown README + any other custom data (repo web URL, rating, whatever data we can safely store and use, assuming it doesn't violate GDPR).

For reference, here is the VS Code extensions image in #139:

.

The latter maintains a list of selected editor components and will mass add/remove components from the batch installation dialog as users add/remove packages of functionality from the installation. Individual components can also be browsed and added/removed as desired.

Here is the original Visual Studio Installer:

I'm imagining that we could setup our own packages-over-addons setup with a VS Code-like README renderer. Something along these lines:

And this would be something that can appear both in the AssetLib tab and as a popup when users are creating a new project. Packages would be pulled from the Godot Asset Library as well as from local user files.

Describe implementation detail for your proposal (in code), if possible:

1. I can publish individual addons (with just a single script even) with minimal effort (just pressing a few buttons).
   1. Editor can generate an addon for me (Remove "addon" versus "plugin" inconsistencies by adopting a single name for both #118).
   2. Editor can access third-party addon APIs for me (Enable plugins to integrate arbitrary 3rd-party data sources to the AssetLib tab #12).
   3. Editor can publish an addon for me to any supported API through a common interface (N/A).
      • Would require some plugin to enable users to interact with third-party APIs and publish to them. Things like this could even be part of an "Addon Creators" Package installable from the Project Manager.
      • Godot would need to be able to, at the very least, directly submit new asset version submissions to a selected remote online vendor (which defaults to the Asset Library, pending Asset versions instead of asset edits [Request For Comments]  godot-asset-library#143, and which gets other API support out-of-the-box with Enable plugins to integrate arbitrary 3rd-party data sources to the AssetLib tab #12 once their plugins are installed).
2. Users can easily browse addons, access their source repositories, and get their information.
   1. Editor AssetLib is redesigned to support rendering Markdown content (Add full Markdown support for use in browsing, creating, and editing projects and addons #139).
   2. Addons could expose direct documentation for included classes.
      1. Script annotations help here, but are not required (Annotations in GDScript godot#20318)
      2. Use script annotations and script reflection data (already exposed in 3.2) to generate XML documentation data. Then we can program the editor to auto-import that documentation data via a commonly defined directory structure. DocData would need to only integrate docs for script classes that are successfully registered. Any time script classes update, DocData would also need to be rebuilt. Start with a cached copy of the ClassDB-only DocData and then re-add all script class data as needed (Enable scripts to generate XML API documentation for the editor's GUI #177).
   3. See Add full Markdown support for use in browsing, creating, and editing projects and addons #139 for an example of what this could look like in the AssetLib editor tab.
3. Users can identify and automatically install all dependencies related to an addon.
   1. Addons would need a standardized, safe-to-copy-paste-install internal structure where Godot can validate whether a directory constitutes a valid addon because of its structure. Only when an addon directory has structured data can the editor apply installation logic to them universally, without humans having to manually check off which files are to be copied into a project (Establish convention for sharing libraries #62).
      • This might also involve the need for some kind of .gdignore file to filter out some files from bundling into an addon.
   • Problems with preventing "engine bloat" and the AssetLibrary godot#19486 discussion must be resolved.
   • godotengine/godot#19178 discussion must be resolved.
   • At the very least, we would need an agreed-upon dependency acquisition mechanism, regardless of whether it ends up being a full-blown management tool or just a "curl these URLs, kthxbai!"
4. Users can view collections of assets related to topics in which they are interested and can easily install/uninstall them en masse (this Issue).
   1. See mockup above.
   2. Packages would maintain a list of optional and required addons/sub-projects/whatever to be associated with them (will call them just "addons" for now).
   3. The GAL would need to be refactored to use a batch install process. In it, users would be able to select a set of addons to install (a list of checkboxes in the GUI).
      1. Every time the list of packages/addons is updated, rebuild the list.
      2. Allow users to view a flat list and search that list for addons directly (the regular GAL behavior).
      3. Allow users to view the checkboxes grouped into packages. If an addon is present in more than one package, that is fine (more about seeing what stuff is associated with a package overall). You still would only install that addon once.
      4. Potentially multithread the download and installation process?
      5. If someone clicks on an individual addon in the flat or grouped list, the currently active tab on the right should update to display that addon's information (README, CHANGELIST, name, other data).
      6. Users should be able to open new tabs in the addon viewer to quickly switch between cached versions of the data (without needing to re-fetch it).
      7. A loading icon should display between requesting a new addon's information and displaying it.
      8. The visualization of an addon's information should have no relevance to whether that addon is checked or unchecked in the flat/grouped list.

If this enhancement will not be used often, can it be worked around with a few lines of script?:

This enhancement is largely based on modifications to the Asset Library, the Editor's AssetLib tab, and introducing a new resource: AssetPackage. It also changes the Project Manager and is dependent on standardization changes in the addon ecosystem. It would revolutionize the accessibility and growth of addons, so yes, it would be used often.

Is there a reason why this should be core and not an add-on in the asset library?:

Tons of changes to NOT script-accessible stuff. See above.

[willnationsdev]

@Calinou While you are working on the Godot Asset Library Laravel implementation and work on adding asset version tracking, I request that you leave room for non-assets to also be version tracked (like these Packages) in the future. You might already be planning to do that, but just in case.

[Calinou]

As for the "addon collection" thing, can this be achieved by publishing a dummy package that lists a bunch of dependencies? I think some npm packages do this. In any case, I'd prefer going for a simple solution 🙂

[willnationsdev]

I would agree, except that I don't know if we want typical addons supporting "optional" dependencies.

[willnationsdev]

@Calinou Maybe we could make the underlying data in the database support both required and optional dependencies, and make addons and packages use the same configuration file to define themselves, and then have a flag set within that config file to say whether it is a package or not. And then, if it is a package, it will additionally search for and use an optional dependencies field which, if defined for an addon, just won't do anything. So the expectation would be...

1. An addon has a config file, content, an optional plugin script, and a required dependencies Dictionary (identifier => version range string).
2. A package has the same config file, ignores any content (if package flag in config is set), has no plugin script (because there is no content), and has both a Dictionary of required and optional dependencies.

Does that sound good? Any suggested changes? Do we like this idea at all?

[Calinou]

Additional thought: I wonder if assets should be able to declare scenes/scripts to be automatically added as singletons when installing them. The name and path to the scene/script would have to be written down somewhere in a dictionary property. Removing the asset should also remove the singleton declaration.

This would make assets centered around singletons significantly faster to set up (which is good for prototypes or game jams).

[willnationsdev]

@Calinou That sounds good to me.

It looks like there's someone else interested in contributing to improving the assetlib's frontend. Figured I'd let you know.

Also, do we know anything about if/when/how dependency management will be implemented? I remember there was an issue with a lot of discussion about it, but I don't think a consensus was every quite reached and idk if reduz has discussed those ideas on IRC at any point. I figured, if there is a data-driven way to extract the dependencies of an asset, then we'd want to be able to display those dependent asset links within an asset's webpage. However, we don't want Godot dependent directly on the Asset Library, so there'd have to be some way of referencing an asset that works both locally in Godot and in the Asset Library (fully qualified namespaces maybe? godotengine/godot#21215).

[Calinou]

Also, do we know anything about if/when/how dependency management will be implemented?

No, it seems reduz is way too busy to discuss this right now.

[nonunknown]

@Calinou As for the "addon collection" thing, can this be achieved by publishing a dummy package that lists a bunch of dependencies? I think some npm packages do this. In any case, I'd prefer going for a simple solution slightly_smiling_face

Related -> #607

[Calinou]

@nonunknown Out of curiosity, how does your proposal differ from this one?

[nonunknown]

it does not differ, it complements this one, specifying how plugins dependency should work :D

[me2beats]

I need my plugin Y to not even start installing without plugin X installed and active.
That is, when I click the Install.. button (before the list of files to be copied to the project opens), I want Godot to find some deps.cfg file in the addon folder (in the downloaded zip package), where I've listed the dependencies - plugins which can be downloaded from the asset lib, which should be installed and activated automatically, and only after that my plugin (Y) may be installed.

Could this proposal solve this?

[Calinou]

Could this proposal solve this?

Yes, but nothing has been implemented yet.

[willnationsdev]

@me2beats @Calinou There's also the fact that the mere decision of whether to implement dependency management at all hasn't actually been decided from what I've seen (unless there's a consensus in a proposal I missed somewhere). There's also this alternative proposal that reduz made a while back about creating a sub-project system and replacing addons with projects wholesale, so if you had dependencies, you'd literally just include them a copy of those dependencies directly in your project. So, there'd be a project tree just like there'd be a scene tree (except, you know, without the magic of scene instancing for projects).

Basically, I haven't seen any actual decisions made on how to move forward with resolving any of this mess, or any action taken to implement any of the proposed solutions, and it makes community-driven addon/plugin development stick in the mud and move like molasses.

[Calinou]

Basically, I haven't seen any actual decisions made on how to move forward with resolving any of this mess, or any action taken to implement any of the proposed solutions, and it makes community-driven addon/plugin development stick in the mud and move like molasses.

I would like to see some kind of dependency management in Godot. However, we need to make every add-on a proper plugin before we can consider adding a way for plugins to declare dependencies. Having dependencies definable only for editor plugins isn't very useful. It's also likely we need to implement #554 first.

Another issue is that dependency resolution can get quite complex if you allow multiple dependency levels – and it can encourage npm-style micropackages to pop up, with all their issues. It would be possible to limit dependencies to a single level to prevent that, but it would also make the system less useful.

[me2beats]

I would like to see some kind of dependency management in Godot. However, we need to make every add-on a proper plugin before we can consider adding a way for plugins to declare dependencies. Having dependencies definable only for editor plugins isn't very useful. It's also likely we need to implement #554 first.

Say there's kenney particle pack.
Its content is just png images.
Does this addon need to have dependencies?
I think no.
On the other hand, this addon itself can be a dependency. But should it have plugin scripts in such a case? I don’t think so.
An asset config file might make sense though.

In my opinion the plugin / asset should just have

• version number (this could be in the config file of the asset)
• any identifier by which it can be downloaded from the asset lib (this could also be in the config file)
• the ability to view/update/delete this asset through the asset manager (can be implemented in the future; it could be a separate tab in the AssetLib - named like "Installed Assets" or "Asset Manager")

This system should only be supported for those assets/plugins, all files of which are in a separate asset folder: res://addons/some-plugin
This sets some restrictions for plugins, limits it's structure flexibility and may not be very convenient for the user, but on the other hand it would save you a lot of headaches, for example, when you need to update an asset to a new version, you just update the contents of the asset folder.

Another issue is that dependency resolution can get quite complex if you allow multiple dependency levels – and it can encourage npm-style micropackages to pop up, with all their issues. It would be possible to limit dependencies to a single level to prevent that, but it would also make the system less useful.

I don't see a problem with dependency levels - in my view it just has to be incompatible with the dependency system: each addon should be still installed in addons in a separate folder.
several plugins in one addon should be prohibited (at least for the first time).

This does not mean that all plugins must follow this rule.
If a plugin doesn't follow this rule, it just would be incompatible with the dependency system.

[willnationsdev]

However, we need to make every add-on a proper plugin before we can consider adding a way for plugins to declare dependencies.

But should it have plugin scripts in such a case? I don’t think so.
An asset config file might make sense though.

That's effectively what he's saying. With the current system, if a given addon has any global script classes, the editor can only toggle on/off those scripts if the addon in question has an EditorPlugin defined for it, thereby making it show up in the ProjectSettings' Plugins tab. And addons themselves aren't required to have any config file, as you mentioned, while plugins do require it.

Considering we are coming up on a compatibility-breaking release, I agree, it is important that we make critical changes that can establish new conventions, even if it doesn't create a full-blown dependency management paradigm. So, requiring a config file for every addon sounds like a good idea to me, and then we can have things in the Asset Library tab that allow you to "manage" the addons present in the res://addons/* directories.

(Edit: actually, I guess, technically speaking, how it is arranged now with the ProjectSettings configuring the management of addon directories (currently as plugins), and then having a shortcut to open that management tab from the Asset Library is probably the best way to do that.)

I don't see a problem with dependency levels - in my view it just has to be incompatible with the dependency system

This actually sounds like a really good short-term solution imo. Until the sub-project stuff is implemented (which I would guess would be coming with some sort of 5.0 release), we really need some sort of barebones way of officially stating the set of dependent addons that a given addon requires, just so that it can ensure that those things are also installed.

[frob]

We could adopt the solution the composer project uses. In composer the package library (or addon in this case) the package.json file is optional. For dependencies that don't have their own package.json but are required by a project file the package.json of the project can define the package.

This is detailed here https://getcomposer.org/doc/05-repositories.md#package-2 and adopting this concept could make it so this change doesn't break BC.

[DrMoriarty]

Hi All! Recently I implemented my own plugin management system for my own modules. It mostly designed for process binary precompiled modules for third-party SDKs for iOS and Android. There are 37 production ready packages: analytics, ads, social network integration and so on.

The system has dependencies, autoloading gd wrappers, export hooks, uniform module settings for app keys and so on. It can be used as GUI addon and as CLI utility (for example when you want build your project with CI and don't want to store all binary frameworks in github repo. They can be automatically downloaded before your project was build.)

Full info and package list: https://drmoriarty.github.io/nativelib/
GUI Addon: https://godotengine.org/asset-library/asset/824
CLI utility: https://github.com/DrMoriarty/nativelib-cli

What do you think of it?

[dploeger]

So a year has gone by and I'd like to ask about the state of this. I've heard rumours about "a new asset lib" and would've guessed that this would be part of that. We see a surge of addons and more are about to come it with more widespread adoption of Godot.

Can somebody please update the core team's current stance on this proposal? Thanks!

[Calinou]

Can somebody please update the core team's current stance on this proposal? Thanks!

npm-style dependency handling is not planned for 4.0, as there are too many other things to work on currently. I think the plugin system in general will not change much compared to what we have right now (other than replacing GDNative with GDExtension).

The issues about allowing nested dependencies are still present and haven't been debated much by the core team yet.

I've heard rumours about "a new asset lib" and would've guessed that this would be part of that.

The new asset library is only a web backend – it does not impact how the editor uses assets. In fact, the REST API it exposes is fully compatible with 3.x. However, there is no ETA for deploying it to production due to possible incompatibility with the hosting we're currently using (TuxFamily).

[dploeger]

@Calinou thanks a lot!

For dependencies to work, a list of dependent addons would be required as a field in the backend. So this proposal would be about

• Implement dependency information per addon into the backend (list containing name of addon and a semver range)
• Implement dependency calculation into the frontend
• Implement full installation workflow

Is there already some sort of epic/label with proposal for a new frontend?

I'm asking because I was tinkering with creating a new addon for a new asset lib frontend that could at some point be used as a base for a core implementation.

[Calinou]

Is there already some sort of epic/label with proposal for a new frontend?

Not that I know of. As I said, the feature proposed here still hasn't reached consensus among core developers, so it may not be worth doing yet.

[dploeger]

Ah okay, makes sense.

[willnationsdev]

It just occurred to me. Godot 4.0 is gonna be a compatibility-breaking release, potentially for quite some time. Is there any plan to update/improve the addon/plugin ecosystem for 4.0, or is there gonna be some way to not introduce compatibility breakage for a new system so that it can be part of subsequent 4.x releases?

[Calinou]

It just occurred to me. Godot 4.0 is gonna be a compatibility-breaking release, potentially for quite some time. Is there any plan to update/improve the addon/plugin ecosystem for 4.0, or is there gonna be some way to not introduce compatibility breakage for a new system so that it can be part of subsequent 4.x releases?

Feature freeze is imminent, so I'm afraid we can't make any significant changes now. The only change I'd consider making for 4.0 is #554, but I won't be working on that.

We expect some compatibility breakage to happen in 4.1 regardless, so we still have some leeway (though not as much as 3.x -> 4.0).

[willnationsdev]

Sounds about right. My guess then is that we'd introduce some entirely new means of handling things like this, but set it up so that the old system is still "working", but deprecated (more than likely).

[sandmuel]

As I mentioned in #6704, this shouldn't be too difficult to implement if done through simply compressing the addon's folder into a .zip. Since Godot already has support for reading from zips, this should just be a matter of using this on the plugins, correct? (do let me know if I am wrong, as I have not looked through the source, and am relying solely on how I think it is done)
NOTE: I am not suggesting it be unzipped, but that the zip be read

[loteque]

In keeping with Godot's philosophy of project source being text files, asset-lib should fundamentally be a tool that manipulates and references text files.

• ⌨️ It could be a used through a flag on command line:
  • command: user@box|~/project $> godot --asset-lib-search "3rd person controller"
    • return: godot asset-lib: package(s) found: "3rd-person-controller-plug"
  • command: user@box|~/project $> godot --asset-lib-install "3rd-person-controller-plug"
  • command: user@box|~/project $> godot --asset-lib-require "3rd-person-controller-plug"

really every project should have a dependency manifest, a text based source file that godot --asset-lib-* manipulates and references.

Here is an example of what that could look like:

[package]
name = "project"
version = "0.1.0"

[dependencies]
https://godotengine.org/asset-library/4.3/serve  3rd-person-controller-plug
https://github.com/godotUserJuan/godot-package-list  custom-test-framework

Ultimately it should work in such a way that a beginner never has to touch the Command Line to manage packages, they may use only the asset-lib tab in the editor.

It should provide a CL interface so advanced and professional users may manage their packages in an automated environment or through a keyboard focused interface.

It should be text based because that keeps in line with Godot projects being hostable in and friendly to Version Control Systems such as GIT.

Edit: the more I think about this the more I realize that a package manager for Godot should probably be an external application and probably not be built into the editor at all.
I could see it having an editor plugin giving the package manager presence in the Godot UI and some interop. However, in keeping with the spirit of Godot's newbie friendliness and simplicty, a package manager is added complexity that isn't really required of the platform.

tldr

I do think there should be an official package manager for Godot that is standalone and text based.

[Ivorforce]

I've recently been wondering about dependencies between GDExtensions specifically. If the path is known, it could be linkable using @rpath or @executable_path.

That would require officially endorsing one path for addons, such as /addons/. It may be possible instead for Godot itself to re-link a binary's dependencies by patching its dependencies' paths before loading them, using something like install_name_tool -change.

Anyway, would everyone consider the issue of linking GDExtensions amongst themselves to be a part of this feature proposal, or should I create a new one for it?

[Sythelux]

I want to put in some things I found while looking into the topic.

Unity uses OpenUPM https://openupm.com/, which is basically an NPM extension to manage Unity internal packages. The Unity Package manager got really good lately as well.

Someone started a similar project here for Godot: https://github.com/godot-package-manager

It is not fully up to date, but it is pretty much something that could be fixed. I like the approach. I don't like NodeJS or the NPM repo, but we host a verdaccio in our company and are managing packages in Unity exclusively with that and we don't have to think about npm even once.

we basically would only need to have a bit better dependency management within the project.godot file and a few more additions to the plugin.cfg mainly... a unique identifier (for example those reverse dns identifiers. org.godotengine.some_package) and a dependency management on that basis, which mentioned above already. although I would get more inspiration from toml:

project.godot:

[plugins]
[plugins.myfirstplugin]
install="local"
name="My First Plugin"
version="1.2.3"

[plugins.onefromgit]
install="git+https://github.com/path/to/repo:godot_addon" <- we use a different branch for the addon path in LayerProcGen: https://github.com/Sythelux/LayerProcGen/tree/godot_addon
install="git+https://github.com/path/to/repo#addons/onefromgit" <- alternatively a subpath as is very common in a lot of addons could also be default tbh
name="A Plugin from Git"
version="1.4.2"

[plugin.onefromassetlib_or_npm]
install="com.example.assetlib" <- the new unique identifier, which can be read by assetlib and npm
name="My Cool Plugin"
version="1.6.2-pre.1"

[plugins_scopes]
[plugins_scopes.main]
url="https://godotengine.org/asset-library"
scopes=["*"]

[plugins_scopes.internal]
url="https://company.internal/verdaccio"
scopes=[ "com.example", "internal.company" ]

my question currently would be. can we inject and read content from the plugin.cfg and godot.project without godot it trying to parse and fail and remove it? if it does could we get the engine to not delete those areas?

Edit: for godot.project I think I solved it because I can make a plugin that stores user properties so I can work on it myself. we still would need a way for unique identifiers across the landscape though and I'm still not sure if I can add stuff to a plugin.cfg without the godot editor removing it. (haven't really tried admittedly)

[willnationsdev]

@Sythelux Note that, while this is a third-party utility and isn't technically affiliated with Godot, the GodotEnv repository gives you a CLI that can manage Godot installations and Godot addons. Might try peeking at that as well.

While I haven't tried the addon management myself yet, it looks somewhat robust and well documented: it'll track dependency management between your addons which are just configured as git URL or local symlink targets, although I can't quite tell from the README whether versions are involved/compared. If you want to avoid NPM, then this might be another viable alternative, perhaps with some tweaks/contributions.

I'm still not sure if I can add stuff to a plugin.cfg without the godot editor removing it. (haven't really tried admittedly)

As far as I know (my knowledge is ~2 years out of date, only recently returned to Godot stuff), plugin.cfg is only ever created initially by Godot or scanned by Godot; the Godot Editor does not mutate the file after it has been created at any point, so you should be safe to manipulate it in whatever way you want, though doing so to third-party plugin developers' addons would obviously get overwritten and become lost if you tried to manually update by replacing the files with a freshly downloaded zip/release, etc.