Establish convention for sharing libraries

[erodozer]

Describe the project you are working on:
This applies to any project that would like to install dependency or assets developed by others.

Describe how this feature / enhancement will help your project:
Right now there is no established convention to enforce for managing packages that may contain source code, assets, or plugins. This applies both to how people are expected to create a repository for it to be used by a Godot project, as well as where a godot project should store its downloaded modules.

Show a mock up screenshots/video or a flow diagram explaining how your proposal will work:
An example of how a project could store dependencies, as well as how those dependencies can acceptably be organized to be recognized by Godot.

project/
├── godot_modules/
│   ├── actor-module/
│   │   ├── package.json 
│   │   ├── index.gd
│   │   ├── Actor.tscn
│   │   └── placeholder.png
│   └── simple-scene-manager/
│       ├── lib/
|       |   ├── SceneManager.tscn
|       |   └── SceneManager.gd
│       ├── assets/
|       |   └── transition.png
│       └── package.json
├── assets/
└── scripts/

Describe implementation detail for your proposal (in code), if possible:
This is more of a community pattern to describe and enforce than it is an actual code feature. However, if it's properly enforced, then the engine could also start to depend on the known convention and implement custom loaders for efficient coding, such as how npm modules can be required from root level namespace instead of relative paths.

Without a properly defined convention for sharing modules, what's checked out can become messy. My least favorite pattern I see in repositories is including a subfolder in the git repo with the same name as the repo. This makes paths in code look redundant when loading, ie.

const ActorClass = preload("res://godot_modules/mymodule/mymodule/index.gd").Actor

With the ability to define packages using with a meta file and recognize the convention with the engine you could simplify this to

const ActorClass = preload("dep://mymodule").Actor

Even resources in the file browser could be organized under a new tree for dependencies, and these files can be treated as Read-Only by the editor to prevent accidental tampering. Resource Loading could be referenced with appropriate abbreviated paths too in the TSCN and TRES files.

If this enhancement will not be used often, can it be worked around with a few lines of script?:
Best solution I've found, since not everyone publishes to the Godot Asset Library, is to use git submodules. This also allows for handling any modules that a company may keep internal for personal projects.

Is there a reason why this should be core and not an add-on in the asset library?:
If leveraging a defined convention to redefine how loading is perfromed, it must be implemented at a core level to perform lookups. Using something like npm's package.json to define where "sources" or an "index" exists is helpful as well.

[Calinou]

See also #1205 and godotengine/godot#19486.

I'd be wary about using the package.json name, since many editors and programs will try to detect the add-on as a npm package. Also, we can reuse the ConfigFile class for reading and writing configuration files, so we can use a .cfg extension like we already do with plugin.cfg.

[fire]

Personally I use a different style of directory layout. https://gist.github.com/fire/f3981ad4a48ccae08224d3b82b9f95bc

It's incompatible with this layout.

If you force a layout, I don't know which one to pick..

One alternative that was mentioned was that each addon acts as a .gdignore and as an entire project.

[erodozer]

The layout itself of packages is not one I'd want to see enforced aside from each package gets its own folder installed inside a consistent location, in my example it being godot_modules. Packages should be left to their own on how they want to structure their contents, but encouraged to organize at the root level of their repo so it can be included as a git submodule or zipped up cleanly to be dropped in. Bad layouts that I've seen have been the expectation that you take the repo and overlap it with your own project, instead of nesting it somewhere safe.

Treating each package's structure as a standalone project would be pretty ideal, and matches the typical approach to packaging dependencies as found in other language ecosystems. The concept of "addons" feels fuzzy within Godot as the common interpretation of addons is more like plugins, however some packages may just be asset collections with no scripts. This is why I tend to refer to dependencies as packages or sometimes as modules.

[willnationsdev]

@nhydock

This is why I tend to refer to dependencies as packages or sometimes as modules.

Well, "modules" as a term is a Godot Engine module, as in, the godot/modules directory of the source code.

Packages should be... encouraged to organize at the root level of their repo so it can be included as a git submodule or zipped up cleanly to be dropped in.

I agree. I think establishing this as a convention for "packages", as you put them, would be a good idea. Assuming "package" refers to any redistributable directory containing assets/plugins or engine modules.

Bad layouts that I've seen have been the expectation that you take the repo and overlap it with your own project, instead of nesting it somewhere safe.

I'm definitely guilty of this, but mostly because I was under the impression that that was the convention already, up to this point.

The concept of "addons" feels fuzzy within Godot as the common interpretation of addons is more like plugins, however some packages may just be asset collections with no scripts.

I feel like project/addons is simply "the place for third-party code in a project", much like your godot_modules concept in the post. An addon doesn't necessarily include a plugin and I don't think there is an assumption that they are all plugins. This "common place for storing packages" already exists more or less imo. And if you renamed it to project/packages or something it would just 1) confuse people who were used to it being "addons", and 2) start an identical process of making people consider "packages" as an alternate term for plugins as the vast majority of addons still include a plugin, regardless of what you choose to call them (addon vs. package).

I'm pretty sure reduz still refers to them as addons too.

If anything, what I think would resolve this Issue is simply having an official "this is how you structure an addon repository" page in the documentation - separate from the "how to make a plugin" page, but perhaps with a note on that page that links to it or something.

[Calinou]

For the record, this is how I distribute my add-ons in a way that's compatible with Godot 3.2.x and doesn't require unchecking any files in the asset library's installation dialog:

Separate the add-on code from the demo

The most important concept here is to separate the add-on code from the demo. They must reside in separate repositories, with the add-on code being manually checked into the demo repository to ensure it works out of the box. Make sure to keep the add-on code inside the demo repository up-to-date; for instance, you can synchronize it on every tagged release. You can't use submodules to point to the add-on repository until #554 is implemented.

That said, even if the expected asset library structure is changed to accomodate for Git submodules, keep in mind ZIP downloads will not contain submodules (this is a GitHub limitation). This means downloading the demo from the project manager's Templates tab will no longer result in a functional demo out of the box. Therefore, I would recommend not using Git submodules even if they will be supported in the future.

Add-on

From godot-lod:

├── addons
│   └── lod
│       ├── LICENSE.md
│       ├── lod_cpu_particles.gd
│       ├── lod_cpu_particles.svg
│       ├── lod_cpu_particles.svg.import
│       ├── lod_omni_light.gd
│       ├── lod_omni_light.svg
│       ├── lod_omni_light.svg.import
│       ├── lod_particles.gd
│       ├── lod_particles.svg
│       ├── lod_particles.svg.import
│       ├── lod_spatial.gd
│       ├── lod_spatial.svg
│       ├── lod_spatial.svg.import
│       ├── lod_spot_light.gd
│       ├── lod_spot_light.svg
│       ├── lod_spot_light.svg.import
│       ├── plugin.cfg
│       └── plugin.gd
├── CHANGELOG.md
├── CONTRIBUTING.md
├── icon.png
├── icon.svg
├── LICENSE.md
├── README.md
└── TIPS.md

At the top level, there's a .gitattributes file with the following contents:

# Exclude all top-level files and directories (except addons) from ZIP downloads.
# This makes installing through the AssetLib easier, because no files and folders
# need to be unchecked.

/.editorconfig export-ignore
/.gitattributes export-ignore
/.gitignore export-ignore
/.pre-commit-config.yaml export-ignore
/README.md export-ignore
/LICENSE.md export-ignore
/CHANGELOG.md export-ignore
/TIPS.md export-ignore
/CONTRIBUTING.md export-ignore
/icon.png export-ignore
/icon.svg export-ignore

There is no .gitignore file.

Demo project

From godot-lod-demo:

├── addons
│   └── lod
│       ├── LICENSE.md
│       ├── lod_cpu_particles.gd
│       ├── lod_cpu_particles.svg
│       ├── lod_cpu_particles.svg.import
│       ├── lod_omni_light.gd
│       ├── lod_omni_light.svg
│       ├── lod_omni_light.svg.import
│       ├── lod_particles.gd
│       ├── lod_particles.svg
│       ├── lod_particles.svg.import
│       ├── lod_spatial.gd
│       ├── lod_spatial.svg
│       ├── lod_spatial.svg.import
│       ├── lod_spot_light.gd
│       ├── lod_spot_light.svg
│       ├── lod_spot_light.svg.import
│       ├── plugin.cfg
│       └── plugin.gd
├── default_env.tres
├── fps_label.gd
├── icon.png
├── icon.png.import
├── icon.svg
├── icon.svg.import
├── LICENSE.md
├── lod_mesh_instance.tscn
├── lod_particles.gd
├── project.godot
├── README.md
└── test.tscn

At the top level, there's a .gitattributes file to hide add-on diffs on GitHub. This is optional but recommended:

# The add-on code in this repository is copied from
# <https://github.com/godot-extended-libraries/godot-lod> regularly.
# Please open pull requests for the add-on code itself there, not in
# this demo repository.
/addons linguist-vendored

.gitignore file:

# Godot-specific ignores
.import/
export.cfg
export_presets.cfg

# Imported translations (automatically generated from CSV files)
*.translation

# Mono-specific ignores
.mono/
data_*/

---

Hopefully, this should provide you with a starting point to create easily reusable add-ons that can still be tested easily by anyone from the project manager's Templates tab.

[fire]

Personally I like the requirement Epic Games does on their demo projects to force the contents of the demo to be an easily removable and unique parent folder.

godot-lod-demo:

├── addons
│   └── lod
│       ├── LICENSE.md
│       ├── lod_cpu_particles.gd
│       ├── lod_cpu_particles.svg
│       ├── lod_cpu_particles.svg.import
│       ├── lod_omni_light.gd
│       ├── lod_omni_light.svg
│       ├── lod_omni_light.svg.import
│       ├── lod_particles.gd
│       ├── lod_particles.svg
│       ├── lod_particles.svg.import
│       ├── lod_spatial.gd
│       ├── lod_spatial.svg
│       ├── lod_spatial.svg.import
│       ├── lod_spot_light.gd
│       ├── lod_spot_light.svg
│       ├── lod_spot_light.svg.import
│       ├── plugin.cfg
│       └── plugin.gd
├── godot-lod-demo
│        ├── fps_label.gd
│        ├── lod_mesh_instance.tscn
│        ├── lod_particles.gd
│        └── test.tscn
|         icon.png
|         icon.png.import
│         icon.svg
│         icon.svg.import
│         LICENSE.md
│         project.godot
│         README.md
│         default_env.tres

[Calinou]

@fire The current asset library doesn't allow marking an asset as being both a demo and an add-on. This means you won't be able to see it both in the project manager's Templates tab and the editor's AssetLib tab.

[fire]

That is a defect and I'm going to assume that marking an asset as being both a demo and an add-on will be fixed some day.

I want to propose this structure for that future.

[Calinou]

That is a defect and I'm going to assume that marking an asset as being both a demo and an add-on will be fixed some day.

This is not handled by the asset library rewrite I'm working on yet, so it'd have to be implemented there. Also, it could be a bit confusing as the add-on and demo project would share the exact same description in the preview page.

[fire]

I recommend that the pure addons be made demo projects and all addons should have a demonstration project. However, I leave that up to you. The epic store is able to enforce minimal requirements that we don't have ability to do. So I don't know.

[Calinou]

I set up a template repository for Godot add-ons. You can use as a base for your new repositories: https://github.com/Calinou/godot-addon-template

[willnationsdev]

Should we devise an automated solution for generating the "correctly formatted" structure of an addon project in order to conform to an official convention? And perhaps think of a way of calculating/reporting whether a given directory is a validly formatted addon directory?

I feel like, before you can begin to set any sort of standard about how addons should be formatted, you need to create a set of tools that would actually generate and validate/enforce that structure for end-users.

For example, there could be an interface in the Editor for exporting a subfolder as an addon that would copy the assets to a specified external directory with the correct structure. You could then also create an interface in the AssetLib for tracking local addons, similar to the project manager window, that would give you a report on whether it is formatted correctly.

[erodozer]

If we just want to say no to the git module approach, and instead just focus on only distributing through the asset store, we don't really need to define how an addon project is structured. We should just do like npm does: the git repo can be organized however the project sees fit, but the package.json defines what files are included in the package for redistribution. When uploading to the asset store you wouldn't be uploading a 1:1 copy of the git repository.

Regarding where addons exist within a project when checking them out from the asset store, this is a solved problem of other package managers that we should take inspiration from. The asset store in godot should should always download things to /<project.addon_folder>/<asset_uploader>/<asset_name>/

Asset store user names are unique and should be considered trusted scopes. This makes it safe to check things out by them and prevents other users from uploading packages with the same name and causing collisions if you import both into your project. Of course this is only applicable for when we only have a single registry. If we define the API for the godot asset store in a way that other registries may appear (ie github registry or artifactory support), either we would add the registry url as an additional path label or if there's a collision just check out whichever is found first.

[Jummit]

Here is how I tackled "Package management" for my projects:

My addon repos have the file structure of a Godot project, with all addons the repo provides being inside addons/. This is handy as you can work on the demo and the addon at the same time, and people who want to check it out just have to clone the repo.

I created a python script (https://github.com/Jummit/godot-package-manager) that looks at a list of repos inside godotmodules.txt, downloads them and their dependencies and copies all addons inside to addons/third_party.

This package manager supports cloning the repos and creating symlinks instead of hard copies, which is handy to create pull requests for changes in the downloaded addons. It can also update the addons, which was the main reason I made it.

The addons have to be careful to only use relative paths in scripts and even scenes so they work in both addons and addons/third_party.

I see a lot of talk about git submodules, but I would first look for a solution where everyone, including people who don't use git, can download and use packages.

I think the Godot asset library could work in a similar fashion, keeping a list of assets and copying the contents of the addons folder into addon/third_party/username/addon_name or addons/assets/username/addon_name.

[erodozer]

I decided to upload some addons and docs regarding how I'm just using npm for my packages with godot, tied with github registry to keep everything private and organized from global npm.

https://github.com/nhydock/godot-addons

These are actual packages that I have converted a project to reference and everything works fine and dandy. It's got its upfront quirks, but overall is nice, flexible, everything is just uploaded as simple tar files, and just being able to repurpose existing tooling from well established ecosystems is great.

The initial proposal did not imply directly using npm package.json, but to instead view it as a reference of a good package specification. However, by itself npm is already rather suitable if you choose not to publish using the Godot Asset Store.

As for a Godot specific package manager, I think everything regarding building packages, uploading, and downloading from the godot asset store should also be made available as a CLI exposed by the Godot Editor binary. Once a package spec is established, tooling can be built around it.

[theraot]

Related: #118