-
-
Notifications
You must be signed in to change notification settings - Fork 97
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New Add-On (sub-project) system #1205
Comments
Modified the text above, renamed addons to sub-projects, which makes more sense. |
I am writing my plugin along with 3 other ones that extend my workflow:
Each are in a separate git repo, but I copy them in the HTerrain project because I need them for testing a real environment. I don't do the opposite because those other plugins change way less frequently.
|
Questions:
Thoughts:
|
Sorry if this may sound "dumb" but I'm very inclined to ask: can this be used for, as @Zylann seems to be, somekind of dependency system? E.g.: I'm currently creating a Platform Template for my future projects, and I'm likewise making updates in the Actor Plugin from many other projects, currently I have to update the Platform Actor Plugin manually and likewise updating all projects that shares this dependency. And there are many other "modules" for animations, particle packs etc... Can we break the dependencies into git projects and "patch"(?) every other project to use the up to date version while keeping the |
Awesome idea, I was having some trouble with how to maintain an asset. If it's also a project but this solves everything. @toger5 I don't think that we will have to select anything. Just create as if it was a project. No need for creating addon file i believe. I want to see a way to keep tabs on the plugin.cfg file. I really tend to forget about it. Probably it should be editable and visible in editor. Probably a template plugin can be generated too something to start of having basic features. And as the other guys said will there be support for a dependency system? |
@swarnimarun plugin.cfg is needed to define plugin-specific script stuff though, and is currently the only thing that allows to differentiate an "end-project" and a plugin. |
@Zylann Atleast it can be visible and editable in Editor. |
Does your main scene now have to extend an editor plugin? |
@LeonardMeagher2 plugins don't need to be a scene, so I'd say no |
@Zylann I guess I'm confused, what's the difference between an add-on and a plugin? I imagine if this helps organization or use, it'd be the method used for any add-on including plugins (which I'm not sure there is a difference) |
@LeonardMeagher2 They are not removing plugin.cfg. That was me being silly. And, addons and plugins are the same. |
I suggest we do the following.
Would that work? |
Having two systems in place (asset and package) seems confusing IMO. |
@Zireael07 It's not really two systems imo. Think that the diversity of stuff you can download from the asset library is huge. Programmers may be more interested in doing a set of reusable classes, which need to be installed on a fixed place, while you may just want to download a pack of textures, shaders, materials, animations, etc. and put them on a specific directory in your project, while you expect them to keep working. The system should be as simple as possible, but also as flexible as possible to support most use cases. |
I'd just like to:
|
AIUI there are 3 types of things provided by the asset library:
For templates, the current system works well, because you just need to download it somewhere and start editing. You don't care about updates, because it's quite easy to break things or override the work you did. Dependencies on assets and plugins can still be managed by Godot like it would manage in any project. Plugins can be put in a fixed location, because they are usually self contained. They might depend on another plugins, so we can either use the fixed path or make some special code for other plugins. The problem is one different plugins depend on different versions of the same plugin: should Godot install both or only the most recent? This is the most problematic part of a dependency-resolution system. Assets can be put anywhere, so their path must not be absolute. If assets can depend on others, it's quite easy to become a mess. I can see assets as "sub-projects", but I'm not sure how paths could be properly resolved or how the many |
Okay, I'm liking the idea of plugins as projects, but a bit confused about the plugin vs asset distinction. If I understood correctly: Both will be project plugins? One will have a fixed install? (addons/?) like now but they won't be able to put anything in root? But the others won't, and they can be moved around by the user? Regarding plugin versions, maybe plugins could specify a range their dependencies (plugin A requires ^0.0.1, and plugin B requires ^0.1.0 = 0.1.0 - to anything below 1.0.0 is installed), so only major version changes would cause a problem (plugin A requires ^0.0.1, and plugin B requires ^1.0.0), and in that case, could Godot support dependency nesting? Otherwise just throw an error/warning? Edit: Also will regular projects be able to depend on plugins as well? It would be nice as I mentioned in godotengine/godot#18131 to have project templates as a workaround to the lack of a global plugin system where the project requires x plugins and they're automatically downloaded and installed if missing. |
Maybe relevant: godotengine/godot#17092 |
Sounds really good, love the general approach. One beef I had with the current system, which is why I have two repos for each GDNative project, doesn't seem solved by this so I'm just flagging it if anyone has any ideas. This may be limited to how a sub-project is made available on the asset library then the actual structure of the sub-project itself. For GDNative projects you have the C(++) code that builds the module, and then the module itself which would be really nice to deploy as a sub-project. Obviously I can keep working the way I do now, having the C(++) code in a "source" repo, and the compiled version with all its support and configuration files in an "asset" repo on github. My ideal scenario would be to have one repo but the root of the sub-project being a folder inside of that repo. Incidentally that is how my source repos currently are stored, with a folder called "Demo" containing the files for the asset that I simply copy over to the "asset" repo when deploying it. |
There are many use cases I can think of where you'd want an extension to the editor to persist between projects. The API allows us to modify some global settings, so I wonder what taking this direction means for that notion. I would say that changing the name to subprojects makes it clear that these exist to extend the tools available to your project, and not be seen as integrated as an actual extension to the editor itself. The current system is mainly only cumbersome to git repos in my view because in order to deploy it easily to end-users with the editor's AssetLib integration, your repo must have a structure which is not conducive to working with how most addons are actively developed (as extensions to an existing project). Multiple addons can't easily be in the same necessary folder structure while each having their own repo -- especially on Windows where symlinks aren't always obvious -- and the parent project needs to be put into .gitignore for the addon repo if you want to deploy directly from there to remote, which would also interfere with the parent project's repo, if it exists. I'm not understanding yet exactly how the new system addresses this, but I have a vague idea. It seems "okay". But, I'm for having a fixed location option available. I'm sure sub-projects are great, but I'm also sure there are probably some improvements that could be made to the system that would put a lot of the same power into the hands of people a lot faster. The way it's structured now heavily implies that the majority of addons were expected to be designed for use in monolithic projects already, ironically. Just that they were ones which would've had limited use on an editor-wide scale. Addons in a fixed editor-wide location can cede priority to one installed local to a given project. P.S. Git submodules aren't necessarily for everyone... |
@nobuyukinyuu I agree, global editor extensions would be very useful. I wanted to wait until the system is more or less stable to work on that (or for someone to make a proposal). The problem though, is that many times what it seems like it should be global is not, as example a Tiled or PSD importer should be in the project, so it will work for other users cloning it. @Zylann I am not really sure a package management system should be compatible with cloning git. If you manage packages, you generally need the list of files you installed to be able to update it, which is not the case of git, but I'm still sure we should be able to find a way to do a simple workaround for your use case. @vnen Still, from this thread, it seems to be that package management is a lot more important than I thought it was, and it makes a lot more sense now |
I don't think adding dependencies is a good idea. Addons should be self-contained if possible, and even in situations where it is not feasible it shouldn't be encouraged. Else you end up with situations like NPM where each package depends on at least 10 other packages going into hundreds of packages in some cases. This can get out of hand really quickly. That said it should be possible to have installed addons interact with each other, maybe just a few functions for querying what other addons are active and what their base path is? |
@API-Beast how do I solve this in a user-friendly way, then Zylann/godot_hterrain_demo#1 |
@Zylann You could bootstrap a script file that checks for dependencies and provides the user with the option to install the dependencies for them. It's not functionality that needs to be implemented in the engine itself. If necessary some API functions could be added to simplify the process of writing such a script. |
@API-Beast well yeah, that's exactly what I mean by additional work for all maintainers. |
@reduz yea I agree that having local addons has the huge advantage when working in a team. and it is surprising how many things are needed for everyone working on the projects (your importer example is good!) |
After giving it a bit of thought I have come to the conclusion that this proposal is a bit too complex. Godot always had the advantage that it keeps things simple by working very close to the file system, changing how "res://" paths work would work against it. I would instead suggest to:
|
@API-Beast makes sense to me. But I still don't completely understand reduz's proposal so I might overlook something. this also would be nice so you could still use the |
It would really help if symbolic links in Windows are detected properly. |
I would keep the current addon system, but add a dependency system.
All this info could be in a config file.
This would
As for subprojects, I think this is an interesting idea and I also think it would be wise to implement this as an extra feature at first time, since there are always hidden pitfalls, while the current addon system is generally working and tested. Also, there seems to be a lack of examples of the structure of the subproject and its use, API (at least in the form of pseudocode). |
The directories which contain other (nested) Godot projects are skipped during filesystem scan entirely, the behavior is the same as for having Here's a patch which allows you to display nested projects: diff --git a/editor/editor_file_system.cpp b/editor/editor_file_system.cpp
index 78fd88e50..627c4ea7d 100644
--- a/editor/editor_file_system.cpp
+++ b/editor/editor_file_system.cpp
@@ -684,8 +684,6 @@ void EditorFileSystem::_scan_new_dir(EditorFileSystemDirectory *p_dir, DirAccess
if (f.begins_with(".")) // Ignore special and . / ..
continue;
- if (FileAccess::exists(cd.plus_file(f).plus_file("project.godot"))) // skip if another project inside this
- continue;
if (FileAccess::exists(cd.plus_file(f).plus_file(".gdignore"))) // skip if another project inside this
continue; I also think that's the first step to start implementing this proposal... 🙂 As for me, I've recently stumbled upon limitation of maintaining self-contained parts of a game project. I've started to create various "sandbox" projects just because it allows me to better concentrate on features which are relevant at the moment, and it's much quicker to launch Godot (3-6 seconds compared to 10-20 seconds...) Then, I'd just copy and paste a working implementation to a main project once completed, but then yeah the only thing which prevents this kind of workflow is exactly having to update resource paths within scenes manually. Something tells me that this part of the problem should be straightforward to resolve (if we put plugins/addons concerns aside, of course). |
Implementation for subproject paths resolutionI've come up with proof-of-concept implementation for resolving subproject paths in my fork: godotengine/godot@3.2...Xrayez:subproject-paths (for Godot 3.2, can also make a draft PR for discussion purposes and easier review). The way it's done:
All resources, including those in subprojects, get imported into the main project, and can be referenced just like any other resource. Normally, you shouldn't try to use subproject local paths in the main project. This approach seems to resolve a lot of limitations without much changes. But this universal solution has some caveats:
I think those issues can be resolved as denoted in the proposal text:
All those caveats can be resolved if you make sure that you have unique subproject paths to avoid those path collisions (just like people do with addons). I've likely over-engineered something in the process, so feedback welcome! This is the first time I'm modifying different core parts of the engine at the same time... Also, there are some issues with renaming/fixing dependencies, not sure how to solve yet, probably just have to skip subprojects. |
Hi All! Recently I implemented my own plugin management system for my own modules. It mostly designed for process binary precompiled modules for third-party SDKs for iOS and Android. There are 37 production ready packages: analytics, ads, social network integration and so on. The system has dependencies, autoloading gd wrappers, export hooks, uniform module settings for app keys and so on. It can be used as GUI addon and as CLI utility (for example when you want build your project with CI and don't want to store all binary frameworks in github repo. They can be automatically downloaded before your project was build.) Full info and package list: https://drmoriarty.github.io/nativelib/ What do you think of it? |
@Xrayez, will we see it as a PR? |
I'm not sure about the implementation (may be hacky or too brute-force), and it solves only 50% of the problem (does not integrate with addons system at all), so I don't see the point of submitting a PR, especially given the fact I've had quite a bunch of feature PRs which haven't received any feedback in the past. I'm sure reduz will come up with something regardless. Nonetheless, if the general implementation is approved, then I can continue the work, that's the reason of me writing here instead of creating a PR, and the entire point behind GIP. But I'm reporting that I've been using the patch successfully in my own project so far, there are minor limitations but it works well for my use case. |
@Xrayez, thanks for the info! Maybe we can start with this patch?
|
That patch does not make any good in and of itself because there would be still an issue with resources being read as part of the main project, leading to parse errors in scripts etc, it's just a crucial first step to take. This proposal requires a complete solution, in my opinion. |
@Xrayez, you are right. BTW, I checked your implementation and I like it :) |
The following notes are weaved in throughout the chat thread, but I wanna make sure these current issues are clear... It would be nice if add-ons were a bit more siloed off when it came to the Project settings. It would be a lot of work to have all the "subprojects" have their own ProjectSettings, but I definitely see the benefit and agree that it may be worth the complexity. I could see turning it into a resource, like environment, that shows up in the "FileSystem" window. |
Idea: subprojects could share plugins of the root/main project. This is not quite global plugins (#831, which is already rejected for non-editor plugins I presume), but at the same time allows to reuse plugins locally in subsidiary projects (like sandbox projects used for prototyping stuff before incorporating changes into the main project). I see it would be especially important for the new GDExtension feature because I'd expect features that act like "core" to be present in all projects where I run Godot (to achieve the "built-in" experience you get when compiling Godot with C++ modules), not just where the plugin is installed locally per project. So, I see this as Top-Down vs. Bottom-Up feature: subprojects may be used as plugins for the main project as proposed here, but at the same time root project can pass down plugins to children project (those which are not necessarily plugins). I know the idea may be crazy but it would be wrong not to share it. By the way, I've been using nested projects with godotengine/godot#51895, works fine so far for my own use cases. |
When you nest one godot project in another I made a command line tool that easily allows you to correct these files (and undo that as well), which allows me to work with arbitrarily nested projects without difficulty. If your projects already use git as vcs you can use git submodules to your advantage as well. Hopefully this can hold some developers over until this feature arrives |
After discussing with @bojidar-bg I think we reached some general idea on how the add-on system could be improved. Currently, the add-on system has many limitations:
Most of the feedback for the current system centers on that it should be great if it could be possible to:
So, we've been discussing this for a while and reached the following proposal, which I believe allows to implement this in the simplest way.
Writing an add-on will work as follows. It will not be an add-on in itself, but a sub-project:
Unde the hood, sub-projects can go anywhere within the main project and it will work the folowing way:
So this is pretty much it. This approach is the simplest we could come up with, should require little work from the codebase, and should make much, much simpler the process of creating addons (which are simply sub-projects now). Feedback welcome!
The text was updated successfully, but these errors were encountered: