Allow modules to live outside of app/code directory

[joshdifabio]

Partially resolves #1013.

This PR aims to enable Magento modules to reside in the vendor directory, or indeed anywhere else on disk, without the use of any Composer plugins and without copying or linking any module files.

How does it work?

This PR adds the concept of a module registry to the Magento\Framework\Module package. This module registry provides Magento with the locations of any modules which do not reside in the standard modules directory (currently app/code). This PR does not remove the existing module bootstrapping functionality and aims to be backwards compatible with Magento 2's existing module implementation.

Added in the PR is a ModuleRegistryInterface which exposes two non-static methods which allow Magento to locate modules. Also provided is a concrete Registrar implementation which implements the ModuleRegistryInterface while also implementing a static registerModule() method which third party modules can use to register themselves with Magento. Magento itself does not depend on the static registerModule() method, which will help with automated testing of those Magento classes which depend on ModuleRegistry.

Modules will be installed using Composer but without the use of any Composer plugins. No code will be copied from vendor, or wherever else the modules are located, into the Magento application directory.

What would a module look like with this approach?

To use this functionality, a module will need to call the static Registrar::registerModule() method in order to notify the Magento application of its presence and location.

bootstrap.php

<?php
Magento\Framework\Module\Registrar::registerModule('FooVendor_BarModule', __DIR__);

composer.json

{
    "name": "foo-vendor/bar-module",
    "autoload": {
        "psr-4": { "FooVendor\\BarModule\\": "" },
        "files": [ "bootstrap.php" ]
    }
}

When Composer's vendor/autoload.php is included, autoload_real.php first processes the various namespaces, class paths and include paths defined in all included composer packages and registers the autoloader. Next, autoload_real.php includes all files defined in autoload.files. This means that, in the above example, bootstrap.php will be included by Composer and will have access to any classes which are known to the Composer autoloader.

Why is this good?

Copying files around using Composer plugins adds a lot of complexity and I believe Magento already know most of the related difficulties. It also makes the standard process of downloading and testing a package very difficult:

$ git clone git@github.com:foo_vendor/bar_module.git
$ cd bar_module
$ composer install
$ php vendor/bin/phpunit

However, this proposal enables the above process to work; even if the module is located in the root while the application code is installed to the vendor directory, the bootstrap.php file will be included by Composer and will register the module with Magento. This would mean that we could finally, easily, create modules and build our code on Travis, Scrutinizer, or any of the other brilliant, free code coverage and static analysis tools.

This weekend (2015-04-24 – 2015-04-26) I'll aim to finish this PR.

[sshymko]

Wow, that's the first usage of generators in Magento 2 :)

[joshdifabio]

I've added an in-depth explanation to the OP.

[otoolec]

@joshdifabio using the composer autoload.files looks like a neat way to allow modules to self register with Magento without requiring a master module registry file or limiting discovery to a set of known locations.

I've been wondering what the best way to solve this problem is as well and have documented my thoughts here. Feel free to also look at the other code changes that were needed as they may help you out this weekend!

I'm curious to know how others about the importance of some features:

• Module can reside anywhere
• Module can reside in vendor or app/code
• Module doesn’t need to provide new configuration
• Module can define own internal folder structure (‘src’, ‘tests’, ‘doc’, ‘config’)

[joshdifabio]

@otoolec Your research will be immensely helpful, thank you. Regarding the four questions you've raised, I'll respond with my own thoughts soon.

Unfortunately, I only had time to update the unit tests at the weekend, so the unit build is now passing but the PR is still WIP.

[fooman]

@joshdifabio Thanks for tackling this - much appreciated!

Having something like this work

$ git clone git@github.com:foo_vendor/bar_module.git
$ cd bar_module
$ composer install
$ php vendor/bin/phpunit

would be awesome. It currently feels like we are almost there with Magento but we can't yet take full advantage of composer.

@otoolec in reply to your questions

Module can reside anywhere

I don't think full flexibility is needed and might even make things more complicated (a code review for example).

Module can reside in vendor or app/code

I think this would be a good compromise - acknowledging that Magento won't be able to port over all modules in time while allowing extensions to use vendor (possibly with a long term view that Magento core modules will also be shifted there)

Module doesn’t need to provide new configuration

I think it is still early for M2 extension development so if this is tackled soon the impact would not be that big. It is still in beta after all.

Module can define own internal folder structure (‘src’, ‘tests’, ‘doc’, ‘config’)

Using composer to install we should get this for free. I think it is a good idea to support what the rest of the php world is seemingly moving towards.

[kojiromike]

Module can reside anywhere

There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're [Marius].
–– excerpt from The Zen of Python, by Tim Peters

I think Magento should be opinionated, so I don't think a module should be able to reside anywhere. I think there should be one obvious right place for a module to reside. I think that place is vendor. ;)

Module can reside in vendor or app/code

For now, for backwards compatibility, I can totally get behind needing to keep some modules in app/code. But it should be deprecated – new modules should be in vendor.

Module doesn’t need to provide new configuration

I don't know if I understand this feature. It's hard for me to imagine how modules would work if they couldn't provide new configuration. Does this include layout?
– update: hmm. I think the bootstrap.php approach is good in that modules should not need to write to files outside their vendor directories to be installed, but I think Magento should still have a way to turn off modules that are otherwise installed in vendor.

Module can define own internal folder structure (‘src’, ‘tests’, ‘doc’, ‘config’)

I think this is in the same vein as my Zen of Python quote above in that module developers should be encouraged to adhere to a consistent structure. But technically, I don't know that it matters that much as long as the module is fully encapsulated.

[joshdifabio]

@fooman, @kojiromike: Consider the following workflow which, I would say, is the standard method used to build and test a PHP package:

$ git clone git@github.com:foo_vendor/bar_module.git
$ cd bar_module
$ composer install
$ php vendor/bin/phpunit

In the above scenario, the module which we are testing does not reside in app/code, nor does it reside in vendor; it resides in the root and the dependencies (Magento core code, etc.) are installed to vendor. If modules can only reside in app/code or vendor then this workflow is no longer possible, which I feel would be a big loss to Magento developers.

If, on the other hand, we allow modules to reside anywhere on disk, implemented in this PR, then the above workflow is possible. See the OP to see how this could work.

[joshdifabio]

@otoolec

I'm curious to know how others about the importance of some features:

• Module can reside anywhere

I think this is crucial in order to support the most basic PHP workflow: clone, composer install, test. In this workflow, the code of the module being tested will not reside in app/code or vendor.

• Module can reside in vendor or app/code

I think that allowing modules to reside in app/code will still be useful if/when proper Composer support is added (as proposed in this PR): For many vendors, it won't be practical to create dozens of VCS repositories per project in order to house individual modules. In many cases it will be preferred to keep project-specific modules in the project VCS repository: It would be extremely messy in such cases to commit those modules to vendor.

• Module doesn’t need to provide new configuration

The less boilerplate the better. Developers hate boilerplate.

• Module can define own internal folder structure (‘src’, ‘tests’, ‘doc’, ‘config’)

Do you have any use cases in mind for this?

[alankent]

Interesting thread. Here is my current opinion until I change my mind! ;-)

The reason for supporting app/code (yes, want to change to app/modules - separate discussion) and vendor/ is to support two use cases. The reason for app/code is "code local to my project"; the reason for vendor is "this is downloaded by Composer".

If you want to contribute a bug fix to Magento 2, you should git clone the M2 repo and create a pull request. If you want to create a production site, you SHOULD NOT do it this way. You should use composer to download official released versions of all packages. It makes sense them residing in 'vendor'. The general rule is "don't touch what is in vendor because it is not yours".

However, it has been pointed out that if every module is in a separate repo, you can use Composer to use git to get the module and put it in vendor, plus if you make local edits (e.g. a bug fix) you can easily push it back to the repo you got it from. This is convenient for developers, but assumes you create a separate git repo per module. The current hackathon approach of copying from vendor to app/code makes it harder (and slower) for developers due to a copy of the vendor files having been made.

So I like the idea of supporting two directories, vendor and app/code. The opinionated position is 'vendor is for composer downloads, app/code is for local code'. The fact that a developer can use vendor in another way is a (very useful) side bonus.

Oh, and to be clear, if you check out M2 from the master git repo, all the modules are "local" to that project, which is why they are in app/code. But if you build a real site project, they will all be downloaded under the 'vendor' directory instead. It is important to remember you WILL NOT git clone the master M2 repo when building customer sites.

[kojiromike]

@joshdifabio I agree with you, but we're talking about different scopes. As a developer, it would be helpful to be able to do

composer create-project my/module
cd my_module && vendor/bin/phpunit # or whatever

and work on that module. (I believe composer create-project mimics the workflow you described.) But that's for writing code, running static checks and phpunit, which may have a few lib dependencies (hopefully mostly just interface mocks) on parts of Magento, but don't need a fully running webstore, and those Magento parts must not overwrite your checkout directory. So to revise:

1. You should be able to run unit tests and static checks after a simple composer install in your module.
2. But the vendor directory should still be considered the "right" place for an installed module in a full-fledged Magento webstore.

[joshdifabio]

The reason for supporting app/code (yes, want to change to app/modules - separate discussion) and vendor/ is to support two use cases. The reason for app/code is "code local to my project"; the reason for vendor is "this is downloaded by Composer".

@alankent Thanks for the reply. There is a third use case, which concerns the active development of a module which lives in its own repository, has its own unit tests and which should be built and statically analysed separately from any particular project code. Think community modules.

Consider the following workflow:

$ git clone git@github.com:foo_vendor/bar_module.git
$ cd bar_module
$ composer install
$ php vendor/bin/phpunit

This is the standard, modern way of developing PHP packages. In such cases, the package (in this case a Magento module) which is being developed and tested will not be installed to the vendor directory; instead, it will reside above vendor, while Magento etc. will be installed to vendor as dependencies of the module. This is the workflow PHP developers use to test their code on CI platforms, such as Travis.

This PR makes this workflow possible and also allows modules to live in the vendor or app/code directories, which I agree should be the standard approach in production.

1. You should be able to run unit tests and static checks after a simple composer install in your module.
2. But the vendor directory should still be considered the "right" place for an installed module in a full-fledged Magento webstore.

@kojiromike Agreed.

[alankent]

Gotcha.

Side bar 1: This is actually another good reason to move service contracts out of implementation modules. When testing, if you have a mock, you can reduce the package download count as you don't need any implementations (and their transient dependencies), just the service contract packages.

Side bar 2: The interesting follow on unresolved question to me is how about integration, functional, etc testing. Where do those cross module tests sit and how to invoke them? But I would rather complete the vendor directory goal first and tackle the next step forwards as a separate objective.

[otoolec]

For "Module can define own internal folder structure (‘src’, ‘tests’, ‘doc’, ‘config’)" I was thinking about making it easier to change the structure of a module while still maintaining backwards compatibility for existing modules. The downside is that each module could come up with its own structure making it harder to maintain modules, but I was thinking of this as being similar to modman where the community came up with a structure different from Magento.

For “Module doesn’t need to provide new configuration”. I meant that no changes should be necessary for any existing modules to suddenly be used from the vendor directory vs being copied over. With the proposal in this PR there would be some configuration in composer.json and in bootstrap.php.

For "Module can reside in vendor or app/code" I was thinking about installed modules, but I completely agree that part of the benefit to allowing modules to live in vendor is that unit tests be run directly from a module's repository, since it no longer tries to copy files over as if it were being installed into magento when composer install is run.

[joshdifabio]

For “Module doesn’t need to provide new configuration”. I meant that no changes should be necessary for any existing modules to suddenly be used from the vendor directory vs being copied over. With the proposal in this PR there would be some configuration in composer.json and in bootstrap.php.

@otoolec Thanks for clarifying. Any Magento 2 modules which have already been implemented will have been designed to run from app/code (either committed directly to that path in a project repo or using a Composer plugin which copies them to that location). If this PR was merged, such modules would continue to function as they did before. With that in mind, is it an issue if we implement a new way of bootstrapping modules going forward? I think we just need to look for the best long-term solution which is achievable and implement it; if we want to eliminate the dependency on Composer plugins then that will impact vendors who've already been working on modules; the key point is that the impact can be minimal.

This proposal in no way conflicts with functionality which would search vendor for additional Magento modules, but do we really want to take that approach? Would we assume that any file named module.xml signifies a Magento module? Or would such files have to be located at vendor/[vendor]/[package]/etc/module.xml? Would we first load the file and check its contents before deciding that we've located a Magento module? What if a single Composer package contains multiple modules? (This is a common pattern.) Are we going to search infinitely deep in vendor for module.xml files? Searching vendor for Magento modules seems quite complex and fragile to me. Is that what you had in mind?

[otoolec]

is it an issue if we implement a new way of bootstrapping modules going forward?

I was thinking about that use case because if we have a way to prevent copying existing Magento modules without requiring them to make any changes, then we can stop supporting the Composer plugin instead of having to support it until all the existing extension migrate. That doesn't make it a show stopper, but I was interested in getting feedback on if anyone else cared about that.

What if a single Composer package contains multiple modules? (This is a common pattern.)

For some reason I thought every composer package would contain only a single module with metapackages being used to group modules together. I know that this repository violates that rule by putting all of the modules together and in the replace section, but I wasn't expecting that to happen with production modules so I wasn't thinking about it, but now that you mention it, it makes sense that someone might want to do the same thing for development so we should try to support. Another plus going for your proposal! 👍

[joshdifabio]

I was thinking about that use case because if we have a way to prevent copying existing Magento modules without requiring them to make any changes, then we can stop supporting the Composer plugin instead of having to support it until all the existing extension migrate. That doesn't make it a show stopper, but I was interested in getting feedback on if anyone else cared about that.

As far as I understand, any existing Magento 2 modules will require magento/magento-composer-installer in composer.json and so they'll need updating regardless if we want to remove the Composer plugin completely.

[barryvdh]

Do plug-ins need to be enabled automatically, when required with Composer (eg. with the autoload) wouldn't is be possible to use some kind of ServiceProvider class (Like other frameworks) which would be autoloaded when requested and sets up the module paths (and perhaps other bootstrapping needed). So modules can just be temporarily disabled without changing the composer file, or enable of of multiple modules from 1 package.

[joshdifabio]

Do plug-ins need to be enabled automatically, when required with Composer (eg. with the autoload) wouldn't is be possible to use some kind of ServiceProvider class (Like other frameworks) which would be autoloaded when requested and sets up the module paths (and perhaps other bootstrapping needed). So modules can just be temporarily disabled without changing the composer file, or enable of of multiple modules from 1 package. – @barryvdh

Good question. I certainly think it should be possible to explicitly disable modules, but do we want to enforce an additional step after composer install if this isn't necessary for other reasons (e.g. deploying static assets which modules provide)?

As someone who expects to run composer install or composer update and have Composer install and bootstrap any new dependencies, it would be tedious as a project maintainer to have to take additional, manual steps afterwards. And from the perspective of a module vendor, I'd want the freedom to re-factor a module into two modules, or require other modules using composer.json, without requiring any action from project maintainers beyond a simple composer update.

How about agencies with code suites which are common to many clients; if a module is added to such a suite, do we want agency developers to have to go through many client projects, individually enabling any modules which are newly required as a result of a composer update?

My personal preference is probably quite obvious: bootstrap and enable modules by default, and allow developers to disable them explicitly. But that doesn't mean I'm not open to changing my mind! I can appreciate that there might be reasons to require a manual deployment step after installing new modules.

How do Magento see this issue? @otoolec @alankent

[otoolec]

Should installing a module via composer cause it to be enabled and setup automatically? This would certainly be convenient, but would there be a potential negative impact since modules alter the database when their setup scripts are run? Not sure how everyone feels about having composer update causing database writes as a side-effect.

Right now to get a sample module to show up in a clone of this repository one needs to run the following commands:

composer config repositories.magento composer http://packages.magento.com/
composer require magento/sample-module-minimal:*
php bin/magento module:enable Magento_SampleMinimal
php bin/magento setup:upgrade

It should be possible to create a new command that simplifies some of the above if desired. It could also be possible to automatically enable a module but not run setup:upgrade when the module is installed from composer, but that will most likely break Magento instead of doing anything useful.

Thoughts?

[kojiromike]

I think you're right, @otoolec, that it would be inadvisable to let a module make database changes after just a composer install.

That said, it's worth considering the use case for an integrator managing projects with many module components. Must that integrator have a script that expressly enables all the modules one by one? Writing such a script would be challenging because you'd have to figure out a way to map a package's composer name to its (one or more) Magento module names.

What I'm saying is that there probably should be a way to enable modules on composer install, but I would not want it to be the default behavior.

Secondarily, it would be really nice if there were a convenient way to map a composer package's name to its Magento modules, because I have a feeling that's going to come up again.

[joshdifabio]

Should installing a module via composer cause it to be enabled and setup automatically? This would certainly be convenient, but would there be a potential negative impact since modules alter the database when their setup scripts are run? Not sure how everyone feels about having composer update causing database writes as a side-effect. – @otoolec

Regarding database upgrades; I agree, they shouldn't be executed automatically, particularly as database upgrades are per-environment and most agencies won't be running Composer on UAT or production.

Regarding the default state of modules (enabled or disabled) upon installation, I think I'd prefer them to be enabled by default, and I'll try to explain why.

Firstly, the way extension developers work at present is to distribute their modules such that they are automatically enabled when they are pulled in by Composer. Check firegento for some examples (note that all the modules specify active = true in the module configuration files). The typical approach in Magento 1, which works well, is approximately:

1. Add a package to your project's composer.json file; Composer installs potentially many Magento modules
2. Use magerun to see which modules have pending database upgrades
3. Use magerun to execute those database upgrades manually

Executing database upgrades with a single terminal command is easy enough; Magento developers are used to doing that and it's easy to integrate into the deployment process. Knowing which modules need to be enabled after running Composer would be less straightforward if there were a number of changes. Developers would most likely just enable everything after each update, or they'd forget to do it. Is there value in that approach?

If I add a package requirement to composer.json then it's because I need that package; whether I'm the project maintainer or an extension maintainer.

It could also be possible to automatically enable a module but not run setup:upgrade when the module is installed from composer, but that will most likely break Magento instead of doing anything useful.

True, but developers are used to this. Also, what if I updated my module and added some new setup scripts? If the module was already installed in a project and was updated, it'd still break as it'd already be enabled but the database schema would be out of date. Developers will always need to run setup scripts after updating their code. I think we should make things as simple as possible and have a single workflow.

[vrann]

@joshdifabio corner case situation, though it was mentioned here before: what if you download a package consisting of multiple modules and want just half of them being enabled? Right now we have just one package like that, magento2-community-edition, but I feel this will happen more often once bundle packages will be implemented.

Another concern with the module enabled via composer plugin is that sometime you may need to distinguish these two steps. For example if we build Web UI wizard on top of the composer, most likely you would like to distinguish package download and installation, because more steps can be added in between (like, database backup)

Personally, I would prefer to do what @otoolec proposed to have a wrapper tool on top of composer which can automate package installation and module enable in one step.

[antonkril]

Static calls are hard-coded dependencies, they can not be substituted or decorated. So please avoid them. Other problem is that with such approach there is no strict rule when all module paths are available, because you don't know if all modules called setModulePaths, and module dev doesn't know when it should call setModulePath.

[joshdifabio]

@antonkril thanks for reviewing this.

Static calls are hard-coded dependencies, they can not be substituted or decorated. So please avoid them.

I agree with you, and I avoid them wherever possible and appropriate. Sometimes, however, it is necessary to use them. Refer to one of the static methods in Magento 2 for an example.

In this case, the only way to achieve what I've proposed in this PR is with a static method call. Please take a look at the section entitiled What would a module look like with this approach? in the OP to see why.

Furthermore, I consider a module to be a static entity in the same way that a class is a static entity. While a class has a single location on disk, so does a module. Consider that autoloaders are registered in PHP statically (via spl_autoload_register). Some functionality is static. This PR essentially provides autoloading for modules; the reason we need to implement it ourselves here is that packages/modules aren't first-class citizens in PHP in the same way classes are.

Also note that nothing in the Magento core depends directly on that static method call, so it doesn't affect testability.

Other problem is that with such approach there is no strict rule when all module paths are available, because you don't know if all modules called setModulePaths, ...

I'm not sure I understand the argument. If a module doesn't reside in app/code and doesn't enable itself via the module resolver then yes, it won't be enabled. Similarly, it won't be enabled if it doesn't implement a module.xml. I don't see what's strange about that.

and module dev doesn't know when it should call setModulePath.

Please read the module implementation suggestion in the OP.

[joshdifabio]

There are a few classes in the Magento code base which will only work with modules located in app/code whereas they should use the Magento\Framework\Module package to locate module files. Regardless of the approach we take with bootstrapping/installing modules, if we wish to remove the limitation of having to copy modules to app/code we will need to re-factor those classes. Can I suggest that that work is done as separate PRs which don't attempt to address the issue of bootstrapping/installing modules as a first step? That work seems like a no-brainer, and at present a few of us are duplicating that effort across PRs which seek to address too many issues at once.

Thoughts?

[otoolec]

@joshdifabio I'm all for incremental improvements that get us to where we want to go. I think your pull request is a step in the right direction.

Was there anything else you wanted to add to this PR?

[danslo]

Was this tested? It works for registering the module but as it looks now, it will never find module configuration files.

Magento\Framework\Module\Dir\Reader::getConfigurationFiles calls getRelativePath but this will never be accurate for things living in vendor directory, because $basePath is <path_to_magento>/app/code.

[danslo]

I'm also not quite sure why getModulePaths and getModulePath in Magento\Framework\Module\Registrar are non-static.

[Vinai]

I haven't specifically checked the PR, however I have created a bunch of modules installed directly (and only) into vendor since, for example https://github.com/vinai/VinaiKopp_EavOptionSetup. Works okay.

[joshdifabio]

@danslo There are a number of places in the code base which still need re-factoring in order for this PR to work as intended. I believe it's with @otoolec of Magento but I'm not certain. If it's not being looked at I'm happy to nibble away at some of those issues myself?

Edit:

I'm also not quite sure why getModulePaths and getModulePath in Magento\Framework\Module\Registrar are non-static.

They aren't static because they don't have to be static. Those methods are part of ModuleRegistryInterface which doesn't contain the static registerModule() method. The clients of the ModuleRegistryInterface are Magento classes which need to be able to locate modules; by making those methods non-static it's then possible to inject other implementations into those client classes, for example mock objects to help with testing. registerModule() is static because it has to be, and its clients are modules, not the Magento classes which need to be able to locate modules. So, although all these methods are implemented in ModuleRegistrar, they implement different interfaces and have different clients.

Hope that makes some sense!

[danslo]

@joshdifabio Fair enough on the static discussion, I just never saw an instance of Registrar being passed anywhere, that's kind of the reason for doubt.

After seeing this PR merged I (wrongly?) assumed this was working. I can definitely confirm that di.xml configuration (and likely others) aren't loaded when not in app/code (reasons mentioned above). Vinai's module only has a module.xml, maybe that follows a different codepath.

[joshdifabio]

...I just never saw an instance of Registrar being passed anywhere, that's kind of the reason for doubt.

It should be handled by a preference in di.xml.

After seeing this PR merged I (wrongly?) assumed this was working. I can definitely confirm that di.xml configuration (and likely others) aren't loaded when not in app/code (reasons mentioned above). Vinai's module only has a module.xml, maybe that follows a different codepath.

It was definitely a fair assumption to make! There are a number of places in the code base at which it is still assumed that all module code lives in app/code which need re-factoring, such as the one you've highlighted; this is probably why Magento aren't yet officially recommending the module registrar approach. I'm not sure how high this is on Magento's priority list, but I should have some time this week so I'll submit some PRs myself if the work isn't already done (@otoolec).

[Vinai]

I also assumed the PR was complete!
Maybe compiling a list of places where app/code is still assumed is less work then directly changing them. If so, please post it here then I can commit some time to this, too, without us working on the same areas.

[joshdifabio]

I'll create a new issue after work this evening with a check-list of the areas which require attention, or you can create it now if you'd prefer. We can use that to keep track of what needs doing. I'm sure someone from Magento will let us know what's already been resolved internally.

[Vinai]

Won't have time today unfortunately, but can't wait to help make this PR feature complete. One of the most important changes from my perspective :) 👍

[alankent]

We would love this change, but its not high priority internally due to other commitments. Discipline and cut lists is how we are hitting dates reliably. So we love the work but not likely to happen for 2.0 without this wonderful community contributions continuing.

[joshdifabio]

We would love this change, but its not high priority internally due to other commitments. Discipline and cut lists is how we are hitting dates reliably.

I'm sure anyone who does this for a living can relate. Thanks for the update, keep up the good work.