Clarification of Versioning Scheme (with proposal for modified SemVer)

[navarr]

I'm posting this issue to get it in the minds of the Magento developers as early as possible.

Something that's greatly annoyed me with Magento 1 (which probably won't be an issue with Magento 2 thanks to using composer) has been the version and patch system.

For example, in Magento 1.9.0 you have 1.9.1 and you also have superfluous patches. This makes it difficult to detect customizations to core code by bad developers and other such things. If Magento is no longer going to be releasing patches with the 2.x product, then you can ignore this next paragraph, otherwise, continue on:

As opposed to releasing just patches, I propose that the Magento team release a new version. So for example, if there was a problem in 2.0.0 and you release a patch, the fix should make the version 2.0.1 and the patch is an upgrade from 2.0.0 to 2.0.1 - so there is no longer a need to document all the patches applied and it does not cause undue strain on third party development efforts.

To accompany this, I propose that Magento clarify & use a semver inspired versioning scheme. Magento, being company focused, probably won't want to use SemVer as is (major number changes per breaking change) - I suggest using a modified semver, where Magento has four version numbers:

a.b.c.d

a - Product Version - mostly for marketing purposes, this number would be "2" for a very long forseeable time.
b - Major Version - incremented with incompatible API changes
c - Minor Version - incremented with each new feature added in a backwards-compatible manner
d - Patch Version - incremented with backwards-compatible bugfixes

With patches incrementing the patch version (and ideally those releases tagged in the repo and installable by composer) we shouldn't have any issues with patches making it difficult to detect the true version of Magento.

[piotrekkaminski]

Thanks for the comment. This is very close to what we are planning to do. We have some question how to support (and do we need to support at all) patches that might be customer specific (eg resolve customer issue but have side effect that might not work for everyone) with versioning. However the general direction is to use semvert, hopefully per module.

[buskamuza]

@navarr , just some thoughts from support perspective.

Patches are changes that may be created and applied very fast - in few hours or in one day. Especially if this is a patch for a blocker. Such workflow may bypass general rules (e.g., definition of done) for the development. For example, it may be not covered with tests or documentation (if needed).

Releasing a version required all the rules to be followed + some activities related to the release done (e.g., update version, run all CI builds, publish the repository).

Second one (release activities) can be reduced to very minimum due to having our repository really open now.

The first one is more complicated. We can't make customers wait until we implement tests. We should provide the fix ASAP. Probably, we can collect all the fixes in a current major release branch (e.g., 2.0.*) and implement tests for them before the next release. Here would be great to have enough resources to cover everything that should go to the next release. But even if this will work fine, original customer who reported the bug will have the patch applied instead of new version installed. Also, I'm not sure that customers will agree to upgrade to the new version (that includes fixes for other customers as well, not covered with tests well enough) just for getting one fix.

[navarr]

@buskamuza My proposal was more for "path releases" where the release is incredibly small and implements the last worthwhile version number, and may contain only a single fix - where you can then roll out a patch file to a customer that simply upgrades them from the patch to the new (very minor) version.

Which would provide a very clear upgrade path.

[buskamuza]

where you can then roll out a patch file to a customer
Is this still the same patch you'd like to get rid of?

In case we're working on multiple fixes for multiple customers how the repository with patch releases should look like?

2.0.0
|
2.0.0-patch1 (fix for customer A)
|
2.0.0-patch2 (fix for customer B, includes fix for customer A because it's the same branch)

or

2.0.0
|  \
|   2.0.0-patch1 (fix for customer A)
|     |
2.0.0-patch2 (fix for customer B)
|   /
2.0.1 - release that includes all patches

Other option?

P.S.: I absolutely support usage of normal Git/Composer tools instead of patches and understand the problem with patches. Just want to see the strategy more clear.

[choukalos]

@navarr Thanks for the input. Per @piotrekkaminski comment our current proposal thats being vetted is pretty close to what you're recommending. Here's what we're thinking:

Use ~Semantic Versioning aka Major.Minor.Patch as the version # with rules/automated check on what constitutes a major/minor version change. We'll have some guidelines on what the version #'s mean in terms of backwards compatibility and hooked into the install tool to flag extension issues before install or upgrade. To start we'll just version all of M2 with one version # ( so really a hybrid versioning ) with the future plan of perhaps versioning by module ( that would be semantic versioning ) later on if that make sense.

For a packaging/release/patching structure we'd do pretty much what you laid out. Create release branches off main, with patches effectively patch point (i.e. 2.0.x ) off the release branch. That way it'll be easy to get the latest release's code w/all known patches included. It might look something like this:

main
|
+------ 2.1.0 ----- 2.1.1
|
+------ 2.0.0 ------ 2.0.1 --- 2.0.2

Where 2.0.1 & 2.1.1 are patches created to address issue X, 2.0.2 is to address issue Y (already solved in 2.1.0 branch & mainline ).

• If you need a fix for Y but don't care about X is it okay to just get it as a bonus? (i.e. you git pull 2.0.2 )
• If release notes include which patches are included (for the main release) will that be clear for merchants who are upgrading from say 2.0.1 to 2.1.1 (or perhaps even larger minor version gap?)
• Other feedback, concerns?

Thanks,
Chuck

[navarr]

That sounds very good. My primary concern is that in general the Magento 1.x way of "patching" a system creates versions of Magento that are completely unknown.

My specific use case for this was writing a system that detects core code modifications (when inheriting a project) so that we can see what's been done and if we need to correct it. Patches make it impossible to compare against a version.

So long as patches end up being a version (such as 2.0.1, 2.0.2) I'm happy, as it provides a known-version state to compare to.

I personally feel that if you need a fix for Y and get a fix for X that's an added bonus and should be perfectly okay.

I imagine the reason most people would be hesitant to upgrade would be massive changes, so a SemVer patch number that increments with each patch, and if desired patch files that upgrade between those numeric increments, would create both a known state of the application and would ease the client into doing non-harmful upgrades.

[alankent]

We have multiple parties internally to all get in alignment. This one affects both Support and Engineering. The Support org needs to get patches out quickly, but does not like per customer patches either (harder to track and maintain). With all the test automation in Magento 2 I am hopeful we can ship fast patches but through official channels. All the work into modularity and better APIs and contracts between modules is to also make it safer to do patch updates. There are other techniques like releasing new features turned off in configuration if we need something for a specific customer. I am hopeful the new norm for Magento 2 will be a lot better for everyone in this area.

[ktomk]

@alankent: Is this still unfinished or can you present some binding scheme in written form?

[alankent]

We are definitely committed to Composer, and plan to follow standard practices there (to avoid problems).

However it is not 100% finished. For example, @moleman #1680 pointed out we might not have dev releases on mainline and composer playing together right yet.

But none of the above discussion has changed. See also a brain dump of related threads in http://alankent.me/2015/08/09/composer-and-connect-and-patches-oh-my/

[ktomk]

@alankent: Are you that committed to composer that the use of composer autoloader will be 100% unhindered? - That's merely the only question I would have right now while you put composer on topic.

And if you still have issues with XSDs in different directories: http://php.net/libxml-set-external-entity-loader might help

• Composer and Connect and Patches! Oh My! (Aug 2015; by Alan Kent)

[alankent]

Very sorry, I missed this in my email flood! I am not sure what you mean by "composer autoloader will be 100% unhindered". Let me say the code we have today is on GitHub. We are doing some work to try and leave more stuff under 'vendor' by GA, but that will be progressive (not a full solution by GA). We plan to keep working on this over time.

And we have a solution for XSDs using paths relative to modules thanks. I am not sure if it will make the GA cut or slip to 2.1, but it is well under way. As we get closer and closer to GA, we are trying to get as much in as possible, but need to let things bake as well. I predict a minor traffic jam of merging last code drops, then letting things settle through final QA phases etc. We want to do this one, but it is not a GA blocker.

So we are definitely trying to be more and more "Composer friendly" - that is a commitment.

[ktomk]

@alankent

Regarding your question about the composer autoloader: by "composer autoloader will be 100% unhindered" I just wanted to write that vendor/autoload.php is required and those autoloaders registered by it aren't hindered. I hope this makes it more clear. And I hope that's covered by what you coined "composer friendly".

[buskamuza]

Hi @ktomk , vendor/autoload.php is already used by Magento application - https://github.com/magento/magento2/blob/develop/app/autoload.php#L17
Could you, please, clarify what exactly is wrong with the current implementation? Thanks.

[ktomk]

@buskamuza: Wrong is such a broad term, where should I start where should I end? And so profound. Let's just call nothing wrong at all and just imagine you want to add your own requirements to Magento 2 (and those requirements are inheriting from Magento 2 components) as composer requirements. AFAIK this straight forward operation ain't yet possible.

This btw. is something I would call Magento 2 being composer friendly for if it worked.

[alankent]

Sorry, I still don't quite understand. Could you provide a concrete illustration of what you think cannot be done today that you want to do? "Inheriting from Magento 2 components" - your module can declare a dependency on a M2 module, and you can then inherit from classes or interfaces in that module. Hence you can extend M2 with additional functionality. There is also dependency injection, plugins, events, etc - lots of ways to extend M2. That is certainly supported today, and fundamental to the whole purpose of M2. If you could provide a more concrete example that would be really helpful to give a more concrete response. @buskamuza was just asking for what you thought was "wrong" in M2 that we can improve to meet your needs.

[buskamuza]

@ktomk , thanks for the response. So, as I understand, you have a module "ktmok/module-a", which depends on Magento 2 core module "magento/module-catalog" and you want in your Magento installation to make dependency (require) to your module "ktmok/module-a" in Composer configuration.
I'm not sure why you can't do it. Or maybe you have different example.
In your module "ktmok/module-a" you add require for "magento/module-catalog" in composer.json, so it will be considered by Magento and Composer as a dependency.
So, there are two ways of using Magento 2 now:

1. Cloning this Git repository. It is useful for the case, when you want to contribute some fix/feature to the core. Still, you can add your "ktmok/module-a" module to app/code and it should work fine. Magento will use information from composer.json to see that it requires "magento/module-catalog". Composer just doesn't need to know anything, as it does not install module "ktmok/module-a". Autoload should work because there is this section in the root composer.json - https://github.com/magento/magento2/blob/develop/composer.json#L225
2. Deploying Magento using composer create-project magento/project-community-edition (it's based on https://github.com/magento/magento2-community-edition). If you want to add your module "ktmok/module-a" to this installation, you just do "composer require ktmok/module-a" and it will be added to requirements in the root composer.json, deployed, and autoload will automatically work, if you define it in your module's composer.json.

Actually, in any of the options you can add your module either via putting it in app/code or by requiring it in composer.json.

If there is some specific issue with the workflow you have or what I described is totally different, we'll be interested in more information about it to ensure we cover all the cases.

[ktomk]

Thanks for the feedback of you two and thanks for your concrete examples, it would require me to bring one up, too, which I currently am not able to due to time constraints. Also I don't want to hijack this older issue, so it's then perhaps better placed in an issue of it's own. I'm also glad to see your commitment.

[buskamuza]

Thanks, @ktomk . Feel free to post a new ticket with your use case.