Moving content from collections in Ansible 2.10 to collections currently outside, or: should we add new collections to 2.10.x?

[felixfontein]

To continue the discussion from today's community meeting.

We'd ❤️ your feedback!

• Under what circumstances would this feature be useful to you?
• Is there anything you expected to see, but didn't?
• Does this have a direct negative impact on your user or developer workflow in any way?

[geerlingguy]

Philosophically, I'm opposed, as it will further expand Ansible's footprint, and even with redirections make it so canonical FQCN module names would be different in the same version of Ansible.

Pragmatically, I could maybe be convinced, but I'd still rather those kinds of changes be in more major-ish releases.

[felixfontein]

If we don't allow new collections in 2.10.x, that means that we cannot deprecate content that should be moved, and that means that the content will stay in the original collection for 2.11 (and will get deprecated from 2.11.0 if the new collection has been added to Ansible 2.11). That means that users manually have to switch to the new collection via FQCNs in 2.11, and only in 2.12 redirections will be active.

On the other hand, if we add these collections to 2.10.x, we can already deprecate the content in the old collection at that point of time, and remove it in 2.11.0 and thus add redirections already in 2.11.0.

(Moved as an answer to this comment.)

[felixfontein]

How about: only allow new collectins which mostly consist of content that's currently contained in Ansible 2.10? This would allow us to move things out of community.general / community.network (or also community.kubernetes) without opening the flood gates :)

[felixfontein]

See "Strawman proposal 2" in https://etherpad.opendev.org/p/ansible-contributor-summit-october-2020-morning:

• add new collections to Ansible 2.10, but do not update meta/runtime.yml of community.general for community.general 1.x.y releases to redirect to them (they would be available via explicit FQCN: community.postgres.postres_user but the old FQCN and short name would use the community.general module)
• remove modules from community.general after the last 1.x.0 minor release (see release schedule: end of November 2020), and add redirects to meta/runtime.yml. Thus in 2.0.0, they will be removed and redirect.
• bugfixes could be backported from the new collection to the stable-1 branch of community.general, depending on whether the maintainers want to do that

[felixfontein]

We discussed about this in today's community meeting. I tried to write out the plan we came up with, with some of the reasons why the plan is at it is, so that everyone can take a look and comment on it:

Reasoning

1. Make development easier: an own repository makes it easier to collaborate, and less anonymous.

2. Improve CI: currently a large percentage of the CI time for community.general is spent on the docker and postgresql modules. Moving them out speeds up CI for all plugins/modules that remain, especially for changes that trigger a full CI run (for example changes to certain kind of plugins).

3. Documentation can be improved a lot: instead of a huge list of content for community.general, you get a precise list of plugins for postgresql, for docker, ... (For example, https://docs.ansible.com/ansible/2.10/collections/community/general/ contains docker-related content in three different plugin categories: connection, inventory, and modules.)

4. Get rid of collection dependencies:

   • Most users of community.general do not want ansible.netcommon, ansible.posix, community.kubernetes and google.cloud.
   • And most users of community.network do not want check_point.mgmt and fortinet.fortios.

   The plan sketched below will drop all dependencies (except ansible.netcommon for community.network, which is needed for a lot of network infrastructure).

Compatiblity

No matter what we do, we have to annoy someone. If we don't move anything, we annoy collection developers and maintainers (see reasons at the top) but also users in the long term. If we do the move, we either annoy 2.10+ users or 2.9 users. We try to look ahead. Here is what we plan to do and how it affect these different groups:

Ansible-base 2.10+

After removal from 2.0.0, meta/runtime.yml redirects will be added. If the user installs the new collections, everything will work as before.

If the user installs Ansible 2.10+, this is fully transparent as the required collections come with it.

Ansible 2.9 (or 2.8, if applicable)

Ansible 2.9 does not support meta/runtime.yml and its redirects. Thus users must use FQCNs, or the collections: keyword. This causes trouble when moving and redirecting content as described above:

1. If Ansible 2.9 users already installed these collections and added FQCNs to their roles/playbooks, they will stop working. Fixing this is simple: assuming the module names haven't changed during the move, one could simply do a serach and replace with community.general.docker_ → commuity.docker.docker_.
2. If Ansible 2.9 users already installed these collections and used the collections: keyword, they simply have to add more collections there - otherwise they fall back to the modules included with Ansible 2.9.

An alternative would be to not remove the modules/plugins from the old collection, but instead to deprecate them, and point to the new collections in the deprecation message. The disadvantages would be:

• Ansible 2.10+ users need to manually use FQCNs for all moved modules if they want to use the latest versions, and not the deprecated ones. Ansible-base 2.11's ansible_builtin_runtime.yml might get updated, which allows the old short-names to point to the correct new versions, but that will only be included in Ansible 2.12, i.e. mid next year (extrapolating the current schedule).
• Ansible 2.9 users still have to update their collections/roles. The only advantage they have is that they get a deprecation and it won't break right away when they upgrade to community.general/community.network 2.0.0.

2.9 vs. 2.10

So no matter how we move things, it will break for 2.9 users if we replace modules/plugins by redirects, and it will break/annoy 2.10+ users if we deprecate the existing plugins/modules without adding redirects.

Note that the sooner the move is done, the fewer 2.9 users are affected, and the fewer roles/playbooks these have to change!

Since 2.10 is the future, and in the long term the redirects are better, we prefer to add redirects and break things for 2.9 users.

Moving plan

community.general:

• move postgresql modules to community.postgresql (now exists)
• move docker modules and plugins to community.docker (now exists)
• move google modules and plugins to community.google
• move kubevirt modules and plugins to community.kubevirt (repo exists)
• move oc connection plugin to community.okd (already exists)
• move nios modules to infoblox.nios_modules (already exists)
• vendor helper function ismount from ansible.posix that is used by the aix_filesystem module to drop the last remaining collection dependency for community.general (merged PR)

community.network:

• move fortios modules and plugins to community.fortios
• community.network.cp_publish already exists as check_point.mgmt.cp_mgmt_publish
• move routeros modules and plugins to community.routeros (now exists)

Deadline

The new collections should be created and published by end of December, so they can be included in Ansible 2.10.x before 2.11.0 is released, and community.general/community.network can be updated accordingly so their 2.0.0 releases contain the redirects and no longer have the content.

[tadeboro]

This seems like a good compromise: only allow moving existing Ansible 2.10 stuff into new collections during the 2.10.x releases and allow external collections to be included in Ansible 2.11.

I also think that focusing on Ansible 2.10 is the way to go. And as already said before, the sooner the moves get done, the less chance of users getting into trouble.

[felixfontein]

I've created #128 to track the state of the collections. I will no longer update the above answer.

[rockaut]

I would love to see migrations like postgres to their own collection. I can see and understand how it complicates things but I would also like don't don't pull things I just don't need at all.

[tadeboro]

Note that if you install the ansible Python package, you already get all those things bundled. So they are essentially free in this case. But ansible-base users will indeed experience reduced download sizes with the move.

[abenokraitis]

Are the adding of features/functionality typically part of the criteria for patch releases as part of Semantic Versioning? I would claim it is not, because PATCH versions are only valid when you make backwards compatible bug fixes. The adding of Collections with all redirects and runtime.yml entries in place (for backward compatibility) would warrant only a MINOR release at best.

IMO, moving the Ansible distribution versioning to a Fedora model will no longer require Ansible distro maintainers from keeping track of minor/patch inclusion criterial at the global level, and leave that to the Collection maintainers to keep up.

[felixfontein]

@abenokraitis Ansible does not use semantic versioning; its patch releases are effectively minor releases in semver terms already since they contain new minor releases of collections. So having a new collection in it won't make the situation worse.