role versioning

[bcoca]

Proposal: role versioning

Author: Brian Coca @bcoca

Date: 2016/07/33

• Status: New
• Proposal type: core design
• Targeted Release: future release

Motivation

Users handling multiple versions of the same 'role'.

Currently there are several workarounds ,mainly adding version into role name or using multiple role paths, but both of these are problematic, specially in restricted environments.

Problems

• Multiple plays, each requiring different versions of the same role and restricted role paths.
• Installing, upgrading and downgrading of roles.
• Choosing which version of an installed role is the 'default' when only specifying the role name.

Solution proposal

• ansible-galaxy will install roles using name + version always, then linking to the 'role name' to provide 'unversioned' access. This allows for multiple versions to be installed in parallel and still serve 'unversioned role references' and the selection of which version of the role serves these requests (think alternatives or library slots).
• On install, If an existing role is found in the 'bare name' move and try to determine version, making this backwards compatible (appending .unkown + sequence in case version detection fails).
• removing roles should detect multiple versions and prompt for 'all' or a specific version(s)
• When referencing a role in a play, ansible-playbook should now also check if version is specified and use that if found, optionally falling back to the 'unversioned' name depending on config flag.
• option to galaxy to remove 'old roles' on install (upgrade?), this is not clear cut as version can be a commit SHA and order there is not related to sorting, clear 'versions 1.1, 1.2' can use loose versioning comparisons.
• ansible-galaxy should also be able to change the 'base role name' to point to specific versions, this solves the issue when the 'latest' is not actually the one existing plays should be using, again this mimics the 'alternatives' functionality.

Dependencies

roles unified spec (name, version, url, method/scm) to call/install a role, use the same in play, tarball naming and/or requirements.yml (name is only required field and the rest should have same defaults as today).

Testing

This would just need to expand on current galaxy role install testing and allow for several versions of the test roles to be installed and referenced by plays.

Documentation

ansible-galaxy options and man page at a minimum, it would also be helpful to add a 'managing roles' page to the docsite.

[bcoca]

cc @tima @willthames

[srgvg]

Would this solution provide a means to select role version from inventory?

In my way of looking at the problem, I focus on multi tenancy, where e.g. I'd manage multiple - say - gitlab roles for different customers, each having its own version in a test environment and in production.

Imagine a generic tomcat role for deploying war based apps, where one has multiple customers/organisations running multiple applications (different wars) in different environments (e.g. dev test prod).

Running the different roles by creating a different playbook for each case will not scale.

Being able to do a dynamic role include based on inventory vars would solve a big part of my use case.

Thoughts?

[bcoca]

not part of this proposal, but related to this AND roles revamp where you can 'call roles as a task'

[willthames]

Thoughts:

• Easy one first, typo in the date
• Where is role src specified - is that in requirements.yml still or part of the playbook role specification?
• Will this cover auto-install and idempotent role installation? Will it only auto install if using role version in playbook?
• Fastest role installation would then be versioned roles linking to either sha or tag (tag just a pointer to a sha really) which wouldn't be touched if version already exists (might need a super --force in case tag has changed - certainly normal --force shouldn't replace a tagged version)
• Asking for other tree-ish references (e.g. HEAD, master etc.) should check src to see if current commit ID is different and upgrade if --force specified.
• One thought is for requirements.yml to be includable, and perhaps merged with playbook role stanza - that allows src/version to be kept separate from playbook, and role ordering to be in the playbook (things like maven separate src/version from actual usage, this would just mimic that).
• How is 'best' role chosen when unversioned is specified - newest commit? But newest commit could be 'dev-test' branch which might not be what is expected (although that's why people should always use a versioned, tagged role)
• You say you might use loose version comparison - is there any restriction on naming (e.g. v1.1, v1.2 rather than 1.1, 1.2)

[bcoca]

@willthames

• not typo (see other proposals), just checking on who is paying attention
• both, the role spec ONLY requires name, the rest are optional and have sane defaults (version= latest,src=<default galaxy server>, method=<calculated from src|default to git>)
• it does not cover it, but it does enable several approaches to it.
• we don't always have a SHA (tarball install for example)
• tree/commit checks will depend on the handlers for each method, possible with git, hg and svn handlers, but not tarball or others (I was not explicit here, but I'm leaving methods open to expansion).
• "includable" requirements is one way, someone also proposed a 'galaxy plugin' that would do similar, we don't need to decide that here as what i'm proposing works with both
• 'best role' is up to user to decide, too many factors depending on workflows for us to decide for them
• loose version is the best i can think of w/o putting too strict of a convention, people do weird things to 'version', i'm open to alternatives as long as they don't restrict existing workflows too much.

[geerlingguy]

I wanted to crosslink this issue with the galaxy project issue ansible/galaxy-issues#168 ('Versioning of roles') to consolidate discussion.

It seems like we would need to make sure there's a defined standard for versioning (e.g. PEP-440) before we can implement this entirely.

[arbabnazar]

@bcoca date is wrong, I think it should be Date: 2016/07/23 instead of Date: 2016/07/33

[tima]

@arbabnazar: @bcoca said "not typo (see other proposals), just checking on who is paying attention" here.

Don't worry you'll get used to his sense of humor. ;)

[arbabnazar]

Thanks @tima

[tima]

I understand the need for this feature and that the status quo has its limitations that could be better addressed. While we can't do nothing, I have to say I am a bit wary we're trying to do too much here though -- it's already a slippery slope (see the PEP-440 spec @geerlingguy already mentioned for instance) no sense putting on a rocket pack.

I don't like the idea of role versioning being embedded in plays/playbooks. Having seen a lot of large organizations workflows, embedding version info won't be practical especially when working with Tower as it exists today. Like it or not, a roles file (i.e. requirements.yml) needs to stay. Not even sure including the roles file in the playbook is a good idea. I can think of a few scenarios (especially under the roles_include feature coming in v2.2) where this could become problematic.

Also, forcing them to create arbitrary variables and pass them via ansible-playbook to fill in the spec or select a different roles file seems worse in that it promotes a solution that will give users problems.

I'm concerned all of these versioning rules and constraints will put too much of a burden on existing users that don't need/care about versioning. Perhaps a separate "fetch" method is needed that doesn't care or consider versioning? Or the users that have setup their own semantics pulling directly from a git repo and are using tags? Not sure how many of them are out there, but I know that's been a recommendation to date when it comes to versioning roles.

Not mentioned here explicitly (it was at the recent contributor conference) is how handle role dependencies especially when using multiple roles that call for the same dependency but different (potentially incompatible) versions. ugh. It hurt to even type that scenario out.

I have a couple of broad questions that aren't entirely clear and mean a lot to the folks I've worked with.

What is the developer workflow under this proposal? Example -- my organization creates a role for our internal use that is shared by multiple teams. I need to make a change and test it before putting it into QA and then into prod. What's my workflow?

What is the migration path from what users have now (scm methods, tags, commits, no specific versioning semantics etc) to this enhanced versioned roles galaxy? How does this get implemented without breaking ever existing user and forcing them to adopt the overhead that versioning would bring? I'm not sure we can responsibly do this in one fell swoop.

[bcoca]

What is the developer workflow under this proposal?

The hope of this proposal is to leave that up to the developer, none is mandated and this should be flexible enough for everyone to keep their current workflows.

What is the migration path from what users have now (scm methods, tags, commits, no specific versioning semantics etc) to this enhanced versioned roles galaxy?

No change should be needed.

The versioning semantics are OPTIONAL to use in plays, this proposal is backwards compatible by keeping current semantics and adding the optional version to the role spec, keeping the spec universal across plays and requriements.yml makes it very easy to keep things consistent.

As for enforcing PEP-X in versions, I'm not in favor of it, this is meant to adapt to existing workflows not create a new restricted one. I plan on having 'best effort' when a version is provided (hence my proposal for using the looseversion lib), in the end it is up to the user to decide to use versions (at all) and how to use them.

I'm just providing the facility for easy management of them on install and reference of versions, this is not directing workflows or usage of it as much as it is possible to stay neutral when dealing with versions.

[steenzout]

I agree with @tima with not embedding version numbers into playbook plays.

I see ansible roles as an analogy to a Python package and therefore have applied similar principles to implement roles and manage the ansible environment.

if I install a different version of requests, my Python code will still do import requests, right?

I think the same should hold true for playbooks and the way they import/include a role.

of course, I should run unit and integration tests to see if I can actually upgrade the version.

I also see the ansible package ansible==x.x.x.x + set of role versions { role_name==versionX, role_name==versionY } as the equivalent of a virtualenv.

I actually use a simple virtualenv-like approach to setup playbook environment(s) which everyone in my organization must use to run any playbook and this has guaranteed the same playbook results even when they are executed by different people with different laptop/server environments.

I do exactly the same thing in ansible and run the role unit tests (and playbook integration tests for all playbooks that are affected by the change) against my role every time the environment changes: upgrade to higher version of a role or change ansible version.

I have yet to face a problem where I am blocked by 2 roles that need/use different/incompatible versions of another role but whenever that happens I can easily solve the problem with another virtualenv-like environment (which is something I already do anyway since it's not easy to migrate all playbooks at the same time to a higher ansible version) or just by going to my test suite and see where / which playbooks break.

I have had seen more problems with ansible version upgrades (maybe because I have tests) then with role dependencies mismatches.

my 2 cents...

[tima]

I'm not sure how this works for galaxy (the community site) if it's optional, but I'll stop there for now.

If the role and versioning info were in the playbook, is the expectation that a playbook runner would do a version check and possible download before executing?

[srgvg]

@steenzout Id like to add this: the difference between roles and libraries - or python packages in you analogy - is a typical software project will need exactly one vspecific version of a library, whilst Ansible potentially manages different applications instances with different versions of those applications, that at some point diverge enough to need different version of its roles to deploy those different instances.

Those different versions of roles might be validated to different versions of ansible, which makes @steenzout 's approach interesting. I'd like to hear more on that!

[bcoca]

@steenzout, @tim

Again, version in playbooks is OPTIONAL, depends on your workflow, not everyone has one that requires it but some do. This proposal allows for all current workflows and adds the POSSIBILITY of a few more.

Not everyone wants to deal with multiple ansible/venvs, the versioning allows for alternate workflows that can simply just add an optional version to a role to test against:

  roles:
    - name: myrole
      version: "{{roleversion|default(omit)}}"

You CAN supply role version, if you don't, it goes to default behavior on matching ONLY on name, this makes it trivial to test against 'devel' or other versions of a role and not have to be switching them in/out constantly.

Galaxy does not need to change, these versions should work independent of it as they are normally in the SCM (github/git/svn) or tarball name.

"Auto-downloading" roles is a different feature, this proposal does not force it nor prevent it, it does make easier to implement in several ways.

[steenzout]

@srvg I actually setup my roles tests to catch ansible version upgrade problems (through travis-ci.org using tox and virtualenv) while doing idempotency and checkmode (something I haven't seen anyone else do and it's key to be able to perform an audit before applying changes) testing.

locally I actually run my tests on multiple linux distributions and against multiple releases.

example here.

I have a private slack domain; if you wish to talk further I can send you an invite.

[tima]

@bcoca I understand it's optional but what if some (public) galaxy roles implement versions, perhaps differently, while others don't bother -- what does that look like? what's the user experience there?

I also understand auto-downloading is a separate feature, but its still related. If I embed my roles and version info in a playbook how does the auto-downloader work with that? What's the user experience? I'm thinking the roles manifest a project uses (and needs to fetch/maintain) and how my plays are put together and execute are separate concerns and should be kept that way. Mixing the two puts too much on the user to keep things from getting messy and potentially setting up unrealistic expectations.

[srgvg]

@steenzout yeah, making sure test mode and idempotency is fully operational are key to good roles. I'd love to have a further chat with you, my contact info is in my profile. love your handle btw ;)

[steenzout]

@bcoca this version parameter could also be used when specifying a role dependency on the dependencies attribute of the role's meta/main.yml file, correct?

[bcoca]

@tima the experience can be the same as now or different, this feature does not lock us into any way of doing it. People can already specify versions, just not in a consistent way, this only changes it to be consistent, predictable and upgradeable.

@steenzout yes, this will become the ONE way to reference roles, anywhere you do so, you CAN add version.

[xrow]

If you would also allow us to decide if the role comes wiht or without the .git folder would be a plus. I would like to do a change and commit instanly. Can`t see this in the spec.

[andyfeller]

@willthames thanks for highlighting one of the esoteric pain points around ansible-galaxy install:

Fastest role installation would then be versioned roles linking to either sha or tag (tag just a pointer to a sha really) which wouldn't be touched if version already exists (might need a super --force in case tag has changed - certainly normal --force shouldn't replace a tagged version)

ansible-galaxy lumps tag, branch, and sha together as version and assumes it can never change, which is not the case for branches. This has caused me pain personally as I want to separate inventories by branch with their own requirements.yml file and use git hooks to call ansible-galaxy to manage their roles. Would be happy to see this spun into its own issue.

sample workflow to highlight and reproduce issue I speak of:

andyfeller@localhost ~ $ ansible-galaxy --version
ansible-galaxy 2.2.0.0
  config file = 
  configured module search path = Default w/o overrides
andyfeller@localhost ~ $ mkdir foobar
andyfeller@localhost ~ $ cd foobar/
andyfeller@localhost foobar $ cat << EOF > ansible.cfg
> [defaults]
> roles_path = ./roles
> EOF
andyfeller@localhost foobar $ cat ansible.cfg 
[defaults]
roles_path = ./roles
andyfeller@localhost foobar $ cat << EOF > requirements.yml
> ---
> - src: geerlingguy.apache
>   version: 1.7.4
> EOF
andyfeller@localhost foobar $ cat requirements.yml 
---
- src: geerlingguy.apache
  version: 1.7.4
andyfeller@localhost foobar $ ansible-galaxy install -r requirements.yml 
- downloading role 'apache', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-apache/archive/1.7.4.tar.gz
- extracting geerlingguy.apache to /Users/andyfeller/foobar/roles/geerlingguy.apache
- geerlingguy.apache was installed successfully
andyfeller@localhost foobar $ cat roles/geerlingguy.apache/meta/.galaxy_install_info 
{install_date: 'Thu Dec 29 20:29:41 2016', version: 1.7.4}
andyfeller@localhost foobar $ vim requirements.yml 
andyfeller@localhost foobar $ cat requirements.yml 
---
- src: geerlingguy.apache
  version: 2.0.0
andyfeller@localhost foobar $ ansible-galaxy install -r requirements.yml 
- geerlingguy.apache is already installed, skipping.
andyfeller@localhost foobar $ cat roles/geerlingguy.apache/meta/.galaxy_install_info 
{install_date: 'Thu Dec 29 20:29:41 2016', version: 1.7.4}

[willthames]

@andyfeller: Those are two separate issues

1. ansible-galaxy doesn't update roles that already exist unless you use --force. I don't think that's news.
2. ansible-galaxy was updated to include support for 'tree-ish's in addition to tags. I think that was a good choice (disclosure, I put a lot of effort into getting that change in, even if I didn't write it). However, your workflow sounds slightly impossible from a git point of view, rather than a ansible-galaxy point of view. If you want to use a branch as a version and then run ansible-galaxy, should it update to latest when the branch changes or not? Having a moving version is bad practice for production, but useful in development.

I don't really fully understand what you're trying to do, as your example seems to be completely unrelated to your use-case.

[andyfeller]

Will, Thanks for taking a moment to reply! 1. You're right; this is the way ansible-galaxy works, however I believe pointing to a branch should pull current HEAD of the branch rather than what was HEAD whenever it was installed 2. I want to use a branch for an inventory/environment each with their own playbook and requirements.yml file. All roles would be managed externally in their own got repositories no different than all of the roles on ansible galaxy. This would allow each inventory/environment to have different versions of playbooks and roles without impacting one another. This is similar to how the puppet community architects best practices with r10k as mentioned. In order to pull this off, git hooks for post-checkout, post-merge, and post-rewrite would need to be in place to update the roles. The problem now is that ansible-galaxy doesn't care about enforcing the revision hash of what was installed, which requires force installing. For mature operations teams, this can be a lot of roles to redoenload simply for changing branches or reverting changes. Regards, Andy

…

Sent from my iPhone

On Dec 29, 2016, at 9:01 PM, Will Thames ***@***.***> wrote: @andyfeller: Those are two separate issues ansible-galaxy doesn't update roles that already exist unless you use --force. I don't think that's news. ansible-galaxy was updated to include support for 'tree-ish's in addition to tags. I think that was a good choice (disclosure, I put a lot of effort into getting that change in, even if I didn't write it). However, your workflow sounds slightly impossible from a git point of view, rather than a ansible-galaxy point of view. If you want to use a branch as a version and then run ansible-galaxy, should it update to latest when the branch changes or not? Having a moving version is bad practice for production, but useful in development. I don't really fully understand what you're trying to do, as your example seems to be completely unrelated to your use-case. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[jtyr]

As discussed on IRC with @bcoca, it would be really useful to introduce another role keyword to specify the Galaxy user of the role. The reason for that is that if I develop a role which was primarily developed for internal use, the dependencies are not in the right format (they are missing the Galaxy user name there). If I would like to share the role on Galaxy portal, I would have to change all the dependency names to the Galaxy format (user:role) otherwise the Galaxy import will fail.

I'm proposing to use the namespace keyword for that as that's in sync with the Galaxy code. Then the following role definition:

roles:
  - jtyr:myrole

would be equivalent to this:

 roles:
    - name: myrole
      namespace: jtyr

[jtyr]

@xenithorb I don't think it's right to put src or scm into the dependencies in meta file for roles you share on Galaxy. The name should currently be the Gallaxy format (user:role) which is the problem I'm trying to solve by the additional namespace key.

[abadger]

As part of the installer proposals we also talked about versioning. Proposed there to make a standard version format (semantic versioning) and only use those for any new features.

[bcoca]

I would only use semantic versioning for determining version for upgrades and downgrades, but still allow for hash/tag/branch as a way to install a 'different' version, just not support 'order comparissons' there.

[leigh-j]

Main change above pull request does is transform the set of roles from a list to a dict when working what gets installed currently keyed of name but could be changed to combination of name and version or something else.

I am not sure that the role versioning needs to get more complex than this, with the above fix that ensures warnings are given on all occasions, when a role collision occurs (trying to install different versions to the same location) Its now possible for developers to be aware of collisions and fix as required.

Fixing the case where two different thirdparty roles depend on different versions of the same role will be a significant overhaul of role handling and adding that complexity might not be worth the effort that patching of said roles would fix.

If it can be done though creating a dict of roles will be easier to manage than a list.

[bcoca]

possible implementation on ansible-galaxy side ansible/ansible#37880

[geerlingguy]

Just wanted to also put a note about collections—they do have explicit versions with I guess a little more organization around versioning than roles... I wonder if we could implement features like ansible-galaxy collection update [collection name] nowadays?

[andyfeller]

@geerlingguy : can you differentiate collections by inventory?

[Fobhep]

are there already any updates regarding this?

[pneumono]

So... has this just been abandoned? It seems wild to me there's no straightforward way to upgrade.

[bcoca]

At this point i would propose a simper approach, add support for handling role versioning the same as collections (semantic versioning) but not require roles to have versions (backwards compatibility), sadly some features would depend on the role versioning scheme.

[C0rn3j]

Adding an optional definition for versioning seems like a good compromise, is there anything preventing it from getting it done?

Anything that will let ansible-galaxy role install have an --update keyword, even if existing roles will need to add a version keyword.

[bcoca]

@C0rn3j mostly someone needs to dedicate time to come up with specific changes and convince rest of core to merge them.