RFC separating plugins from nornir "core"

[dbarrosop]

This is a request for comments from the community.

As the project grows the community contributes plugins for different use cases. This is great and it's certainly good news for the community but it puts a burden on the core maintainers. Some of these plugins relate to systems which we, the core maintainers, are no experts of so review is slow and, in case of bugs/issues, we can't help much about it.

For this reason I am proposing the following:

1. Ship nornir without any plugins whatsoever, just the engine framework.
2. Plugins will live in their own repo by theme. For instance, nornir-napalm may contain the connection plugin for napalm + its tasks. nornir-netmiko may contain the connection plugin for netmiko + its tasks. nornir-netbox, the inventory plugin for netbox, etc...

This means that if you want to use nornir with netmiko and netbox as the inventory you would have to install both nornir, nornir-netmiko and nornir-netbox. For instance:

pip install nornir nornir-netmiko nornir-netbox

This has a few extra benefits:

1. Installation footprint is as small/big as you need it to be. For instance, no need to install napalm and its tons of dependencies if you don't plan to use it.
2. It sets a pattern where people can own the plugins under their own organization as it's not required to live in the nornir repo. As a matter of fact this is not necessary today either but the pattern is to contribute everything to the core repo.
3. Plugins will be able to be developed/iterated faster as they will not depend on nornir's release schedule

This has two main issues; overhead of having to package everything and discoverability.

To solve those issues we will:

1. Maintain a directory where people can submit information about their plugins. This information will be rendered alongside the docs.
2. Maintain a cookiecutter repo contributors can leverage to get all the boilerplate fixed for them.

As the "core maintained" plugins will be separated from the "core" nornir repo those plugins will leverage (1) and (2) as well ensuring there are no first and second class plugins here.

[ktbyers]

Yes, I think it is a good idea.

Looking at both NAPALM and Ansible, the burden of contributed code starts to become a huge issue and creates big conflicts on either side (i.e. maintainers are frustrated that they can't manage the amount of work and contributors are frustrated that fixes/improvements don't happen).

I think we should probably consider some of the issues we ran into when we broke the napalm project into separate repositories. In other words, we probably need some way to declare plugin-core version so we don't have to try to make core and plugins move in sync in some way (which basically becomes impossible). Which maybe is some meta information that both core contains and the plugin must have like nornir_plugin_version where Core lists a range of supported plugin versions and the individual plugin indicates the version it supports. With the expectation that Nornir Core would support a range.

We should also think more about plugin testing and potentially have boilerplate for testing (that is just included in the cookiecutter in some way).

I wonder if for discoverability we want a bit more. Might be interesting to research any project that has done this well. Potentially meta information on the plugin side that could be pulled in/read via tooling so that if there was a command-line tool you could list the plugins and other information. Anyways might be worth researching more.

[lykinsbd]

I really like this idea, and there's some tools built into pkgutil to help us do this in a dynamic fashion as well. pkgutil.iter_modules() would let us discover all the nornir-<plugin> modules.

[dbarrosop]

I think we should probably consider some of the issues we ran into when we broke the napalm project into separate repositories

Fortunately here the scope is very different. Plugins should be compatible (guaranteed as per our semver rules) for a major version. The problem with napalm was that even non-breaking changes required coordination. So basically what that means is that plugins should have as a dependency the major version they were written for in their requirements.txt

Potentially meta information on the plugin side that could be pulled in/read via tooling so that if there was a command-line tool you could list the plugins and other information

Yeah, I like that idea. What I was thinking was that repos with plugins would have some file with well-known metadata (in a yaml or similar). In the docs we would maintain a list of repos with plugins and then pull the metadata from those repos daily or so.

would let us discover all the nornir- modules

Don't think we need anything so fancy. Today you import netmiko tasks as from nornir.plugins.tasks.networking import netmiko_send_command, after this hypothetical change the path could easily be from nornir_netmiko.plugins.tasks... instead so there is really not much to do there. Same applies to inventory plugins. The tricky bit will be that nornir_netmiko will have to register itself as a connection plugin but that shouldn't be more complex than it is today as you only need to call Connections.register

[carlmontanari]

Does poetry have something like setup.py extras?

If so, could have all the core plugins (stuff that exists now) already "attached" to the project in that way

i.e. "netbox": ["nornir-netbox>=2.0]

You could then do pip install nornir[netbox] -- or of course you could just pip install nornir-netbox as the extras would just point to that package anyway.

This would be a super minor thing but could be an easy way to tie/associate the "core" plugins to the project directly, while leaving all the "community" stuff just as the normal pip install nornir-mywidget.

If nobody likes that idea/that doesn't exist in poetry, just pretend I said nothing :)

Otherwise, I think its all a cool idea! I had been reading about plugins a bit after Dmitry mentioning it recently -- seems like cool stuff!

[dmfigol]

@carlmontanari poetry does have extras, but I am not sure we want to go down that path, because it makes plugins defined there first-class citizens, while others become second-class. We want to avoid this bias.

[ubaumann]

I agree with most of the points but we still have some risks. For example we could end up with many plugins needing different versions of nornir and I'm afraid the developing of new versions of nornir will slow down because "breaking" changes have a bigger impact of all the plugins someone started and we will think twice if a change should be done or not.

Also I am afraid many people will start a new plugin because they have a good idea but will never come to a stable version. And if many plugins are in bad quality the whole nornir image will be affected.

[dbarrosop]

1. Regarding breaking changes and plugins requiring different versions of nornir. We follow semver rules so we don't do breaking changes, ever, unless there is a major release. So as long as the plugins pin a major version everything should be good.
2. Re plugins sprawl and bad quality, this is something that can happen today. There is nothing stopping you from writing your own plugins and publishing them somewhere so what we are saying here is that we will move out of the core repo plugins that some of the core maintainers of nornir will maintain anyway. In any case, this is a valid concern and thanks for bringing it up, thinking about it we could enforce quality via the central directory:
   1. If you want to add a plugin to the central directory we will review and make sure it's documented and tested
   2. The documentation will be pulled and merged into nornir's own documentation as part of the documentation build process
   3. The tests will be run as part of the documentation build and add a "coverage" metric

Hopefully this will help ensuring a minimum amount of quality in the plugins that we refer to in our docs, or at least give you a sense of how much you can/want to trust it. Obviously, if they are not there we can't do much about it but that's no different from today.

[dbarrosop]

Hello everybody, I have compiled a list plugins in #504. I'd appreciate if people could take a look and confirm if they are interested (assuming they have volunteered) or to let me know if they are interested in helping even in the case they weren't.

Thanks!

[ogenstad]

We should think of how this impacts the documentation, everything could of course be spread out into its own RTD repo, or we could do some integrations.

There's also the issue of very small plugins, like the tcp_ping one, or the two that handles Jinja rendering. Should all of those live in their own repos or should we have some base plugins repo?

[dbarrosop]

Yeah, I have been thinking about this and short-term I am going to add a "plugins" section to www.nonir.tech with a table that will serve as central repository for plugins. You can see a prototype here:

https://nornir.tech/nornir/plugins/

The table is populated from:

https://github.com/nornir-automation/nornir.tech.src/blob/master/data/nornir/plugins.yaml

Medium-Long term we could expand this and have a central repo for documentation too but I think we need to be realistic and start small or we won't get this done.