Pluggable Providers TODO

[durandom]

This is a list of all open and accomplished tasks to split out providers from the main ManageIQ core repo.

please add things you know of as comments and I'll update the list

Extract all providers

• amazon
• azure
• vmware
• openstack
• ovirt / redhat
• hawkular
• openshift
• kubernetes
• ansible_tower
• embedded_ansible
• foreman
• google
• microsoft
• nuage

Config / Settings

• provider generator generator for providers #12209
• provider specific settings [WIP] Settings for providers #10944
• provider event definitions Ems event groups - allow provider settings (deeper_merge edition) #14177
• provider task_final_events
• provider specific purge timers Add purge timer for archived entities #14322 (comment)
• safeguard overwriting of settings fix indents so we don't overwrite the main settings.yml manageiq-providers-openstack#53

Seeding

• Dialogs allow seeding of dialogs from plugins #14668
• MiqDialogs allow MiqDialog to be seeded from plugins... #14653
• OrchestrationTemplates move service dialogs from core manageiq-providers-azure#46 (comment)
• Reports Allow plugins to bring their own miq_reports #19391
• Scan Profiles Pluggable ScanItems #19388

Pluggability

• Register VMDB plugins Register VMDB plugins #12221
• generate a csv of features supported across all models generate a csv of features supported across all models #11281
• worker definitions Pluggable Providers TODO #14840 (comment)
• Bring-your-own-logger (re: Pluggable Providers TODO #14840 (comment))

Couplings

• move amazon authenticator from core to amazon repo move amazon authenticator from core to amazon repo #14492

Specs

• use base classes in ManageIQ core
• use shared examples in providers convert floating_ip spec to shared example #11414
• dont use spec_helper from core in the providers

UI

• manageiq-ui-classic is not really being tracked here but see this document and this repo https://github.com/martinpovolny/miq_plugin_example
• Pluggable provider creation and credential checking API driven, pluggable, provider creation and validation #18818
• Remove hard-coded provider class names

to be discussed / concretized

• Fleecing logic Remove oVirt provider code manageiq-gems-pending#118 (comment)
• Namespace Orchestration Templates - Where to place shared orchestration template class? #12454

cc @Fryguy @blomquisg @bdunne @gmcculloug @gtanzillo @chessbyte
@miq-bot add_label pluggable providers

[chessbyte]

@durandom - @juliancheal and I extracted a bunch of providers recently (Ansible Tower, Google, Foreman, Microsoft SCVMM). Please update ch

[NickLaMuro]

@durandom et. al.

I think something that should also be considered and added to the list is the workers, and having the definitions of those living in the providers themselves versus being part of a list in MiqServer::WorkerManagement::Monitor::ClassNames (I have moved the real definition around a bit, but basically the constants defined there is what I am referring to).

This would be a bit tricky to implement, and definitely something I am still pondering on myself. This might also be related to the "base MIQ models" effort that @jrafanie and others are looking into, and would affect how we define plugins as well.

[jrafanie]

I feel like providers should be able to register and enable themselves (as two steps), what models/worker classes/etc. they provide, and insert rails boot hooks if they're an engine. It's still important for them to not load all the things since we can't sustain all of our provider code and libraries being loaded in the same process. The rails engine code must not load all the things. It should only do minimal work and delay loading any gem code until it's actually used.

[Fryguy]

.subclasses/descendants can get us all of the leaf workers, and so we can probably change ClassNames to do something like

MiqEmsRefreshWorker.subclasses + 
MiqEmsMetricsCollectorWorker.subclasses +
...

[NickLaMuro]

@Fryguy That works if you want to load the world first before loading miq_worker_types, but right now in run_single_worker.rb, I am loading it by itself. This allows the command line to be snappy when doing -l, as well as not requiring everything from every provider just to boot up a worker.

Doing what you suggests means we need to load the world to determine if this is a valid worker type.

[Fryguy]

The monitor has to monitor everything, so not sure why it loading the world is an issue. I would argue that if we extract worker management from MiqServer into a standalone object, then only the evmserver should be loading that class, and then accessing MiqServer won't load the world via the worker management.

[NickLaMuro]

The monitor has to monitor everything...

That doesn't mean it has to load everything to monitor it (make sure it is running, make sure it hasn't stalled, etc.).

It needs to know about what it is at most, not what it is doing and why it is doing it. We are just re-organizing the monolith if our expectation is that it is just going to continue to load everything.

[juliancheal]

@miq-bot assign juliancheal

[Fryguy]

We also probably want custom Loggers to live in their own repo, which might be necessary in certain cases (see #17228 )

[Fryguy]

Pluggable configuration validators should be allowed since we are allowing plugins to bring their own settings. (see #17168 (comment) )

[miq-bot]

This issue has been automatically marked as stale because it has not been updated for at least 6 months.

If you can still reproduce this issue on the current release or on master, please reply with all of the information you have about it in order to keep the issue open.

Thank you for all your contributions!

[JPrause]

@juliancheal is this still a valid issue. If not can you close.
If there's no update by next week, I'll be closing this issue.

[JPrause]

Closing issue. If you feel the issue needs to remain open, please let me know and it will be reopened.
@miq-bot close_issue

[agrare]

#16162 should be a good first pass on getting hard-coded model names from providers out of core/ui

[NickLaMuro]

@agrare / @juliancheal So I gave this a bit more thought since last week, and here are a few items I think could be added to this list:

• Gemify PostgresAdmin
• Bring back mounting code in to "core"
• A "core gem"
• Third-party plugin support (without bundler?)

The first two are items related to the database backup work I did last summer, and specifically with the mounting code, there are pieces for the s3 and swift portions that really should live in the provider plugins for them specifically. Carboni specifically mentioned wanting to do the PostgresAdmin bit, and I tend to agree with him there. I think we would want to make it a bit more generic than it is currently, but that should be doable.

The "core gem" idea is something @Fryguy had two summers ago, but I think that will help decouple things better. Whether it is "one gem", or a few (such as manageiq-base-models and manageiq-core) is something I haven't considered much yet, but was having multiple was brought up at options when we discussed it last.

The last one might require the core gem as a per-requisite (or a manageiq-base-provider of some sort), but the idea would be that you can put a self-contained gem in a directory somewhere to add extra providers to MIQ. While it might end up having the "npm problem" (multiple versions of the same library) by not using bundler, I think having a plugin bring all of it's dependencies (vendored and probably namespaced) avoids the core needing to provide it for them. If we go this route, I think we would need to encourage limiting dependencies to only what is needed for the provider (so aws-sdk for amazon for example).

Obviously dependencies like the core gem wouldn't need to be shipped with the providers gems, and we can assume the , and if we go more the route described here, this might be less of an issue when it comes to memory bloat. One thing we could also do for plugin compatibility is do something like Rails does for migrations:

https://guides.rubyonrails.org/v5.2/active_record_migrations.html#migration-overview

And have classes with a version in them. That way developers can write to a model spec in core, and we can deprecate/remove those versions after a few newer versions have been released.

---

Anyway, sorry for the 📖, but I can possibly try experimenting with the last two and comment on my findings in @juliancheal 's #16162 PR.

[agrare]

As a lot of progress has already been made and we are starting another round of pluggability, I am going to close this issue and start a new one that represents the current state of pluggability a little better, please move any further discussion to #19440