META - Extension mechanism enhancements

[iameskild]

"Composition over Inheritance"

• The current structure is heavily inheritance based (NebariStage --> NebariTerraformStage --> KubernetesInfrastructureStage). This means there is a higher likelihood of changes that are made to base classes causing breaking changes for their inherited class.
• How can we break up the stages into smaller, reusable components?
• What are the interfaces we want to design?
  • render
  • deploy
  • destroy

Tasks

Preview Give feedback

1. [ENH] Add stage plugins explicitly (instead of implicitly) #1939
   area: dependencies 📦 area: user experience 👩🏻‍💻 +2
2. Create Keycloak provider object #1940
   area: integration/keycloak area:developer-experience 👩🏻‍💻 type: enhancement 💅🏼 +3
3. Create Traefik provider object #1941
   area:developer-experience 👩🏻‍💻 type:extension +2
4. [EHN] Create a more robust way of tracking dependencies between stages #1942
   area: dependencies 📦 type: enhancement 💅🏼 type:extension +3
5. Break up stage-07 (kubernetes services) into their own stage plugins #1943
   area: k8s ⎈ type: enhancement 💅🏼 type:extension +3

[sblair-metrostar]

Thanks for starting this discussion, @iameskild. Really excited to get this extensions rewrite out the door. Ken and I will be spending some time this week testing all this in preparation for release, at least on the AWS side. Aside from any issues that come out of that, here are some of my thoughts from having worked with the plugins so far.

Make users explicitly determine the stage plugins they want to use instead of registering all plugins that are pip installed

Completely agree with this point. Auto-installing plugins based on what's found in the environment has revealed several logical problems already.

• Managing multiple environments with different plugin configurations gets messy and is really easy to screw up. Several times I've accidentally installed a plugin in an environment I didn't mean to.
• Possibly a subcommand to handle installing and uninstalling plugins. Can update the nebari-config.yaml, handle cleanup and uninstallation, etc...

Create Keycloak (auth) provider object (to create new Keycloak clients, roles, etc)

While I agree a reusable component that would serve to configure Keycloak clients and whatnot for downstream plugins would be valuable, my concern with creating a provider (presumably without Terraform) would just lead to reinventing the Terraform provider. Would like to maybe discuss what the failings of the Terraform approach are and brainstorm solutions to those before reinventing any wheels.

• Difficult to nebari destroy Keycloak stages if the cluster is unavailable or the Keycloak API is inaccessible. Should just be able to skip stages like this anyway since there's no need to deconfigure Keycloak if I'm just destroying the environment anyway.
• Maybe have nebari render capable of generating shared Terraform modules that a Nebari Keycloak class can reference and be reused in plugin stages.

"Composition over Inheritance"

I think this one is going to be critical for breaking out of the Terraform mold, as the current inheritance based approach coupled with the limitations in managing subcomponents is naturally leading us to a single stage + Terraform lowest common denominator pattern. Being able to more easily split plugins into subcomponents would improve testability and maintainability, as well as expanding options for non-Terraform solutions where appropriate.

• Base stage class that defines properties/methods for composition of plugins.
• Add subcomponents to a list or something that the base class is responsible for orchestrating the render/deploy/destroy behaviors, shared configuration, output composition, whatever.

Confirm that the DNS auto-provision is working as expected

Suggest maybe moving toward something like External DNS as a more versatile solution. The Cloudflare auto-provision is helpful for my specific case but is overly specific to Cloudflare and can't realistically be expanded to cover other DNS providers without a ton of work.

Other issues/items I'd add to the conversation:

• Can't install a plugin more than once since 1) the package is detected in the environment and can only be pip install once, and 2) the name is hard coded in the plugin so multiple instances would conflict anyway. As an example, I'd like to be able to create a generic Helm chart plugin to replace the old helm extensions but lets me fix some of its limitations (namespacing, private repos, secrets, whatever).
• A stage plugin package should be treated as a single component by Nebari. Currently, if my plugin exports multiple subcomponents (AWS Terraform stage, Keycloak, etc...) they are treated as separate stages by Nebari and have no inherent connection to each other besides the packaging. This relates to the composition/inheritance issue, so moving away from an inheritance model would be a big step.
• No clear way to uninstall a plugin aside from destroying the whole environment. This ties into the point about pivoting away from automagically registering plugins that are present in the environment toward a more deterministic configuration based solution. Will need a way to have the code for a plugin present in the system but not assumed to want it deployed, also probably a subcommand to manage uninstalling it. Should have something cleaner though so it can be managed with CI/CD pipelines.

I'll probably have more to add to this, just wanted to get some initial thoughts down.

[dharhas]

Can we start splitting these into individual issues and linking them above. It feels like several of these coould be worked on independently and a few need to be coupled and worked on by one person. This is a good meta issue but have smaller individual tasks should help us divide and conquer and move forward.

Great discussion by the way I like where this is going.

[pavithraes]

Can we start splitting these into individual issues and linking them above. It feels like several of these coould be worked on independently and a few need to be coupled and worked on by one person. This is a good meta issue but have smaller individual tasks should help us divide and conquer and move forward.

+1, I suggest using GitHub's tasklists - it's a little buggy but convenient to convert tasks into issues as required + track them.

[iameskild]

Okay the open tasks have been converted to individual issues.

---

Originally we also included:

• Clean up init and --guided-init
  • Add more comprehensive tests these
• Confirm that the DNS auto-provision is working as expected
  • Add more comprehensives tests

For item one:

• @sblair-metrostar wrote a lot of tests for init in nebari init unit tests #1931
• I have open open issue to do the same for --guided-init ([META] Write comprehensive tests for --guided-init #1944)

For item two:

• I cleaned up the DNS auto-provision in Add cleanup step for AWS integration test, ensure diable_prompt is passed through #1921