Decouple ModuleTemplates from Channels

[c-pius]

Context

Side quest from: #1681

It has been requested to decouple ModuleTemplates from channels. ModuleTemplates will always define concrete versions of a module. Those version are created once and never changed. The assignment of these module versions to channels shall happen at a different place.

It is proposed that an additional CR will be introduced that maps a given module version to a channel.

Questions to be clarified:

1. have 1 CR per module version and channel, or group multiple assignments (or even all) together into one resource?
   • how does this affect the watch mechanism?
   • how does this affect the sync to the SKR
2. does the new CR for channel assignments need to be synced to the SKRs?
   • likely yes, since Dashboard/CLI need to be able to display available channels, but to be confirmed
   • can the sync be optimized, e.g., using a dedicated controller insead of the Kyma controller?
     • note that previously, it has been requested that KLM is the only KCP component that should write into the SKR. Maybe this decision is worth being re-evaluated to increase performance?
3. does the Key Assumption hold that once a ModuleTemplate is available, this version is installable by the end user?
   • or do we need a separate mapping to determine which versions are installable directly

Goals: align on above questions so we can feed this back to #1681

Decision

To be proposed

Consequences

To be proposed

Attachments

Given ModuleTemplates like

# ModuleTemplate
apiVersion: operator.kyma-project.io/v1beta2
kind: ModuleTemplate
metadata:
  name: api-gateway-2.3.1
  namespace: kcp-system
spec:
  moduleName: api-gateway
  version: 2.3.1
  ## ...

And Module Status in KymaCR like

# KCP Kyma
apiVersion: operator.kyma-project.io/v1beta2
kind: Kyma
metadata:
  name: default
  namespace: kcp-system
spec:
  channel: regular
status:
  activeChannel: regular
  modules:
  - channel: regular
    fqdn: kyma-project.io/module/api-gateway
    manifest:
      apiVersion: operator.kyma-project.io/v1beta2
      kind: Manifest
      metadata:
        name: 0c979a42-f521-41d8-b5ee-eabbaff6abd1-api-gateway-1209152227
        namespace: kcp-system
    name: api-gateway
    resource:
      apiVersion: operator.kyma-project.io/v1alpha1
      kind: APIGateway
      metadata:
        name: default
        namespace: ""
    state: Ready
    template:
      apiVersion: operator.kyma-project.io/v1beta2
      kind: ModuleTemplate
      metadata:
        generation: 12
        name: api-gateway-regular
        namespace: kcp-system
    version: 2.5.0

And ModuleChannel CR like

# ModuleChannel
apiVersion: operator.kyma-project.io/v1beta2
kind: ModuleChannel
metadata:
  name: api-gateway-regular
  namespace: kcp-system
spec:
  channelName: regular
  moduleName: api-gateway
  moduleVersion: 2.5.0

We could change the watch for ModuleTemplates roughly as follows:

watch(obj) {
  moduleChannel := getModuleChannel(obj)
  kymas := getKymas()

  affectedKymas := getAffectedKymas(kymas, moduleChannel)
  return createReconcileRequestsForObjects(affectedKymas)
}

getAffectedKymas(kymas, moduleChannel) {
  affectedKymas := []
  for kyma := range kymas {
    for statusModule := range kyma.status.modules {
      if statusModule.channel == moduleChannel.channelName {
        affectedKymas.push(kyma)
      }
    }
  }

  return affectedKymas
}

In addition, Kyma reconciliation could be changed so that instead of looking up module templates directly, it must first lookup the ModuleChannel CR, from there determine the module version currently assigned to the channel, and with that fetch the correct module template.

[c-pius]

Feedback in arch round today was to group by module. A full list may be to cumbersome to work with in the module repo.

[jeremyharisch]

Context
It has been requested to decouple ModuleTemplates from channels. ModuleTemplates will define specific versions of a module, and these versions will be immutable after creation. The assignment of module versions to channels will be managed via a new CR.

Proposal

1. CR structure for module-channel assignments

   • Based on the feedback from the Architecture round (04.09.2024), it’s clear that smaller, more focused CRs are preferred over a single large CR containing all module-channel assignments. This leads to a more modular and manageable solution.
   • Proposed Option: Use a ChannelAssignment CR
     • Structure: The ChannelAssignment CR contains a list of version-channel pairs, where each item specifies a module version and its assigned channel.

     apiVersion: operator.kyma-project.io/v1beta2
     kind: ChannelAssignment
     metadata:
       name: Eventing
     spec:
       assignments:
         - version: "1.0.0"
           channel: "regular"
         - version: "1.1.0"
           channel: "fast"
         - version: "1.2.0"
           channel: "experimental"

     • Pros:
       • Granular control: Each module team can manage its own ChannelAssignment CR, giving them ownership over their module's version-channel assignments.
       • Cleaner separation: The smaller CRs offer modularity and prevent a single, large CR from becoming a bottleneck.
     • Cons:
       • Potential overhead: Managing multiple smaller CRs could add complexity and impact performance, particularly during sync operations.
       • Syncing to SKRs: Each CR needs to be synced to the SKRs, which could increase the number of resources to monitor.
   • How does this affect the Watch mechanism?
     • The current watch mechanism watches ModuleTemplates, since ModuleTemplate will be immutable in future, this watch mechanism is obsolete.
     • We need a new watch mechanism which watches the ChannelAssignment CRs, making sure if a ChannelAssignment CR gets updated (i.e. new version added, channel switched, etc.), all corresponding Kyma CRs get queued for the Kyma reconcile loop.
     • What are the corresponding Kymas which should be queued? All Kymas which have the Module enabled with a specified channel. If a Kyma Module is enabled with a specific version, then this Kyma must not be requeued.
     • Code Example TBD

2. Syncing ChannelAssignments to SKRs

   • It is confirmed that Busola (dashboard) and the Kyma CLI need to display available modules and corresponding channels/versions using resources from within the SKR cluster. Currently, these tools use synced ModuleTemplates to determine available modules, but with the removal of channel from ModuleTemplate.spec, the ChannelAssignments will need to be synced as well to make sure the channels are shown as well.
   • Possible Approaches:
     • Option 1: Busola and the CLI could rely solely on ChannelAssignments to show which modules are available, but this would not cover user-installable modules. TBD: Discuss specific usecase, if it is still valid.
     • Option 2 (Preferred): Continue syncing both ModuleTemplates and ChannelAssignments, ensuring Busola and the CLI have access to complete data (both managed and user-installed modules).
   • Optimizing the Sync:
     • The current sync process for ModuleTemplates happens inside the Kyma Controller (check here). Adding ChannelAssignments to this process could slow down the Kyma reconciliation loop.
     • Proposal: Introduce a dedicated controller for syncing ChannelAssignment CRs and ModuleTemplate CRS to the SKRs, separate from the Kyma Controller. This improves:
       • Separation of concerns: The Kyma reconciliation loop doesn’t depend on ModuleTemplates or ChannelAssignments, so syncing these resources via a separate controller improves performance.
       • Focus on syncing: This controller could handle both ModuleTemplates and ChannelAssignments, adding their condition to the Kyma CR while keeping the Kyma reconcile loop efficient. This could leed to raceconditions, when two reconcile loops try to update the same Kyma CR, but this is somehow expected in a K8s application.
       • Main Reason: Reduces load on the Kyma controller and allows the sync operation to be optimized independently. Separation of concerns

3. ModuleTemplate installability

   • Assumption: Once a ModuleTemplate is available, it is installable. However, some modules may need additional checks or conditions before they are installable by the end user.
   • Proposal: Modify the ChannelAssignment CR to include all installable managed module versions (whether or not they are linked to a specific channel).
     • The channel field becomes optional for modules that are available but not tied to a specific channel. For example, earlier versions or special cases could be included here without channel assignments.
     • Potential rename: Consider renaming the CR to something like ModuleVersion, as it would contain both channel-assigned and unassigned versions.
   • Example:

     apiVersion: operator.kyma-project.io/v1beta2
     kind: ChannelAssignment
     metadata:
       name: Eventing
     spec:
       assignments:
         - version: "0.1.1"
         - version: "0.1.2"
         - version: "1.0.0"
           channel: "regular"
         - version: "1.1.0"
           channel: "fast"
         - version: "1.2.0"
           channel: "experimental"

   • This structure allows for flexibility in defining which versions are installable, regardless of their channel assignment.

Open Points

• Watch Mechanism: How is it affected? What needs to be changed if we move the sync to a separate controller?

[c-pius]

Thanks for the write up. Only minor comments at this point in time

1. CR structure for module-channel assignments

• should we include the module name also as part of the spec?

3. ModuleTemplate installability

• if plain versions are added, we need to find another name for the CR. Overall no strong opinion, but I think we could also start by allowing to install all versions that have a template. If in the future we find that we need to allow it explicitly, we could either add this explicit info, or think about alternatives like labels/annotations preventing the template to be installed directly and synced to the SKR.

[c-pius]

Summary from refinement 2024-09-09:

• decision for explicitly listing installable versions postponed
  • only relevant when supporting installations of specific versions in the future
  • at that point in time: a) check with module teams if explicit listing is desired, b) check implications on ModuleTemplate sync and other boundary conditions
  • until then: design the new CR so that it can be extended with the explicit versions list
• watch mechanism on channel changes still desired

Proposal:

• name: ModuleReleaseMeta
  • specific enough that it hosts any kind of metadata related to a module release
  • generic enough to add further info in the future
• candidates for extension:
  • list of versions
  • deprecation info

apiVersion: operator.kyma-project.io/v1beta2
kind: ModuleReleaseMeta
metadata:
  name: eventing
spec:
  moduleName: eventing
  channels:
  - channel: regular
    version: 1.0.0
  - channel: fast
    version: 1.1.0
  - channel: experimental
    version: 2.0.0
## future
  versions:
  - version: 0.0.9
    deprecationDate: 2024-09-09
    deprecationMessage: v0.0.9 has been deprecated, use a newer version instead
  - version: 1.0.0
  - version: 1.1.0

[jeremyharisch]

Follow-Up Issues:

• Introducing new CRD: [ModuleCatalog] Introduce ModuleReleaseMeta CRD #1848
• Introduce Logic for new CRD: [ModuleCatalog] Implement Logic for ModuleReleaseMeta CRD in Kyma Reconcile Loop #1849
• Adapt watch mechanism: [ModuleCatalog] Adapt Watch Mechanism to Monitor ModuleReleaseMeta CR Changes #1850
• Create SyncResources controller: [ModuleCatalog] Extend Catalog sync to handle both ModuleTemplate and ModuleReleaseMeta CRs #1851