Backwards incompatibility between release 0.2.0 and 0.3.0 k8s-snap enabled features and non-dependent config flags for the features

[ader1990]

Hello,

The k8s-snap installed using the cluster-api-k8s v0.2.0 has the following feature states as defaults:

        # CONFIGURABLE
	EnableDefaultDNS TRUE
	EnableDefaultLocalStorage TRUE
	EnableDefaultMetricsServer TRUE
	EnableDefaultNetwork TRUE

        # NONCONFIGURABLE
       EnableDefaultLoadBalancer FALSE
       EnableDefaultGateway FALSE
       EnableDefaultIngress FALSE

The k8s-snap installed using the cluster-api-k8s v0.3.0 has the following feature states as defaults:

        # CONFIGURABLE
	EnableDefaultDNS TRUE
	EnableDefaultLocalStorage TRUE
	EnableDefaultMetricsServer TRUE
	EnableDefaultNetwork TRUE
        EnableDefaultLoadBalancer TRUE
        EnableDefaultGateway TRUE
        EnableDefaultIngress TRUE

This is just a screenshot of what is configured by default and you can see the backwards incompatible behaviour, which is not a problem per se, but it is worth to be documented.

The problem though relies on the v0.3.0 deployment matrix, as the EnableDefaultGateway is set by default on TRUE, irrespective of whether the EnableDefaultNetwork config value. If someone sets EnableDefaultNetwork = FALSE, without also setting the EnableDefaultGateway = FALSE, the deployment will fail, as the k8s bootstrap will fail due to EnableDefaultGateway depedency on EnableDefaultNetwork = TRUE.

For example, a better way to implement the dependent flags would be to either throw an error here or to enable the DefaultGateway only if DefaultNetwork is enabled: https://github.com/canonical/cluster-api-k8s/blob/v0.3.0/bootstrap/api/v1beta2/ck8sconfig_types.go#L253, basically making a tree of depedencies maybe?

[HomayoonAlimohammadi]

Hi @ader1990 and thank you so much for bringing attention to this issue.
I believe, as you mentioned it as well, it's more of a snap issue rather than a CAPI issue. But we'll make sure to address this in the future.