packaging management components, clusterclass-based creation #2581

vuil · 2022-06-08T06:44:34Z

What this PR does / why we need it

Changes pertaining to packaging management components and enabling cluster creation using a clusterclass based Cluster resource definition (CCluster.yaml)
Packaged and ClusterClass based cluster LCM
and some github workflow changes

Which issue(s) this PR fixes

Related-to #1339
Related-to #1340
Related-to #1341
Fixes #1531

Describe testing done for PR

export OCI_REGISTRY=...
export COMPONENTS="pkg/v2/tkr/controller/tkr-source pkg/v2/tkr/controller/tkr-status pkg/v1/sdk/features addons pkg/v2/tkr/webhook/infra-machine pkg/v1/sdk/capabilities pkg/v2/tkr/webhook/tkr-conversion pkg/v2/tkr/webhook/cluster/tkr-resolver addons/pinniped/tanzu-auth-controller-manager"
export BUILD_VERSION="v0.21.0"

make docker-build
make docker-publish
make local-registry
make kbld-image-replace
make package-push-bundles-repo PACKAGE_REPOSITORY=management
make clean-registry

Release note

TBA

PR Checklist

x Squash the commits into one or a small number of logical commits
x Use good commit messages
x Ensure PR contains terms all contributors can understand and links all contributors can access

Additional information

Special notes for your reviewer

github-actions · 2022-06-08T06:56:18Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/2581/20220608065616/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

hack/kind/Testing_Packages.md

.github/workflows/main.yaml

codecov · 2022-06-08T07:57:41Z

Codecov Report

Merging #2581 (cdd2e07) into main (ce1df4f) will increase coverage by 0.03%.
The diff coverage is 43.03%.

@@            Coverage Diff             @@
##             main    #2581      +/-   ##
==========================================
+ Coverage   42.72%   42.75%   +0.03%     
==========================================
  Files         404      405       +1     
  Lines       40151    40516     +365     
==========================================
+ Hits        17153    17323     +170     
- Misses      21386    21569     +183     
- Partials     1612     1624      +12

Impacted Files	Coverage Δ
pkg/v1/config/clientconfig.go	`37.28% <ø> (ø)`
pkg/v1/tkg/client/client.go	`80.64% <ø> (ø)`
pkg/v1/tkg/client/cluster.go	`14.31% <0.00%> (-1.38%)`	⬇️
pkg/v1/tkg/client/config.go	`6.54% <0.00%> (-0.19%)`	⬇️
pkg/v1/tkg/client/init.go	`0.00% <0.00%> (ø)`
pkg/v1/tkg/tkgconfigbom/bom.go	`49.80% <0.00%> (-0.40%)`	⬇️
pkg/v1/tkg/tkgconfigupdater/client.go	`100.00% <ø> (ø)`
pkg/v1/tkg/tkgconfigupdater/ensure.go	`14.70% <0.00%> (-0.17%)`	⬇️
pkg/v1/tkg/tkgctl/init.go	`0.00% <0.00%> (ø)`
pkg/v1/tkg/client/common.go	`13.14% <36.00%> (+7.01%)`	⬆️
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ce1df4f...cdd2e07. Read the comment docs.

vuil · 2022-06-10T14:14:45Z

/test install-vc7

github-actions · 2022-06-10T14:21:14Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/2581/20220610142111/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

alfredthenarwhal · 2022-06-10T17:57:27Z

@vuil: /test install-vc7
Commit: fe4594f

Build failed! Build no: 2156

github-actions · 2022-06-10T20:27:24Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/2581/20220610202722/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

vuil · 2022-06-11T07:37:52Z

/test install-vc7

alfredthenarwhal · 2022-06-11T07:53:05Z

@vuil: /test install-vc7
Commit: e37bae2

Tests failed! Build no: 2167

files following the naming convention clusterclass-xxx.yaml are treated as cluster class files and are excluded from the checksum computation. also - renamed function to use more inclusive language Signed-off-by: Vui Lam <vui@vmware.com>

* Enable CI on package-based-lcm branch Co-authored-by: Sai Bandi <saib@vmware.com> Co-authored-by: Anuj Chaudhari <anujc@vmware.com>

- installs dependencies Signed-off-by: Vui Lam <vui@vmware.com>

…azure, clusterclass-vsphere - Add base for clusterclass-docker - Add base for clusterclass-aws - Add base for clusterclass-azure - Add base for clusterclass-vsphere - Add clusterclass packages to package-values.yaml file -- Script to sync CC yaml changes into infra specific package directories -- Update cc yamls in cc packages upstream here are actually locations to various cc-specific ytt templates/overlays in pkg/v1/providers they are being brought over by the sync_cc.sh script. -- Minor makefile and kbld-config.yaml to unblock build of the management package repository -- Fix BASTION_HOST_ENABLED in cc -- Add resource rules to the service account -- Fix package-install name

also: Remove commented code from Azure ClusterClass

This change fixes a nesting issue for defaults in clusterclass variables. The default was set on the variable and not the openAPISchema for the variable. Effects aws, azure and vsphere

…nzu#1984)

Adds a new ClusterClassSelector interface. This interface takes in the cluster config and determines what clusterclass to use and returns the name. Returns empty string if it can't determine the cluster class.

Signed-off-by: Vui Lam <vui@vmware.com>

…g is enabled - Refactored codebase to use more generic implementation for workload and management cluster creation - Return CCluster.yaml for workload clusterif user is providing legacy configuration file when and no customizations are performed by user - Always use ClusterClass based cluster creation for management-cluster deployment when `package-based-lcm` feature-flag is enabled

…-flag - Added 'features.cluster.auto-apply-generated-clusterclass-based-configuration' feature-flag to support this usecase.

Gate the cluster creation with ClusterClass with feature-flag `features.global.package-based-lcm` Authored-by: Anuj Chaudhari <anujc@vmware.com>

…-flag - Added 'features.cluster.auto-apply-generated-clusterclass-based-configuration' feature-flag to support this usecase.

vuil · 2022-06-11T15:34:32Z

/test install-vc7

github-actions · 2022-06-11T15:45:37Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/2581/20220611154535/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

alfredthenarwhal · 2022-06-11T16:00:57Z

@vuil: /test install-vc7
Commit: 9e13af4

Tests failed! Build no: 2168

vuil · 2022-06-11T20:46:30Z

/test install-vc7

alfredthenarwhal · 2022-06-11T23:50:19Z

@vuil: /test install-vc7
Commit: 9e13af4

Tests failed! Build no: 2169

@ytt

Files removed by ./packages/tkg-clusterclass/hack/sync-cc.sh /Users/fgold/go/src/github.com/vmware-tanzu/tanzu-framework/packages/tkg-clusterclass/hack/../../../pkg/v1/providers/infrastructure-aws/v1.2.0/cconly/base.yaml /Users/fgold/go/src/github.com/vmware-tanzu/tanzu-framework/packages/tkg-clusterclass/hack/../../../pkg/v1/providers/infrastructure-aws/v1.2.0/cconly/overlay.yaml /Users/fgold/go/src/github.com/vmware-tanzu/tanzu-framework/packages/tkg-clusterclass/hack/../../../pkg/v1/providers/infrastructure-vsphere/v1.1.0/cconly/base.yaml /Users/fgold/go/src/github.com/vmware-tanzu/tanzu-framework/packages/tkg-clusterclass/hack/../../../pkg/v1/providers/infrastructure-vsphere/v1.1.0/cconly/overlay.yaml /Users/fgold/go/src/github.com/vmware-tanzu/tanzu-framework/packages/tkg-clusterclass/hack/../../../pkg/v1/providers/infrastructure-azure/v1.0.1/cconly/base.yaml /Users/fgold/go/src/github.com/vmware-tanzu/tanzu-framework/packages/tkg-clusterclass/hack/../../../pkg/v1/providers/infrastructure-azure/v1.0.1/cconly/overlay.yaml ./hack/verify-dirty.sh Verify uncommitted files... The following files are uncommitted. Please commit them or add them to .gitignore: - packages/tkg-clusterclass-aws/bundle/config/upstream/aws-clusterclass.yaml - packages/tkg-clusterclass-azure/bundle/config/upstream/azure-clusterclass.yaml - packages/tkg-clusterclass-vsphere/bundle/config/upstream/overlay-windows.yaml - packages/tkg-clusterclass-vsphere/bundle/config/upstream/vsphere-clusterclass.yaml Diff: diff --git a/packages/tkg-clusterclass-aws/bundle/config/upstream/aws-clusterclass.yaml b/packages/tkg-clusterclass-aws/bundle/config/upstream/aws-clusterclass.yaml deleted file mode 100644 index 8e4f5b2d..00000000 --- a/packages/tkg-clusterclass-aws/bundle/config/upstream/aws-clusterclass.yaml +++ /dev/null @@ -1,49 +0,0 @@ -#@ load("@ytt:overlay", "overlay") -#@ load("@ytt:yaml", "yaml") -#@ load("@ytt:data", "data") - ---- -apiVersion: cluster.x-k8s.io/v1beta1 -kind: ClusterClass -metadata: - name: tkg-aws-clusterclass - namespace: default -spec: - controlPlane: - ref: - apiVersion: controlplane.cluster.x-k8s.io/v1beta1 - kind: KubeadmControlPlaneTemplate - name: tkg-aws-kcp - machineInfrastructure: - ref: - kind: AWSMachineTemplate - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 - name: tkg-controlplane-aws-template - infrastructure: - ref: - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 - kind: AWSClusterTemplate - name: tkg-aws-cluster-template - workers: - machineDeployments: - - class: tkg-worker - template: - bootstrap: - ref: - apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 - kind: KubeadmConfigTemplate - name: tkg-aws-bootstrap-worker-template - infrastructure: - ref: - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 - kind: AWSMachineTemplate - name: tkg-aws-worker-template - variables: - - name: AWS_REGION - required: true - schema: - openAPIV3Schema: - type: string - #@ if data.values.AWS_REGION: - default: #@ data.values.AWS_REGION - #@ end diff --git a/packages/tkg-clusterclass-azure/bundle/config/upstream/azure-clusterclass.yaml b/packages/tkg-clusterclass-azure/bundle/config/upstream/azure-clusterclass.yaml deleted file mode 100644 index 2b10ed63..00000000 --- a/packages/tkg-clusterclass-azure/bundle/config/upstream/azure-clusterclass.yaml +++ /dev/null @@ -1,35 +0,0 @@ -apiVersion: cluster.x-k8s.io/v1beta1 -kind: ClusterClass -metadata: - name: tkg-azure-clusterclass - namespace: default -spec: - controlPlane: - ref: - apiVersion: controlplane.cluster.x-k8s.io/v1beta1 - kind: KubeadmControlPlaneTemplate - name: tkg-azure-kcp - machineInfrastructure: - ref: - kind: AzureMachineTemplate - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 - name: tkg-controlplane-azure-template - infrastructure: - ref: - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 - kind: AzureClusterTemplate - name: tkg-azure-cluster-template - workers: - machineDeployments: - - class: tkg-worker - template: - bootstrap: - ref: - apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 - kind: KubeadmConfigTemplate - name: tkg-azure-bootstrap-worker-template - infrastructure: - ref: - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 - kind: AzureMachineTemplate - name: tkg-azure-worker-template diff --git a/packages/tkg-clusterclass-vsphere/bundle/config/upstream/overlay-windows.yaml b/packages/tkg-clusterclass-vsphere/bundle/config/upstream/overlay-windows.yaml deleted file mode 100644 index ecdf083c..00000000 --- a/packages/tkg-clusterclass-vsphere/bundle/config/upstream/overlay-windows.yaml +++ /dev/null @@ -1,58 +0,0 @@ -#@ load("@ytt:overlay", "overlay") -#@ load("@ytt:data", "data") - -#@ if data.values.IS_WINDOWS_WORKLOAD_CLUSTER: - -#@overlay/match by=overlay.subset({"kind":"ClusterClass"}) ---- -apiVersion: cluster.x-k8s.io/v1beta1 -kind: ClusterClass -metadata: - namespace: #@ data.values.NAMESPACE -spec: - controlPlane: - ref: - name: tkg-vsphere-kcp - machineInfrastructure: - ref: - name: tkg-vsphere-control-plane - machineHealthCheck: - #@ if data.values.ENABLE_MHC or data.values.ENABLE_MHC_CONTROL_PLANE: - nodeStartupTimeout: #@ data.values.NODE_STARTUP_TIMEOUT - unhealthyConditions: - - type: Ready - status: Unknown - timeout: #@ data.values.MHC_UNKNOWN_STATUS_TIMEOUT - - type: Ready - status: "False" - timeout: #@ data.values.MHC_FALSE_STATUS_TIMEOUT - #@ end - infrastructure: - ref: - name: tkg-vsphere-infrastructure - namespace: #@ data.values.NAMESPACE - workers: - machineDeployments: - #@overlay/match by=overlay.index(0) - - class: tkg-worker - machineHealthCheck: - #@ if data.values.ENABLE_MHC or data.values.ENABLE_MHC_WORKER_NODE: - nodeStartupTimeout: #@ data.values.NODE_STARTUP_TIMEOUT - unhealthyConditions: - - type: Ready - status: Unknown - timeout: #@ data.values.MHC_UNKNOWN_STATUS_TIMEOUT - - type: Ready - status: "False" - timeout: #@ data.values.MHC_FALSE_STATUS_TIMEOUT - #@ end - template: - bootstrap: - ref: - name: tkg-vsphere-md-0 - namespace: #@ data.values.NAMESPACE - infrastructure: - ref: - name: tkg-vsphere-worker - namespace: #@ data.values.NAMESPACE -#@ end diff --git a/packages/tkg-clusterclass-vsphere/bundle/config/upstream/vsphere-clusterclass.yaml b/packages/tkg-clusterclass-vsphere/bundle/config/upstream/vsphere-clusterclass.yaml deleted file mode 100644 index 1d7e199c..00000000 --- a/packages/tkg-clusterclass-vsphere/bundle/config/upstream/vsphere-clusterclass.yaml +++ /dev/null @@ -1,35 +0,0 @@ -apiVersion: cluster.x-k8s.io/v1beta1 -kind: ClusterClass -metadata: - name: tkg-vsphere-clusterclass - namespace: default -spec: - controlPlane: - ref: - apiVersion: controlplane.cluster.x-k8s.io/v1beta1 - kind: KubeadmControlPlaneTemplate - name: tkg-vsphere-kcp - machineInfrastructure: - ref: - kind: VsphereMachineTemplate - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 - name: tkg-controlplane-vsphere-template - infrastructure: - ref: - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 - kind: VSphereClusterTemplate - name: tkg-vsphere-cluster-template - workers: - machineDeployments: - - class: tkg-worker - template: - bootstrap: - ref: - apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 - kind: KubeadmConfigTemplate - name: tkg-vsphere-bootstrap-worker-template - infrastructure: - ref: - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 - kind: VsphereMachineTemplate - name: tkg-vsphere-worker-template Signed-off-by: F. Gold <fgold@vmware.com>

vuil · 2022-06-12T02:04:55Z

/test install-vc7

github-actions · 2022-06-12T02:17:25Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/2581/20220612021723/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

navidshaikh · 2022-06-12T17:47:26Z

@vuil: /test install-vc7
Commit: cdd2e07

Tests passed! Build no: 2170

(build successful but CI signal failed because of a network flake)

.github/workflows/tkg_integration_tests.yaml

anujc25

Looks good

vuil requested review from a team as code owners June 8, 2022 06:44

vmwclabot added the cla-not-required label Jun 8, 2022

vuil commented Jun 8, 2022

View reviewed changes

hack/kind/Testing_Packages.md Show resolved Hide resolved

vuil commented Jun 8, 2022

View reviewed changes

.github/workflows/main.yaml Show resolved Hide resolved

vuil force-pushed the pblcm-1d-packaging-createc branch from bed9946 to 9fe902d Compare June 10, 2022 14:10

vuil changed the title ~~WIP: packaging management components, clusterclass-based creation~~ packaging management components, clusterclass-based creation Jun 10, 2022

vuil force-pushed the pblcm-1d-packaging-createc branch from 9fe902d to e37bae2 Compare June 10, 2022 20:15

Sai Manoj Bandi and others added 12 commits June 11, 2022 08:29

Implement feasibility check for using CC during legacy invocation

4343006

Make provider checksum ignore clusterclass files

6dbdc2e

files following the naming convention clusterclass-xxx.yaml are treated as cluster class files and are excluded from the checksum computation. also - renamed function to use more inclusive language Signed-off-by: Vui Lam <vui@vmware.com>

Enable CI on package-based-lcm branch (vmware-tanzu#2120)

7cc87ec

* Enable CI on package-based-lcm branch Co-authored-by: Sai Bandi <saib@vmware.com> Co-authored-by: Anuj Chaudhari <anujc@vmware.com>

github workflow updates to support clusterclass clustergen tests

39d9533

- installs dependencies Signed-off-by: Vui Lam <vui@vmware.com>

Install clusterclass to tkg-system namespace

73eb6af

also: Remove commented code from Azure ClusterClass

Patch CNI ingress rules for AWS cluster as part of clusterclass

39ec92d

Fix docker clusterclass package building issue

8d49825

Add Windows Template changes

97dc0a9

Fix variable structure error in base clusterclass (vmware-tanzu#2420)

c5fa8e7

This change fixes a nesting issue for defaults in clusterclass variables. The default was set on the variable and not the openAPISchema for the variable. Effects aws, azure and vsphere

Converting StorageClass to a package that can be installed (vmware-ta…

d9fb438

…nzu#1984)

Add scripts to create kind cluster for testing packages

0c83820

Chandra Pamuluri and others added 13 commits June 11, 2022 08:29

Implement 'tanzu cluster create -f ccluster.yaml'

2e440f9

Add functionality to auto determine cluster class

ee4a864

Adds a new ClusterClassSelector interface. This interface takes in the cluster config and determines what clusterclass to use and returns the name. Returns empty string if it can't determine the cluster class.

Respect management image's imagerepository override

ebd095c

Signed-off-by: Vui Lam <vui@vmware.com>

TKGs support for create cluster with clusterclass config file

f5c713c

Add auto deploy workload cluster using CCluster yaml based on feature…

b026501

…-flag - Added 'features.cluster.auto-apply-generated-clusterclass-based-configuration' feature-flag to support this usecase.

Gate the cluster creation on feature flag

5d9b655

Gate the cluster creation with ClusterClass with feature-flag `features.global.package-based-lcm` Authored-by: Anuj Chaudhari <anujc@vmware.com>

issue 2203, create cluster with ccluster.yaml support for TKGS

ce513bc

Add auto deploy workload cluster using CCluster yaml based on feature…

bdce907

…-flag - Added 'features.cluster.auto-apply-generated-clusterclass-based-configuration' feature-flag to support this usecase.

Add script to [hack]patch the management cluster

d67cfd5

Add CC based integration tests for AWS

8fdb08d

Update github action to modify bom

def1680

Fix linter errors

9e13af4

vuil force-pushed the pblcm-1d-packaging-createc branch from e37bae2 to 9e13af4 Compare June 11, 2022 15:32

vuil commented Jun 13, 2022

View reviewed changes

.github/workflows/tkg_integration_tests.yaml Show resolved Hide resolved

anujc25 approved these changes Jun 13, 2022

View reviewed changes

anujc25 added the ok-to-merge PRs should be labelled with this before merging label Jun 13, 2022

vuil merged commit c79de3d into vmware-tanzu:main Jun 13, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

packaging management components, clusterclass-based creation #2581

packaging management components, clusterclass-based creation #2581

vuil commented Jun 8, 2022 •

edited

Loading

github-actions bot commented Jun 8, 2022

codecov bot commented Jun 8, 2022 •

edited

Loading

vuil commented Jun 10, 2022

github-actions bot commented Jun 10, 2022

alfredthenarwhal commented Jun 10, 2022

github-actions bot commented Jun 10, 2022

vuil commented Jun 11, 2022

alfredthenarwhal commented Jun 11, 2022

vuil commented Jun 11, 2022

github-actions bot commented Jun 11, 2022

alfredthenarwhal commented Jun 11, 2022

vuil commented Jun 11, 2022

alfredthenarwhal commented Jun 11, 2022

vuil commented Jun 12, 2022

github-actions bot commented Jun 12, 2022

navidshaikh commented Jun 12, 2022 •

edited

Loading

anujc25 left a comment

packaging management components, clusterclass-based creation #2581

packaging management components, clusterclass-based creation #2581

Conversation

vuil commented Jun 8, 2022 • edited Loading

What this PR does / why we need it

Which issue(s) this PR fixes

Describe testing done for PR

Release note

PR Checklist

Additional information

Special notes for your reviewer

github-actions bot commented Jun 8, 2022

codecov bot commented Jun 8, 2022 • edited Loading

Codecov Report

vuil commented Jun 10, 2022

github-actions bot commented Jun 10, 2022

alfredthenarwhal commented Jun 10, 2022

github-actions bot commented Jun 10, 2022

vuil commented Jun 11, 2022

alfredthenarwhal commented Jun 11, 2022

vuil commented Jun 11, 2022

github-actions bot commented Jun 11, 2022

alfredthenarwhal commented Jun 11, 2022

vuil commented Jun 11, 2022

alfredthenarwhal commented Jun 11, 2022

vuil commented Jun 12, 2022

github-actions bot commented Jun 12, 2022

navidshaikh commented Jun 12, 2022 • edited Loading

anujc25 left a comment

Choose a reason for hiding this comment

vuil commented Jun 8, 2022 •

edited

Loading

codecov bot commented Jun 8, 2022 •

edited

Loading

navidshaikh commented Jun 12, 2022 •

edited

Loading