Skip to content
This repository has been archived by the owner on Jun 28, 2023. It is now read-only.

Build and package everything related to TCE using a core main branch #918

Merged
merged 1 commit into from
Jul 14, 2021

Conversation

davidvonthenen
Copy link
Contributor

@davidvonthenen davidvonthenen commented Jul 1, 2021

What this PR does / why we need it

This builds and packages TCE using a branch called tce-main (https://github.com/vmware-tanzu/tanzu-framework/tree/tce-main) that preserves standalone clusters but using a commit starting from main (vmware-tanzu/tanzu-framework@23164b1). As you can see, the diff between the two is minimal (vmware-tanzu/tanzu-framework@23164b1...tce-main)... in other words is only adds code and does not modify. If we want to update to a later version of main, we just need to rebase tce-main to that commit, test it works, and then update the branch.

The changes include:

  • most are deleting the package plugin
  • pretty much everything else is updating code to the latest dependencies and folder structure, or
  • removing code referencing the obsolete repos (tkg-providrer, tkg-cli, and tanzu-plugins)

IMPORTANT: This PR also removes code and references to the experimental package plugin originating from this repo and instead pulls the package plugin from the core repo instead.

This PR obsoletes this PR:
#730

Which issue(s) this PR fixes

NA

Describe testing done for PR

Verified that make release still works and won't break the release.

Verified tanzu plugin list works:

[vonthd@centos8 tce]$ tanzu plugin list
  NAME                LATEST VERSION  DESCRIPTION                                                        REPOSITORY  VERSION       STATUS         
  alpha               v1.3.1          Alpha CLI commands                                                 core                      not installed  
  audit               v0.1.0          Run an audit request on an organization                            advanced                  not installed  
  builder                             Build Tanzu components                                                         v1.4.0-dev.0  installed      
  cluster             v1.3.1          Kubernetes cluster operations                                      core        v1.4.0-dev.0  installed      
  clustergroup        v0.1.0          A group of Kubernetes clusters                                     advanced                  not installed  
  data-protection     v0.1.0          Backup, restore, or migrate cluster data.                          advanced                  not installed  
  integration                         Get available integrations and their information from registry.    advanced                  not installed  
  kubernetes-release  v1.3.1          Kubernetes release operations                                      core        v1.4.0-dev.0  installed      
  login               v1.3.1          Login to the platform                                              core        v1.4.0-dev.0  installed      
  management-cluster  v1.3.1          Kubernetes management cluster operations                           core        v1.4.0-dev.0  installed      
  package                             Tanzu package management                                                       v1.4.0-dev.0  installed      
  pinniped-auth       v1.3.1          Pinniped authentication operations (usually not directly invoked)  core        v1.4.0-dev.0  installed      
  policy              v0.1.0          Permission management for resources                                advanced                  not installed  
  standalone-cluster                  Create clusters without a dedicated management cluster                         v0.7.0-dev.1  installed      
  test                                Test the CLI                                                                   v1.4.0-dev.0  installed      
  tmc-cluster         v0.1.0          A TMC managed Kubernetes cluster                                   advanced                  not installed  
  workspace           v0.1.0          A group of Kubernetes namespaces                                   advanced                  not installed  

and you can invoke the standalone cluster plugin (but did not deploy a cluster):

[vonthd@centos8 tce]$ tanzu standalone-cluster help
Create clusters without a dedicated management cluster

Usage:
  tanzu standalone-cluster [command]

Available Commands:
  completion    generate the autocompletion script for the specified shell
  create        create a standalone workload cluster
  delete        delete a standalone workload cluster

Flags:
  -h, --help            help for standalone-cluster
  -v, --verbose int32   Number for the log level verbosity(0-9)

Use "tanzu standalone-cluster [command] --help" for more information about a command.

Special notes for your reviewer

NA

Does this PR introduce a user-facing change?

NA

resolves: #508

EDIT: updated with tanzu-framework links and references

Copy link
Contributor

@joshrosso joshrosso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like something is going on with plugin builder or LDFLAGs for the standalone plugin.

TANZU_CLI_NO_INIT=true GOPRIVATE="github.com/vmware-tanzu-private/*,github.com/vmware-tanzu/*,github.com/dvonthenen/*,github.com/joshrosso/*" go run -ldflags "-s -w -X "github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildDate=$(date -u +"%Y-%m-%d")" -X "github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildSHA=$(git rev-parse --short HEAD)" -X "github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildVersion=$(git describe --tags --abbrev=0)" -X 'main.BuildEdition="tce"'" github.com/vmware-tanzu-private/core/cmd/cli/tanzu \
        plugin install all --local ./artifacts
Error: version cannot be empty for plugin "standalone-cluster"

✖  version cannot be empty for plugin "standalone-cluster"
exit status 1
make: *** [Makefile:246: install-cli-plugins] Error 1

@jpmcb
Copy link
Contributor

jpmcb commented Jul 6, 2021

✖ version cannot be empty for plugin "standalone-cluster"

Hmmm I saw something similar when building the multiple go modules in #735. I was able to narrow it down to this chunk in the core main builder code: https://github.com/vmware-tanzu-private/core/commit/d5d49dad5327afe06700429557d8288ab208a1a6#diff-baf814ba21730ad0b108762105ce1339828946c47899def96f26c67fa92ddf04R38-R42 where it wasn't properly pulling the build metadata off the version

It wasn't successful when using the auto-generated tag v0.6.0-dev.2 even though that is a valid semver. I was able to reproduce this in the go playground. Looking deeper into it, replacing the - with the + makes the semver.Build call not just return an empty string. This can be seen in this playground snippet.

But then I wasn't able to reproduce it locally and it stopped failing in CI ... So definitely a flake, but wasn't able to investigate further

@joshrosso
Copy link
Contributor

joshrosso commented Jul 6, 2021

+ would be the valid semver identifier anyway, right? Should we just start using that?

@nrb
Copy link
Contributor

nrb commented Jul 6, 2021

fwiw on my macOS Catalina machine, it succeeds.

ℹ  🐱 - $ /usr/local/bin/go build -o /Users/nrb/projects/tce/artifacts/standalone-cluster/v0.7.0-dev.1/test/tanzu-standalone-cluster-test-windows_amd64.exe -ldflags -s -w -X github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildDate=2021-07-06 -X github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildSHA=933f4ba -X github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildVersion=v0.7.0-dev.1 -X 'main.BuildEdition=tce' -X 'github.com/vmware-tanzu-private/core/pkg/v1/tkg/buildinfo.IsOfficialBuild=' -tags  ./cli/cmd/plugin/standalone-cluster/test
ℹ  🐱 - $ /usr/local/bin/go build -o /Users/nrb/projects/tce/artifacts/standalone-cluster/v0.7.0-dev.1/test/tanzu-standalone-cluster-test-linux_386 -ldflags -s -w -X github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildDate=2021-07-06 -X github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildSHA=933f4ba -X github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildVersion=v0.7.0-dev.1 -X 'main.BuildEdition=tce' -X 'github.com/vmware-tanzu-private/core/pkg/v1/tkg/buildinfo.IsOfficialBuild=' -tags  ./cli/cmd/plugin/standalone-cluster/test
ℹ  🐱 - $ /usr/local/bin/go build -o /Users/nrb/projects/tce/artifacts/standalone-cluster/v0.7.0-dev.1/test/tanzu-standalone-cluster-test-linux_amd64 -ldflags -s -w -X github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildDate=2021-07-06 -X github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildSHA=933f4ba -X github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildVersion=v0.7.0-dev.1 -X 'main.BuildEdition=tce' -X 'github.com/vmware-tanzu-private/core/pkg/v1/tkg/buildinfo.IsOfficialBuild=' -tags  ./cli/cmd/plugin/standalone-cluster/test
ℹ  🐱 - $ /usr/local/bin/go build -o /Users/nrb/projects/tce/artifacts/standalone-cluster/v0.7.0-dev.1/test/tanzu-standalone-cluster-test-linux_arm64 -ldflags -s -w -X github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildDate=2021-07-06 -X github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildSHA=933f4ba -X github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildVersion=v0.7.0-dev.1 -X 'main.BuildEdition=tce' -X 'github.com/vmware-tanzu-private/core/pkg/v1/tkg/buildinfo.IsOfficialBuild=' -tags  ./cli/cmd/plugin/standalone-cluster/test
ℹ  🐱 - $ /usr/local/bin/go build -o /Users/nrb/projects/tce/artifacts/standalone-cluster/v0.7.0-dev.1/test/tanzu-standalone-cluster-test-darwin_amd64 -ldflags -s -w -X github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildDate=2021-07-06 -X github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildSHA=933f4ba -X github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildVersion=v0.7.0-dev.1 -X 'main.BuildEdition=tce' -X 'github.com/vmware-tanzu-private/core/pkg/v1/tkg/buildinfo.IsOfficialBuild=' -tags  ./cli/cmd/plugin/standalone-cluster/test
ℹ  🐱 - $ /usr/local/bin/go build -o /Users/nrb/projects/tce/artifacts/standalone-cluster/v0.7.0-dev.1/test/tanzu-standalone-cluster-test-darwin_arm64 -ldflags -s -w -X github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildDate=2021-07-06 -X github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildSHA=933f4ba -X github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildVersion=v0.7.0-dev.1 -X 'main.BuildEdition=tce' -X 'github.com/vmware-tanzu-private/core/pkg/v1/tkg/buildinfo.IsOfficialBuild=' -tags  ./cli/cmd/plugin/standalone-cluster/test
========
✔  successfully built local repository
TANZU_CLI_NO_INIT=true GOPRIVATE="github.com/vmware-tanzu-private/*,github.com/vmware-tanzu/*,github.com/dvonthenen/*,github.com/joshrosso/*" go run -ldflags "-s -w -X "github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildDate=$(date -u +"%Y-%m-%d")" -X "github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildSHA=$(git rev-parse --short HEAD)" -X "github.com/vmware-tanzu-private/core/pkg/v1/cli.BuildVersion=$(git describe --tags --abbrev=0)" -X 'main.BuildEdition=tce' -X 'github.com/vmware-tanzu-private/core/pkg/v1/tkg/buildinfo.IsOfficialBuild=""'" github.com/vmware-tanzu-private/core/cmd/cli/tanzu \
		plugin install all --local ./artifacts

[COMPLETE] installed plugins at "/Users/nrb/Library/Application Support"/tanzu-cli/. These plugins will be automatically detected by tanzu CLI.

[COMPLETE] installed tanzu CLI at /Users/nrb/go:/usr/share/gocode//bin/tanzu. Move this binary to a location in your path!
make build-all  2054.19s user 610.77s system 769% cpu 5:46.54 total

@stmcginnis
Copy link
Contributor

Build succeeds for me. Attempted to deploy on Ubuntu 21.04 failed, but we have known issues there so I don't think this should block things. Would be good to get this through so we can identify and fix any side effects of the change.

@davidvonthenen
Copy link
Contributor Author

yea, we definitely need to fix the bom reference and kind image reference... i just haven't touched that part of the code and am unfamiliar with what i might need to do

Copy link
Contributor

@joshrosso joshrosso left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plugin installs are now impacted by the file ~/.config/tanzu/config.yaml. 2 questions we should figure out for this PR:

  1. Should we be cleaning out this config file when doing local builds to ensure configuration does not stick around?

  2. When should we run make set-unstable-versions such that our plugins with a - or + suffix are included?

For #2, i solved it by adding

diff --git a/hack/build-tanzu.sh b/hack/build-tanzu.sh
index 1d7b6ba..f23cf21 100755
--- a/hack/build-tanzu.sh
+++ b/hack/build-tanzu.sh
@@ -47,6 +47,7 @@ sed -i.bak -e "s/\$(shell git describe --tags --abbrev=0 2>\$(NUL))/${BUILD_VERS
 # make configure-bom
 # build and install all "core" CLI plugins
 # (e.g. management-cluster, cluster, etc)
+make set-unstable-versions
 make build-install-cli-all
 # by default, core only builds admins plugins for the current platform. we need darwin also.
 GOHOSTOS=linux GOHOSTARCH=amd64 make build-plugin-admin

@stmcginnis stmcginnis mentioned this pull request Jul 8, 2021
@davidvonthenen
Copy link
Contributor Author

Just fixing the release-docker build and i think we should be good

@davidvonthenen davidvonthenen force-pushed the build-core-main branch 3 times, most recently from 01034e0 to 524552c Compare July 8, 2021 21:03
@jpmcb
Copy link
Contributor

jpmcb commented Jul 8, 2021

Build with make build and everything compiled / built for me!!

Deploying a standalone cluster:

I get the following error when trying to deploy a management cluster:

Setting up standalone cluster...
Validating configuration...
Using infrastructure provider docker:v0.3.19
Generating cluster configuration...
Error: failed to initialize standalone cluster.
Cause: unable to set up management cluster: unable to build standalone cluster configuration: unable to get template: unable to parse raw artifact bytes: no kind "TemplateDefinition" is registered for version "providers.tanzu.vmware.com/v1alpha1" in scheme "pkg/runtime/scheme.go:101"

I get the same error with the CLI and the UI. I'm guessing this is related to the BOM issue?

Deploying a management cluster:

This also fails for me, but it gets further than the standalone-cluster. I think this is related to #891 since I'm getting the same error. So, with this bump, are we not getting the newest providers?

❯ k get pods -A --kubeconfig /home/jmcb/.kube-tkg/tmp/config_8QWPGcNE
NAMESPACE            NAME                                                                  READY   STATUS             RESTARTS   AGE
cert-manager         cert-manager-7b7f88644d-6c972                                         0/1     Pending            0          13m
cert-manager         cert-manager-cainjector-7cd8d6b475-dnlj8                              0/1     Pending            0          13m
cert-manager         cert-manager-webhook-5f9b95bd4-p5q55                                  0/1     Pending            0          13m
kube-system          coredns-68d49685bd-8pb9k                                              0/1     Pending            0          15m
kube-system          coredns-68d49685bd-jwpqn                                              0/1     Pending            0          15m
kube-system          etcd-tkg-kind-c3jm93n69kf5tpa109a0-control-plane                      1/1     Running            0          15m
kube-system          kindnet-s2zrg                                                         0/1     Error              4          15m
kube-system          kube-apiserver-tkg-kind-c3jm93n69kf5tpa109a0-control-plane            1/1     Running            0          15m
kube-system          kube-controller-manager-tkg-kind-c3jm93n69kf5tpa109a0-control-plane   1/1     Running            0          15m
kube-system          kube-proxy-r5hql                                                      0/1     CrashLoopBackOff   7          15m
kube-system          kube-scheduler-tkg-kind-c3jm93n69kf5tpa109a0-control-plane            1/1     Running            0          15m
local-path-storage   local-path-provisioner-78776bfc44-xvv2h                               0/1     Pending            0          15m

@davidvonthenen
Copy link
Contributor Author

davidvonthenen commented Jul 9, 2021

Build with make build and everything compiled / built for me!!

Deploying a standalone cluster:

I get the following error when trying to deploy a management cluster:

Setting up standalone cluster...
Validating configuration...
Using infrastructure provider docker:v0.3.19
Generating cluster configuration...
Error: failed to initialize standalone cluster.
Cause: unable to set up management cluster: unable to build standalone cluster configuration: unable to get template: unable to parse raw artifact bytes: no kind "TemplateDefinition" is registered for version "providers.tanzu.vmware.com/v1alpha1" in scheme "pkg/runtime/scheme.go:101"

I get the same error with the CLI and the UI. I'm guessing this is related to the BOM issue?

Deploying a management cluster:

This also fails for me, but it gets further than the standalone-cluster. I think this is related to #891 since I'm getting the same error. So, with this bump, are we not getting the newest providers?

❯ k get pods -A --kubeconfig /home/jmcb/.kube-tkg/tmp/config_8QWPGcNE
NAMESPACE            NAME                                                                  READY   STATUS             RESTARTS   AGE
cert-manager         cert-manager-7b7f88644d-6c972                                         0/1     Pending            0          13m
cert-manager         cert-manager-cainjector-7cd8d6b475-dnlj8                              0/1     Pending            0          13m
cert-manager         cert-manager-webhook-5f9b95bd4-p5q55                                  0/1     Pending            0          13m
kube-system          coredns-68d49685bd-8pb9k                                              0/1     Pending            0          15m
kube-system          coredns-68d49685bd-jwpqn                                              0/1     Pending            0          15m
kube-system          etcd-tkg-kind-c3jm93n69kf5tpa109a0-control-plane                      1/1     Running            0          15m
kube-system          kindnet-s2zrg                                                         0/1     Error              4          15m
kube-system          kube-apiserver-tkg-kind-c3jm93n69kf5tpa109a0-control-plane            1/1     Running            0          15m
kube-system          kube-controller-manager-tkg-kind-c3jm93n69kf5tpa109a0-control-plane   1/1     Running            0          15m
kube-system          kube-proxy-r5hql                                                      0/1     CrashLoopBackOff   7          15m
kube-system          kube-scheduler-tkg-kind-c3jm93n69kf5tpa109a0-control-plane            1/1     Running            0          15m
local-path-storage   local-path-provisioner-78776bfc44-xvv2h                               0/1     Pending            0          15m

I can give this a try on RHEL8...

On CentOS8 with a management deploy to AWS using latest commit off main, stalls on certmanager trying to come up... verified this is working using a simple build off core/main:

Validating configuration...
Using infrastructure provider aws:v0.6.6
Generating cluster configuration...
Setting up bootstrapper...
Bootstrapper created. Kubeconfig: /home/vonthd/.kube-tkg/tmp/config_8ElCK9T6
Installing providers on bootstrapper...
Fetching providers
Installing cert-manager Version="v1.1.0"
Waiting for cert-manager to be available...
Installing Provider="cluster-api" Version="v0.3.19" TargetNamespace="capi-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.19" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.19" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="infrastructure-aws" Version="v0.6.6" TargetNamespace="capa-system"
Start creating management cluster...
Saving management cluster kubeconfig into /home/vonthd/.kube/config
Installing providers on management cluster...
Fetching providers
Installing cert-manager Version="v1.1.0"
Waiting for cert-manager to be available...

@davidvonthenen
Copy link
Contributor Author

davidvonthenen commented Jul 12, 2021

updated this PR to use tanzu-framework latest... will update the initial PR comments to reflect this

EDIT: updated to remove all "core" references... everything builds fine

@jpmcb
Copy link
Contributor

jpmcb commented Jul 12, 2021

I'm still getting the same kube-proxy crash loop backoff from #891

On kernel:

OS: Pop!_OS 21.04 x86_64
Kernel: 5.11.0-7620-generic

Did we bump to the latest providers in this PR or is this something we should tackle outside this uber PR?

@davidvonthenen
Copy link
Contributor Author

I'm still getting the same kube-proxy crash loop backoff from #891

On kernel:

OS: Pop!_OS 21.04 x86_64
Kernel: 5.11.0-7620-generic

Did we bump to the latest providers in this PR or is this something we should tackle outside this uber PR?

We are running off the latest on tanz-framework main... the other repo was just marked read only on Friday and the new repo isn't accepting changes until some things are worked out. so I'm guessing they are probably out of sync.

@davidvonthenen davidvonthenen force-pushed the build-core-main branch 2 times, most recently from 933e599 to 9d52d11 Compare July 12, 2021 22:06
@jpmcb
Copy link
Contributor

jpmcb commented Jul 13, 2021

Trying this again this morning, but getting the following error:

❯ CLUSTER_PLAN=dev tanzu management-cluster create -i docker

Validating the pre-requisites...
Error: configuration validation failed: unable to verify cluster name uniqueness

I remember someone saying something maybe about removing some of the configs before this will work. But removing ~/.config/tanzu didn't resolve this

@davidvonthenen
Copy link
Contributor Author

Trying this again this morning, but getting the following error:

❯ CLUSTER_PLAN=dev tanzu management-cluster create -i docker

Validating the pre-requisites...
Error: configuration validation failed: unable to verify cluster name uniqueness

I remember someone saying something maybe about removing some of the configs before this will work. But removing ~/.config/tanzu didn't resolve this

also be sure to stop and remove orphaned kind clusters. i got burned by that recently.

@stmcginnis
Copy link
Contributor

Trying this again this morning, but getting the following error

There's no name provided on that command line. Copy/paste error, or was that actually omitted?

@jpmcb
Copy link
Contributor

jpmcb commented Jul 13, 2021

I must have had something funky going on in my environment. Did make build-all which removes a number of config folders and it's working now with

CLUSTER_PLAN=dev tanzu management-cluster create -i docker

@stmcginnis I got that command from josh and it's going now and looks like it's getting some generated cluster name

@github-actions github-actions bot added owner/docs Work executed by VMware documentation team area/release-eng labels Jul 13, 2021
make builds succeed

update build options

Updates so that make release functions correctly

Building of core main

fix version

more updates

latest updates rebased from the rebase on top of main branch

update go mod

successfully built off main

remove install option not needed

update core main hash

fix package darwin

fix packaging

rebase on main

update the oops

updates to edition

remove bom workarounds

default value

conform to core plugin source layout

fix UI working

fix makefile

fix linting

fix hack/packages

build linux explicitly to cover building on other platforms

enable non-GA builds

fix release-docker build

second fix to release-docker

update with latest lint fixes

remove the go.mod from the packages util

go mod tidy

remove directory for docker build

update core references to latest

remove boms dir

update to use tanzu-framework

remove references to core

rebase on latest main

fix sha saying dirty

fix make check
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
owner/core-eng Work executed by TCE's core engineering team owner/docs Work executed by VMware documentation team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Introduce proper model for BOM/Provider/TKR customization in TCE and Move atop tanzu-framework
5 participants