E2E usability updates #7353

kaovilai · 2024-01-24T19:29:36Z

Set GOBIN so Makefile don't modify $PATH on go install
Fix realPath resolving when cloud credentials is prefixed by ~ for home dir
Add Velero NS to get/delete call from CLI to ignore NS set in ~/.config/velero/config.json
Discover dockerconfig in home dir
Typo updates
e2e README.md updates

Signed-off-by: Tiger Kaovilai tkaovila@redhat.com

Thank you for contributing to Velero!

Please add a summary of your change

Does your change fix a particular issue?

Fixes #(issue)

Please indicate you've done the following:

Accepted the DCO. Commits without the DCO will delay acceptance.
Created a changelog file or added /kind changelog-not-required as a comment on this pull request.
Updated the corresponding documentation in site/content/docs/main.

Changes to make e2e tests work in my local env with kind cluster

make test-e2e CREDS_FILE=x BSL_BUCKET=x CLOUD_PROVIDER=kind OBJECT_STORE_PROVIDER=aws BSL_PREFIX=velero BSL_CONFIG="region=us-east-1" GINKGO_FOCUS="Basic\]\[ClusterResource";

Ran 3 of 54 Specs in 137.958 seconds
SUCCESS! -- 3 Passed | 0 Failed | 0 Pending | 51 Skipped
PASS

test/testdata/storage-class/kind.yaml

kaovilai · 2024-01-25T01:12:32Z

/kind changelog-not-required

codecov · 2024-01-25T01:26:24Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (99e0c48) 61.65% compared to head (b1d95cf) 61.65%.
Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #7353   +/-   ##
=======================================
  Coverage   61.65%   61.65%           
=======================================
  Files         262      262           
  Lines       28479    28479           
=======================================
  Hits        17558    17558           
  Misses       9689     9689           
  Partials     1232     1232

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

test/util/velero/velero_utils.go

kaovilai · 2024-01-25T04:30:35Z

@danfengliu lmk if there is any problematic changes here. I can also split into smaller PRs if required.

danfengliu · 2024-01-25T05:58:03Z

@kaovilai I will run nightly against this PR, it seems no need to split it for now.

danfengliu · 2024-01-25T08:05:29Z

Hi @kaovilai , there're some cross parts with this PR 7292, so if this PR is merged first ( nightly CI has support IRSA, so I prefer to have PR 7292 go first, sorry for letting you wait for this), you might need to rebase main branch and solve some conflicts. And please help to review my PR, we're going to cover IRSA in nightly test.

kaovilai · 2024-01-25T14:08:14Z

Sounds good

mateusoliveira43 · 2024-01-25T15:25:38Z

test/perf/Makefile

maybe for another PR, but can't test/e2e/Makefile and test/perf/Makefile be merged to avoid duplication?

@qiuming-best , please help to provide your thought on this question, is there any block things from merging these 2 Makefile?

Yes, maybe it could be merged into one, but may be more confusing to use.

I originally set the performance Makefile apart from the e2e Makefile for both tests, which all need lots of input parameters, and also both of them have some unique parameters. To not confuse I make them independent.

kaovilai · 2024-01-26T01:49:20Z

.github/workflows/e2e-test-kind.yaml

+        focus:
+          # tests to focus on, use `|` to concatenate multiple regexes to run on the same job
+          - Basic\]\[ClusterResource
+          - Basic\]\[StorageClass
+          - PrivilegesMgmt\]\[SSR|ResourceModifier
+          - ResourceFiltering
+          - NamespaceMapping\]\[Single\]\[Restic|NamespaceMapping\]\[Multiple\]\[Restic


Adding more coverage to the PR e2e

test/util/kibishii/kibishii_utils.go

danfengliu · 2024-02-02T09:32:06Z

Logic in line 78~88 should be moved to here, because this logic only triggered by function VeleroInstall , but most of tests will not install Velero if it’s not needed or Velero instance exists by previous test. so objectStoreProvider might not be set in these tests.

For example in backup deletion test, veleroCfg was set to the initial empty value here.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

danfengliu · 2024-02-04T08:35:28Z

test/util/velero/install.go

-		args = append(args, "--features", options.Features)
-		if strings.EqualFold(options.Features, FeatureCSI) && options.UseVolumeSnapshots {
-			if strings.EqualFold(cloudProvider, "azure") {
+	if strings.EqualFold(cloudProvider, "azure") {


Suggested change

if strings.EqualFold(cloudProvider, "azure") {

if len(options.Features) > 0 {

args = append(args, "--features", options.Features)

if strings.EqualFold(options.Features, FeatureCSI) && options.UseVolumeSnapshots &&

strings.EqualFold(cloudProvider, "azure") {

fmt.Println("Start to install Azure VolumeSnapshotClass ...")

if err := KubectlApplyByFile(ctx, "../util/csi/AzureVolumeSnapshotClass.yaml"); err != nil {

return err

If options.Features is not empty, arg --features must be set with that value no matter what provider it is.

Current code blocked data mover pipeline due to args --features is missing.

danfengliu · 2024-02-04T09:39:55Z

Logic in line 78~88 should be moved to here, because this logic only triggered by function VeleroInstall , but most of tests will not install Velero if it’s not needed or Velero instance exists by previous test. so objectStoreProvider might not be set in these tests.
For example in backup deletion test, veleroCfg was set to the initial empty value here.

Sorry I also made a mistake, after re-running nightly against your latest code, test failed due to missing --provider , this is cause by moving line 78~88 to another location, this logic should not be deleted from the original place, because it is used when isStandbyCluster is true, please recover line 78~88, which means we have this logic in 2 places, you can consider wrap it to a function.

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

…th resolving when cloud credentials is prefixed by `~` for home dir Use `~/.docker/config.json` if REGISTRY_CREDENTIAL_FILE not defined and skip step if does not exists since it is optional Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Set `GOBIN` so Makefile don't modify $PATH on `go install` Fix realPath resolving when cloud credentials is prefixed by `~` for home dir Use `~/.docker/config.json` if REGISTRY_CREDENTIAL_FILE not defined and skip step if does not exists since it is optional Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Add kind testdata storageclass Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Add kind testdata storageclass Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> log `Start to install Azure VolumeSnapshotClass ...` only on azure when csi is enabled Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Add BSL_CONFIG example and notes Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Makefile: Set `GOBIN` for `_output/...` Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> README spacing Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> StandbyClusterObjectStoreProvider typo Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Specify velero namespace during get/delete command Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Use object stores rather than cloudProvider for bucket queries Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Remove debug print Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> simplify NS get changes, add velero NS to `DeleteBackupResource` Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Skip file system backups on kind which uses hostPath volumes Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Add StorageClass change test to PR kind e2e Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Add more tests to pr Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Add NS mapping to PR e2e Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Add `SKIP_KIND` to some jobs containing volumes Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Remove kind from kibishii tests Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Label volume resource policies as restic, skip restic/snapshot tests, add more tests Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> TTLTest is a snapshot test Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Remove non working tests Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Resolve vmware-tanzu#7353 (comment) Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> address https://github.com/vmware-tanzu/velero/pull/7353/files#r1477218762 Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com> Address vmware-tanzu#7353 (comment) Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

danfengliu

LGTM

github-actions bot added the has-e2e-tests label Jan 24, 2024

kaovilai force-pushed the e2e-updates branch 3 times, most recently from b0e615e to 84c5dbd Compare January 24, 2024 19:53

kaovilai changed the title ~~Set GOBIN so Makefile don't modify $PATH on go install~~ E2E usability updates Jan 24, 2024

kaovilai force-pushed the e2e-updates branch from 84c5dbd to 12288ce Compare January 24, 2024 22:33

kaovilai commented Jan 25, 2024

View reviewed changes

test/testdata/storage-class/kind.yaml Outdated Show resolved Hide resolved

github-actions bot added the kind/changelog-not-required PR does not require a user changelog. Often for docs, website, or build changes label Jan 25, 2024

kaovilai marked this pull request as ready for review January 25, 2024 01:31

github-actions bot assigned kaovilai Jan 25, 2024

github-actions bot requested review from Lyndon-Li and reasonerjt January 25, 2024 01:31

kaovilai force-pushed the e2e-updates branch from 547aff2 to e4e926a Compare January 25, 2024 03:56

kaovilai commented Jan 25, 2024

View reviewed changes

test/util/velero/velero_utils.go Outdated Show resolved Hide resolved

test/util/velero/velero_utils.go Outdated Show resolved Hide resolved

kaovilai mentioned this pull request Jan 25, 2024

Ability to ignore Velero client configuration file for e2e #7355

Closed

reasonerjt requested review from danfengliu and ywk253100 and removed request for reasonerjt and Lyndon-Li January 25, 2024 04:55

mateusoliveira43 reviewed Jan 25, 2024

View reviewed changes

kaovilai commented Jan 26, 2024

View reviewed changes

danfengliu reviewed Jan 26, 2024

View reviewed changes

test/util/kibishii/kibishii_utils.go Outdated Show resolved Hide resolved

kaovilai mentioned this pull request Jan 26, 2024

E2E: Stateless Workload Alternative to RunKibishiiTests that can run on KinD (no FSB/Snapshot) #7360

Closed

kaovilai force-pushed the e2e-updates branch from 520727e to de9b310 Compare February 2, 2024 04:49

kaovilai force-pushed the e2e-updates branch from de9b310 to e6750d5 Compare February 2, 2024 05:49

kaovilai added a commit to kaovilai/velero that referenced this pull request Feb 2, 2024

Address vmware-tanzu#7353 (comment)

33f3f7e

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

kaovilai requested a review from danfengliu February 2, 2024 22:07

danfengliu suggested changes Feb 4, 2024

View reviewed changes

kaovilai added a commit to kaovilai/velero that referenced this pull request Feb 5, 2024

Resolve vmware-tanzu#7353 (comment)

dd8c925

Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>

kaovilai force-pushed the e2e-updates branch from dd8c925 to b1d95cf Compare February 5, 2024 03:17

kaovilai requested a review from danfengliu February 5, 2024 03:17

danfengliu requested a review from blackpiglet February 7, 2024 02:03

danfengliu approved these changes Feb 7, 2024

View reviewed changes

blackpiglet approved these changes Feb 7, 2024

View reviewed changes

blackpiglet merged commit 3b8370e into vmware-tanzu:main Feb 7, 2024
66 checks passed

shubham-pampattiwar mentioned this pull request Oct 21, 2024

Nominate Tiger Kaovilai as a maintainer #8328

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

E2E usability updates #7353

E2E usability updates #7353

kaovilai commented Jan 24, 2024 •

edited

Loading

kaovilai commented Jan 25, 2024

codecov bot commented Jan 25, 2024 •

edited

Loading

kaovilai commented Jan 25, 2024

danfengliu commented Jan 25, 2024

danfengliu commented Jan 25, 2024 •

edited

Loading

kaovilai commented Jan 25, 2024

mateusoliveira43 Jan 25, 2024

danfengliu Jan 26, 2024

qiuming-best Jan 26, 2024

kaovilai Jan 26, 2024

danfengliu commented Feb 2, 2024

danfengliu Feb 4, 2024

danfengliu Feb 4, 2024

kaovilai Feb 4, 2024

danfengliu commented Feb 4, 2024

danfengliu left a comment

-	if strings.EqualFold(cloudProvider, "azure") {
+	if len(options.Features) > 0 {
+		args = append(args, "--features", options.Features)
+		if strings.EqualFold(options.Features, FeatureCSI) && options.UseVolumeSnapshots &&
+		      strings.EqualFold(cloudProvider, "azure")  {
+			fmt.Println("Start to install Azure VolumeSnapshotClass ...")
+			if err := KubectlApplyByFile(ctx, "../util/csi/AzureVolumeSnapshotClass.yaml"); err != nil {
+				return err

E2E usability updates #7353

E2E usability updates #7353

Conversation

kaovilai commented Jan 24, 2024 • edited Loading

Please add a summary of your change

Does your change fix a particular issue?

Please indicate you've done the following:

kaovilai commented Jan 25, 2024

codecov bot commented Jan 25, 2024 • edited Loading

Codecov Report

kaovilai commented Jan 25, 2024

danfengliu commented Jan 25, 2024

danfengliu commented Jan 25, 2024 • edited Loading

kaovilai commented Jan 25, 2024

mateusoliveira43 Jan 25, 2024

Choose a reason for hiding this comment

danfengliu Jan 26, 2024

Choose a reason for hiding this comment

qiuming-best Jan 26, 2024

Choose a reason for hiding this comment

kaovilai Jan 26, 2024

Choose a reason for hiding this comment

danfengliu commented Feb 2, 2024

danfengliu Feb 4, 2024

Choose a reason for hiding this comment

danfengliu Feb 4, 2024

Choose a reason for hiding this comment

kaovilai Feb 4, 2024

Choose a reason for hiding this comment

danfengliu commented Feb 4, 2024

danfengliu left a comment

Choose a reason for hiding this comment

kaovilai commented Jan 24, 2024 •

edited

Loading

codecov bot commented Jan 25, 2024 •

edited

Loading

danfengliu commented Jan 25, 2024 •

edited

Loading