tanzu-auth-controller-manager unit test failing due to v1_cascade_controller_test assertion failing

[codegold79]

Bug description

Not sure what's causing this unit test to fail, but an assertion is not passing. Also, Ginkgo is warning that some feature(s) are deprecated.

See test logs here: https://github.com/vmware-tanzu/tanzu-framework/runs/6535358353?check_suite_focus=true

@sabbey37 or @ankeesler might be able to help with fixing this.

=== RUN   TestController
Running Suite: Controller Suite
===============================
Random Seed: 1653120162
Will run 34 of 34 specs

••
------------------------------
• Failure [1.045 seconds]
Controller
/home/runner/work/tanzu-framework/tanzu-framework/addons/pinniped/tanzu-auth-controller-manager/controllers/v3_cascade_controller_test.go:22
  ClusterBootstrap Secret
  /home/runner/work/tanzu-framework/tanzu-framework/addons/pinniped/tanzu-auth-controller-manager/controllers/v3_cascade_controller_test.go:58
    when random values are added to the secret
    /home/runner/work/tanzu-framework/tanzu-framework/addons/pinniped/tanzu-auth-controller-manager/controllers/v3_cascade_controller_test.go:94
      they are preserved [It]
      /home/runner/work/tanzu-framework/tanzu-framework/addons/pinniped/tanzu-auth-controller-manager/controllers/v3_cascade_controller_test.go:109

      Timed out after 1.001s.
      Expected success, but got an error:
          <*errors.errorString | 0xc0000573f0>: ***
              s: "Assertion in callback at /home/runner/work/tanzu-framework/tanzu-framework/addons/pinniped/tanzu-auth-controller-manager/controllers/v3_cascade_controller_test.go:114 failed:\nExpected\n    <map[string][]uint8 | len:1>: ***\n        \"values.yaml\": [105, 100, 101, 110, 116, 105, 116, 121, 95, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 95, 116, 121, 112, 101, 58, 32, 110, 111, 110, 101, 10, 105, 110, 102, 114, 97, 115, 116, 114, 117, 99, 116, 117, 114, 101, 95, 112, 114, 111, 118, 105, 100, 101, 114, 58, 32, 118, 115, 112, 104, 101, 114, 101, 10, 116, 107, 103, 95, 99, 108, 117, 115, 116, 101, 114, 95, 114, 111, 108, 101, 58, 32, 119, 111, 114, 107, 108, 111, 97, 100, 10],\n    ***\nto equal\n    <map[string][]uint8 | len:1>: ***\n        \"values.yaml\": [105, 100, 101, 110, 116, 105, 116, 121, 95, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 95, 116, 121, 112, 101, 58, 32, 110, 111, 110, 101, 10, 105, 110, 102, 114, 97, 115, 116, 114, 117, 99, 116, 117, 114, 101, 95, 112, 114, 111, 118, 105, 100, 101, 114, 58, 32, 118, 115, 112, 104, 101, 114, 101, 10, 116, 107, 103, 95, 99, 108, 117, 115, 116, 101, 114, 95, 114, 111, 108, 101, 58, 32, 119, 111, 114, 107, 108, 111, 97, 100, 10, 115, 119, 101, 101, 116, 101, 115, 116, 95, 99, 97, 116, 58, 32, 108, 105, 111, 110, 101, 108],\n    ***",
          ***
          Assertion in callback at /home/runner/work/tanzu-framework/tanzu-framework/addons/pinniped/tanzu-auth-controller-manager/controllers/v3_cascade_controller_test.go:114 failed:
          Expected
              <map[string][]uint8 | len:1>: ***
                  "values.yaml": [105, 100, 101, 110, 116, 105, 116, 121, 95, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 95, 116, 121, 112, 101, 58, 32, 110, 111, 110, 101, 10, 105, 110, 102, 114, 97, 115, 116, 114, 117, 99, 116, 117, 114, 101, 95, 112, 114, 111, 118, 105, 100, 101, 114, 58, 32, 118, 115, 112, 104, 101, 114, 101, 10, 116, 107, 103, 95, 99, 108, 117, 115, 116, 101, 114, 95, 114, 111, 108, 101, 58, 32, 119, 111, 114, 107, 108, 111, 97, 100, 10],
              ***
          to equal
              <map[string][]uint8 | len:1>: ***
                  "values.yaml": [105, 100, 101, 110, 116, 105, 116, 121, 95, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 95, 116, 121, 112, 101, 58, 32, 110, 111, 110, 101, 10, 105, 110, 102, 114, 97, 115, 116, 114, 117, 99, 116, 117, 114, 101, 95, 112, 114, 111, 118, 105, 100, 101, 114, 58, 32, 118, 115, 112, 104, 101, 114, 101, 10, 116, 107, 103, 95, 99, 108, 117, 115, 116, 101, 114, 95, 114, 111, 108, 101, 58, 32, 119, 111, 114, 107, 108, 111, 97, 100, 10, 115, 119, 101, 101, 116, 101, 115, 116, 95, 99, 97, 116, 58, 32, 108, 105, 111, 110, 101, 108],
              ***

      /home/runner/work/tanzu-framework/tanzu-framework/addons/pinniped/tanzu-auth-controller-manager/controllers/v3_cascade_controller_test.go:115
------------------------------
1.6531201702942212e+09	DEBUG	v1alpha3 tanzu auth cascade controller	adding secret for reconciliation	***"secret namespace": "some-name-pinniped.tanzu.vmware.com-package", "secret name": "pinniped-namespace"***
1.6531201702942524e+09	INFO	v1alpha3 tanzu auth cascade controller.reconcile	starting reconciliation	***"request object": "pinniped-namespace/some-name-pinniped.tanzu.vmware.com-package"***
1.653120170294354e+09	DEBUG	v1alpha1 tanzu auth cascade controller	secret is not an addon Type or does not have the given label	***"secret namespace": "pinniped-namespace", "secret name": "some-name-pinniped.tanzu.vmware.com-package", "label": "pinniped"***
1.6531201702957263e+09	DEBUG	v1alpha3 tanzu auth cascade controller.reconcile	pinniped-info configmap not found, setting Data to nil	***"request object": "pinniped-namespace/some-name-pinniped.tanzu.vmware.com-package"***
1.6531201702990901e+09	DEBUG	v1alpha3 tanzu auth cascade controller.reconcile	creating or patching secret	***"request object": "pinniped-namespace/some-name-pinniped.tanzu.vmware.com-package", "secret namespace": "pinniped-namespace", "secret name": "some-name-pinniped.tanzu.vmware.com-package", "cluster namespace": "pinniped-namespace", "cluster name": "some-name"***
1.6531201703088045e+09	DEBUG	v1alpha3 tanzu auth cascade controller	adding secret for reconciliation	***"secret namespace": "some-name-pinniped.tanzu.vmware.com-package", "secret name": "pinniped-namespace"***
1.6531201703088193e+09	INFO	v1alpha3 tanzu auth cascade controller.reconcile	finished reconciling secret	***"request object": "pinniped-namespace/some-name-pinniped.tanzu.vmware.com-package", "secret namespace": "pinniped-namespace", "secret name": "some-name-pinniped.tanzu.vmware.com-package", "cluster namespace": "pinniped-namespace", "cluster name": "some-name", "result": "updated"***
1.653120170308866e+09	INFO	v1alpha3 tanzu auth cascade controller.reconcile	starting reconciliation	***"request object": "pinniped-namespace/some-name-pinniped.tanzu.vmware.com-package"***
1.6531201703088667e+09	DEBUG	v1alpha1 tanzu auth cascade controller	secret is not an addon Type or does not have the given label	***"secret namespace": "pinniped-namespace", "secret name": "some-name-pinniped.tanzu.vmware.com-package", "label": "pinniped"***
1.6531201703107054e+09	DEBUG	v1alpha3 tanzu auth cascade controller.reconcile	pinniped-info configmap not found, setting Data to nil	***"request object": "pinniped-namespace/some-name-pinniped.tanzu.vmware.com-package"***
1.6531201703136806e+09	DEBUG	v1alpha3 tanzu auth cascade controller	adding secret for reconciliation	***"secret namespace": "some-name-pinniped.tanzu.vmware.com-package", "secret name": "pinniped-namespace"***
1.6531201703136923e+09	DEBUG	v1alpha1 tanzu auth cascade controller	secret is not an addon Type or does not have the given label	***"secret namespace": "pinniped-namespace", "secret name": "some-name-pinniped.tanzu.vmware.com-package", "label": "pinniped"***
1.65312017031512e+09	DEBUG	v1alpha3 tanzu auth cascade controller.reconcile	creating or patching secret	***"request object": "pinniped-namespace/some-name-pinniped.tanzu.vmware.com-package", "secret namespace": "pinniped-namespace", "secret name": "some-name-pinniped.tanzu.vmware.com-package", "cluster namespace": "pinniped-namespace", "cluster name": "some-name"***
1.6531201703203166e+09	DEBUG	v1alpha3 tanzu auth cascade controller	adding secret for reconciliation	***"secret namespace": "some-name-pinniped.tanzu.vmware.com-package", "secret name": "pinniped-namespace"***
1.6531201703203568e+09	INFO	v1alpha3 tanzu auth cascade controller.reconcile	finished reconciling secret	***"request object": "pinniped-namespace/some-name-pinniped.tanzu.vmware.com-package", "secret namespace": "pinniped-namespace", "secret name": "some-name-pinniped.tanzu.vmware.com-package", "cluster namespace": "pinniped-namespace", "cluster name": "some-name", "result": "updated"***
1.6531201703203828e+09	INFO	v1alpha3 tanzu auth cascade controller.reconcile	starting reconciliation	***"request object": "pinniped-namespace/some-name-pinniped.tanzu.vmware.com-package"***
1.6531201703204596e+09	DEBUG	v1alpha1 tanzu auth cascade controller	secret is not an addon Type or does not have the given label	***"secret namespace": "pinniped-namespace", "secret name": "some-name-pinniped.tanzu.vmware.com-package", "label": "pinniped"***
1.653120170321902e+09	DEBUG	v1alpha3 tanzu auth cascade controller.reconcile	pinniped-info configmap not found, setting Data to nil	***"request object": "pinniped-namespace/some-name-pinniped.tanzu.vmware.com-package"***
1.653120170326538e+09	DEBUG	v1alpha3 tanzu auth cascade controller.reconcile	creating or patching secret	***"request object": "pinniped-namespace/some-name-pinniped.tanzu.vmware.com-package", "secret namespace": "pinniped-namespace", "secret name": "some-name-pinniped.tanzu.vmware.com-package", "cluster namespace": "pinniped-namespace", "cluster name": "some-name"***
1.6531201703292966e+09	INFO	v1alpha3 tanzu auth cascade controller.reconcile	finished reconciling secret	***"request object": "pinniped-namespace/some-name-pinniped.tanzu.vmware.com-package", "secret namespace": "pinniped-namespace", "secret name": "some-name-pinniped.tanzu.vmware.com-package", "cluster namespace": "pinniped-namespace", "cluster name": "some-name", "result": "unchanged"***
1.6531201713305516e+09	DEBUG	v1alpha3 tanzu auth cascade controller	secret is being deleted, skipping reconcile	***"secret namespace": "some-name-pinniped.tanzu.vmware.com-package", "secret name": "pinniped-namespace", "secret namespace": "pinniped-namespace", "secret name": "some-name-pinniped.tanzu.vmware.com-package"***
1.6531201713305771e+09	DEBUG	v1alpha1 tanzu auth cascade controller	secret is not an addon Type or does not have the given label	***"secret namespace": "pinniped-namespace", "secret name": "some-name-pinniped.tanzu.vmware.com-package", "label": "pinniped"***
•••••••••••••••••••••••••••••••


Summarizing 1 Failure:

[Fail] Controller ClusterBootstrap Secret when random values are added to the secret [It] they are preserved 
/home/runner/work/tanzu-framework/tanzu-framework/addons/pinniped/tanzu-auth-controller-manager/controllers/v3_cascade_controller_test.go:115

Ran 34 of 34 Specs in 13.166 seconds
FAIL! -- 33 Passed | 1 Failed | 0 Pending | 0 Skipped

You're using deprecated Ginkgo functionality:
=============================================
Ginkgo 2.0 is under active development and will introduce several new features, improvements, and a small handful of breaking changes.
A release candidate for 2.0 is now available and 2.0 should GA in Fall 2021.  Please give the RC a try and send us feedback!
  - To learn more, view the migration guide at https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md
  - For instructions on using the Release Candidate visit https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#using-the-beta
  - To comment, chime in at https://github.com/onsi/ginkgo/issues/711

  You are using a custom reporter.  Support for custom reporters will likely be removed in V2.  Most users were using them to generate junit or teamcity reports and this functionality will be merged into the core reporter.  In addition, Ginkgo 2.0 will support emitting a JSON-formatted report that users can then manipulate to generate custom reports.

  If this change will be impactful to you please leave a comment on https://github.com/onsi/ginkgo/issues/711
  Learn more at: https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#removed-custom-reporters

To silence deprecations that can be silenced set the following environment variable:
  ACK_GINKGO_DEPRECATIONS=1.16.5

--- FAIL: TestController (13.19s)
FAIL
coverage: 74.7% of statements
FAIL	github.com/vmware-tanzu/tanzu-framework/addons/pinniped/tanzu-auth-controller-manager/controllers	13.221s
FAIL
make: *** [Makefile:494: test] Error 1
Error: Process completed with exit code 2.

I didn't notice these tests were failing before merging PR 1906:

Affected product area (please put an X in all that apply)

• APIs
• Addons
• CLI
• Docs
• IAM
• Installation
• Plugin
• Security
• Test and Release
• User Experience

Expected behavior
All unit tests should pass and Ginkgo deprecation warning should not appear.

Steps to reproduce the bug

Version (include the SHA if the version is not obvious)

Environment where the bug was observed (cloud, OS, etc)

Relevant Debug Output (Logs, manifests, etc)

[ankeesler]

@codegold79 - thanks for reporting. I think Sarah has done some research on a failure similar to this. The failure that she looked at ended up being a flake, and she applied a longer timeout. I haven't looked deeply enough at this particular failure yet, but maybe it is the same thing. In the meantime - do you want to rerun the job to see if it passes when run a second time?

[codegold79]

@codegold79 - thanks for reporting. I think Sarah has done some research on a failure similar to this. The failure that she looked at ended up being a flake, and she applied a longer timeout. I haven't looked deeply enough at this particular failure yet, but maybe it is the same thing. In the meantime - do you want to rerun the job to see if it passes when run a second time?

Re-running failed jobs now: https://github.com/vmware-tanzu/tanzu-framework/actions/runs/2362236794
Let's see how it goes 🤞

UPDATE (May 23, 2022)

The second try failed.

Also, the same unit test failed when a new PR #2329 got merged today. See https://github.com/vmware-tanzu/tanzu-framework/runs/6560117849?check_suite_focus=true

Evidence seems to be pointing away from test flakiness.

[ankeesler]

mmm. ok. i'll take a look.