ginkgo v2.3.0 is failing all CI builds (and local ones as well)

[marten-seemann]

example run: https://github.com/lucas-clemente/quic-go/actions/runs/3229723185/jobs/5287486320

[onsi]

v2.3.0 added a new flag to the ginkgo library and the ginkgo cli. I believe what's happening here is you are installing the latest version of the cli but using a version of the ginkgo library that is different.

rather than go install github.com/onsi/ginkgo/v2/ginkgo@latest you should install the version tracked in your go.mod with go install github.com/onsi/ginkgo/v2/ginkgo (assuming you are in a directory with a go.mod)

[marten-seemann]

Seems to work: quic-go/quic-go#3591. Thank you!

[SaschaSchwarze0]

v2.3.0 added a new flag to the ginkgo library and the ginkgo cli. I believe what's happening here is you are installing the latest version of the cli but using a version of the ginkgo library that is different.

rather than go install github.com/onsi/ginkgo/v2/ginkgo@latest you should install the version tracked in your go.mod with go install github.com/onsi/ginkgo/v2/ginkgo (assuming you are in a directory with a go.mod)

@onsi is the limitation that the Ginkgo CLI can only run test suites with the same Ginkgo version in go.mod documented somewhere ? I don't find it here: https://onsi.github.io/ginkgo/#installing-ginkgo.

From a usability pov, we have scenarios were we have to run a set of test suites that come from different teams. We'd have to install different Ginkgo versions for many of them (and not just Ginkgo 1 and 2 what we do today).

[onsi]

You have to use the same version of the cli as the version of the ginkgo library in the package's go.mod - I can update the docs to make that clearer.

Go test parses flags in such a way that an extra flag causes the test to exit/fail and, unfortunately, that isn't something i can control.

More than that, however, the protocol between the cli and library can change from version to version (whether major/minor/patch) so you do need to use the same version.

[rmikhael]

Forgive me, but any change that is not backward compatible is not a well-thought out change.
We have 10s of repos that are using the same

` go install github.com/onsi/ginkgo/v2/ginkgo@latest

go install github.com/onsi/gomega/...`

Suddenly they are all failing to CI, and we have to fix all of them?

If you needed to make such a change, it should have been a /v3/ change.

[onsi]

I'm going to:

(a) look into adding some mechanism to the ginkgo cli to have it emit a clear version mismatch error if users try to run one version of ginkgo on a library that has a different version of ginkgo in it's go.mod.

(b) update the docs to make it super clear that you need to use the same version and provide guidance on how best to do that.

This was not a breaking change to Ginkgo in that existing test suites will continue to work without requiring any updates. Ginkgo does not have a formal compatibility contract between one version of the cli and a different version of the library. The two are packaged and shipped together as one repo and the requirement is that users use the same version of the cli as in their go.mod. Running go install github.com/onsi/ginkgo/v2/ginkgo@latest while convenient, breaks this requirement.

This particular change introduces a new feature (interrupts and timeouts) without breaking existing test suites and a lot of care was put into introducing this substantial set of new functionality in a way that integrates cleanly with existing code. I've made similar changes in the past that entailed adding new cli flags and will likely make more such changes in the future.

So - I appreciate this is inconvenient but I believe doing the work to ensure a version match between the cli and the library will only need to be done once. And I don't believe that a feature of this kind warrants a major release which would require folks to rewrite their code (at least their import statements) and would make the adoption of the current supported version of Ginkgo even harder to sustain.

[onsi]

I've just shipped Ginkgo v2.3.1. The CLI can now detect if it is being run against a package that has a version mismatch. If it detects the mismatch if prints out a (hopefully helpful) message and proceeds to attempt to run the specs.

With this change, anyone who is using @latest in their CI setup or toolchain will at least see a message that points them in the right direction.

[rmikhael]

Truly appreciate your dedication to this repo.
We will keep learning about golang tooling forever I guess.
I will adjust all our Makefile(s) to target a fixed version, instead of using @latest. So owner can update both together.

[onsi]

❤️ thanks @rmikhael and sorry for the fire drill this morning with all those repos! hopefully we've shortened the feedback loop with the new cli logging so future users can figure this out sooner!

[ripienaar]

For most go install github.com/onsi/ginkgo/v2/ginkgo will not just work I am afraid, because the cli has dependencies the library doesnt have and so the install fails even in the directory with the project go mod. You have to go get a few packages first, and you have to do so at the right version.

github.com/onsi/ginkgo/v2/ginkgo/internal@v2.4.0
github.com/onsi/ginkgo/v2/ginkgo/generators@v2.4.0

[blgm]

Hi @ripienaar. I'm aware of two things that work well and I have seen lots of folks use:

1. Is to add github.com/onsi/ginkgo/v2/ginkgo to a tools.go file and run Ginkgo via go run github.com/onsi/ginkgo/v2/ginkgo
2. Is to install the Ginkgo CLI with go install github.com/onsi/ginkgo/v2/ginkgo@latest

I would discourage folks from specifically adding github.com/onsi/ginkgo/v2/ginkgo/internal, github.com/onsi/ginkgo/v2/ginkgo/generators, etc... to their go.mod file because you then end up depending on implementation details of Ginkgo which could change in the future.

[ripienaar]

Yes, I agree its really bad to add those - but I think it's worth pointing out that the advise here is not sufficient and doesnt work for most.

Thanks for the tip on tools.go, thats nice but quite annoying for sure