gomod

[dcormier]

Fixes #366 (I think).

Once merged, additional tags need to be added to the repo:

• redis/v3.0.0
• redisx/v3.0.0

This will make this repo support go modules, and follows the official guidance of bumping the major version when there are released tags that are not actually for go mod.

It maybe worth considering not having the redis and redisx modules in the same repo. Before taking this change might be a good opportunity to do that since consumers will have to change the import path, anyway.

At the very least, it may be worthwhile to move the redisx package out of this repo, since it's intended to be experimental and should probably remain v0.

[stevenh]

I may be missing something but why not just add the v3 tag?

Update I was missing something, just read the fine print here:
https://github.com/golang/go/wiki/Modules#releasing-modules-v2-or-higher

[dcormier]

The CI failures here are from go1.7 and 1.8. Versions prior to go1.9.7 do not support minimal module compatibility.

Additionally, at this point, these versions are well outside the window of support from the golang team anyway. Only the 2 most recent major versions are supported.

Is it a requirement to continue to support these older versions?

[Songmu]

I'm glad to support Go Modules.

Wouldn't it be okay to place only one go.mod at the top level of the repository like the following. I think that is easier to understand.

654073a

In this way, we just only add v3.0.0 tag.

Certainly, redisx is an experimental package, so it's hard to follow the semantic version and keep compatibility. But, isn't this package exceptionally acceptable for incompatible fixes?

[dcormier]

Wouldn't it be okay to place only one go.mod at the top level of the repository like the following. I think that is easier to understand.

654073a

In this way, we just only add v3.0.0 tag.

It looks like that would work just fine. Please go that route if it's your preference.

But, isn't this package exceptionally acceptable for incompatible fixes?

I'm not certain what you mean, here, but I think you're asking if it's OK to not semantically version redisx and continue to have incompatible fixes there. If it's in this current repo, I don't see a way to avoid it being semantically versioned because this repo already has version tags on it. If it's moved out to another repo (say, github.com/gomodule/redisx), you can make it a go module there without ever tagging it with a version. That way it would be implied to be v0, and incompatible fixes would be permitted there. Russ Cox's guidance on that:

v0 is omitted from import paths because - according to semver - there are no compatibility guarantees at all for those versions. Requiring an explicit v0 element would do little to ensure compatibility; you'd have to say v0.1.2 to be completely precise, updating all import paths on every update of the library. That seems like overkill. Instead we hope that developers will simply look at the list of modules they depend on and be appropriately wary of any v0.x.y versions they find.

[stevenh]

Ok I think we've let this one sit long enough. I've just put a PR to remove 1.7.x and 1.8.x validation which had their last release at the beginning of 2017.

If anyone has any objections to this please say now, otherwise I intend to merge: #465 and then this PR which I'm hoping @dcormier can rebase.

Please use 👍 or 👎 to express your preference!

[dcormier]

Rebased.

[dcormier]

Is there anything else preventing this from moving forward?

[stevenh]

Just giving people time to respond, its only been a few days; end of the week or beginning of next.

[stevenh]

So I've been testing this morning how this impacts existing projects and I must be missing something as without changing consumer import to includes the /v3 suffix it simply doesn't work.

go get -u
go: github.com/stevenh/gomodtest2 upgrade => v3.0.0+incompatible
go: downloading github.com/stevenh/gomodtest2 v3.0.0+incompatible
go: finding module for package github.com/stevenh/gomodtest2/mypkg
go: finding module for package github.com/stevenh/gomodtest2/mypkg
main.go:4:2: module github.com/stevenh/gomodtest2@latest found (v3.0.0+incompatible), but does not contain package github.com/stevenh/gomodtest2/mypkg

This is a terrible user experience as it will literally break everyones projects, did I miss something?

[dcormier]

No, you didn't miss something. Unfortunately, that is how gomod works. The docs linked in my comment at the beginning of this PR lay it out. Unfortunately, they are lengthy. But since this repo has version tags, the repo has to play by gomod's rules if consumers are to get the version of this package that one would expect.

As a reminder, the state this repo is currently in is that if you're using a supported version of go (currently go1.13+) and go a go get -u github.com/gomodule/redigo/redis to try to add this package to your project, you'll be given v2.0.0, from March 15, 2018. Rather than the most recent tag, v1.7.0, from January 19, 2019.

You have to jump through additional hoops to get the most recently tagged version of this package, anyway.

[LasTshaMAN]

As a reminder, the state this repo is currently in is that if you're using a supported version of go (currently go1.13+) and go a go get -u github.com/gomodule/redigo/redis to try to add this package to your project, you'll be given v2.0.0, from March 15, 2018. Rather than the most recent tag, v1.7.0, from January 19, 2019.

I can corroborate that, tried to go get the latest version and ended up with v2.0.0 :(

[stevenh]

I was afraid of that, I think the guide could do with clarifying when it comes to the user side as it gives the impression it will just work which is not the case.

I kind of expected it given the internal URL's had to change but it's definitely not 100% clear.

So given this I think we need to get an update to this to detail the changes people need front an center in the README, seems like the only tool out there that that does this currently is https://github.com/marwan-at-work/mod, there's thread here about making a core tool.

Is this something you have some time do to @dcormier?

[dcormier]

I've updated the readme to give the correct import path and docs link for once the redis/v3.0.0 tag is added to the repo and the new version is added to pkg.go.dev.

If you'd like some additional text, feel free to provide some. You can either reply with it (and where you want it) and I'll update the readme in this PR, open another PR against my branch and I'll merge it, or just update it after this PR is merged.

For what it's worth, as a golang module user, I don't expect every module author to be providing instructions on how to make the tools consume the latest major version upgrade. I just expect repos to be tagged properly, and, if it's present, to show the correct import path in their readme or package-level godoc. It's on me, a golang module user, to know that I have to update my import paths to take major version upgrades.

Relevant docs here:

If the module is version v2 or higher, the major version of the module must be included as a /vN at the end of the module paths used in go.mod files (e.g., module github.com/my/mod/v2, require github.com/my/mod/v2 v2.0.1) and in the package import path (e.g., import "github.com/my/mod/v2/mypkg"). This includes the paths used in go get commands (e.g., go get github.com/my/mod/v2@v2.0.1. Note there is both a /v2 and a @v2.0.1 in that example. One way to think about it is that the module name now includes the /v2, so include /v2 whenever you are using the module name).

And, in the same section:

In general, packages with different import paths are different packages. [...] This is also true if different import paths are due to different major versions appearing in the import path. Thus example.com/my/mod/mypkg is a different package than example.com/my/mod/v2/mypkg, and both may be imported in a single build, which among other benefits helps with diamond dependency problems and also allows a v1 module to be implemented in terms of its v2 replacement or vice versa.

Available versions for a module are discoverable in a repo itself, and on pkg.go.dev (example, another).

[stevenh]

Thanks for doing this I've added a suggestion to help our users upgrade to this new way of working. I agree for new users the go get ... is all that's needed but given this will come as a shock for existing consumers detailing out how they can easily fix this I think is important.

[stevenh]

Thanks @dcormier I've asked some colleagues to review, make sure I've not missed something.

Once that's done I'll look to get it merged.

[darkliquid]

LGTM

[stevenh]

Thanks to bmills in the other thread, looks like go 1.14 has an enhancement that's relevant here:
https://golang.org/doc/go1.14#incompatible-versions

With 1.14 and later we could enable go mod support without moving to v3 thoughts?

[dcormier]

If it works, that's the easy way, for sure.

[stevenh]

Feels like we should go that route, its less compatible needs people to be on 1.14 but with that it totally eliminates the need for switching to v3 which feels like a pain for a huge amount users.

Thoughts?

[stevenh]

More information:

go get github.com/gomodule/redigo/redis@v1.8.0
go: found github.com/gomodule/redigo/redis in github.com/gomodule/redigo v1.8.0

go get github.com/gomodule/redigo/redis@latest
go: downloading github.com/gomodule/redigo/redis v0.0.0-20200429221454-e14091dffc1b
go: github.com/gomodule/redigo/redis latest => v0.0.0-20200429221454-e14091dffc1b

So why does the explicit version work but latest doesn't?

Note that the hash latest is picking up (e14091d) is older than v1.8.0 (898fbd0)

I have the horrible feeling that now the proxy has seen a go.mod at the lower level there's no going back :(

[urandom2]

So why does the explicit version work but latest doesn't?

It works because you are asking for a package at a given vcs tag: at v1.8.0 there exists the requested package. You are circumventing the normal resolution flow.

I have the horrible feeling that now the proxy has seen a go.mod at the lower level there's no going back :(

Yeah, I was wanting confirmation from @bcmills to see if this was a known issue. But that is the way it looks, once you go submodule you cannot go back?

That being said, this exact behaviour does seem like a bug, since an untagged, pre-release v0.0.0 is preferred over the valid v1.8.0. It just leads to a host of edge cases where a branch/commit/pr/pre-release tag exists and it takes precedent over the valid release.

[stevenh]

Indeed, any small slip and its impossible to recover, which is far from ideal.

[bcmills]

The fix for the accidental nested modules is to remove the packages from the “latest” version of them, so that the go command will notice they they don't actually provide the imports it's looking for and keep trying other paths.

One easy way to do that is with a (presumably off-master) commit that provides trivial go.mod files in those subdirectories but deletes all of the directories and files. (The proxy will be able to fetch that commit as long as it is reachable from any branch or tag —not just a SemVer tag — and you can fetch it using go get -d with the commit hash as the version.)

[bcmills]

You might also be able to achieve the same effect by changing the module line in the go.mod files so that it no longer matches the module path being fetched. (Such a module can only be used in a replace directive for the declared path.)

[stevenh]

Have tried this approach but didn't seem to work.

To clarify I created a branch with just a redis/go.mod redisx/go.mod no other files this is here:
github.com/gomodule/redigo/commit/393804bd3349bc80396be3ef10afe3ce339e646f

I then used go get -d as below

user@localhost:~/code/test2$ go get -d github.com/gomodule/redigo/redis@393804bd3349bc80396be3ef10afe3ce339e646f
go: downloading github.com/gomodule/redigo/redis v0.0.0-20200430125810-393804bd3349
go: downloading github.com/gomodule/redigo v1.8.1-0.20200430125810-393804bd3349
go: github.com/gomodule/redigo/redis 393804bd3349bc80396be3ef10afe3ce339e646f => v0.0.0-20200430125810-393804bd3349
user@localhost:~/code/test2$ go get -d github.com/gomodule/redigo/redisx@393804bd3349bc80396be3ef10afe3ce339e646f
go: downloading github.com/gomodule/redigo/redisx v0.0.0-20200430125810-393804bd3349
go: github.com/gomodule/redigo/redisx 393804bd3349bc80396be3ef10afe3ce339e646f => v0.0.0-20200430125810-393804bd3349

Then tried to get the mainline (v1.8.0)

user@localhost:~/code/test2$ go clean -modcache
user@localhost:~/code/test2$ go get github.com/gomodule/redigo@v1.8.0
go: downloading github.com/gomodule/redigo v1.8.0
user@localhost:~/code/test2$ go get github.com/gomodule/redigo/redis
go: downloading github.com/gomodule/redigo/redis v0.0.0-20200429221454-e14091dffc1b
go: github.com/gomodule/redigo/redis upgrade => v0.0.0-20200429221454-e14091dffc1b

This is still picking up the original nested go.mod version.

Did this agree with the process you had in mind @bcmills ?

[stevenh]

Also seeing the following if I use @latest

user@localhost:~/code/test2$ go get -d github.com/gomodule/redigo/redis@latest
go: downloading github.com/gomodule/redigo v1.8.0
go: downloading github.com/gomodule/redigo/redis v0.0.0-20200429221454-e14091dffc1b
go: github.com/gomodule/redigo/redis latest => v0.0.0-20200429221454-e14091dffc1b

user@localhost:~/code/test2$ go get -d github.com/gomodule/redigo/redisx@latest
go: downloading github.com/gomodule/redigo/redisx v0.0.0-20200429221454-e14091dffc1b
go: github.com/gomodule/redigo/redisx latest => v0.0.0-20200429221454-e14091dffc1b
go get: github.com/gomodule/redigo/redisx@v0.0.0-20200429221454-e14091dffc1b requires
        github.com/gomodule/redigo/redis@v1.7.1: reading github.com/gomodule/redigo/redis/redis/go.mod at revision redis/v1.7.1: unknown revision redis/v1.7.1

[bcmills]

@stevenh, here's what I see at the moment:

example.com$ go version
go version devel +c76befe0 Thu Apr 30 08:07:47 2020 +0000 linux/amd64

example.com$ go mod init example.com
go: creating new go.mod: module example.com

example.com$ go get github.com/gomodule/redigo
go: downloading github.com/gomodule/redigo v1.8.0
go: github.com/gomodule/redigo upgrade => v1.8.0

example.com$ go list -m all
example.com
github.com/davecgh/go-spew v1.1.0
github.com/gomodule/redigo v1.8.0
github.com/pmezard/go-difflib v1.0.0
github.com/stretchr/objx v0.1.0
github.com/stretchr/testify v1.5.1
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405
gopkg.in/yaml.v2 v2.2.2

example.com$ go get github.com/gomodule/redigo/redis
go: downloading github.com/gomodule/redigo/redis v0.0.0-20200429221454-e14091dffc1b
go: github.com/gomodule/redigo/redis upgrade => v0.0.0-20200429221454-e14091dffc1b
go get github.com/gomodule/redigo/redis: ambiguous import: found package github.com/gomodule/redigo/redis in multiple modules:
        github.com/gomodule/redigo v1.8.0 (/tmp/tmp.Pk8oDwqIzC/_gopath/pkg/mod/github.com/gomodule/redigo@v1.8.0/redis)
        github.com/gomodule/redigo/redis v0.0.0-20200429221454-e14091dffc1b (/tmp/tmp.Pk8oDwqIzC/_gopath/pkg/mod/github.com/gomodule/redigo/redis@v0.0.0-20200429221454-e14091dffc1b)

example.com$ rm go.mod

example.com$ go mod init example.com
go: creating new go.mod: module example.com

example.com$ go get github.com/gomodule/redigo/redis
go: github.com/gomodule/redigo/redis upgrade => v0.0.0-20200429221454-e14091dffc1b

example.com$ go get github.com/gomodule/redigo/redis@393804bd3349bc80396be3ef10afe3ce339e646f
go: github.com/gomodule/redigo/redis 393804bd3349bc80396be3ef10afe3ce339e646f => v0.0.0-20200430125810-393804bd3349

So the v1.8.0 seems to be successful, but the proxy doesn't seem to have registered v0.0.0-20200430125810-393804bd3349 as latest yet. However, I think that's just a matter of waiting for the proxy's cache TTL to expire:

example.com$ rm go.mod

example.com$ export GOPROXY=direct

example.com$ go mod init example.com
go: creating new go.mod: module example.com

example.com$ go get github.com/gomodule/redigo/redis
go: found github.com/gomodule/redigo/redis in github.com/gomodule/redigo v1.8.0

[bcmills]

I think there is a problem with the proxy.golang.org cache failing to update @latest when the module no longer exists at head. I'm going to escalate this to the proxy.golang.org maintainers.

[bcmills]

Aha! I think I've got it. You can tag commit 393804bd3349bc80396be3ef10afe3ce339e646f as redis/v0.0.0-do-not-use and redisx/v0.0.0-do-not-use and then fetch those tags through the proxy.

Then that tagged version will show up in https://proxy.golang.org/github.com/gomodule/redigo/redis/@v/list, and the go command will prefer it over https://proxy.golang.org/github.com/gomodule/redigo/redis/@latest.

[dcormier]

I apologize for this mess. I know it wasn't working right before (with gomod picking up the old v2.0.0 tag), but I shouldn't have put this forward. I should have just left it as an issue.

Feel free to completely ignore this and not respond to it at all:

<rant>
To be honest, and excuse my language, but gomod is a shitshow. It's just seems so overcomplicated. I mean, look how long this page is!

We're suddenly forced to adopt it in some cases to get packages to work right, but it has so many gotchas that it can be tricky to adopt for a package that existed prior to gomod. And if you don't know all the rules, it's easy to break them and end up in a bad situation with no obvious way out. And gomod suddenly changes the behavior of some preexisting mechanisms (like git tags).

Golang users shouldn't be reliant on members of the golang team to jump in and give direction. It's great that they do that, and I very much appreciate it, but that it's needed seems like a bad sign.

Not to mention that it provides no way to discover that a new major upgrade is available for a package.

I miss dep.
</rant>

[dcormier]

@bcmills, does v0.0.0-do-not-use have a special meaning that's documented somewhere?

[bcmills]

@dcormier, to be fair, the Wiki recommends against multi-module repos, and https://blog.golang.org/migrating-to-go-modules explicitly says “If you have existing version tags, you should increment the minor version.”

(And yeah, we know the reference documentation needs work. That's golang/go#33637, and @jayconrod in particular is actively working on it.)

[bcmills]

v0.0.0-do-not-use is just a pre-release version that no one is likely to mistake for a release intended for public consumption. (There is nothing special about it beyond the fact that it is a valid semantic version.)

[stevenh]

Aha! I think I've got it. You can tag commit 393804bd3349bc80396be3ef10afe3ce339e646f as redis/v0.0.0-do-not-use and redisx/v0.0.0-do-not-use and then fetch those tags through the proxy.

Then that tagged version will show up in https://proxy.golang.org/github.com/gomodule/redigo/redis/@v/list, and the go command will prefer it over https://proxy.golang.org/github.com/gomodule/redigo/redis/@latest.

To confirm my understanding I need to tag create the two tags redis/v0.0.0-do-not-use and redisx/v0.0.0-do-not-use temporarily while we perform go get to force the proxy to remove the old records and these tags and that commit can then be removed?

[bcmills]

Correct. The proxy won't remove the old records per se, but it will permanently cache the existence of those versions (and the contents at those versions), and the go command will try them instead of the (also still cached) bogus pseudo-version.

[stevenh]

Looks like we have a winner!

user@localhost:~/code/test2$ go clean -modcache
user@localhost:~/code/test2$ go get -d github.com/gomodule/redigo
go: downloading github.com/gomodule/redigo v1.8.0
go: github.com/gomodule/redigo upgrade => v1.8.0

Thank you so much for your help @bcmills it would have taken ages to fix that without your help!

Feels like this is a new go mod FAQ entry :)

[domino14]

Help what is happening.

$ go clean -modcache
$ go get -d github.com/gomodule/redigo
go: finding github.com/gomodule/redigo v2.0.0+incompatible
go: downloading github.com/gomodule/redigo v2.0.0+incompatible
go: extracting github.com/gomodule/redigo v2.0.0+incompatible

That is the wrong version right?

[bcmills]

@domino14, you need to be using Go 1.14 or higher for go get to ignore the +incompatible version.

[stevenh]

@domino14 your path is wrong you want:

go get github.com/gomodule/redigo/redigo