cmd/go: suppress warning messages about symlinks

[cespare]

I'm converting a large source tree which used vendoring in GOPATH world into a Go module. An issue I've encountered is that go mod tidy and go mod why print some warnings about symlinks.

These look like:

$ go mod tidy
warning: ignoring symlink /path/to/some/symlink/dir
...

Where a warning line is printed for every symlinked directory in the repo.

These warnings make these 'go mod' commands unpleasant to use (you have to scroll through all the junk to see the relevant bits, if any, at the bottom).

None of the Go code is in symlinked directories. The symlinks are for other projects in other languages. For example, one use of symlinks is as a cheap way to share certain JS/CSS assets between multiple sub-projects.

Would it be possible to simply delete these warnings? Alternatively, could we remove the warning if the symlinked directory doesn't contain any Go code?

I tested this behavior in both Go 1.13.4 and tip (bf3ee57).

/cc @bcmills @jayconrod for cmd/go, and @rsc who originally wrote these warning messages as far as I can tell.

[jayconrod]

It looks like this warning was introduced in CL 46425, long before modules. It's emitted any time a pattern is expanded (all or anything with ...) and a symlink to a directory is encountered. go mod tidy is doing that internally.

I don't think these warnings should be deleted entirely, but making them less spammy would be good. Not sure how to do that though. Checking for .go files in a symlinked directory is probably not enough, since the directory itself may not have any .go files, but it may have subdirectories that do. I think Kubernetes' vendor/k8s.io directory works like that (with symbolic links to staging). Traversing subdirectories after a symbolic link could get us into a loop though.

[cespare]

@jayconrod yeah, I've run into this in the past when doing things like go list ./....

However, I think the spam is more of a problem with modules:

• go list is used a lot less in the day-to-day workflow in the GOPATH world than go mod commands are used in the modules world.
• With go list ./... or similar, it's more understandable that you're asking the Go tool to please go discover all the packages. With go mod tidy it's less clear.

What about suppressing the warnings for go mod subcommands only? (And keeping them for go list?)

[jayconrod]

Suppressing these warnings for go mod subcommands seems reasonable to me. @bcmills WDYT?

[bcmills]

If they're not appropriate for go mod, they're probably not appropriate for go list either. (And the converse: if they're useful for go list, then they're probably also useful for go mod.)

Given that folks can already explicitly prune out subdirectories using go.mod files, I'm curious about the directory structure that leads to this sort of situation. Are the non-Go symlinks in sibling directories to the Go source files, or in subdirectories of Go package directories?

As an alternative, would it make sense to do the go.mod pruning check before we do the symlink-pruning check?

[cespare]

If they're not appropriate for go mod, they're probably not appropriate for go list either. (And the converse: if they're useful for go list, then they're probably also useful for go mod.)

I explained above why I think it's reasonable to treat these differently, but if we have to treat them the same, then I'd err on the side of no spam.

Given that folks can already explicitly prune out subdirectories using go.mod files, I'm curious about the directory structure that leads to this sort of situation.

We have a large monorepo with projects in several languages. Before it was all a single GOPATH with vendored source. I'm converting it to be a single module instead.

Adding empty go.mod files to a bunch of project directories just to suppress this warning is probably a nonstarter.

Are the non-Go symlinks in sibling directories to the Go source files, or in subdirectories of Go package directories?

Currently, only sibling (or "cousin" or "cousin Nth removed") directories.

I didn't mind ignoring these warnings (or 2>/dev/null) when running go list in the GOPATH world. It just doesn't seem reasonable that I have to continue doing so with commands like go mod tidy, which is part of my normal workflow.

[agnivade]

Just to add a small point, we have a situation where a Go directory is linked with a symlink. (I know .. I know ..). So removing these warning would certainly help.

[bcmills]

Currently, only sibling (or "cousin" or "cousin Nth removed") directories.

So, could you move the go.mod file to the root of the Go source tree, instead of the repo root? (Or does it need access to testdata in sibling directories?)

[cespare]

So, could you move the go.mod file to the root of the Go source tree, instead of the repo root?

There is no single Go source tree. Everything's mixed. What I meant was only that no Go package directory or any descendent thereof contains symlinks.

├─go.mod
├─go.sum
├─go─project─1/
├─go─project─2/
├─non─go─project─1/
│ └─symlink
└─sub/
  ├─go─project─3/
  ├─non─go─project─2/
  │ └─symlink
  └─symlink

[sebd71]

As workaround you can create an empty go.mod file in each directory you want to ignore.

touch no-go-project-1/go.mod
touch sub/no-go-project-2/go.mod

But this is not a perfect workaround because you ignore sub/symlink with this method.

See go wiki modules https://github.com/golang/go/wiki/Modules#can-an-additional-gomod-exclude-unnecessary-content-do-modules-have-the-equivalent-of-a-gitignore-file

[cespare]

@sebd71 as I wrote above:

Adding empty go.mod files to a bunch of project directories just to suppress this warning is probably a nonstarter.

It's simply not reasonable for us to add lots of go.mod files to non-Go projects, especially in an ongoing way (adding and deleting them as we add and delete symlinks).

[sebd71]

I'm agree with you and that's why I said that it's not a perfect workaround.
I'm also facing the same problem.
A good solution would be to add in root go.mod "exclude mydir/*" statement, but unfortunately it seems not supported for now.

[cespare]

@bcmills @jayconrod would you accept a PR for this once the tree opens? We still haven't migrated our code to modules and this tooling wart is definitely a sticking point.

[jayconrod]

Discussed with @matloob, @bcmills offline. It seems like the best thing to do here is only print this warning when there's an explicit argument that has a ..., and the go command does not follow a symbolic link to a directory when expanding that pattern. The warning should not be printed for all, nor should it be printed for go mod tidy.

@cespare Does that work for you?

I'd be happy to review a CL for this. We can review any time, but it won't be merged until the tree opens for 1.16.

[cespare]

@cespare Does that work for you?

Absolutely. Thanks!

[gopherbot]

Change https://golang.org/cl/311890 mentions this issue: cmd/go: show warnings about symlinks only for patterns containing ...