cmd/go: track tools/tooling updates to support modules

[flibustenet]

Many Go tools that previously worked with GOPATH need to be upgraded in order to work with modules.

Note: Many of these tools have been deprecated with modules, and their features have been incorporated into gopls. This is indicated below.

This is a tracking issue for the conversion of tools to work with modules.

• aligncheck - Merged - opennota/check#43
• bundle - (x/tools/cmd/bundle: use caller's module version, not $GOPATH #32031)
• counterfeiter - Please support modules ! maxbrunsfeld/counterfeiter#90
• deadcode
• delve - Working
• dupl
• eg
• errcheck - Beta - Support go modules kisielk/errcheck#148
• fillstruct - available through gopls, or cmd/fillstruct: broken without modules davidrjenni/reftools#31
• gas
• goa/goagen - support building code outside of GOPATH for go module support goadesign/goa#1914
• go-langserver - use gopls instead
• go-outline - also available through gopls
• go-symbols
• gocode - available through gopls or https://github.com/stamblerre/gocode
• goconst
• gocyclo - doesn't need an update, see cmd/go: track tools/tooling updates to support modules #24661 (comment)
• godef - available through gopls or github.com/rogpeppe/godef
• godoc - x/tools/cmd/godoc: add module support #33655
• godoctor - available through gopls
• gofmt - working and also available through gopls
• gogrep - working (Switch to go/packages for module support mvdan/gogrep#39)
• go-fuzz - (preliminary Go modules support dvyukov/go-fuzz#274)
• gogetdoc - working (typecheck with gopackages zmb3/gogetdoc#40)
• goimports - working and also available through gopls
• golint - Deprecated - x/lint: freeze and deprecate #38968
• gometalinter - Deprecated - Archive gometalinter alecthomas/gometalinter#590
• gomobile
• gomodifytags
• gomvpkg
• gopkgs - unnecessary with gopls
• goplay
• gorename - available through gopls
• goreturns - available through gopls
• gosimple
• go-sumtype - (Go modules support BurntSushi/go-sumtype#9)
• goswagger - Go module aware scan go-swagger/go-swagger#1991
• gotags
• gotests
• gotext
• gotype (@ramya-rao-a confirmed this was not required for vscode-go)
• gotype-live - available through gopls
• govet - also available through gopls
• gqlgen - (Add support for go modules 99designs/gqlgen#514)
• guru - replaced by gopls
• impl
• ineffassign
• keyify
• lll - Working
• maligned
• megacheck
• misspell
• mockery - (Support Go 1.11 modules vektra/mockery#218)
• motion
• nakedret
• safesql
• staticcheck - also available through gopls
• stringer
• structcheck - Merged - opennota/check#43
• unconvert - merged - unconvert: use x/tools/go/packages mdempsky/unconvert#32
• unparam - mvdan/unparam@0cc8df2
• unused
• varcheck - Merged - opennota/check#43

[rsc]

Yes, this is an issue we're very aware of. Will leave this open to track it.

[myitcv]

@rsc - just to check this issue effectively encapsulates what we discussed in golang-dev?

[myitcv]

Changed the title so that this issue matches on use of the words "tool", "tools" or "tooling"

Will keep a list of tools here too (please edit):

• Requiring build output:
  • guru
  • gocode
  • stringer
• Requiring source code
  • goimports
  • godef
  • godoc.org (support Go modules gddo#567)

[gopherbot]

Change https://golang.org/cl/105855 mentions this issue: x/vgo: add deplist command to get build information about (test) deps

[rsc]

See https://go-review.googlesource.com/c/tools/+/116359 for a sketch of a start.
/cc @alandonovan

[gopherbot]

Change https://golang.org/cl/125296 mentions this issue: go/build: invoke go command to find modules during Import, Context.Import

[myitcv]

@alandonovan @ianthehat - is it worth creating a number of issues, specific to each tool that's being worked on?

I keep directing people to this umbrella issue... but maybe there's merit in getting a bit more granularity on things? Equally, maybe not.

So I'm definitely wouldn't push back if you said "let's keep things here" 👍

[sam3d]

How about the case of goimports importing packages that are both:

• Within the module's local directory
• Outside of the system scoped $GOPATH

---

Assuming the following directory structure:

./
├── go.mod
├── main.go
└── test
    └── test.go

With ./go.mod containing:

module example.com

goimports will happily leave alone and even format ./main.go:

package main

import (
	"log"

	"example.com/test"
)

func main() {
	log.Print("Hello, world!")
	test.Start()
}

With ./test/test.go:

package test

import "fmt"

func Start() {
	fmt.Println("Started properly")
}

---

However, if the "example.com/test" import is removed from main.go, it wont get added or found again. If its import path is changed in any way to render it incorrect (say to "example.com/tes"), goimports simply deletes the line.

It's odd - goimports can clearly tell that package test exists when it's been provided already, but it can't actively seek and add it if it hasn't been.

Is this something that requires looking into the module cache for, or is this a different issue?

[ianthehat]

goimports will assume that the import line "example.com/test" imports a package called test if it cannot find any sources for that package, and it knows the test symbol is used, so it won't delete the line. It does not mean that it knew the package existed...

goimports scans your GOPATH and GOROOT, anything not there it will not find right now.

It will be made to use x/tools/go/packages plus some extra stuff that is beyond the scope of that package. When it does, this case will work (does not involve the module cache, just the list of packages in the current module, which you can already get with go mod -packages

[sam3d]

Thank you for explaining @ianthehat! I've misunderstood how goimports handles symbols.

[metakeule]

I have two questions:

1. will x/tools/go/packages move to the standard library with Go 1.12?
2. I can't see any information about the version of a plugin, so how is a tool that assists in importing (like GoSublime) supposed to get a list of possible imports (with versions)?

[metakeule]

Also: How to prevent that import completion will get really slow with larger projects? (see mdempsky/gocode#32)

[myitcv]

@metakeule

will x/tools/go/packages move to the standard library with Go 1.12?

The plan is for it to move to the standard library at some point, yes, once it has stabilised (ref).

I can't see any information about the version of a plugin

This is similar to the goimports "problem"; see #26717

How to prevent that import completion will get really slow with larger projects?

Many of the problems describe in mdempsky/gocode#32 relate to speed problems brought about by the use of the -source flag, which drives the use of the source importer. go/packages will work using the underlying build cache and so should not suffer these problems.

But in case lag does become an issue for gocode with its current architecture, go/packages will be offering support for long-running programs. So it's conceivable that gocode's cache will switch to be module-based, where it effectively "watches" for relevant changes and refreshes its cache accordingly. Its cache will then be primed for fast responses to ad-hoc queries.

cc @dominikh (in case there's any overlap of interest here with your work)

[thepudds]

dvyukov/go-fuzz was listed above as not supporting modules, but updated list to indicate it does now support modules (dvyukov/go-fuzz#274).

The current approach for go-fuzz works reasonably well for fuzzing modules, but it is not the "forever" solution for go-fuzz. In particular, the current approach will stop working if or when GOPATH mode goes away completely (because a portion of the implementation still relies on being able to disable module mode temporarily, though that is more-or-less transparent to the user).

[gopherbot]

Change https://golang.org/cl/255053 mentions this issue: cmd/bundle: expand test coverage to all modes

[powerman]

Maybe it's time to update this list? I'm pretty sure most of mentioned linters works just fine today with modules (at least within golangci-lint). Probably other things has changed too since Feb.

[stamblerre]

Just went through and updated a bit. A lot of the tools in this list have been replaced by gopls as well.

[fzipp]

I am the author of gocyclo. It does not need an upgrade to support modules, it works on a file-by-file basis.

[mdempsky]

maligned (github.com/mdempsky/maligned) is updated to support modules. (It's now implemented as an x/tools/go/analysis.Analyzer.)

[matttproud]

Not sure if this is useful for anyone, but I briefly looked at porting eg, gomvpkg, gorename from package loader to package packages as a spare time activity to take advantage of user-providable GOPACKAGESDRIVER drivers for custom workspaces (due to gopls having (temporarily) disabled support for overrides).

Basic findings (naive), which I am writing for myself should I ever wonder this again:

Porting the tools does not look terribly difficult, but it appears that loader's support for file-orientation (*loader.Conf).CreateFromFiles as opposed to package-orientation makes this a little bit trickier the old behavior. I presume one would have better luck forking and pivoting the tool interface behavior versus updating them in situ.

[josharian]

I ported eg already. But I see to my horror that I never hit submit. I’ll plan to do that when I’m next at a computer. https://go-review.googlesource.com/c/tools/+/258304 might be useful.

[mvdan]

At the risk of being too off-topic, if you found eg useful, you might want to take a look at https://github.com/rsc/rf.

[thepudds]

@matttproud

I briefly looked at porting eg, gomvpkg, gorename

And just to briefly expand on @mvdan's comment, https://github.com/rsc/rf is also useful if you wanted a replacement for gomvpkg and gorename.

Here is a sample use, where you can see it doing eg-like example-based refactoring, as well as moving packages and renaming identifiers:

527a189

rf is effectively a superset of multiple tools on the main list at the start of this issue. (It is still WIP as far as I understand, but still usable).

[matttproud]

I appreciate the plug for rf. I am mainly looking for something that 1. uses package packages under the hood (it does) and 2. at a maturity level higher than experimental (not there yet) since I will want to point a large user base of mixed skills at. :-). I’ll follow what’s happening there closely; I promise.

[mdempsky]

Apropos to this conversation, rsc/rf originally used x/tools/go/packages, but for performance switched to a custom loader (which does not support GOPACKAGESDRIVER, at least currently): rsc/rf@8880f26

[mvdan]

I wonder if there's value in keeping this issue open indefinitely. It's been three years, and practically every maintained Go tool now supports modules. For those that haven't done it yet, it probably makes sense to open an issue on their trackers and continue there. And if the tool isn't maintained anymore, it's always an option to fork and continue development.

Perhaps we keep this issue open for a little longer (a year?), but I personally wouldn't go beyond that. The point of this issue was to coordinate work, keep track of progress, and potentially identify issues with supporting modules. I think we've succeeded at that :)

[mvdan]

It seems like we all agree this issue has served it's purpose, and should be closed soon.

If you're the author or maintainer of a tool that still needs to add support for Go modules, please file an issue in your issue tracker about it, and link to it in a comment here for others to follow. Discussion and progress should continue there.

I'll leave this issue open for another four weeks, to leave time for those comments, and any remaining input. Thanks!

[ainar-g]

Apologies if this is not the right place to ask, but as far as I know, there still isn't a tool like gomvpkg that would allow one to move a package within a module and also update all references to it within the module?

[mvdan]

@ainar-g I think rf would satisfy your need, as per this test, though note that it's not ready for general use and lacks documentation.

[ainar-g]

@mvdan, thanks, but I've already tried rf and have seen that example. The example seems to be about moving an identifier between packages and not packages themselves. There is also an example of moving files between packages, but renaming a package file-by-file is rather… suboptimal. I've tried several other scripts, and all of them fail with various errors.

I guess it's mv + ed + vim time again.

[mvdan]

I'll leave this issue open for another four weeks, to leave time for those comments, and any remaining input. Thanks!

Four weeks elapsed, so closing. Thanks all for the work that happened here!