x/tools: clean up dependencies

[matloob]

The following directories in x/tools have dependencies outside tools:

• cmd/tip (a whole bunch of stuff)
• cmd/godoc (appengine, cloud) also redigo through internal/memcache
• cmd/html2article (x/net/html)
• cmd/present (transitively through playground)
• playground/... (appengine, websocket)
• blog (transitively through present)
• godoc (golang.org/x/net/ctxhttp) also redigo through internal/memcache) (edit 2019-02-20: internal/memcache dependency is gone)
• internal/memcache (github.com/gomodule/redigo)

We should be able to remove the dependencies fairly easily, especially now that the website repo exists. Tools is a relatively low level repo and shouldn't have many external dependencies, especially on appengine and cloud.

[dmitshur]

I will investigate this and update the original issue with a plan for how this can be resolved.

Edit: The following is the latest version of my investigation.

golang.org website group:

• cmd/tip (a whole bunch of stuff) - has moved to x/build
• cmd/godoc (appengine, cloud) - the golang.org aspect (which is what imports appengine, cloud) is moving to x/website as part of issue doc, x/tools/cmd/godoc, x/website: move golang.org content into its own repository #29206

cmd/present (used for serving talks.golang.org) group:

• cmd/present (appengine, x/net/websocket transitively through playground and playground/socket) - in appengine mode, it’s used to serve talks.golang.org
• playground (appengine) - used by x/blog, x/tour and x/talks (also cmd/godoc, but that’s moving to x/website); can consider moving it to golang.org/x/playground/handler
• playground/socket (x/net/websocket) - used by x/tour (in local mode) and x/tools/cmd/present (both in local mode, and in appengine mode)

Others:

• cmd/html2article (x/net/html) - still need to decide what to do with it

• blog (only stdlib imports) - no longer an issue, it only imports stdlib

• godoc/redirect (x/net/ctxhttp) - can trivially stop using x/net/ctxhttp by manually implementing (a subset of) what ctxhttp.Get does; see below:

godoc/redirect diff

diff --git a/godoc/redirect/redirect.go b/godoc/redirect/redirect.go
index b4599f6a..cf4972f7 100644
--- a/godoc/redirect/redirect.go
+++ b/godoc/redirect/redirect.go
@@ -18,8 +18,6 @@ import (
 	"strings"
 	"sync"
 	"time"
-
-	"golang.org/x/net/context/ctxhttp"
 )
 
 // Register registers HTTP handlers that redirect old godoc paths to their new
@@ -250,7 +248,11 @@ func isGerritCL(ctx context.Context, id int) (bool, error) {
 	// https://gerrit-review.googlesource.com/Documentation/rest-api-changes.html#get-change.
 	ctx, cancel := context.WithTimeout(ctx, 5*time.Second)
 	defer cancel()
-	resp, err := ctxhttp.Get(ctx, nil, fmt.Sprintf("https://go-review.googlesource.com/changes/%d", id))
+	req, err := http.NewRequest(http.MethodGet, fmt.Sprintf("https://go-review.googlesource.com/changes/%d", id), nil)
+	if err != nil {
+		return false, err
+	}
+	resp, err := http.DefaultClient.Do(req.WithContext(ctx))
 	if err != nil {
 		return false, err
 	}

[bradfitz]

This bug is making the assumption that repo-level dependencies are a problem.

Implicit in this bug report is that you assume that repo == module, and that having multiple modules in a repo is bad / not an option.

Before we clean things up, can we at least agree on the problem?

[dmitshur]

The following is my understanding of the situation.

I understand that repositories that contain more than one module (multi-module repositories) incur a certain complexity cost. Repositories containing a single module are a simpler solution, one that is preferable to a more complex one when both options are viable.

There are some situations where it is necessary and justifiable to use multi-module repositories.

Whether to make a repository multi-module or not is a decision that needs to be carefully thought through before committing to it, especially if picking the more complex solution. The importance of making an appropriate decision is amplified for golang.org/x subrepos, since many people look at the subrepos for inspiration and best practices to follow.

Historically, x/tools had been used as a default repository for placing many tools related to the Go project, and ended up housing packages roughly fitting into the following 3 distinct categories:

1. libraries for working with Go code (go/types, astutil, imports, etc.),
2. commands that work with Go code (cmd/goimports, cmd/gorename, etc.),
3. tools that support the Go project in production (cmd/godoc in appengine mode that powers golang.org, godoc/short in appengine mode that imports cloud.google.com/go/datastore to implement the golang.org/s URL shortener, cmd/tip which powers tip.golang.org, x/tools/playground package in appengine mode that imports google.golang.org/appengine, etc.)

By now, many of the production use cases have been or are in the process of being moved out into separate repositories (x/website, x/blog, x/playground, x/build, etc.).

This issue outlines some of the remaining packages that support our production uses, and in turn pull in many external dependencies like appengine and cloud.

As I understand, the goal of this issue is to investigate how viable it is to move the relatively few remaining pieces such that the x/tools repository can end up containing packages from categories 1 and 2 only. Then we can evaluate whether it makes sense to move those packages to the subrepositories where they fit better (e.g., x/tools/blog -> x/blog subrepo), and in turn make it possible for x/tools repository to consider the option of becoming a single Go library+command module.

It may still turn out after doing this investigation that it is justifiable for x/tools to become a multi-module repository, but I think it'd be shame if that decision is forced on it simply because there are currently some remaining packages that happen to import external dependencies for production uses.

/cc @hyangah @matloob

[matloob]

This bug is making the assumption that repo-level dependencies are a problem.

I think module-level dependencies are a problem, not necessarily repo-level dependencies.

Maybe we already agree on this? But here's my thinking:

I don't think that module-level dependencies are always a problem, but I think it's important to be careful about which dependencies modules take on. Tools "proper" (categories 1 and 2 in @dmitshur's post) is intended to be used by many different tools created by the community. When a project or organization is taking on a dependency on tools, they need to download and perhaps vet/audit or and add to their organization module proxy all the modules. I don't think organizations should need to pull in cloud or appengine or even x/sys to use the code in tools.

Implicit in this bug report is that you assume that repo == module, and that having multiple modules in a repo is bad / not an option.

Multi-module repositories are more complex than single-module repositories. There are some cases where multi-module repositories' complexity isn't a big issue: for instance a single repo that contains several independent top level directories which are modules isn't too hard to work with.

On the other hand if tools itself was a module, and the repo contained another module, say, at cmd/tip, there's a lot more complexity. Users need to remember that dependencies from the cmd/tip module to the tools module would use the code in the module cache, not that in the tools repo checked out and immediately surrounding the code. And if dependency on the "outer" module is at a version before the "inner" go.mod was added, there could be two paths that satisfy the same import, and users need to reason which one will be used. Moving code between modules or changing module boundaries have even more complexities.

We might be able to work around these issues, but to me it seems like it's worth doing the (relatively minimal) cleanup work to avoid those issues.

Before we clean things up, can we at least agree on the problem?

I'm probably doing a poor job of explaining this, so maybe we should meet offline to discuss, and share our conclusions here after?

[bradfitz]

Multi-module repositories are more complex than single-module repositories. There are some cases where multi-module repositories' complexity isn't a big issue: for instance a single repo that contains several independent top level directories which are modules isn't too hard to work with.
...
On the other hand if tools itself was a module, and the repo contained another module, say, at cmd/tip, there's a lot more complexity.
...
And if dependency on the "outer" module is at a version before the "inner" go.mod was added, there could be two paths that satisfy the same import, and users need to reason which one will be used. Moving code between modules or changing module boundaries have even more complexities.

Thanks. That's a reasonable argument for how this would bite people.

Does that argument still apply to putting go.mod files in leaf package main directories, like in cmd/html2article that don't have child directories with non-main packages?

And what's the policy going forward for which x/foo => x/bar deps are allowed?

In std we have go/build/deps_test.go. Are we looking to have a similar policy?

[matloob]

Does that argument still apply to putting go.mod files in leaf package main directories, like in cmd/html2article that don't have child directories with non-main packages?

It applies if those leaf package mains depend on other packages in the same repo: for instance, golang.org/x/tools/cmd/gorename is a leaf package that has dependences into tools (primarily on golang.org/x/tools/refactor/rename, which contains most of its logic).

On the other hand, cmd/html2article has no dependencies into any other packages in tools: its only dependencies are on golang.org/x/net/html and golang.org/x/net/html/atom. So I think it might be safe to isolate in its own module, especially since it's stable and unlikely to take on imports on tools in the future.

[matloob]

I wonder if @ianthehat has any opinions on multi-module repos in the case of a leaf package which has no dependencies on any modules in the rest of the repo.

[ianthehat]

Every single case of multi-module repositories I have seen so far has led to confusion of some form for some people, at this point I want to see a really strong reason to use them rather than searching for reasons not to.

[bradfitz]

So in the end state that is being targeted here, what are the rules of when you're allow to depend on things?

It'd be slightly sad if tools in x/tools could only depend on packages in x/tools.

If all the tools move to their own modules, then x/tools is no longer tools but rather x/ast-util-packages-for-other-tools-to-depend-on-from-elsewhere.

I'm fine breaking things up and creating new repos(==modules), but I want to understand where we're going first.

[ianthehat]

The trouble is there are three things conflated in this repository.

1: Packages that are essentially one step above the stdlib, things like /go/packages /go/analysis /go/ssa /imports etc which you should be able to depend on without pulling in the world.
2: Actual tools, main packages, which should be free to pull in just about anything you like.
3: Random stuff that ended up here because it was the easiest place to drop it

Everything in group 3 should just go somewhere else, the difficulties are in deciding what to do about the effect that group 2 has on the users of group 1.
I would argue that the contents of group 2 all needs to be versioned separately from group 1 anyway, and thus should be in a different module. That means either a multi-module repository, or moving them out to a separate repositor(y|ies).

I actually think we should create a single new repository for the important go code handling libraries and move them out of x/tools so x/tools can go back to being the binaries.