cmd/go: [modules + integration] go mod discover, discover a set of unpacked modules within a directory

[nim-nim]

This report is part of a series, filled at the request of @mdempsky, focused at making Go modules integrator-friendly.

Please do not close or mark it as duplicate before making sure you’ve read and understood the general context. A lot of work went into identifying problems points precisely.

Needed feature

Go needs an official go mod discover command that processes a directory of unpacked Go code and identifies the valid Go modules within this directory.

Constrains

• the output should be a machine-readable list of:
  • module path
  • module version
  • filesystem path
• the command should warn if different filesystem paths declare the same module and version.
• the feature should also be exposed as a function in the official Go API, returning a go object (map, list, struct…)
• discovery should not consult a cache or the internet, just the provided directory
• the directory may include info files next to the corresponding mod files, or not

Motivation

• existing software baselines have been created in unpacked GOPATH/vendor mode,
• go allows creating go.mod files in other places than the root of the project
• go allows nested modules
• go allows module names that do not match the project url (and mandates them for versions >1)
• niceties like testdata or vendor mean some go.mod files are not considered as module descriptor but just data, so a basic find won’t work
• future language evolutions will probably add more twists to the concept of “valid module”

All of this make module discovery a lot of work that should be handled by common tooling.

[mdempsky]

It would help if you could trim down the redundant boilerplate text from each of your posts into a single sentence + link for more details (e.g., a Gist or meta-issue or something). The added verbosity makes it more difficult to quickly identify the unique details of each individual issue. Thanks.

[mdempsky]

Go needs an official go mod discover command that processes a directory of unpacked Go code and identifies the valid Go modules within this directory.

What does it mean for a Go module to be "valid" here?

E.g., you suggest go.mod files within testdata and/or vendor subdirectories are not valid. Why is that the case?

[nim-nim]

It would help if you could trim down the redundant boilerplate text from each of your posts into a single sentence + link for more details (e.g., a Gist or meta-issue or something). The added verbosity makes it more difficult to quickly identify the unique details of each individual issue. Thanks.

Done. Thanks for the comment. Do ping me if you find more things to improve.

[nim-nim]

@mdempsky

Go needs an official go mod discover command that processes a directory of unpacked Go code and identifies the valid Go modules within this directory.

What does it mean for a Go module to be "valid" here?

Short term, probably just that the module descriptor is parse-able by Go without errors. Long term, whatever sanity rules the language wants to impose on its modules.

E.g., you suggest go.mod files within testdata and/or vendor subdirectories are not valid. Why is that the case?

For vendor, that's because the current Go module design seems to ignore vendoring unless passed specific mode switches (I may have misunderstood this part, I didn't look at vendoring closely, our integration best practices call for vendor removal as the first integration step). And if you don't ignore vendoring you have to handle module collisions because the same module will be vendored multiple times in different vendor directories.

For testdata, that's because it is common and expected for projects to ship broken files inside testdata. The broken files can include broken module descriptor files, or module descriptors for a tree of broken packages go mod tidy can make no sense of. I had actually forgotten this part, before testing my toy implementation on real-world Go projects, and see it explode on broken and non-conformant module descriptors inside the testdata of some projects.

Either way ignoring vendor and testdata by default while walking the directory tree is not limitating. You can still point the command directly to the vendor or testdata directory, since it will not have arrived here via a walk the walk protections won't apply.

[mdempsky]

Short term, probably just that the module descriptor is parse-able by Go without errors.

Today you can check this by running go mod edit -json path/to/go.mod >/dev/null and seeing if it exits 0 or 1.

[nim-nim]

@mdempsky you forgot the find part, then the tesdata/vendor magic rules, add them to the mix and you'll find out it's not so simple

[mdempsky]

you forgot the find part,

There's nothing Go-specific about that, but okay:

find . -name go.mod -print0 | xargs -0 -n1 sh -c 'go mod edit -print "$1" >/dev/null && echo "$1"' --

then the tesdata/vendor magic rules,

You defined "valid" as "just that the module description is parse-able by Go without errors," and I identified a command to distinguish that. But it sounds now like you want something more nuanced than that. It would help if you would clearly and concisely state precisely what you're asking for.

Alternatively, identify some sample Go code repositories and the go.mod files within that should or should not be considered "valid".

[nim-nim]

@mdempsky I defined the discovery feature in the first message, you are trying to do bits of it separately, the result is still bits of it. The more nuanced part is just applying all the Go source code rules. There are already quite a lot of them on vendor and testdata. There will be probably more in the future. Will you maintain your shell snippet when the language adds more rules, when it is already taking more than a line?

Your snippet is also not outputting the modules names corresponding to the mod files. That's another thing which that can not be deduced trivially from the go.mod file path (and sometimes when it looks like it can be deduced from the file path, it's still wrong, so deducing is dangerous)

[seankhliao]

doesn't look like there's anything to do here.