cmd/go: modfetch.Unzip checks that all filenames are acceptable, causing problems for test fixture directories

[dgodd]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.11beta2 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/dgodd/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/dgodd/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/golang1.11beta2"
GOTMPDIR=""
GOTOOLDIR="/usr/local/golang1.11beta2/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/53/t04mmfdn0qq68l978jgz64cc0000gn/T/go-build523317176=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

If you import a module which includes files which don't pass module.CheckFilePath then you can't use that module. A good example of this is https://github.com/bmatcuk/doublestar. Doublestar has a test directory with several very strangely named files to make sure that it works on such directories. However, that means I can not use go modules to import doublestar

module github.com/dgodd/gomodfilenames

require (
  github.com/bmatcuk/doublestar v1.0.9
)

What did you expect to see?

I expect to be able to import doublestar. I'm not entirely sure what should change. Possibly modules should be able to specify files/directories which will not be imported. If such a system existed then doublestar could state that the test directory should not be extracted by modfetch.Unzip, this seems to me to be good since as a user of the module I don't need it anyway.

I should mention that go get github.com/bmatcuk/doublestar does work.

What did you see instead?

go: extracting github.com/bmatcuk/doublestar v1.0.9
-> unzip /Users/dgodd/go/src/mod/cache/download/github.com/bmatcuk/doublestar/@v/v1.0.9.zip: malformed file path "test/a☺b": invalid char '☺'
go: import "github.com/dgodd/gomodfilenames" ->
	import "github.com/bmatcuk/doublestar": cannot find module providing package github.com/bmatcuk/doublestar

[bcmills]

Possibly modules should be able to specify files/directories which will not be imported.

You should be able to put a go.mod file in any directory you want to exclude from the parent module.
(That makes it logically a separate module, although I'm not 100% certain that that will exclude it from the zip file.)

[michilu]

I got the same problem in my case:

go: extracting github.com/antlr/antlr4 v0.0.0-20180728001836-7d0787e29ca8
-> unzip go/src/mod/cache/download/github.com/antlr/antlr4/@v/v0.0.0-20180728001836-7d0787e29ca8.zip: malformed file path "runtime/Cpp/demo/Windows/antlr4-cpp-demo/antlr4-cpp-demo-vs2015.vcxproj..filters": double dot

I think to should be acceptable the normal cases that like the double dot.

[jamie-digital]

A decent compromise would be for such packages to create their fixtures when the tests are run, if they don't already exist, rather than checking them into source control. The same already applies for fixtures with unusual mode bits, since ZIP archives don't necessarily preserve them.

[dgodd]

Thank you all. It turns out that @bcmills suggestion works wonderfully. I have created a PR to doublestar (bmatcuk/doublestar#16) and am closing this issue. Thank you all.

[vitarb]

Please note that creating mod files in those repos would be slow and painful process to follow for consumers as some packages might be not under active development.
Also it seems that we are putting restriction that any package that has special symbols in file names (even those that are not critical for core build) that are illegal on SOME platform couldn't be used on ANY platform.
Also although solution that was proposed above may help in many cases, creating mod files won't help if there are go files in there (either mixed or in sub-folders) and that module needs to be consumed at the end.