go mod doesn't work for github.com/gomarkdown/markdown/html

[kjk]

What version of Go are you using (go version)?

$ go version
go version go1.11 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

OS X 10.13.6

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/kjk/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/kjk/src/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/2k/p2_4052s70vd5_cfdm0k0l740000gn/T/go-build479826685=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

• make sure you're OUTSIDE OF GOPATH
• git clone https://github.com/kjk/modtest.git
• cd modtest
• go mod init github.com/kjk/modtest
• go build

What did you expect to see?

This should build.

What did you see instead?

$ go build
go: finding github.com/gomarkdown/markdown/ast latest
go: finding github.com/gomarkdown/markdown/html latest
go: finding github.com/gomarkdown/markdown latest
main.go:8:2: unknown import path "github.com/gomarkdown/markdown/html": cannot find module providing package github.com/gomarkdown/markdown/html

For some reason go mod has issue with github.com/gomarkdown/markdown/html even though it works for github.com/gomarkdown/markdown and github.com/gomarkdown/markdown/ast.

The source of the problem seems to be:

$ go get -v github.com/gomarkdown/markdown/html
go: finding github.com/gomarkdown/markdown/html latest
go: finding github.com/gomarkdown/markdown latest
Fetching https://github.com?go-get=1
Parsing meta tags from https://github.com?go-get=1 (status code 200)
go get github.com/gomarkdown/markdown/html: no matching versions for query "latest"

There is no problem when running same go get when in GOPATH.

[josephlr]

@kjk I just looked into this, I was able to get your example working. I had to make a tiny modification as htmlHighlight is not defined, but after commenting out that line, I was able to run:

> git clone "https://github.com/kjk/modtest.git"
> cd modtest
> go mod init # To create a blank go.mod
> go mod tidy # To fill in the contents of go.mod
> go build
> ./modtest
Hello

[josephlr]

Full output:

➜ git clone https://github.com/kjk/modtest.git
Cloning into 'modtest'...
remote: Counting objects: 7, done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 7 (delta 0), reused 4 (delta 0), pack-reused 0
Unpacking objects: 100% (7/7), done.
➜ cd modtest
➜ go build
main.go:7:2: cannot find package "github.com/gomarkdown/markdown/ast" in any of:
        /snap/go/2635/src/github.com/gomarkdown/markdown/ast (from $GOROOT)
        /home/joe/dev/go/src/github.com/gomarkdown/markdown/ast (from $GOPATH)
main.go:8:2: cannot find package "github.com/gomarkdown/markdown/html" in any of:
        /snap/go/2635/src/github.com/gomarkdown/markdown/html (from $GOROOT)
        /home/joe/dev/go/src/github.com/gomarkdown/markdown/html (from $GOPATH)
➜ vim main.go
➜ go mod init
go: creating new go.mod: module github.com/kjk/modtest
➜ go build
go: finding github.com/gomarkdown/markdown/html latest
go: finding github.com/gomarkdown/markdown/ast latest
go: finding github.com/gomarkdown/markdown latest
go: downloading github.com/gomarkdown/markdown v0.0.0-20180907191918-6fda95a9e93f
➜ ./modtest
Hello

[kjk]

@josephlr I fixed the compilation error in the code.

However, I believe you did this inside GOPATH (as indicated by (from $GOROOT) messages).

As I've indicated, you have to checkout the code outside of GOPATH.

Try repro steps after e.g. cd $HOME.

[josephlr]

@kjk Nope this was done outside of my GOPATH. The go build errors were just what happens if no go.mod file is present.

[josephlr]

Just tried this on darwin/amd64, got the same result. go mod init && go build && ./modtest produced no errors for me (building outside of GOPATH).

[kjk]

@josephlr Are you using go 1.11 or master? Did you set GO11MODULE env variable to something? (I didn't so it's auto).

I just double-checked and am still getting an error:

Krzysztofs-MacBook-Pro:src kjk$ git clone https://github.com/kjk/modtest.git
Cloning into 'modtest'...
remote: Counting objects: 10, done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 10 (delta 1), reused 7 (delta 1), pack-reused 0
Unpacking objects: 100% (10/10), done.
Krzysztofs-MacBook-Pro:src kjk$ cd modtest
Krzysztofs-MacBook-Pro:modtest kjk$ go mod init
go: creating new go.mod: module github.com/kjk/modtest
Krzysztofs-MacBook-Pro:modtest kjk$ go build
go: finding github.com/gomarkdown/markdown/ast latest
go: finding github.com/gomarkdown/markdown/html latest
go: finding github.com/gomarkdown/markdown latest
main.go:8:2: unknown import path "github.com/gomarkdown/markdown/html": cannot find module providing package github.com/gomarkdown/markdown/html
Krzysztofs-MacBook-Pro:modtest kjk$ go mod tidy
go: finding github.com/gomarkdown/markdown/html latest
go: finding github.com/gomarkdown/markdown latest
Krzysztofs-MacBook-Pro:modtest kjk$ go build
go: finding github.com/gomarkdown/markdown/html latest
go: finding github.com/gomarkdown/markdown latest
main.go:8:2: unknown import path "github.com/gomarkdown/markdown/html": cannot find module providing package github.com/gomarkdown/markdown/html
Krzysztofs-MacBook-Pro:modtest kjk$ go get -v github.com/gomarkdown/markdown/html
go: finding github.com/gomarkdown/markdown/html latest
go: finding github.com/gomarkdown/markdown latest
Fetching https://github.com?go-get=1
Parsing meta tags from https://github.com?go-get=1 (status code 200)
go get github.com/gomarkdown/markdown/html: no matching versions for query "latest"

[kjk]

Also, what does go get -v github.com/gomarkdown/markdown/html do?

$ go get -v github.com/gomarkdown/markdown/ast
go: finding github.com/gomarkdown/markdown/ast latest
go: finding github.com/gomarkdown/markdown latest

$ go get -v github.com/gomarkdown/markdown/html
go: finding github.com/gomarkdown/markdown/html latest
go: finding github.com/gomarkdown/markdown latest
Fetching https://github.com?go-get=1
Parsing meta tags from https://github.com?go-get=1 (status code 200)
go get github.com/gomarkdown/markdown/html: no matching versions for query "latest"

https://github.com?go-get=1 is suspicious and only happens for html sub-package, not for ast sub-package.

[kjk]

And just to be sure, do GO111MODULE=on go get -u -v github.com/gomarkdown/markdown/html.

It works with GO111MODULE=off and fails with on or auto if not in GOPATH.

modtest kjk$ GO111MODULE=off go get -u -v github.com/gomarkdown/markdown/html
github.com/gomarkdown/markdown (download)

modtest kjk$ GO111MODULE=on go get -u -v github.com/gomarkdown/markdown/html
go: finding github.com/gomarkdown/markdown/html latest
go: finding github.com/gomarkdown/markdown latest
Fetching https://github.com?go-get=1
Parsing meta tags from https://github.com?go-get=1 (status code 200)
go get github.com/gomarkdown/markdown/html: no matching versions for query "latest"

[josephlr]

@kjk Still no luck reproducing your bug on darwin/amd64

I'm outside of GOPATH, no Go environment variables set (except GOPATH), using go 1.11 installed from Homebrew latest. I even went an manually deleted the entire package cache. Here's my full output:

joe@Lust ➜  git clone https://github.com/kjk/modtest.git
Cloning into 'modtest'...
remote: Counting objects: 10, done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 10 (delta 1), reused 7 (delta 1), pack-reused 0
Unpacking objects: 100% (10/10), done.
joe@Lust ➜  cd modtest
joe@Lust ➜  bash
bash-3.2$ exit
exit
joe@Lust ➜  GO111MODULE=on go get -u -v github.com/gomarkdown/markdown/html
go: creating new go.mod: module github.com/kjk/modtest
go: finding github.com/gomarkdown/markdown/html latest
go: finding github.com/gomarkdown/markdown latest
go: downloading github.com/gomarkdown/markdown v0.0.0-20180907191918-6fda95a9e93f
joe@Lust ➜  go build
joe@Lust ➜  ./modtest
Hello
joe@Lust ➜  pwd
/Users/joe/Documents/programming/funstuff/modtest
joe@Lust ➜  echo $GOPATH
/Users/joe/Documents/programming/go

Other versions of software that could help:

git --version
git version 2.16.2

wget -V
GNU Wget 1.19.4 built on darwin17.3.0.

-cares +digest -gpgme +https +ipv6 +iri +large-file -metalink +nls
+ntlm +opie -psl +ssl/openssl

Wgetrc:
    /Users/joe/.wgetrc (user)
    /usr/local/etc/wgetrc (system)
Locale:
    /usr/local/Cellar/wget/1.19.4_1/share/locale
Compile:
    clang -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/usr/local/etc/wgetrc"
    -DLOCALEDIR="/usr/local/Cellar/wget/1.19.4_1/share/locale" -I.
    -I../lib -I../lib -I/usr/local/opt/openssl/include -DNDEBUG
Link:
    clang -DNDEBUG -lidn2 -L/usr/local/opt/openssl/lib -lssl -lcrypto
    -ldl -lz ftp-opie.o openssl.o http-ntlm.o ../lib/libgnu.a -liconv
    -lintl -Wl,-framework -Wl,CoreFoundation -lunistring

Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
Please send bug reports and questions to <bug-wget@gnu.org>.

[kjk]

@josephlr Interesting. I tried to reproduce on another mac machine and couldn't.

So it's something specific to a setup on one machine but I have no ideas on how to investigate this further.

They both have similar setup, all programs from homebrew, kept up-to-date.

[agnivade]

@bcmills

[myitcv]

@kjk please can you try the following:

export GOPATH=$(mktemp -d)
cd $(mktemp -d)
git clone https://github.com/kjk/modtest.git
cd modtest/
go mod init
go build

[kjk]

That works, but I already said it works inside GOPATH.

Krzysztofs-MacBook-Pro:src kjk$ export GOPATH=$(mktemp -d)
Krzysztofs-MacBook-Pro:src kjk$ cd $(mktemp -d)
Krzysztofs-MacBook-Pro:tmp.uw4hgRPe kjk$ git clone https://github.com/kjk/modtest.git
Cloning into 'modtest'...
remote: Counting objects: 10, done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 10 (delta 1), reused 7 (delta 1), pack-reused 0
Unpacking objects: 100% (10/10), done.
Krzysztofs-MacBook-Pro:tmp.uw4hgRPe kjk$ cd modtest/
Krzysztofs-MacBook-Pro:modtest kjk$ go mod init
go: creating new go.mod: module github.com/kjk/modtest
Krzysztofs-MacBook-Pro:modtest kjk$ go build
go: finding github.com/gomarkdown/markdown/ast latest
go: finding github.com/gomarkdown/markdown/html latest
go: finding github.com/gomarkdown/markdown latest
go: downloading github.com/gomarkdown/markdown v0.0.0-20180907191918-6fda95a9e93f

[myitcv]

@kjk

That works, but I already said it works inside GOPATH.

But this test was outside of GOPATH; notice GOPATH and your working directory are two different temporary directories, so by definition you are outside of GOPATH.

This therefore sounds like an issue with your build/module cache (most likely the latter).

Perhaps take a backup of $GOPATH/pkg/mod (in case we want to investigate further) and then run go clean -modcache and see whether your original problem persists.

[bcmills]

I'm not able to reproduce the failure either.

I'm curious what happens if you run:

go get github.com/gomarkdown/markdown@v0.0.0-20180907191918-6fda95a9e93f
go mod verify

~$ cd $(mktemp -d)

$ git clone https://github.com/kjk/modtest.git
Cloning into 'modtest'...
remote: Counting objects: 10, done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 10 (delta 1), reused 7 (delta 1), pack-reused 0
Unpacking objects: 100% (10/10), done.

$ cd modtest

modtest$ go mod init github.com/kjk/modtest
go: creating new go.mod: module github.com/kjk/modtest

modtest$ go build
go: finding github.com/gomarkdown/markdown/html latest
go: finding github.com/gomarkdown/markdown/ast latest
go: finding github.com/gomarkdown/markdown latest
go: downloading github.com/gomarkdown/markdown v0.0.0-20180907191918-6fda95a9e93f

modtest$ go version
go version devel +b0a1c5df98 Wed Aug 8 12:50:33 2018 -0400 linux/amd64

[kjk]

go clean -modcache fixed this.

So the cache was busted to the point of messing things up. Knowing what I know now, most likely /Users/kjk/go/pkg/mod/github.com/gomarkdown/markdown\@v0.0.0-20180907191918-6fda95a9e93f and/or corresponding /Users/kjk/go/pkg/mod/cache/vcs/${sha1} and/or /Users/kjk/Library/Caches/go-build was corrupted.

Unfortunately by removing the cache I also lost all the info needed to investigate it.

A hint from earlier debug tries:

modtest kjk$ go get -v
go: finding github.com/gomarkdown/markdown/html latest
go: finding github.com/gomarkdown/markdown/ast latest
go: finding github.com/gomarkdown/markdown latest
Fetching https://github.com?go-get=1
Parsing meta tags from https://github.com?go-get=1 (status code 200)
github.com/gomarkdown/markdown/html
github.com/gomarkdown/markdown/ast
go build github.com/gomarkdown/markdown/html: no Go files in

no Go files in indicates that the git repository was corrupted because there are .go files in https://github.com/gomarkdown/markdown/tree/master/html

Also the error message is truncated or just badly phrased.

Some other thoughts.

-n flag for -modcache seems broken

$ go clean -n -modcache
cd /Users/kjk/src/modtest
rm -f modtest modtest.exe modtest.test modtest.test.exe main main.exe

I'm pretty sure go clean -modcache deletes more than that.

debugging tools for this area are not adequate

-v doesn't provide enough information.

GODEBUG allows printing a bit more information but not enough and it's not documented anywhere in full (as in: relevant to go toolchain as a whole; it's scattered in runtime docs etc.)

In retrospect it looks like local copy of github.com/gomarkdown/markdown in /Users/kjk/go/pkg/mod/cache/vcs/ was busted but I only figured it out after running strace (well, dtruss) on go get step (after I fixed it by cleaning the cache).

To properly debug this there should be a switch that makes go tool print all the files it's inspecting, all git commands its executing etc.

Realistically this bug can be closed because I don't see a way of making progress on debugging this further.

As a data point:

• it's possible to corrupt modules cache in a way that is not detected
• earlier I've been running 1.11 from master post beta 2 so corruption might have been introduced by that version and subsequently the root cause of such corruption might have been fixed before release

[bcmills]

I'm pretty sure go clean -modcache deletes more than that.

Yeah, go clean needs quite a bit of work for 1.12. (See also #27469, #27458, #27310, and #26991.)

earlier I've been running 1.11 from master post beta 2 so corruption might have been introduced by that version and subsequently the root cause of such corruption might have been fixed before release

That could certainly do it. go1.11beta2 was a pretty rough build for modules (sorry about that!).