go build -o should be using -i

[mattetti]

I noticed that we rebuild all packages every time a user saves a project with build on save on. I think it would be best to use the -i flag to only build packages that changed.

The -i flag installs the packages that are dependencies of the target.

My understanding is that installed packages aren't rebuilt. I can see that building a project with a slow to build dependency such as github.com/mattn/go-sqlite3 is much much faster when the project is built with -i.

@ramya-rao-a what do you think?

[ramya-rao-a]

@mattetti

My understanding is that installed packages aren't rebuilt.

Is there any official documentation for this?

[mattetti]

https://golang.org/cmd/go/#hdr-Compile_packages_and_dependencies The -i flag installs the packages that are dependencies of the target. Unfortunately I am not aware of documentation explaining the ramifications of having packages installed. Here is a Quora explaination: https://www.quora.com/What-is-the-difference-between-build-and-install-in-Go Basically what I saw is that when using -i in build or test, the dependency packages are cached and reused if they haven't changed. You can test that using an expensive pack to build such as sqlite3. Once you build it once with -i, other builds can use the cache.

…

On Thu, Dec 15, 2016, 18:37 Ramya Rao ***@***.***> wrote: @mattetti <https://github.com/mattetti> My understanding is that installed packages aren't rebuilt. Is there any official documentation for this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#673 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAAcc16nX5skf1Zy71OCq8JAHwAd-1Vks5rIflUgaJpZM4LBHQH> .

[ramya-rao-a]

@derekperkins mentioned https://github.com/joshuarubin/zb in another issue. This claims to run faster builds. It comes down to build vs install there as well

[mattetti]

I don't know if we should go all the way to pushing something like zb but yes zb uses some of the same techniques. -i really shouldn't have any down sides, besides making for faster builds, especially on save.

…

On Thu, Dec 15, 2016, 19:08 Ramya Rao ***@***.***> wrote: @derekperkins <https://github.com/derekperkins> mentioned https://github.com/joshuarubin/zb in another issue. This claims to run faster builds. It comes down to build vs install there as well — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#673 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAAcSP9mLKu3ISNySbO7eTukSPi1Vnmks5rIgCmgaJpZM4LBHQH> .

[mattetti]

I'll look at adding this flag and send a PR, @ramya-rao-a would you mind allocating this issue to me?

[ramya-rao-a]

Done

[ramya-rao-a]

@mattetti

I was doing some reading up on go build in general.
One of the build flags is -a which according to the docs will force rebuilding of packages that are already up-to-date.

If there is such a flag, then it doesnt make sense that go build -o is rebuilding all packages while building current package (Save triggers go build -o on current package)

I'd prefer to see some documentation on why go build -i is faster than go build -o before making this a default experience. Can we add setting for this change so that people can opt-in to use go build -i?

[mattetti]

@ramya-rao-a indeed the -a flag is a full rebuild of everything and it's recommended not to use it since it also rebuilds the system packages which might have permission differences.

go build -o doesn't always rebuild all packages used by the currently build package, it builds all packages that aren't "installed" and therefore not tracked. If you run go build -i once, the packages that haven't changed won't be rebuilt (which is the workaround I currently use).

I could add a setting to not use -i but I am not aware of any reasons why you wouldn't that on. Well, that's not true, there is one case you don't want that on: it's for main package. Library packages are installed in a cache, but main packages go to the bin folder which might not be something a user wants.

As you are reading more about go build, I wonder if you can find for downsides of using -i, if there are any, then we should add a preference, otherwise it's probably better not to since it would give users an option they don't need.

Oh, and before you ask, I think the reason why the go team didn't setup go build to use -i automatically is probably because they didn't want to bother looking up the package being built, but that's just a guess.

[mattetti]

Note that the hope if that the PR is merged, the problem reported here and in #686 should be addressed.

I did some investigation, zb checks if a package is a command by using go/tool and go/tool uses the package name like in my PR.
zb: https://github.com/joshuarubin/zb/blob/master/lib/dependency/gopackage.go#L30

Go/tool:
https://golang.org/src/go/build/build.go?s=13875:13909#L389

// IsCommand reports whether the package is considered a
// command to be installed (not just a library).
// Packages named "main" are treated as commands.
func (p *Package) IsCommand() bool {
	return p.Name == "main"
}

I looked at the difference between go build and go build -i by using the -x flag.
I built a package of mine: github.com/go-audio/examples/format-converter.
In both cases, the compiler tool prepares temp folders:

mkdir -p $WORK/github.com/go-audio/examples/format-converter/_obj/
mkdir -p $WORK/github.com/go-audio/examples/format-converter/_obj/exe/

But in the -i case, another folder is created:

mkdir -p $GOPATH/pkg/darwin_amd64/github.com/go-audio/

And then the compiled dependency is pushed there (wav pkg archive):

mv $WORK/github.com/go-audio/wav.a $GOPATH/pkg/darwin_amd64/github.com/go-audio/wav.a

I didn't dig much further but my limited understanding is that archive file is to be used next time instead of being recompiled. I'm sure Go does some extra check to make sure it hasn't changed but that's the gist of it.

[mattetti]

Here are some interesting threads talking about go install/go build -i:

https://www.reddit.com/r/golang/comments/2qpzm8/question_on_go_build_i_and_rebuilding_a_library/
Here the issue mentioned is that go install doesn't check for updates once a dependency package was rebuilt. So if I'm rebuilding pkg a that depends on pkg b and I updated b without reinstalling it, then pkg a will import the old build of b. If all packages had to be rebuilt, we would need to walk the dependency tree and try to rebuild them all one by one.

This is a project that says can make your compilation 7x faster. It's really only a wrapper around go install: https://github.com/kovetskiy/go-fast

This is an interesting discussion about go build and go install and why one should always use go install.

Dave puts it simply:

go build builds your package then discards the results.
go install builds then installs the package in your $GOPATH/pkg directory.

However, his suggestion is to just use go install, I believe it has the same behavior than go build -i. I'll look a bit more into it and see if we should make go install the default and offer an option to use go build instead.

[mattetti]

It doesn't look like there is a difference between go build -i and go install:
https://twitter.com/davecheney/status/819443033053609984

[ramya-rao-a]

We should invite Dave to this discussion :)

[mattetti]

@davecheney do you have a recommendation for us?
VsCode has a feature to optionally auto build the package you are editing. I was suggesting to add the -i flag to cache dependencies.

Q1: is there a downside to installing non main packages ? (As in should we offer a way to turn that off)
Q2: is go install a better approach?
Q3: any other recommendations to make actions at save time faster (go build, vet, lint...)?

[ramya-rao-a]

Another note: The main intention of running go build at every save, is to show compiler errors if any.
What is being debated here is that when dependent packages have not changed, which is faster go build -o or go build -i

[davecheney]

My opinion is use go install. Dominic or Damian would be better people to ask about tool recommendations.

…

On Thu, 12 Jan 2017, 18:28 Matt Aimonetti ***@***.***> wrote: @davecheney <https://github.com/davecheney> do you have a recommendation for us? VsCode has a feature to optionally auto build the package you are editing. I was suggesting to add the -i flag to cache dependencies. Q1: is there a downside to installing non main packages ? (As in should we offer a way to turn that off) Q2: is go install a better approach? Q3: any other recommendations to make actions at save time faster (go build, vet, lint...)? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#673 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcAz_u3LgbkFuGTOv_l-hE7mu6xMciks5rRdY1gaJpZM4LBHQH> .

[davecheney]

In the scope of one run, neither is faster. In the scope of subsequent builds, the latter will be faster.

…

On 12 Jan 2017, at 18:32, Ramya Rao ***@***.***> wrote: Another note: The main intention of running go build at every save, is to show compiler errors if any. What is being debated here is that when dependent packages have not changed, which is faster go build -o or go build -i — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ankon]

Just to confirm/clarify:

https://twitter.com/ankoninchen/status/817391388107096064

I did that. If I understand this ticket correctly then the go extension will also trigger a full compile on saving? In my case though this isn't that expensive because most of my packages are installed the moment I start the debugger for the first time?

[ramya-rao-a]

@ankon
Yes, once this changes are in, all dependent packages of the file you are editing would be up to date. So when delve runs the build using the buildflags: -i you send via launch.json, that would not be as expensive as it might be today.

[ramya-rao-a]

@mattetti I did some more digging and my findings match yours. PR approved and merged :)

[ramya-rao-a]

@mattetti More on exactly what you have been saying: Below is from https://twitter.com/MorganGeek/status/821981043527974914

Builds can be too slow, because the go command doesn’t cache build results as aggressively as it should. Many people don’t realize that go install saves its work while go build does not, and then they run repeated go build commands that are slow because the later builds do more work than they should need to. The same for repeated go test without go test -i when dependencies are modified

[ramya-rao-a]

The latest update (0.6.53) has this change.

Thanks @mattetti

[atombender]

Wondering if it's worth resurrecting this. Often you do work on a main package, and if you do, compilation accidentally becomes much slower because -i isn't used.

This behaviour wasn't clear to me, and so I didn't understand why my builds were getting super slow. I work on a tool that pulls in the Kubernetes client, which is huge and takes a long time (10-20 seconds with something like 4 parallel processes) to compile, and this is executed on every save. (At one point my entire machine stalled because a thundering herd of 20-30 compile processes were suddenly running, though I've not been able to reproduce this. Does the Go extension kill the in-flight compile on the next save?) But if I compile with -i, it takes 2 seconds to build.

Why can't -i be used for all builds? It seems to combine just fine with -o. As I understood the above discussion, the problem is that this will also install the binary, but couldn't we simply use -o /dev/null to avoid this?

[ramya-rao-a]

As discussed here: #1013 (comment) Yes, we can use go build -i for main packages as well.

We always use -o. So that will take care of the generated binary.

Fixed with 2f398ae

[atombender]

Awesome. Any reason it's writing to a file and not /dev/null? Windows compatibility?

[ramya-rao-a]

Must be. That was before my time.
If we do use /dev/null that's another if/else to support Windows.