-
Notifications
You must be signed in to change notification settings - Fork 670
Build in a container by default, using go 1.5 and vendored dependencies. #1861
Conversation
Why the use of gvt? What's wrong with just GOVENDOREXPERIMENT plus git submodules (or subtrees if you prefer)? |
This is using GOVENDOREXPERIMENT. Matthias doesn't want the dependancies checked in (ruling out manually fetching the dependancies or using subtree). I don't want to use submodules, as they are a pain. The requirements were to pin a specific version of each dependancy, but have them fetched at build time. |
Could you elaborate on how submodules are a pain? It's possible to make their use very inconspicuous in this context. The advantage of using submodules is that there is no need to learn the conventions of any tools beyond go and git. (I'd guess subtrees might work as well, but I'm not very familiar with them.)
I notice that gvt is only referred to from circle.yml. Presumably it needs to be run to retrieve the dependencies when doing a non-circle build? How does that happen? |
3e60a7f
to
9df8518
Compare
I'm assuming your suggest is we subtree in every dependancy? And the do a git submodule sync/update/whatever in the makefile to ensure they are always on the right version? We've been using gvt in scope & the service and we've found it massively easier than dealing with submodules. The submodule for tools has already causes problems for some people (Bryan, I think).
This PR is marked work in progress; we were just discussing how to do that in the net room. The PR should now include a mechanism (via an uptodate file) to ensure gvt is run correctly. |
7b8e383
to
9dd4855
Compare
You have collapsed the WEAVEUTIL_EXE rule into SIGPROXY_EXE, DOCKEPLUGIN_EXE, etc rule. That has two undesirable consequences:
|
You removed the |
re: travis: $(EXES) yes I removed that as I removed all the travis build stuff. Will re-add as |
This also fixes #1850, doesn't it? http://docs.weave.works/weave/latest_release/building.html needs updating. |
Just the TestHTTPCancel failure now, which @bboreham is working on. |
Well, we've been using submodules for vendoring in ambergreen, and it is so inconspicuous that it is hard for me to see how anything could be massively easier.
Without knowing details I can't respond. If it goes beyond simple unfamiliarity, then maybe more details would be relevant. Like git in general, submodules have a learning curve. But that will be true of any vendoring tool as well. The difference is that some people are already familiar with submodules, and experience gained with them is widely reusable. Also, from https://golang.org/s/go15vendor :
|
I'll give the submodules a try. |
e9a2b9c
to
af043cc
Compare
$(WEAVEUTIL_EXE): | ||
go build $(BUILD_FLAGS) -o $@ ./$(@D) | ||
$(NETGO_CHECK) | ||
|
This comment was marked as abuse.
This comment was marked as abuse.
Sorry, something went wrong.
Linting should be containerised. |
ifeq ($(BUILD_IN_CONTAINER),true) | ||
tests: $(BUILD_UPTODATE) $(VENDOR_UPTODATE) tools/.git | ||
$(SUDO) docker run $(RM) $(RUN_FLAGS) \ | ||
-v $(shell pwd):/go/src/github.com/weaveworks/weave \ |
This comment was marked as abuse.
This comment was marked as abuse.
Sorry, something went wrong.
I am not keen on eventually having all our main Makefile targets look like: ifeq $(BUILD_IN_CONTAINER,true)
target:
$(SUDO) docker run ... target
else
target:
the real build commands
fi How can we avoid that? |
I can't think of another way, maybe @dpw can. I've made it so we only have one big if now, and only two docker runs (ie not one for every target).
Done. Added golint to container.
Sorry. Done.
(for tests) go test --help says:
Working on figuring out if this is true or not. |
yup. what can you do? |
tests lint: | ||
$(SUDO) docker run $(RM) $(RUN_FLAGS) \ | ||
-v $(shell pwd):/go/src/github.com/weaveworks/weave \ | ||
-v $(shell pwd)/.pkg:/go/pkg \ |
This comment was marked as abuse.
This comment was marked as abuse.
Sorry, something went wrong.
This comment was marked as abuse.
This comment was marked as abuse.
Sorry, something went wrong.
re: This is interests: https://code.google.com/p/go/issues/detail?id=3896 Basically, sticking a |
Maybe not a massive speed up, but a welcome one:
|
So this PR contains five changes:
AFAICT, the only real dependencies are 1->2, and 4->5. Correct? Mixing all these in a single commit, in a single PR, makes it rather more difficult to review & iterate on these. I shall try to extract a few things into separate master commits / PRs. I have already done that for a couple of small refactors. |
@dpw the link you provided also has the following paragraph, right after the one you quoted:
I've just confirmed running |
Yeah, I thought I did, but clearly I didn't. (I wondered if I found some contorted way to persuade So I suspect I just did a regular |
5f3b6c7
to
6bf8878
Compare
Say I've started depending on a new library which in turn depends on a bunch of other libs. What do I run to add the new dependencies? How do I update a specific dependency to a new version? go-dockerclient springs to mind, which we need to update in order to be able to use recent docker features. And what happens to the dependencies of that when I do the update? How does one update all dependencies to the most recent version? Note that this should also remove any dependencies that are no longer needed (as well as, obviously, add any new ones). It's the kind of thing I envisage us doing every now and then, i.e. we'd run such an update, do a bunch of tests and pin any breakages to earlier working versions. |
The answer to all of these, with the PR as it stands, is manually. Using It's worth noting that gvt isn't a panacea. The gvt rebuild is slow, and I don't think any of these are frequent operations, so I think we should On Monday, 11 January 2016, Matthias Radestock notifications@github.com
|
That's not a great answer. I don't want us to accumulate cruft, or be stuck with ancient versions. Lack of tooling encourages that. Surely you didn't produce the .gitmodules and associated "Subproject commit" files manually. So in my, possibly incredibly naive view, it should be easy to remove all those artifacts and re-create them from scratch, using whatever horrific script you cooked up to create them in the first place. |
Yes, I did: go list -f '{{join .Deps "\n"}}' ./prog/* | grep -E '^(github.com|k8s.io)' | grep -v 'github.com/weaveworks/weave' | sort -u > deps
cd vendor
cat ../deps | while read repo; do git submodule add -f --depth 1 https://$repo.git $repo; done It choked on some of the golang.org imports, which I fixed up manually. And it misses some of the test dependancies, which I added manually. It wouldn't be too hard to put together some scripts which do this properly. |
2477f3c
to
50384c9
Compare
FROM golang:1.5.2 | ||
ENV GO15VENDOREXPERIMENT 1 | ||
RUN apt-get update && apt-get install -y libpcap-dev python-requests time | ||
RUN go get github.com/FiloSottile/gvt; touch /tmp/.donotremove |
This comment was marked as abuse.
This comment was marked as abuse.
Sorry, something went wrong.
4650090
to
91224be
Compare
- Use submodules for vendoring. - Don't go clean, now that everything is in .pkg. - sudo -E as we need the environment variables. - Install -race version of std so the tests can run without sudo. - Cache the build image on circle, use the rebuild-image script from tools.
91224be
to
c098aef
Compare
Rebased, squashed, and remove the incorrectly vendored golint and gvt source. |
I was interested to find that this build is faster for small changes. For example, after having done a full build:
On my machine the old build takes 6.9s and the new takes 4.1s. This appears to be because the |
As discussed elsewhere, |
fdc52a6
to
6684ae8
Compare
…n-container Build in a container by default, using go 1.5 and vendored dependencies.
Fixes #1657, fixes #1656, fixes #1850. I think it also fixes #1563.
This PR:
go get
to prove the vendored dependancies are all thereTODO: