non-vendor mode?

[tve]

I'm frequently finding myself in a use-case where I'm working on multiple repositories simultaneously and the current Go vendor model just doesn't work. It would be awesome if glide supported a non-vendor mode where it updates the gopath instead of the vendor directory. I know this may be outside of glide's mission and it would be better if the Go vendor stuff worked for this use-case, but I thought I'd raise the issue here anyway.

Example use-case: I'm working on a library in one repo checkout out into my gopath and an app that uses the library in another repo checked out in my gopath. As I'm adding functions to the app I'm adding stuff to the library. Both have a number of shared dependencies. I don't know how to make this use-case work with vendor dirs 'cause that inevitably ends up with two copies of the same dependency, only in the app's vendor dir and one in the library's vendor dir or in the gopath if the library doesn't have a vendor dir.

The way I make things work is to cleanly manage my gopath manually (e.g. create a gopath for this overall project) and not use vendoring at all, which means that I recreate the dependencies listed in glide.yaml manually using go get. It would be awesome if I could tell glide install to put stuff into the gopath instead of the vendor subdir.

[mattfarina]

@tve working on two interconnected things can be a real problem. It's also a problem for resolvers that want to make changes to the repo as a way to workout a version.

Yet, we do need to make this work.

How often do you make a change in the GOPATH that needs to be reflected in vendor/?

What kind of workflow would you like to use to switch in and out of a mode where you're using a local copy?

[tve]

Whoa, a bunch of more use-cases :-)

One thing that occurred to me is that when using GOPATH I believe the Go tools still honor the vendor dir in imported packages. What this means is that if I'm building package A that uses package B in the GOPATH and B has a vendor dir then B's vendored dependencies will be used. This runs a bit counter glide's philosophy of flattening the vendored packages. Sadly with the removal of the vendor experiment env var one can't turn off pulling in the vendor dirs anymore.

How often do you make a change in the GOPATH that needs to be reflected in vendor/?

When iterating it can be ~hourly or less. For example, while testing my app I may run into a bug in the library. I switch my focus to the library, fix the bug, add a test case, run the tests, then switch back to the app. If I'm OK pushing to github I could push the lib, switch back to the app, run glide up -u and move on. But I don't always want to push, specially when the switch of focus wasn't for a cut&dry bug fix but to add a feature.

What kind of workflow would you like to use to switch in and out of a mode where you're using a local copy?

That's a good question. I would assume that for any project I'm checking out I can run glide up or glide install. I don't check in the vendor dir, but I know that others do. The flow I would use:

• check out the repos I work on into my (messy) gopath
• blow away the vendor dirs (in my case a fresh checkout wouldn't have any)
• run glide install --gopath-only in each project to ensure everything in the gopath meets glide.yaml requirements
• run go build or go test as desired
  And when I'm done I would:
• run glide install in my application project and expect the vendor dir to be populated as usual

I believe the main issue with the above flow is that if the user checks the vendor dir into version control then doing an add/commit requires ignoring the fact that vendor is missing and doing a checkout (for example to switch branch) requires remembering to blow away vendor again.

[mattfarina]

@tve you'll be impacted by #547. You might want to take a look at that.

[tve]

Thanks for the heads-up. Do I understand correctly that I could solve my use-case by overriding the cache location for the library to point to where I have it checked out in my gopath? This way glide pulls that into the vendor dir of the app? But I'd have to run glide up -u each time, with the resulting delay while it tries to update every dependency I use (really slow for me)?

[mattfarina]

@tve A few things...

• The -u flag on glide up is going away. No longer needed. The intermediate cache is going to be used which should make things a lot faster for those who had been doing glide up -u.
• Things from the local system (using the file:// URI) will be pulled into the cache before being exported to their final destination. In the cache resolution occurs safely without impacting an external system.
• Only committed changes are passed around.

In theory you can use the mirror to point to dev in the GOPATH and it should be pretty quick because the intermediate cache means only unknown commits are passed around now. But, keep in mind, If you pin something and put the app without the dependency having the commit pushed too it will be a pain for others.

To share some numbers. Prior to the change I ran glide up -u on a project at it took 2:53.17 in total time. After that change (with a warm cache) glide up (with the deprecated -u built in) took 51.432 in total time.

There is also the option of circumventing the things while developing. In your vendor/ folder you could delete the dependency you're developing on and then go will just resolve it to the GOPATH location. When done switch back to having it in the vendor/ folder.

[tve]

Mhh, sounds painful, I guess I'll have to build some tool that rsyncs stuff into the app vendor directory :-(. Having to commit and run glide each time I switch context and then having to squash commits so they're not a total mess is not attractive to me.

The last paragraph you wrote "There is also the option of circumventing the things while developing..." is actually very incorrect, that's the first thing I tried. For example, suppose both the app and the library use the log15 logger or use context.Context. Well the app will use the package from its vendor dir and the library will use a different copy either from its vendor dir or the gopath, depending on whether the lib has a vendor dir. At that point you know what happens: you have two copies of log15.Root() and anything that sticks something into the context using a package-global var as key ends up not working across lib & app. In short: a nice corner of debugging hell.

If you remove the -u flag what is the method for pulling new versions from the upstream repos of dependencies?

[mattfarina]

@tve have you tried symlinking the location in your GOPATH to the corresponding location in vendor/ for this kind of development?

[sdboyer]

Mhh, sounds painful, I guess I'll have to build some tool that rsyncs stuff into the app vendor directory :-(. Having to commit and run glide each time I switch context and then having to squash commits so they're not a total mess is not attractive to me.

Yeah, the optimal workflow for this use case really is to just let it source directly from GOPATH - no need to commit for it to be used in your build. As you noted, though, that creates an issue with shared deps; if the secondary project is sitting on GOPATH, it won't be able to access the deps placed under the main project's vendor/.

This is really the kind of case where just dropping in a symlink would be ideal, but then you've got the go toolchain's weirdness about symlinks (so I'm not actually even sure that that would work?). Honestly, absent some changes in the toolchain, I wonder if the easiest solution here might be to just fall back to rsync if available, else sloooow copying. (man do i wish someone would write a fast, native Go impl of recursive copying)

[tve]

I'll try the symlinking. I swore never to use symlinks again in the GOPATH after having too many issues, but that's now over a year ago, so the wounds are no longer open and I can try again :-O.

The issue with using the GOPATH is the removal of the GOVENDOREXPERIMENT variable. It means I can't disable vendoring, which is what I'd like to do so the GOPATH is what's being used and I can manage that. Not being able to disable vendoring means that any dependency in the GOPATH that happens to have a vendor subdir goes off into its own namespace. So I'd have to diligently delete all vendor subdirs in the GOPATH, which now means I no longer have clean checkouts, which means tools complain at the next update, and it's downhill from there.

OK, symlinks, here I come...

[sdboyer]

Yep, the nested vendor dir is why I'd prefer to have a copying/rsync solution - that way, at least, we can just exclude the vendor dir from what's pulled over. No way to do that with symlinks.

[mattfarina]

@tve if symlinks don't work and an rsync style solution is best... it might be worth writing a glide plugin to sync a project from the GOPATH in a cross platform way. I'm really curious about your experience.

I could see glide sync github.com/foo/bar that pull over the code minus the VCS and vendor directories. Would need to consider the way to handle glide config files in doing so (and I'm sure Sam will help with that). Just pondering for now.

[jbarefoot]

@sdboyer on #608 you said you would "love to see someone put together even a slapdash approach". Well I have the slapdash-iest of approaches for your consideration. :)

I've written a pair of complementary bash scripts to handle this via symlinks (in the attached zip). I call it glide-link as it is similar conceptually to npm link. It works by renaming the appropriate directories and creating a symlink as follows (leading slashes below are just to reinforce that these are directories):

1. Rename /dependency/vendor to /dependency/vendor_glide-link_bak
2. Rename /current_package/dependency to /current_package/dependency_glide-link_bak
3. Sym link /current_package/dependency -> /dependency

It does some sanity checks on directory existence as well, and glide-unlink just reverses the above. This is hacktastic but does what I want. The way I use it is to drop the 2 scripts into a directory somewhere, chmod +x them both, then add to your PATH.

Do you see a way this would break?

glide-link.zip

[revett]

@jbarefoot works a treat 🎉

Have you got these in Github? I would like to share them with the team & in a blog post.

[jbarefoot]

Thanks for testing it out @revett. No, I don't have them in github anywhere yet--I was waiting for opinions on whether this should just be part of the glide CLI. If that seems best to other folks, I can re-write these in Go as glide commands. On the other hand, if folks just want the scripts in their own repo for now for collaborative improvement, I'm fine with that too. @mattfarina what would you advise here?

[eriknelson]

Is this officially on the roadmap? npm-link style functionality would be great. Doesn't have to work the same, but I'm thinking specifically about a simple workflow with a bin importing a separate lib project, and I'm actively working on both (as described in the golang wiki). Is there a recommended way to do this with glide today, or is this multi-repo setup at odds?

In the meantime, @jbarefoot's scripts are working well for me. Thanks!

[revett]

We've been using @jbarefoot's solution at Vidsy this year.

Repo: https://github.com/revett/glide-link

[ohcrfpv]

my solution tool.sh

#!/bin/bash
for folder in $(pwd)/vendor/*; do
	temp_package=`basename $folder`
	echo $temp_package
        ln -sf $(pwd)/vendor/$temp_package $(pwd)/src/$temp_package
done

[fermin-silva]

Since I do not want to mess around with symlinks, my manual approach so far is to delete the dependency from the vendor directory altogether.
If it's not found in the vendor dir, go build looks for dependencies in the $GOPATH, where all my projects lie, and thus finds my still-not-pushed shared dependency which I can edit on the fly.

Of course every time I do glide up or glide install, glide puts my shared dependency back into the vendor folder, but I delete it again manually