Repository size over 500mb

[ivucica]

Hi,

The total directory size for $GOPATH/src/google.golang.org/api is 692.7MB, out of which $GOPATH/src/google.golang.org/api/.git is 548.1MB.

Even a single checkout is large, but if someone keeps different projects in different $GOPATHs, this means storing multiple copies of half-a-gigabyte directory. go get doesn't support shallow clones and doing so would mean involving wrappers around git.

Even if generated code cannot be reduced in size, should alternatives be explored for google.golang.org/api, such as splitting it into multiple Git repositories and restarting the history every once in a while?

[codyoss]

Hey,

We are aware that the project is quite large right now and have had discussions on this topic in the past. I will re-bring up the topic again and see if anything has changed.

such as splitting it into multiple Git repositories

I don't know if we would do this, but we have talked about splitting into multiple modules in the past.

restarting the history every once in a while

This would be tricky and a potentially dangerous. I am not sure how this would work with git tags and modules.

[ivucica]

I don't know if we would do this, but we have talked about splitting into multiple modules in the past.

I forgot about the existence of modules -- I'm firmly in the self-managed GOPATH world, happily ignoring the existence of modules. In my old-school world, breaking things up would mean simply nuking the old ${GOPATH}/src/google.golang.org/api folder and letting go get resolve the new Git repositories, with the HTML pages serving one per API.

Thinking about the Go modules world (which I have a passing familiarity with), I'd be concerned about how the lockfiles would behave.

Still, the 600mb+ disk usage, even if the binary uses just one of the many APIs, is rather concerning for my non-corp use, on my tiny VPS. :-)

restarting the history every once in a while

This would be tricky and a potentially dangerous. I am not sure how this would work with git tags and modules.

I'd be more concerned if go get uses git pull to update packages.

[tbpg]

Go modules, by default with recent versions of Go, are distributed as zip files of just the source code for that particular version.

When in GOPATH mode, repositories are downloaded with git clone (which clearly isn't ideal).

So, if you're using modules, the size of the total git repository does not matter -- only the size of the source at the version you are asking for.

That's one of the main reasons using modules (with a proxy! The default for recent Go versions) is so much faster than GOPATH mode -- there is way less to download. Modules also understand go.mod files, which list the dependencies of your project, so the go command can be smarter about what code actually needs to be downloaded. The result is that when using modules, you download less stuff and the stuff you do download is smaller.

---

For an introduction to modules and how to use them, see https://golang.org/blog/using-go-modules. In a nutshell, you can start using modules for your project today by doing:

cd $GOPATH/src/path/to/your/project
go mod init
go test ./...

If you then look at the go.mod file that go mod init created and go test populated, it will list all of the dependencies being used and their versions. If you look in $GOPATH/pkg/mod/cache/, you can see a read-only cache of modules, so that you don't have to download dependencies separately for every project.

You'll also note a go.sum file next to the go.mod. This is a (big!) list of dependencies and their hashes. In combination with the Go sum DB (http://sum.golang.org/), this verifies the integrity of your dependencies. That way, you can be sure your dependencies haven't been modified (either intentionally or unintentionally).

---

A question that comes up frequently is "If dependencies aren't checked out in GOPATH anymore, how to I modify them? The cache is read-only...?" You can check the repo out wherever you want in your system then use a replace to point to that directory. https://github.com/rogpeppe/gohack exists to make this way easier.

[tbpg]

I wrote a blog post expanding on why modules are faster than GOPATH: https://dev.to/tbpalsulich/why-go-modules-are-faster-than-gopath-blj

Closing this. We may take some action to reduce the repo size in the future, but modules solve much of the issues today. So, we're not planning to change anything for the moment.