cmd/go: auto cache clean fail

[josharian]

I am in the process of running some toolchain experiments. (Results will probably be sent to golang-dev once it is complete.) This involves compiling executables tens of thousands of times, each time slightly differently.

It turns out that this wreaks havoc on the cache: Every single compilation results in a bunch of new cache entries, which over the course of 24 hours ate my hard drive. (And in the process, caused me to lose my results so far.) I'm several hours into go cache -clean. I have 360gb free and counting.

GOCACHE=off is no longer supported.

What should I have done (or better, what should cmd/go do) to prevent runaway cache growth? A hard size limit of 100gb for the cache would have saved me a world of trouble.

cc @bcmills @jayconrod

[bcmills]

I'm several hours into go cache -clean.

That seems surprising. Could you file a separate issue for go cache -clean being slow? 🙂

[bcmills]

What should I have done (or better, what should cmd/go do) to prevent runaway cache growth?

Compiling executables tens of thousands of times with tiny variations is a very unusual mode of usage. It doesn't seem entirely unreasonable to expect you to run go cache -clean manually every so often during that experiment...

That said, if there is some portable way we can detect free space, it might be a good idea for the go command to periodically check the ratio of free space to cache usage and clean if it gets too high.

[jayconrod]

I think this is a dup of #29561.

Definitely unfortunate behavior though. I don't think we can put an upper bound on cache size, but we ought to have a smarter eviction policy that takes cache size and free space into account.

For reference, the current cache eviction policy is hard-coded in cmd/go/internal/cache. We scan for deletable files at most once per 24 hours. We delete files that haven't been used in 5 days.

[josharian]

That seems surprising. Could you file a separate issue for go cache -clean being slow? 🙂

I'm not sure that it is slow. I think the problem is that it had to delete 400gb of smallish files. rm -rf would be similarly slow.

[josharian]

It doesn't seem entirely unreasonable to expect you to run go cache -clean manually every so often during that experiment...

One other sad note, just for posterity. Running go cache -clean concurrently with builds causes those builds to fail. So I'm now having to coordinate pausing and restarting all the other compilation jobs in order to clean the cache occasionally.

[jayconrod]

@josharian That doesn't seem like good behavior either. What should go clean -cache do with concurrent builds?

• We could have a "clean" lock that would block go build and other commands until go clean -cache finishes.
• go clean -cache could leave files created after it started. This might be hard to do reliably and portably though.

[bcmills]

I think that locking scheme is possible to implement using the existing cmd/go/internal/lockedfile API.

go build would obtain a read-lock on the file (os.O_CREATE|os.O_RDONLY)
go clean would obtain a write-lock on the file (os.O_CREATE|os.O_WRONLY)

We don't currently rely on file-locking for the cache, but since this would only prevent a build / clean race perhaps it's not so bad.

[josharian]

I'll file a new issue to discuss, thanks!

[josharian]

#31948