cmd/go: size of go-build directory increasing rapidly #68872
Comments
sorry this didnt help, my problem is somewhat different. |
|
I found a workaround: I am comparing the timestamps and deleting the files that were created after the first go run invocation. This is working good for me now. |
|
@deoxyimran Do you have a reproducing case that we can run to try to diagnose this? Which files are being created after the first go run? Are they C object files? |
cagedmantis
added
GoCommand
|
Yesterday night my timestamp compare and delete script failed to work. It deleted more files than it should have! Sigh I have to rebuild again....I guess I shouldn't mess around with the build artifacts... How does this caching mechanism even work? Are the old build artifacts slowly replaced by new ones as time goes on? So many questions now!
Yes numerous C binaries. The library in question is just go bindings for gtk4 by diamondburned. |
|
If each invocation of If it's a case of small changes causing the cache to grow a lot, the same command can be run to see what packages are getting recompiled/reindexed, and where they're being copied to in the build cache. You can then look at the size of the cached blobs (the ones that end with You might also find this script helpful for getting a list of the packages that are taking up the most space in the cache.
The build system will calculate a hash for each action based on its inputs, then check the build cache to see if the result of the action is already cached; if not, it performs the action and stores the result in the cache. So at the basic level it only adds new files to the cache, it doesn't update old ones. Old and new artifacts for the same package will live side by side in the same cache. However, the cache system does have an additional pass where it will trim the cache every 24 hours (or more precisely, at the end of the build if it's been 24 hours or more since the last trim). It trims the cache by removing all files that are more than 5 days old in order to keep the cache from growing in size forever. |
|
Thanks a lot for the valuable info, will definitely try out the commands and the script tonight to see if i find anything useful! |
|
This is what I am getting after adding the -x and -v arguments: import configpackagefile context=/home/darkc0der/.cache/go-build/cb/cbe665db303eb8a94673f011510252e270a60312249539dd802e2c1f6b5235e8-d import configpackagefile fmt=/home/darkc0der/.cache/go-build/7d/7d9cea467775f38ca753cf33fed8d5056e8910dfc15d287e1902ff0ba3437318-d |
|
I tried the script but it doesnt list the sizes of all the packages? Also how do i know which binary maps to which package? This way I will be able to delete my desired package's build artifact. |
|
I read through the whole script, I think now I have a better understanding how go's building, caching and linking works. Thanks again! |
|
@deoxyimran Let us know if we could provide more information. I'm going to set this as WaitingForInfo for now. |
matloob
added
WaitingForInfo
|
Timed out in state WaitingForInfo. Closing. (I am just a bot, though. Please speak up if this is a mistake or you have the requested information.) |
|
I am back. Sorry for delayed response but the bot made the right decision to close this issue since the problem is not affecting me anymore it seems. I noticed the build size goes back to a more manageable size when it gets too big, maybe some automatic background process spawns up and cleans up the excessive junk? I dont know when the trimming happens, but it sure saved quite a bit of space and hence not relying on the scripts anymore. I am marking this issue as SOLVED. |
Go version
go version go1.21.6 linux/amd64
Output of
go envin your module/workspace:What did you do?
I was building and running my source with go run command.
What did you see happen?
A project of mine requires a third party lib that uses a lot of cgo calls under the hood. I have already built all the packages during the first go run. Subsequent go run invocations were fast. But the cache size kept grewing even after all the modules built. How do i solve this issue? I only have a limited amount of disk space and I can't afford to clean the build cache every time the cache gets too big (sometimes exceeds 1gigs) and rebuild everything from scratch; this would waste a lot of time and resources. Do I have to live with this? Is there no workaround for this? Helpful to note that that cache grew by 1MB for every two or three go run invocations after the first invocation. Thank for for your time.
What did you expect to see?
The cache size stays relatively constant through multiple go run invocations.
The text was updated successfully, but these errors were encountered: