Include OS and CPU architecture in CircleCI cache keys

[cameel]

Recently b_osx job has been failing in some of our PRs. Specifically external PRs that have been inactive for some time.

Failed to unarchive cache

Error untarring cache: Error extracting tarball /var/folders/1b/gl7yt7ds26vcyr1pkgld6l040000gn/T/cache3415993642 : usr/local/Homebrew/Library/Homebrew/shims/scm: Can't remove already-existing dir tar: Error exit delayed from previous errors. : exit status 1

"Error untarring cache" when running a macOS job suggests to include {{ arch }} in cache key. The reason for failures is probably that there has been an OS update after these PRs were last updated and the cache is simply no longer valid.

This PR adds {{ arch }} to cache keys on macOS and also on Windows just in case.

[chriseth]

Looks good for now.

[cameel]

Looks like {{ arch }} did not help. It must be something else.

[chriseth]

Wait but the build worked, did you forget to add arch to the test run?

[cameel]

Hmm... I'm guessing that build went fine because the key was not present in the cache. The t_ jobs restore it and that's what went wrong. Not sure why though. I'm trying to SSH into the machine but for some reason I get "permission denied".

[cameel]

b_osx rerun is failing too which confirms that there's nothing special about t_ jobs here.

[cameel]

Back to draft. I need to experiment with it a bit.

[cameel]

We'll have to wait for CircleCI for a proper fix (I think this is a problem in restore_cache and not in our config) but I have a workaround. Removing /usr/local/Homebrew/Library/Homebrew/shims/scm/ lets restore_cache finish without errors.

The dir contains only two files: git (some wrapper script over git) and svn (symlink to git). Removing them should be safe since they will be restored from cache anyway.

I'm leaving the {{ arch }} workaround in too. We probably do not need it but it does not break anything and this way we won't run into the problem described in the article.

[cameel]

After digging deeper, I think that this is a bug in restore_cache that got triggered by a recent change in Homebrew. Homebrew/brew#12170 replaced the /usr/local/Homebrew/Library/Homebrew/shims/scm/ directory with a symlink to shared/ in the same parent dir. CircleCI has an older version of Homebrew already installed in the image but we install (and cache) a newer one. restore_cache runs tar to unpack the cache, which refuses to overwrite an existing dir with a symlink during unpacking. In this case the overwrite is legit (we want everything replaced with the cached content) but restore_cache does not take this scenario into account and just lets tar fail.

[christianparpart]

Do you mean the rm -r below? If so, what about rm -rf?

[cameel]

No, the rm is the workaround :) Works without --force.

It's restore_cache that can't remove the files and it's provided by CircleCI so we have to wait for a fix from them.