Use my template ?

[cartazio]

https://github.com/cartazio/ralist/blob/cd2a6c65baf4b4ee7f8d316e57e282087f89c07e/.github/workflows/hs-matrix.yml

Or provide it as an example.

I still wanna tweak some things. But it has a lot of nice things like working caching. Plus a small digital memorial for my recently passed grand parent

[gelisam]

My condolences.

like working caching

Can you explain the "fall back when the freeze plan is different" bit? does it mean that if the freeze file is different from the cache, it will not start from scratch and will instead pick a random cache entry which uses the same OS and GHC?

[omelkonian]

Can you explain the "fall back when the freeze plan is different" bit? does it mean that if the freeze file is different from the cache, it will not start from scratch and will instead pick a random cache entry which uses the same OS and GHC?

Exactly.

[amesgen]

Can you explain the "fall back when the freeze plan is different" bit? does it mean that if the freeze file is different from the cache, it will not start from scratch and will instead pick a random cache entry which uses the same OS and GHC?

Exactly.

A bit more precise: The newest cache entry with same OS + GHC is used. See here for the docs.

---

Also, I would suggest some minor updates: https://gist.github.com/amesgen/f39e84a911cfc843bf9f7298f1dd22d9 (also see the diff)

• GHC patch versions on windows are already handled: 2e698e5
• actually use haskell/actions/setup
• unify caching using the cabal-store output

[cartazio]

these are really good suggestions, i'll have a go at iterating on these later today/this week

also instead of using the hash of the freeze file, i noticed in the new haskell-ci style template that they have github.sha, which would perhaps be better behaved for rebuilds? see https://github.com/haskell-CI/haskell-ci/pull/416/files#diff-73f8a010e9b95aa9fe944c316576b12c2ec961272d428e38721112ecb1c1a98bR162

also thanks @amesgen for answering samuel's question, i didnt have time this past weekend ..

[hazelweakly]

The current best practice in order to get the cache behaving most efficient with cabal is to cache everything by utilizing cabal-cache rather than the native caching mechanism (but the same advice applies with caching the cabal store and/or dist-newstyle)

The subtle bit is in the key. What you want is you want to always upload the new cache in an append only manner for hermetic caches like cabal's (or bazel's, etc), but this is not possible with github's cache by default as they don't upload the new cache on a cache-hit. So what you do is:

1. Set the cache key to something that will always fail. github.sha is particularly nice because it gives you a perfect hit should you rebuild the same exact commit again in CI (which is usually what you want)

2. Set the fallback to the "real" key. This is the one that should be something like ${{ runner.os }}-${{ matrix.ghc }}-${{ hashFiles('cabal.project.freeze') }}. But, given that it's harmless to have bad stuff in the cache, what I tend to do is set multiple cache fallbacks, so I'll have the first one be on the os + ghc + cabal.project.freeze file, and the second on the runner os + ghc, so overall the entire thing looks like

   - uses: actions/cache@v2
     with:
       path: |
         ${{ steps.setup-haskell.outputs.cabal-store }}
         dist-newstyle
       key: ${{ runner.os }}-${{ matrix.ghc }}-${{ hashFiles('cabal.project.freeze') }}-${{ github.sha }}
       restore-keys: |
         ${{ runner.os }}-${{ matrix.ghc }}-${{ hashFiles('cabal.project.freeze') }}-${{ github.sha }}
         ${{ runner.os }}-${{ matrix.ghc }}-${{ hashFiles('cabal.project.freeze') }}-
         ${{ runner.os }}-${{ matrix.ghc }}-

   If you're looking at this and saying to yourself "wow, that looks particularly verbose and error-prone", that's because it is. Now that github exposes the cache API in nodejs, it'll be possible to implement the cache step entirely in the action should someone set cache: true, so I'd like to do that as a future improvement.

   In practice, it's likely sufficient to drop the cabal.project.freeze hash in the middle, but it makes it so that you don't lose the cache if multiple people are working on somewhat disjoint git histories at the same time, so it's a tradeoff.

[hazelweakly]

To answer the original question at the top, I'd be perfectly happy to expand out the examples and include something very similar to yours.

I plan to eventually have an examples directory where all the examples live and run most of them automatically in github actions (as well as having the simplest ones in the README). I'll be sure to include an example utilizing haskell-CI as well.

[omelkonian]

   If you're looking at this and saying to yourself "wow, that looks particularly verbose and error-prone", that's because it is. Now that github exposes the cache API in nodejs, it'll be possible to implement the cache step entirely in the action should someone set `cache: true`, so I'd like to do that as a future improvement.

Glad to hear you want automatic caching, since I already did that in setup-agda, so it won't be too much trouble making a PR with this.

[andreasabel]

@hazelweakly wrote:

2. But, given that it's harmless to have bad stuff in the cache, what I tend to do is set multiple cache fallbacks,

So, if I understand this correctly, some cache (usually the most recent) will always restored, and the updated build dirs will always be recached.

Isn't this risking that the build dirs grow out of hand, accumulating old builds? cabal does not clean up old builds, and it is eager to rebuild stuff with a new hash if some dependency changed somewhere, or a stone fell off the volcano on Mars. So, wouldn't the cache entry grow fatter and fatter until it squeezes other entries out of the cache or busts the GitHub cache limit in the end?

Would be nice to get some stats on the cache contents after a 1000 CI runs...

[cartazio]

i think github evicts caches after a month, but i'm not current on the details

…

On Tue, May 17, 2022 at 1:14 PM Andreas Abel ***@***.***> wrote: @hazelweakly <https://github.com/hazelweakly> wrote: 2. But, given that it's harmless to have bad stuff in the cache, what I tend to do is set multiple cache fallbacks, So, if I understand this correctly, some cache (usually the most recent) will always restored, and the updated build dirs will always be recached. Isn't this risking that the build dirs grow out of hand, accumulating old builds? cabal does not clean up old builds, and it is eager to rebuild stuff with a new hash if some dependency changed somewhere, or a stone fell off the volcano on Mars. So, wouldn't the cache entry grow fatter and fatter until it squeezes other entries out of the cache or busts the GitHub cache limit in the end? Would be nice to get some stats on the cache contents after a 1000 CI runs... — Reply to this email directly, view it on GitHub <#7 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAABBQWXYOYRJU3DLQX5TILVKPHYDANCNFSM4UZYINRQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[andreasabel]

i think github evicts caches after a month

This would not help if you use CI continuously (e.g. in Agda it runs daily) and the cached directory accumulates old builds. Github cannot shrink a cache entry (how would it know what to discard?), it can only remove it. So if your entries grow to big, caching becomes dysfunctional.

[cartazio]

I think each time you save it’s a different hash digest key. …

…

On Thu, May 19, 2022 at 10:34 AM Andreas Abel ***@***.***> wrote: i think github evicts caches after a month This would not help if you ruse CI continuously (e.g. in Agda it runs daily) and the *cached directory accumulates old builds*. Github cannot shrink a cache entry, it can only remove it. So if your entries grow to big, caching becomes disfunctional. — Reply to this email directly, view it on GitHub <#7 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAABBQQYX7GMP6ZNBPPWM3LVKZGPDANCNFSM4UZYINRQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>