Skip to content

Fix rerun log cache handling #2450

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Dec 8, 2021
Merged

Fix rerun log cache handling #2450

merged 3 commits into from
Dec 8, 2021

Conversation

jneira
Copy link
Member

@jneira jneira commented Dec 8, 2021

  • The cache of tasty logs to not repeat succesful tests between attempts of the same run was not working correctly
  • new makes rerun all tests again, as succesful one in one attempt are "new" for the next one
  • Logs are saved in each package root dir, so the cache should be **./.tasty-rerun-log*
  • func-tests and wrapper-tests are from the same hls top level package so at least one of them needs a specific name (did it for wrapper tests)
  • I've checked a second rerun does not run any effective test here: https://github.com/jneira/haskell-language-server/actions/runs/1550353254

@pepeiborra
Copy link
Collaborator

pepeiborra commented Dec 8, 2021

The approach of preserving rerun logs across test runs doesn't seem sound. Counterexample:

  1. I send the empty PR and wait for it to pass all the tests (you would hope)
  2. I send a commit that breaks X tests, but since X had already passed it doesn't rerun.
  3. The commit gets reviewed, approved and merged
  4. HEAD is now broken

Let me know if I am misunderstanding how it works

@jneira
Copy link
Member Author

jneira commented Dec 8, 2021

Let me know if I am misunderstanding how it works

Sorry i missed to stress the more important thing: the cache is v1-${{ runner.os }}-${{ matrix.ghc }}-test-log-${{ github.sha }} so it is per os, ghc version and commit. If the test suite passes for a concrete commit i think it is quite safe to skip it in the next attempt. It only can be affected by external things like dependencies (and we have pinned the hackage index) or other things which might be transient.

I mentioned it in the original pr #2408 which was closed and included in #2393:

  • To try to alliviate flakiness. Even if we get to reduce the number of flaky tests i think we could keep this as tests always can fail due to external factors
  • The trick is cache .tasty-rerun-log per commit and workflow matrix. So f.e. linux+ghc-8.6.5+commit is succesful and windows+ghc-9.0.1 fails the previous one will be succesfull without actually running any test (which might fail this time)
  • Test options are --rerun-update --rerun-filter failures,exceptions,new, if there is no .tasty-rerun-log, the situation in the first attempt of the run, it will execute all tests. All succesives test executions will filter out the succesful ones, even between different run attempts

@jneira jneira added the merge me Label to trigger pull request merge label Dec 8, 2021
@mergify mergify bot merged commit ddacbce into haskell:master Dec 8, 2021
pepeiborra pushed a commit that referenced this pull request Dec 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
merge me Label to trigger pull request merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants