Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration tests are flaky and frequently fail #237

Open
dmikusa opened this issue Jan 1, 2024 · 1 comment
Open

Integration tests are flaky and frequently fail #237

dmikusa opened this issue Jan 1, 2024 · 1 comment
Labels
type:bug A general bug

Comments

@dmikusa
Copy link
Contributor

dmikusa commented Jan 1, 2024

Expected Behavior

  1. Integration tests should consistently pass
  2. We should not have integration tests. It's difficult to mock out other buildpacks and external factors for a reliable integration test. If we can't make it reliable, then we should just remove it.

Current Behavior

See https://github.com/paketo-buildpacks/apache-tomee/actions/runs/7378265147/job/20073179955#step:9:99 as an example.

Possible Solution

  • Make test reliable

OR

  • Remove test

Steps to Reproduce

Pick one of the integration tests & re-run it. It seems to fail around 50% of the time.

Motivations

It causes hassles when evaluating PRs and wastes time. No one needs that.

@dmikusa dmikusa added the type:bug A general bug label Jan 1, 2024
@loewenstein
Copy link

It is fair to evaluate individual test cases for their weight-to-worth ratio. However, I don't think that removing testing entirely - and removing a whole test category like integration tests comes really close to that I believe - is a good idea.

@c0d1ngm0nk3y @pbusko @modulo11 @phil9909 we do run integration tests on component and composite buildpack level and don't experience a failure rate like @dmikusa mentions, right? Do we have anything to share to help improve the test success rate for Paketo?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:bug A general bug
Projects
None yet
Development

No branches or pull requests

2 participants