Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a thorough test on **all** compatible versions #857

Open
johnnychen94 opened this issue Jan 1, 2020 · 1 comment
Open

Add a thorough test on **all** compatible versions #857

johnnychen94 opened this issue Jan 1, 2020 · 1 comment

Comments

@johnnychen94
Copy link
Member

johnnychen94 commented Jan 1, 2020

In #805 I once proposed to test the nightly stage on the master branch of all dependencies, it could help catch some errors before releasing a new version without requiring new CI resources. It turned out that the proposal isn't really useful in practice if we choose to tag our versions frequently.

Recent devs showed that many bugs are due to incompatible versions that we claim compatible. E.g., JuliaImages/ImageCore.jl#110, JuliaGraphics/Colors.jl#390, JuliaGraphics/Colors.jl#383

That's not good news for both users and developers. Code reviews can help catch some of them but to guarantee none of them passes silently, we might need a really thorough test on it.

The plan is simple:

  • generating all possible Manifest.tomls for all versions of dependencies we claim compatible
  • test all of them using CI

The only and really big challenge is we don't have enough CI resources to support this.

There could be thousands of possible combinations if we literally test all of them and it's impractical, hence we need to take some strategy to reduce the number, for example:

  • versions that older than three months are not under consideration
  • if there're many patch versions, only test the latest, e.g., test v0.8.7 and ignore v0.8.0-v0.8.6.
  • only trigger the CI when we trigger the registrator

I need to do some statistics on this strategy and see if it reduces the number to a practical one.

Right now, this isn't an urgent issue, and it might not be the best time to tackle this.

@johnnychen94 johnnychen94 changed the title [RFC] Add a thorough test on **all** compatible versions Add a thorough test on **all** compatible versions Mar 28, 2021
@johnnychen94
Copy link
Member Author

For the record, https://github.com/SciML/SciMLBase.jl/blob/master/.github/workflows/Downstream.yml is a good example to tackle half of this issue; that it can test all compatible Images versions from ImageCore. We may also need a reversed version to test all compatible ImageCore versions from Images.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant