Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test pythons > 3.9 in CI #2164

Merged

Conversation

ADBond
Copy link
Contributor

@ADBond ADBond commented May 2, 2024

This adds runs for python versions 3.10, 3.11, and 3.12 to the backend tests in CI (and so is sort-of a long-delayed follow-up to #1090).

The previous issue with adding these was something to do with numpy versions. It seems that in the intervening time, the issue related to 3.10 + 3.11 has been resolved (I think this is probably to do with us updating the lockfile - I think the version of numpy we had locked had some issues on certain platforms with these versions).

The version of numpy we now have in the lockfile though doesn't work with python 3.12 - it seems that poetry didn't recognise this incompatibility. So I have added numpy explicitly as a dependency - the minimum version (for python < 3.12) is the minimum accepted by our current minimum pandas version. For python 3.12 I have added the minimum numpy release supporting it - and updated the lockfile with these new updates.

I realise this results in quite a number of checks now, but they are fairly quick, so don't think this should be a huge burden. You can see the checks running in this PR on my fork - I have cherry-picked everything from that branch, except for a dummy code change I added to trigger CI.

Couple of other things here:

  • spark tests don't run on 3.12, as pyspark does not work with this version. There is a fix to add this compatibility added to pyspark, but it seems from the discussion there that this will not be arriving before Spark 4
  • I have turned fail-fast off, so that tests failing for one version don't cancel the runs in other versions. I'm not sure there is a huge gain in failing fast, and this way we can spot more easily whether failures are due to a single version, or failing for the backend in general. Previously, for instance, the spark test failed for 3.12, cancelling all the others, which made it unclear whether other versions were affected
  • I considered consolidating these four workflows into a single one, but decided not to as a) was not sure how to deal with the postgres service dependency in one version only, and b) the above pyspark issue makes it a bit more fiddly. We can probably figure out something a bit neater down the line if wanted.

Closes #1496.

Copy link
Member

@RobinL RobinL left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great - thanks

@ADBond ADBond merged commit 1eb4d61 into moj-analytical-services:splink4_dev May 8, 2024
25 checks passed
@ADBond ADBond deleted the test-all-pythons-ci branch May 8, 2024 15:20
@ADBond ADBond mentioned this pull request Jul 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants