Use 3.11 for Github actions #6551

Pierre-Sassoulas · 2022-05-08T18:35:18Z

Type of Changes

	Type
✓	✨ New feature
✓	📜 Docs

Description

Closes #5920

DanielNoord · 2022-05-08T20:31:46Z

This is still broken. dill isn't 3.11 compatible yet. You can test this locally, but for some reason in Github Actions we don't fail but instead hang.

DanielNoord · 2022-05-22T16:29:08Z

dill has released a new version but we still need an astroid update for these tests to pass.

Pierre-Sassoulas · 2022-06-18T19:10:13Z

dill is still broken it seems.

DanielNoord · 2022-06-18T20:10:47Z

They are releasing at the end of June.

hugovk · 2022-06-22T11:16:07Z

py311 also needs adding to tox.ini

Pierre-Sassoulas · 2022-06-27T17:55:33Z

Milestone for the required release of dill : https://github.com/uqfoundation/dill/milestone/17

Pierre-Sassoulas · 2022-06-29T07:21:04Z

Re-trying as dill 0.3.5.1 was released and it look like the milestone we had to follow is not the one I thougth.

Pierre-Sassoulas · 2022-06-29T07:31:42Z

We do need the next version of dill.

DanielNoord · 2022-06-29T07:32:43Z

We also need astroid 2.12 😄

AdamWill · 2022-07-11T16:13:08Z

There's an astroid 2.12.1 now, which passes all its own tests on Python 3.11. Current git HEAD dill is good with 3.11, but a release hasn't been tagged yet.

AdamWill · 2022-07-11T16:51:07Z

With a recent snapshot of dill, and astroid 2.12.1, I'm down to 43 failures in the tests when building pylint 2.14.4:

https://kojipkgs.fedoraproject.org/work/tasks/1561/89381561/build.log

edit: the ones related to deprecated modules/methods are likely downstream-specific (we exclude similar tests from 3.7 and 3.8 in our pytest command), others are more likely to also affect upstream.

DanielNoord · 2022-07-11T17:09:18Z

We are working on getting support for astroid 2.12.x in pylint, progress of which you can track in #7153.

dill seems to have missed its release date on the 27th on June. I thought about sending them a reminder, but I think they know we're waiting on their release so I don't want to bug them too much.

AdamWill · 2022-07-11T19:12:10Z

OK, so with new astroid and dill and #7153 , here's some really-python3.11 issues:

telnetlib is deprecated in 3.11, that breaks access_attr_before_def_false_positive.py
asyncio.coroutine was removed in 3.11, that breaks iterable_context_py3.py
I think due to changes in Enum upstream, we're now getting an unexpected super-init-not-called message for enum_self_defined_member_5138

AdamWill · 2022-07-11T21:35:11Z

Hmm, this is a fun one. the pylint output for the syntax_error.py test has changed, which breaks both that test and import_error (which imports it). It changed from just "invalid syntax" to "expected '('". The new error seems if anything better? But not sure why it changed, or how to allow the output to differ across Python versions (if that's desired).

DanielNoord · 2022-07-11T21:37:57Z

Hmm, this is a fun one. the pylint output for the syntax_error.py test has changed, which breaks both that test and import_error (which imports it). It changed from just "invalid syntax" to "expected '('". The new error seems if anything better? But not sure why it changed, or how to allow the output to differ across Python versions (if that's desired).

This is an annoying issue which I encountered before. We should probably merge #7097 first which deals with a similar issue and then any PR that moves syntax-error functional tests to unittests is much appreciated!

AdamWill · 2022-07-11T21:39:56Z

Thanks. I'm trying to work on fixes for the things I'm sure are python 3.11 (and not just general running-the-tests-downstream stuff, or already-fixed-since-2.14.4 stuff), but I don't know the codebase very well so I may be doing it wrong. :P I'll send a PR when I'm done.

DanielNoord · 2022-07-11T21:44:00Z

Thanks. I'm trying to work on fixes for the things I'm sure are python 3.11 (and not just general running-the-tests-downstream stuff, or already-fixed-since-2.14.4 stuff), but I don't know the codebase very well so I may be doing it wrong. :P I'll send a PR when I'm done.

Much appreciated! If you need any direction feel free to ping me, that can also be on your local fork. It's often much easier with respect to time and effort for me (or other maintainers) to comment on a WIP PR than create one ourselves.

AdamWill · 2022-07-11T21:47:39Z

Thanks! The fork is https://github.com/AdamWill/pylint/commits/py311 , with my attempts at fixes for the telnetlib and asyncio.coroutine issues so far. I'll send a PR when I'm done looking through the other few failures that are left.

edit: oh, and yeah, the syntaxerror change is indeed inherited from Python itself:

[adamw@xps13k pylint (py311)]$ python3 tests/functional/s/syntax/syntax_error.py 
  File "/home/adamw/local/pylint/tests/functional/s/syntax/syntax_error.py", line 1
    def toto # [syntax-error]
             ^^^^^^^^^^^^^^^^
SyntaxError: expected '('
[adamw@xps13k pylint (py311)]$ python3.10 tests/functional/s/syntax/syntax_error.py 
  File "/home/adamw/local/pylint/tests/functional/s/syntax/syntax_error.py", line 1
    def toto # [syntax-error]
             ^^^^^^^^^^^^^^^^
SyntaxError: invalid syntax

so I guess I'll look at your idea of moving it and see if I can do that.

cclauss · 2022-09-25T05:57:11Z

My mistake. On Windoze, it is . venv\\Scripts\\activate

cclauss · 2022-09-25T07:47:34Z

We need a Primer / prepare / 3.7 / Linux (pull_request) if we want Primer / Run / Run / 3.7 (pull_request) to pass.

I think that line 2 of .github/workflows/primer_run_main.yaml can be removed when comparing the URL to the current action.

Pierre-Sassoulas · 2022-09-29T13:06:22Z

I actually did not dare to create our own version 😄 Amazing !

DanielNoord · 2022-09-29T13:09:15Z

@Pierre-Sassoulas I think this is ready for review 😄

I did some other clean up too. Let's use 3.11 everywhere we can, might as well help the testing efforts by checking if other jobs in our CI still pass with it.

DanielNoord · 2022-09-30T10:44:30Z

Failures are expected. Everything else passes, which is also expected. The failures for 3.11 were already fixed in astroid.

cclauss · 2022-09-30T10:46:56Z

Failures expected: --> #6551 (comment)

DanielNoord · 2022-09-30T10:54:10Z

I think that line 2 of .github/workflows/primer_run_main.yaml can be removed when comparing the URL to the current action.

Which line are you referring to here?

We need a Primer / prepare / 3.7 / Linux (pull_request) if we want Primer / Run / Run / 3.7 (pull_request) to pass.

This is not completely true. We need a run on main. This is because we want main to create the venv to make sure that any commits on astroid main are "taken" by the primer run on main and then "distributed" to PRs rather than having PRs use a new version of astroid compared to the main run they are compared against.
This means that any change to the primer cache, workflow, etc. on PRs will always fail to execute as those changes need to be on pylint main first. This is (highly) annoying but I haven't come up with a good solution yet that 1) only allows main to create venvs and 2) exits gracefully on PRs when there is a discrepancy. I don't think there is a good solution for it, sadly.

cclauss · 2022-09-30T11:01:01Z

Which line are you referring to here?

Line 2.

Should the failing test Primer / Run / Run / 3.7 (pull_request) be dropped?

DanielNoord · 2022-09-30T11:17:42Z

This is L2 of the file you mentioned:

https://github.com/PyCQA/pylint/blob/aca8dd546e6ec03d169eb84d7421d4a1427920ce/.github/workflows/primer_run_main.yaml#L2

I don't understand what it has to do with the rest of your comment, sorry!

Should the failing test Primer / Run / Run / 3.7 (pull_request) be dropped?

Why should it? See my explanation above, it works for all other PRs. See also:
https://github.com/PyCQA/pylint/blob/aca8dd546e6ec03d169eb84d7421d4a1427920ce/.github/workflows/primer_run_pr.yaml#L14-L18

We ignore it where possible, but these ignores are not exhaustive. So if there is any file in the PR that is not being ignored then the workflow will still run.

cclauss

LGTM

DanielNoord · 2022-09-30T11:25:32Z

.github/workflows/checks.yaml

@@ -8,8 +8,8 @@ on:
  pull_request: ~

 env:
-  CACHE_VERSION: 27
-  DEFAULT_PYTHON: "3.10"
+  CACHE_VERSION: 29


@Pierre-Sassoulas Btw, we could also make this a repository secret. It would mean that you need to update it in the settings, but it might be easier than having to do so in PRs every time.
Same goes for DEFAULT_PYTHON and PRE_COMMIT_CACHE.

I'd rather make it accessible for everyone for now because I don't want to freeze pylint development if I'm not available. It would help if PyCQA was more democratic and I could add you to the admin group.

Yeah, but for some reason I feel like for pylint this is different. We also struggle with match ... case statements but we still support that version according to our classifier. For astroid I feel like we don't really introspect the ast fully but for pylint we do offer our current functionality completely on 3.11.
The objectives are a bit different, if you understand what I mean. But I'll leave this to you to decide!

I think you're answering the other comment about package metadata ? Let's add the classifier then.

Ah yeah, my bad!

doc/whatsnew/fragments/5920.other

Co-authored-by: Daniël van Noord <13665637+DanielNoord@users.noreply.github.com>

Pierre-Sassoulas · 2022-09-30T12:10:15Z

Shall we backport too ? Maybe release 2.16 immediately ?

DanielNoord · 2022-09-30T12:12:55Z

I have some free time today so I was thinking of tackling some regressions. Notably #7003 which is blocked by #7496.

Either way, let's not release on Friday. That screwed us a bit with 2.15.

Pierre-Sassoulas · 2022-09-30T14:08:01Z

Python 3.11.0 should ship on Wednesday so why not wait until after the tests pass on that?

Because then pylint would not be ready when 3.11 is out 😄 Release before Wednesday would be nice, especially considering Daniel created our own version of dill in order to be ready on time.

cclauss · 2022-09-30T14:09:24Z

My mistake -- Python 3.11 official release is planned for Monday, 2022-10-24.

Use ``dill-pylint`` (our own version of dill) because dill's 0.3.6 release is taking forever. Closes pylint-dev#5920 Co-authored-by: Daniël van Noord <13665637+DanielNoord@users.noreply.github.com>

Pierre-Sassoulas added Maintenance Discussion or action around maintaining pylint or the dev workflow python 3.11 labels May 8, 2022

Pierre-Sassoulas force-pushed the add-3.11-to-CI branch from 931cb02 to edc75cf Compare May 8, 2022 18:36

Pierre-Sassoulas added the Blocked 🚧 Blocked by a particular issue label May 8, 2022

Pierre-Sassoulas force-pushed the add-3.11-to-CI branch from 266a2a9 to e039e4c Compare May 8, 2022 21:09

Pierre-Sassoulas force-pushed the add-3.11-to-CI branch from e039e4c to c56560b Compare June 18, 2022 18:55

Pierre-Sassoulas force-pushed the add-3.11-to-CI branch from c56560b to c9738a4 Compare June 18, 2022 19:14

Pierre-Sassoulas mentioned this pull request Jun 18, 2022

AttributeError: 'code' object has no attribute 'co_endlinetable' with python 3.11 uqfoundation/dill#514

Closed

Pierre-Sassoulas force-pushed the add-3.11-to-CI branch from c9738a4 to 58d9198 Compare June 22, 2022 11:33

Pierre-Sassoulas mentioned this pull request Jun 27, 2022

2.11.6 creates pylint regression pylint-dev/astroid#1669

Closed

Pierre-Sassoulas force-pushed the add-3.11-to-CI branch from 58d9198 to e813226 Compare June 29, 2022 07:20

Pierre-Sassoulas added the Needs astroid update Needs an astroid update (probably a release too) before being mergable label Jun 29, 2022

Pierre-Sassoulas force-pushed the add-3.11-to-CI branch from a075731 to 1166297 Compare September 25, 2022 05:39

Pierre-Sassoulas added 2 commits September 25, 2022 08:52

[github action] Upgrade cache version previous to dill upgrade

f8b9996

[github action] Use the latest dill in the CI for 3.11-dev

7d6667d

Pierre-Sassoulas force-pushed the add-3.11-to-CI branch from 1166297 to 7d6667d Compare September 25, 2022 06:52

DanielNoord added 2 commits September 29, 2022 14:54

Use dill-pylint

d517799

Like this then?

4c6c9ac

Some other fixes

fd316f0

Pierre-Sassoulas removed the Blocked 🚧 Blocked by a particular issue label Sep 29, 2022

cclauss previously approved these changes Sep 30, 2022

View reviewed changes

DanielNoord requested changes Sep 30, 2022

View reviewed changes

Update doc/whatsnew/fragments/5920.other

593b591

Co-authored-by: Daniël van Noord <13665637+DanielNoord@users.noreply.github.com>

Pierre-Sassoulas dismissed cclauss’s stale review via 593b591 September 30, 2022 11:34

Add classifier

48ba923

DanielNoord approved these changes Sep 30, 2022

View reviewed changes

Pierre-Sassoulas merged commit 644f3fd into main Sep 30, 2022

Pierre-Sassoulas deleted the add-3.11-to-CI branch September 30, 2022 19:56

Use 3.11 for Github actions #6551

Use 3.11 for Github actions #6551

Conversation

Pierre-Sassoulas commented May 8, 2022

Type of Changes

Description

DanielNoord commented May 8, 2022

DanielNoord commented May 22, 2022

Pierre-Sassoulas commented Jun 18, 2022

DanielNoord commented Jun 18, 2022

hugovk commented Jun 22, 2022

Pierre-Sassoulas commented Jun 27, 2022

Pierre-Sassoulas commented Jun 29, 2022 • edited Loading

Pierre-Sassoulas commented Jun 29, 2022

DanielNoord commented Jun 29, 2022

AdamWill commented Jul 11, 2022 • edited Loading

AdamWill commented Jul 11, 2022 • edited Loading

DanielNoord commented Jul 11, 2022

AdamWill commented Jul 11, 2022

AdamWill commented Jul 11, 2022

DanielNoord commented Jul 11, 2022

AdamWill commented Jul 11, 2022 • edited Loading

DanielNoord commented Jul 11, 2022

AdamWill commented Jul 11, 2022 • edited Loading

cclauss commented Sep 25, 2022

cclauss commented Sep 25, 2022 • edited Loading

Pierre-Sassoulas commented Sep 29, 2022

DanielNoord commented Sep 29, 2022

DanielNoord commented Sep 30, 2022

cclauss commented Sep 30, 2022

DanielNoord commented Sep 30, 2022

cclauss commented Sep 30, 2022

DanielNoord commented Sep 30, 2022

cclauss left a comment

Choose a reason for hiding this comment

DanielNoord Sep 30, 2022

Choose a reason for hiding this comment

Pierre-Sassoulas Sep 30, 2022

Choose a reason for hiding this comment

DanielNoord Sep 30, 2022

Choose a reason for hiding this comment

Pierre-Sassoulas Sep 30, 2022

Choose a reason for hiding this comment

DanielNoord Sep 30, 2022

Choose a reason for hiding this comment

Pierre-Sassoulas commented Sep 30, 2022

DanielNoord commented Sep 30, 2022

Pierre-Sassoulas commented Sep 30, 2022

cclauss commented Sep 30, 2022 • edited Loading

Pierre-Sassoulas commented Jun 29, 2022 •

edited

Loading

AdamWill commented Jul 11, 2022 •

edited

Loading

AdamWill commented Jul 11, 2022 •

edited

Loading

AdamWill commented Jul 11, 2022 •

edited

Loading

AdamWill commented Jul 11, 2022 •

edited

Loading

cclauss commented Sep 25, 2022 •

edited

Loading

cclauss commented Sep 30, 2022 •

edited

Loading