Reduce use of blacklist/whitelist terminology

[pkolbus]

Steps

• Add a ChangeLog entry describing what your PR does.
• If it's a new feature or an important bug fix, add a What's New entry in doc/whatsnew/<current release.rst>.
• Write a good description on what the PR does.

Description

The terms whitelist/blacklist are considered problematic/oppressive, and usage runs contrary to many projects' goals of fostering an inclusive and welcoming community. This PR aims to reduce usage in cases and ways that are nonbreaking, and especially to allow pylint users to remove usage of these terms in their projects.

For further discussion of the concern see e.g.:

• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6148600/
• https://tools.ietf.org/id/draft-knodel-terminology-00.html
• https://bugs.python.org/issue41521
• Editorial: consider offering alternative to whitelist_externals tox-dev/tox#1491
• https://open.nytimes.com/finding-the-right-words-491b93b8b668

Type of Changes

	Type
✓	🐛 Bug fix
✓	📜 Docs

Related Issue

Closes #3669

[coveralls]

Coverage decreased (-0.03%) to 90.888% when pulling f0079a4 on pkolbus:inclusive-terminology into 2e21eb4 on PyCQA:master.

[coveralls]

Coverage decreased (-0.006%) to 91.423% when pulling 90af4f3 on pkolbus:inclusive-terminology into 0f1245c on PyCQA:master.

[pkolbus]

Note for reviewers- this makes significant progress but there are still a few aspects I'd still like to address. From my viewpoint, these could be either in this PR or a follow-on. I'm interested in your thoughts, or pointers in these areas:

• astroid has some problematic API names (e.g. AstroidManager.extension_package_whitelist); astroid itself would need a PR then pylint is updated to use the new terms.
• the dest names black_list and black_list_re in PyLinter.make_options(); black_list is noted in documentation for plugins. Can probably do this with some aliasing extensions in the options mixin classes, and use of __getattr__ / __setattr__ overrides.
• the term master (primarily as a pylintrc section name) is also considered problematic. This likely requires work in the options mixins, but I'm less certain how to accomplish this for section/group names.

[Pierre-Sassoulas]

Changing the "master" in the options and configuration would break compatibility with all the pylint configuration files ever written. This would impact a huge amount of person and would make a lot of documentation and question on stackoverflow and everywhere else obsolete. The change on blacklist/whitelist also breaks the options given to pylint (only for those using those two options, so it's less disruptive), but it still need to be done as a breaking change for 3.0. I expect those kind of changes to be included in a big refactor of the configuration and option parser to use something modern like click or argparse and pyproject.toml that would affect pylint, but also pylint integration in emacs and other editor. I think you saw for yourself that the OptionMixin are not the best example of straightforward and elegant code either, so there's also that to take into account. We would need a tool to upgrade the configuration automatically and or a proper documentation for the migration. So this change is not going to be easy nor quick (similar to the one from isort 4 to isort 5). Also, as you said, astroid should be modified too and probably be modified first, maybe let's start here as change to astroid only affect pylint :)

[pkolbus]

@Pierre-Sassoulas, thanks for the feedback.

I do understand that pylint is very widely used, so this PR does aim to preserve backwards compatibility and provide a transition path for users. That would also be true for the unfinished parts. For example, if changing the section name to main, pylint would continue to accept master- or even not deprecating master yet but simply accepting main as an equivalent synonym/alias too. Likewise for config.black_list and config.black_list_re. (If there’s a backward compatibility case that I’ve not accounted for, please point out what no longer works but needs to).

I have not actually studied the mixin code in depth yet, but would certainly withdraw my proposal to add the section name and dest name aliases in this PR if those changes damage the readability of the options mixin classes.

Is 3.0 or the option parser refactoring planned already? If so I’m happy to build on (or possibly contribute to) that work, and am also comfortable with including this PR only for 3.0 if backwards compatibility is in doubt or impossible.

I’ll work on the astroid PR soon.

[Pierre-Sassoulas]

I think most maintainer that had to touch it at some point have it on their radar. The thing is touching the OptionProviderMixin is not something we can do easily, because it's so huge, because all checkers inherit from it and because it's using Optparse that is deprecated in favor of argparse since python 3.2. So it's moving slowly.

We need this refactor for something very useful though : per directory configuration. So, I think the easier way to make that possible, would be to contribute here: https://github.com/PyCQA/pylint/projects/1, and probably this one in particular : #2355

[gpshead]

Thanks for this!

[gpshead]

FWIW, I suggest doing this in bite sized pieces. This is already one bite.

make other PRs for other issues. things like [MASTER] in the config file can be addressed, it's a matter of updating the code to look for multiple possible section names for that. but those belong in their own PR to keep things smaller and reviewable.

I also suggest focusing on external user visible things first. as this PR mostly does.

internal API names inside the code can be their own cleanup if desired (like the astroid api name).

End users see command line flags, config files, and lint warning messages every day. A far fewer number of people look within the code.

[pkolbus]

Thanks. I still intend to continue with this initiative, just haven't found the time recently. If you think this PR is too big of a first bite or anything might be too risky, let me know what should be deferred to later PRs.

[Pierre-Sassoulas]

I reviewed the actual changes and it look like this is not making any breaking changes so it could go in 2.7.0 after a rebase on latest master.

We're never going to break compatibility for the old name. That would be forcing maintenance work on pylint users for political reasons. The deprecation warnings imply that we will do that in the future so I think that they have to be removed.

[pkolbus]

Thanks for the feedback. Aside from the political motivation, there’s also value in having clearer terminology — but regardless of the motive, I agree deprecation is a separate concern and isn’t a requirement here.

Will get back to this shortly.

[pkolbus]

This should be ready for review again. I've dropped the pyreverse commit for now. I think it would be better for a future PR so that this one can focus on the most common user-facing use of this term.

Happy for this to land in 2.8 to not disrupt 2.7.x bug fixing. The fix for #3701 in particular might need this PR gets another rebase, so I'll keep an eye on that.

[Pierre-Sassoulas]

The fact that there was an old "dest": "black_list_re", and no problem in the tests make me fearful. I don't think we have integration test where we launch pylint and test the actual parsing 😨. What do you think ? Apart from that I made some comment on naming but this not very important. Great job !

[Pierre-Sassoulas]

Shouldn't this be changed too ?

[pkolbus]

Renaming the dest names (black_list, black_list_re, extension_pkg_whitelist etc.) could be breaking for plugins and other programmatic access since these are the member names in the config object. doc/how_tos/plugins.rst even sets black_list in the example.

It might be possible to clean this up in a way that's still backwards compatible -- I'll need to study optparse further since I'm not familiar with it. If optparse.Values doesn't provide a built-in way to alias the names, we could probably extend optparse.Values and add some @property annotations or __getattr__/__setattr__ to provide compatibility. Or as you've noted previously, maybe it's time for bigger rework of the config system. But because it won't make a difference to most users, this seems like something to discuss and address in a later issue/PR.

[Pierre-Sassoulas]

Sorry, lack of sleep, you're right, we should not touch it (not before 3.0 at least).

[Pierre-Sassoulas]

Thank you for this merge request, that was a lot of work !

[ashb]

Thanks for picking this up @pkolbus! ❤️

[Talador12]

Quoting @gpshead and continuing my discussion with @Pierre-Sassoulas

things like [MASTER] in the config file can be addressed

It looks like this has not yet been addressed. There has been discussion about this here and on #5465

To change the references to MASTER in the code to support MASTER and MAIN requires:

• 4 references to MASTER in run.py
• 2 references to MASTER in lint_module_test.py
• a constant MAIN_CHECKER_NAME found in constants.py
• tests using a lower case master
• documentation using all forms of master

If we are able to support both MAIN and MASTER, then we can update all docs to MAIN and still be backwards compatible. Changing those six MASTER references should be quick, but having a flexible MAIN_CHECKER_NAME does not look as easy. Only 11 references to the variable itself, but the value is inherited into a web of use cases. From constants.py:

# You probably don't want to change the MAIN_CHECKER_NAME
# This would affect rcfile generation and retro-compatibility
# on all project using [MASTER] in their rcfile.
MAIN_CHECKER_NAME = "master"

This is not something requested for political reasons. It is required in many technology organizations to use this terminology. There has already been great work on this effort, but this MASTER to MAIN request is another that should be handled eventually. It is clear that backwards compatibility to old terminology is required, but defaults should use new terminology going forward.

[Pierre-Sassoulas]

@Talador12 please don't spam older issues. I locked #5465 for a reason.

should be handled eventually
should use new terminology

I will add that no one can order open-source contributor around. We're volunteer and we work on what we want to work. Decisions are reached via consensus between maintainers and active contributors. So you're especially in no position to take decision or give order as you never contributed to pylint and just opened your first issue. So be mindful of your phrasing here. I'm ready to do the changes required so that it's transparent for users. Could you assign me #XXXX, please would work better than what you're currently saying.

You can open a new issue for the master to main change that you researched here (thank you) so it can be discussed and is not lost in this old MR. The fact that we won't do breaking changes until it's painless for users will not be up for discussion.