Reduce/remove files from the exclusion list of pre-commit, prospector, yapf, pylint #2207

giovannipizzi · 2018-11-16T08:47:15Z

We now still have a quite long blacklist of files not to be checked by pre-commit hooks (yapf, prospector).
Most of them only have 1/2 things to fix.
A simple script to check how many errors there are is the following (run it in a py3 venv with prospector installed):

import yaml, json, os
from subprocess import run, DEVNULL, PIPE

with open('.pre-commit-config.yaml') as f:
    data = yaml.load(f)

assert data[0]['hooks'][0]['id'] == 'yapf'
files = [_.strip() for _ in data[0]['hooks'][0]['exclude'].split('(')[2].split(')')[0].replace('\n', '').split("|") if '*' not in _]

for file in files:
    if not os.path.isfile(file):
        print("REMOVE ->", file)

print("*"*72)
for file in files[:2]:
    response = run(['prospector', '-o', 'json', file], stdout=PIPE, stderr=DEVNULL, encoding='utf8')
    response_data = json.loads(response.stdout)
    print("{} {}".format(file, response_data['summary']['message_count']))

(of course it can be changed to also show the specific errors etc.)

The idea is that we can already remove and fix most of those with few problems.
As once put in there they will always be checked and must be ok, it's better to first try to remove as many as possible of those that are simple to fix, and leave the complicated ones at the end.

giovannipizzi · 2018-11-16T08:48:11Z

Quick statistics out of my current branch (columns: line number, filename, number of messages);

1 aiida/backends/djsite/db/subtests/djangomigrations.py 
2 aiida/backends/tests/control/test_computer_ctrl.py 
3 aiida/backends/tests/work/utils.py 
4 aiida/orm/authinfo.py 
5 aiida/orm/backend.py 
6 aiida/orm/implementation/calculation.py 
7 aiida/orm/node.py 
8 utils/create_requirements.py 
9 .ci/test_fixtures.py 0
10 aiida/backends/tests/cmdline/commands/test_devel.py 0
11 aiida/backends/tests/cmdline/commands/test_export.py 0
12 aiida/backends/tests/cmdline/commands/test_work.py 0
13 aiida/calculations/plugins/__init__.py 0
14 aiida/control/tests/test_postgres.py 0
15 aiida/parsers/plugins/templatereplacer/__init__.py 0
16 aiida/scheduler/__init__.py 0
17 setup.py 0
18 aiida/backends/__init__.py 1
19 aiida/backends/djsite/__init__.py 1
20 aiida/backends/djsite/db/__init__.py 1
21 aiida/backends/djsite/db/admin.py 1
22 aiida/backends/djsite/db/management/__init__.py 1
23 aiida/backends/djsite/db/subtests/__init__.py 1
24 aiida/backends/djsite/db/views.py 1
25 aiida/backends/djsite/settings/__init__.py 1
26 aiida/backends/djsite/settings/settings_profile.py 1
27 aiida/backends/djsite/settings/wsgi.py 1
28 aiida/backends/profile.py 1
29 aiida/backends/sqlalchemy/models/__init__.py 1
30 aiida/backends/sqlalchemy/tests/__init__.py 1
31 aiida/backends/sqlalchemy/tests/utils.py 1
32 aiida/backends/tests/cmdline/commands/test_calculation.py 1
33 aiida/backends/tests/cmdline/commands/test_group.py 1
34 aiida/backends/tests/test_plugin_loader.py 1
35 aiida/backends/tests/utils/fixtures.py 1
36 aiida/backends/tests/work/daemon.py 1
37 aiida/common/__init__.py 1
38 aiida/common/additions/backup_script/__init__.py 1
39 aiida/common/additions/config_migrations/_utils.py 1
40 aiida/common/exceptions.py 1
41 aiida/common/ipython/__init__.py 1
42 aiida/daemon/timestamps.py 1
43 aiida/orm/calculation/__init__.py 1
44 aiida/orm/calculation/job/__init__.py 1
45 aiida/orm/data/error.py 1
46 aiida/orm/data/float.py 1
47 aiida/orm/data/int.py 1
48 aiida/orm/data/str.py 1
49 aiida/orm/implementation/django/__init__.py 1
50 aiida/orm/implementation/general/__init__.py 1
51 aiida/orm/implementation/sqlalchemy/__init__.py 1
52 aiida/orm/implementation/sqlalchemy/querytool.py 1
53 aiida/plugins/info.py 1
54 aiida/tools/__init__.py 1
55 aiida/tools/data/__init__.py 1
56 aiida/tools/dbexporters/__init__.py 1
57 aiida/tools/dbimporters/plugins/__init__.py 1
58 aiida/workflows/__init__.py 1
59 aiida/workflows/user/__init__.py 1
60 aiida/backends/settings.py 2
61 aiida/backends/tests/cmdline/commands/test_comment.py 2
62 aiida/backends/tests/cmdline/params/types/test_plugin.py 2
63 aiida/common/additions/config_migrations/__init__.py 2
64 aiida/common/links.py 2
65 aiida/orm/implementation/django/backend.py 2
66 aiida/orm/implementation/sqlalchemy/backend.py 2
67 aiida/orm/log.py 2
68 aiida/orm/utils/remote.py 2
69 aiida/parsers/__init__.py 2
70 aiida/parsers/exceptions.py 2
71 aiida/parsers/parser.py 2
72 aiida/scheduler/plugins/direct.py 2
73 aiida/settings.py 2
74 aiida/sphinxext/tests/test_workchain.py 2
75 aiida/sphinxext/tests/workchain_source/demo_workchain.py 2
76 aiida/tools/dbimporters/__init__.py 2
77 aiida/utils/__init__.py 2
78 aiida/utils/cli/__init__.py 2
79 aiida/utils/error_accumulator.py 2
80 aiida/work/run.py 2
81 aiida/backends/djsite/db/migrations/0003_add_link_type.py 3
82 aiida/backends/djsite/db/migrations/0004_add_daemon_and_uuid_indices.py 3
83 aiida/backends/djsite/db/migrations/0005_add_cmtime_indices.py 3
84 aiida/backends/djsite/db/migrations/0006_delete_dbpath.py 3
85 aiida/backends/djsite/db/migrations/0008_code_hidden_to_extra.py 3
86 aiida/backends/djsite/db/migrations/0009_base_data_plugin_type_string.py 3
87 aiida/backends/djsite/db/migrations/0010_process_type.py 3
88 aiida/backends/djsite/db/migrations/0011_delete_kombu_tables.py 3
89 aiida/backends/djsite/db/migrations/0012_drop_dblock.py 3
90 aiida/backends/djsite/db/migrations/0013_django_1_8.py 3
91 aiida/backends/djsite/db/migrations/0016_code_sub_class_of_data.py 3
92 aiida/backends/djsite/db/migrations/0017_drop_dbcalcstate.py 3
93 aiida/backends/djsite/globalsettings.py 3
94 aiida/backends/sqlalchemy/queries.py 3
95 aiida/backends/tests/cmdline/params/types/test_computer.py 3
96 aiida/backends/tests/cmdline/params/types/test_data.py 3
97 aiida/backends/tests/cmdline/params/types/test_group.py 3
98 aiida/backends/tests/cmdline/params/types/test_node.py 3
99 aiida/backends/tests/common/test_datastructures.py 3
100 aiida/backends/tests/example_helpers.py 3
101 aiida/backends/tests/orm/data/remote.py 3
102 aiida/backends/tests/orm/log.py 3
103 aiida/backends/tests/work/class_loader.py 3
104 aiida/common/lang.py 3
105 aiida/common/profile.py 3
106 aiida/common/test_logging.py 3
107 aiida/common/test_utils.py 3
108 aiida/orm/data/bool.py 3
109 aiida/orm/data/singlefile.py 3
110 aiida/orm/implementation/general/comment.py 3
111 aiida/orm/implementation/sqlalchemy/node.py 3
112 aiida/orm/utils/__init__.py 3
113 aiida/tools/dbimporters/plugins/tcod.py 3
114 aiida/utils/find_folder.py 3
115 setup_requirements.py 3
116 aiida/backends/djsite/db/migrations/__init__.py 4
117 aiida/backends/sqlalchemy/models/authinfo.py 4
118 aiida/backends/sqlalchemy/models/comment.py 4
119 aiida/backends/tests/cmdline/commands/test_code.py 4
120 aiida/backends/tests/cmdline/params/types/test_code.py 4
121 aiida/backends/tests/cmdline/params/types/test_workflow.py 4
122 aiida/backends/tests/orm/mixins.py 4
123 aiida/backends/tests/work/test_futures.py 4
124 aiida/common/additions/config_migrations/_migrations.py 4
125 aiida/common/example_helpers.py 4
126 aiida/orm/calculation/inline.py 4
127 aiida/orm/code.py 4
128 aiida/orm/data/folder.py 4
129 aiida/orm/group.py 4
130 aiida/orm/implementation/django/code.py 4
131 aiida/orm/implementation/sqlalchemy/code.py 4
132 aiida/orm/workflow.py 4
133 aiida/parsers/plugins/templatereplacer/doubler.py 4
134 aiida/scheduler/plugins/sge.py 4
135 aiida/utils/test_timezone.py 4
136 utils/validate_pyproject.py 4
137 aiida/backends/djsite/db/migrations/0014_add_node_uuid_unique_constraint.py 5
138 aiida/backends/djsite/db/migrations/0015_invalidating_node_hash.py 5
139 aiida/backends/sqlalchemy/models/user.py 5
140 aiida/backends/sqlalchemy/models/utils.py 5
141 aiida/backends/testimplbase.py 5
142 aiida/backends/tests/daemon/test_client.py 5
143 aiida/backends/tests/test_caching_config.py 5
144 aiida/backends/tests/work/persistence.py 5
145 aiida/common/additions/config_migrations/test_migrations.py 5
146 aiida/orm/data/frozendict.py 5
147 aiida/orm/implementation/django/log.py 5
148 aiida/plugins/factory.py 5
149 aiida/scheduler/datastructures.py 5
150 aiida/utils/which.py 5
151 aiida/backends/djsite/db/migrations/0002_db_state_change.py 6
152 aiida/backends/djsite/db/migrations/0007_update_linktypes.py 6
153 aiida/backends/sqlalchemy/globalsettings.py 6
154 aiida/backends/sqlalchemy/migrations/versions/a514d673c163_drop_dblock.py 6
155 aiida/backends/sqlalchemy/migrations/versions/a6048f0ffca8_update_linktypes.py 6
156 aiida/backends/sqlalchemy/migrations/versions/f9a69de76a9a_delete_kombu_tables.py 6
157 aiida/backends/sqlalchemy/models/group.py 6
158 aiida/backends/sqlalchemy/models/log.py 6
159 aiida/backends/sqlalchemy/tests/migration_test/versions/b947a8821295_add_a_column.py 6
160 aiida/backends/tests/cmdline/params/types/test_calculation.py 6
161 aiida/backends/tests/computer.py 6
162 aiida/common/additions/backup_script/backup_sqlalchemy.py 6
163 aiida/common/extendeddicts.py 6
164 aiida/daemon/runner.py 6
165 aiida/orm/data/orbital.py 6
166 aiida/orm/implementation/sqlalchemy/comment.py 6
167 aiida/tools/data/array/kpoints/__init__.py 6
168 aiida/tools/data/structure/__init__.py 6
169 aiida/tools/dbimporters/plugins/nninc.py 6
170 .ci/polish/workchain.py 7
171 aiida/backends/djsite/db/migrations/0001_initial.py 7
172 aiida/backends/sqlalchemy/__init__.py 7
173 aiida/backends/sqlalchemy/migrations/versions/162b99bca4a2_drop_dbcalcstate.py 7
174 aiida/backends/tests/backup_setup_script.py 7
175 aiida/backends/tests/cmdline/commands/test_user.py 7
176 aiida/backends/tests/work/run.py 7
177 aiida/backends/tests/work/test_runners.py 7
178 aiida/orm/data/array/xy.py 7
179 aiida/orm/data/base.py 7
180 aiida/orm/data/numeric.py 7
181 aiida/orm/data/parameter.py 7
182 aiida/orm/implementation/django/utils.py 7
183 aiida/orm/implementation/sqlalchemy/log.py 7
184 aiida/scheduler/plugins/slurm.py 7
185 aiida/sphinxext/workchain.py 7
186 aiida/tools/dbimporters/plugins/oqmd.py 7
187 aiida/backends/djsite/cmdline.py 8
188 aiida/backends/djsite/manage.py 8
189 aiida/backends/sqlalchemy/migrations/versions/6c629c886f84_process_type.py 8
190 aiida/backends/sqlalchemy/migrations/versions/a603da2cc809_code_sub_class_of_data.py 8
191 aiida/backends/sqlalchemy/models/computer.py 8
192 aiida/backends/sqlalchemy/tests/migration_test/versions/470e57bc0936_create_account_table.py 8
193 aiida/backends/tests/cmdline/params/types/test_identifier.py 8
194 aiida/backends/tests/work/test_process_spec.py 8
195 aiida/common/additions/backup_script/backup_django.py 8
196 aiida/parsers/plugins/arithmetic/add.py 8
197 aiida/scheduler/plugins/lsf.py 8
198 aiida/scheduler/plugins/test_direct.py 8
199 aiida/transport/plugins/test_ssh.py 8
200 aiida/utils/capturing.py 8
201 aiida/utils/delete_nodes.py 8
202 .ci/polish/expression.py 9
203 aiida/backends/djsite/db/subtests/migrations.py 9
204 aiida/backends/sqlalchemy/migrations/versions/0aebbeab274d_base_data_plugin_type_string.py 9
205 aiida/backends/sqlalchemy/migrations/versions/5d4d844852b6_invalidating_node_hash.py 9
206 aiida/backends/sqlalchemy/migrations/versions/89176227b25_add_indexes_to_dbworkflowdata_table.py 9
207 aiida/backends/tests/cmdline/commands/test_workflow.py 9
208 aiida/backends/tests/work/test_launch.py 9
209 aiida/orm/__init__.py 9
210 aiida/orm/implementation/django/authinfo.py 9
211 aiida/orm/implementation/sqlalchemy/authinfo.py 9
212 aiida/plugins/entry.py 9
213 aiida/sphinxext/tests/conftest.py 9
214 aiida/tools/dbimporters/plugins/mpod.py 9
215 aiida/transport/plugins/test_local.py 9
216 aiida/backends/sqlalchemy/alembic_manage.py 10
217 aiida/backends/sqlalchemy/migrations/versions/62fe0d36de90_add_node_uuid_unique_constraint.py 10
218 aiida/backends/testbase.py 10
219 aiida/backends/tests/__init__.py 10
220 aiida/backends/tests/daemon.py 10
221 aiida/common/constants.py 10
222 aiida/orm/implementation/django/comment.py 10
223 .ci/workchains.py 11
224 aiida/backends/sqlalchemy/models/settings.py 11
225 aiida/calculations/plugins/templatereplacer.py 11
226 aiida/common/caching.py 11
227 aiida/common/orbital/__init__.py 11
228 aiida/orm/implementation/__init__.py 11
229 aiida/orm/implementation/general/group.py 11
230 aiida/plugins/loader.py 11
231 aiida/plugins/registry.py 11
232 aiida/utils/queries.py 11
233 aiida/utils/timezone.py 11
234 aiida/backends/sqlalchemy/migrations/versions/59edaf8a8b79_adding_indexes_and_constraints_to_the_.py 12
235 aiida/backends/sqlalchemy/models/base.py 12
236 aiida/backends/tests/work/test_process_builder.py 12
237 aiida/common/datastructures.py 12
238 aiida/common/ipython/ipython_magics.py 12
239 aiida/orm/implementation/django/user.py 12
240 aiida/orm/implementation/sqlalchemy/user.py 12
241 aiida/plugins/utils.py 12
242 fastentrypoints.py 12
243 aiida/backends/sqlalchemy/cmdline.py 13
244 aiida/backends/sqlalchemy/migrations/versions/35d4ee9a1b0e_code_hidden_attr_to_extra.py 13
245 aiida/backends/sqlalchemy/migrations/versions/70c7d732f1b2_delete_dbpath.py 13
246 aiida/backends/sqlalchemy/tests/generic.py 13
247 aiida/backends/sqlalchemy/tests/schema.py 13
248 aiida/backends/tests/dbimporters.py 13
249 aiida/backends/tests/work/test_rmq.py 13
250 aiida/backends/tests/workflows.py 13
251 aiida/orm/data/list.py 13
252 aiida/tools/data/array/kpoints/seekpath.py 13
253 aiida/tools/dbimporters/plugins/mpds.py 13
254 aiida/backends/tests/orm/data/frozendict.py 14
255 aiida/backends/tests/work/test_transport.py 14
256 aiida/calculations/plugins/arithmetic/add.py 14
257 aiida/common/graph.py 14
258 aiida/orm/mixins.py 14
259 aiida/plugins/entry_point.py 14
260 aiida/tools/dbimporters/baseclasses.py 14
261 aiida/utils/cli/options.py 14
262 aiida/backends/sqlalchemy/tests/migrations.py 15
263 aiida/backends/tests/calculation_node.py 15
264 aiida/common/additions/backup_script/backup_setup.py 15
265 aiida/orm/data/remote.py 15
266 .ci/polish/cli.py 16
267 aiida/backends/sqlalchemy/tests/query.py 16
268 aiida/orm/data/__init__.py 16
269 aiida/orm/implementation/sqlalchemy/group.py 16
270 aiida/backends/djsite/queries.py 17
271 aiida/backends/general/abstractqueries.py 17
272 aiida/backends/sqlalchemy/migrations/env.py 17
273 aiida/backends/sqlalchemy/tests/testbase.py 17
274 aiida/backends/tests/cmdline/commands/test_computer.py 17
275 aiida/common/orbital/realhydrogen.py 17
276 aiida/orm/implementation/django/group.py 17
277 aiida/scheduler/plugins/test_lsf.py 17
278 aiida/utils/ascii_vis.py 17
279 aiida/backends/djsite/db/subtests/generic.py 18
280 aiida/backends/tests/orm/utils/loaders.py 18
281 aiida/orm/autogroup.py 18
282 aiida/backends/tests/work/job_processes.py 19
283 aiida/orm/implementation/django/computer.py 19
284 aiida/backends/tests/work/test_utils.py 20
285 aiida/orm/implementation/sqlalchemy/computer.py 20
286 aiida/utils/cli/validators.py 20
287 aiida/backends/sqlalchemy/tests/nodes.py 21
288 aiida/scheduler/plugins/test_pbspro.py 21
289 aiida/tools/dbimporters/plugins/pcod.py 21
290 aiida/workflows/test.py 21
291 aiida/backends/sqlalchemy/tests/session.py 22
292 aiida/sphinxext/tests/workchain_source/conf.py 22
293 aiida/backends/sqlalchemy/models/node.py 23
294 aiida/backends/tests/parsers.py 23
295 aiida/backends/utils.py 23
296 aiida/orm/data/array/kpoints.py 23
297 aiida/scheduler/plugins/test_torque.py 23
298 aiida/common/folders.py 25
299 aiida/tools/dbimporters/plugins/cod.py 25
300 aiida/backends/tests/nodes.py 27
301 aiida/backends/tests/work/process.py 27
302 aiida/common/additions/backup_script/backup_base.py 27
303 aiida/orm/implementation/django/workflow.py 27
304 aiida/scheduler/plugins/test_sge.py 27
305 aiida/orm/data/code.py 28
306 aiida/work/__init__.py 28
307 aiida/daemon/execmanager.py 29
308 aiida/orm/data/array/projection.py 29
309 aiida/backends/djsite/db/subtests/query.py 30
310 aiida/common/archive.py 30
311 aiida/orm/data/upf.py 30
312 aiida/backends/sqlalchemy/utils.py 31
313 aiida/orm/implementation/django/node.py 31
314 aiida/scheduler/plugins/test_slurm.py 31
315 aiida/daemon/workflowmanager.py 33
316 aiida/orm/implementation/sqlalchemy/utils.py 36
317 aiida/backends/sqlalchemy/transition_06dj_to_07sqla.py 39
318 .ci/test_daemon.py 43
319 aiida/backends/sqlalchemy/models/workflow.py 43
320 aiida/orm/implementation/general/node.py 44
321 aiida/backends/djsite/db/subtests/nodes.py 45
322 aiida/orm/utils/loaders.py 45
323 aiida/backends/tests/tcodexporter.py 46
324 aiida/scheduler/plugins/pbsbaseclasses.py 48
325 aiida/scheduler/plugins/pbsbaseclasses.py 48
326 aiida/backends/tests/work/test_workfunctions.py 49
327 aiida/tools/dbexporters/tcod_plugins/__init__.py 50
328 aiida/backends/tests/generic.py 51
329 aiida/workflows/wf_XTiO3.py 54
330 aiida/orm/implementation/sqlalchemy/workflow.py 57
331 aiida/backends/tests/backup_script.py 59
332 aiida/orm/implementation/general/workflow.py 59
333 aiida/work/job_processes.py 60
334 aiida/transport/plugins/test_all_plugins.py 62
335 aiida/backends/tests/base_dataclasses.py 63
336 aiida/workflows/wf_demo.py 68
337 aiida/tools/data/array/kpoints/legacy.py 74
338 aiida/backends/sqlalchemy/migrations/versions/e15ef2630a1b_initial_schema.py 77
339 aiida/tools/dbimporters/plugins/icsd.py 80
340 aiida/common/test_extendeddicts.py 83
341 aiida/transport/plugins/ssh.py 89
342 aiida/orm/data/array/bands.py 91
343 aiida/backends/djsite/db/models.py 93
344 aiida/backends/tests/cmdline/commands/test_node.py 96
345 aiida/orm/data/structure.py 98
346 aiida/backends/tests/cmdline/commands/test_data.py 99
347 aiida/orm/querybuilder.py 107
348 aiida/orm/node/process/calculation/calcjob.py 117
349 aiida/tools/dbexporters/tcod.py 131
350 aiida/backends/tests/export_and_import.py 160
351 aiida/orm/importexport.py 162
352 aiida/backends/tests/work/work_chain.py 171
353 aiida/backends/tests/query.py 189
354 aiida/backends/tests/dataclasses.py 262

About 2/3 have <10 messages

giovannipizzi · 2018-12-03T14:37:13Z

Once all files are processed, we should then check if there are leftover 'disable' that are actually not needed

ltalirz · 2020-02-21T22:29:44Z

Actually, this can be a good first issue for anyone willing to help improve code quality - simply

look at the exclude list in the .pre-commit-config.yaml file
pick one file & remove it from the list
pip install -e .[pre-commit] and pre-commit install
git commit -a - and fix the issues reported by pylint

giovannipizzi added the priority/quality-of-life would simplify development label Nov 16, 2018

giovannipizzi self-assigned this Nov 16, 2018

sphuber added this to the v1.1.0 milestone Dec 3, 2018

yakutovicha self-assigned this Dec 9, 2019

yakutovicha mentioned this issue Dec 9, 2019

Remove items from the pre-commit black list #3625

Merged

giovannipizzi mentioned this issue Dec 10, 2019

Fix two issues in verdi computer configure ssh #3636

Merged

sphuber closed this as completed in #3625 Dec 13, 2019

sphuber reopened this Dec 13, 2019

ltalirz added the good first issue Issues that should be relatively easy to fix also for beginning contributors label Feb 21, 2020

sphuber removed this from the v1.1.0 milestone Feb 28, 2020

sphuber mentioned this issue Jun 24, 2020

Remove all files from the pre-commit exclude list #4196

Merged

sphuber closed this as completed in #4196 Jul 1, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduce/remove files from the exclusion list of pre-commit, prospector, yapf, pylint #2207

Reduce/remove files from the exclusion list of pre-commit, prospector, yapf, pylint #2207

giovannipizzi commented Nov 16, 2018

giovannipizzi commented Nov 16, 2018

giovannipizzi commented Dec 3, 2018

ltalirz commented Feb 21, 2020

Reduce/remove files from the exclusion list of pre-commit, prospector, yapf, pylint #2207

Reduce/remove files from the exclusion list of pre-commit, prospector, yapf, pylint #2207

Comments

giovannipizzi commented Nov 16, 2018

giovannipizzi commented Nov 16, 2018

giovannipizzi commented Dec 3, 2018

ltalirz commented Feb 21, 2020