Pre-commit: move `yapf` and `mypy` config to `pyproject.toml` #4996

sphuber · 2021-06-25T08:06:07Z

The libraries yapf and mypy support the pyproject.toml file for
their configuration starting from v0.31 and v0.9, respectively.

sphuber · 2021-06-25T08:07:13Z

@chrisjsewell to have support for pyproject.toml required updating mypy which incurred a bunch of new errors in the pre-commit. Don't have time to fix this now, but thought to already create the PR

codecov · 2021-06-25T08:24:12Z

Codecov Report

Merging #4996 (e05ef2d) into develop (ead3f39) will decrease coverage by 0.01%.
The diff coverage is 86.21%.

@@             Coverage Diff             @@
##           develop    #4996      +/-   ##
===========================================
- Coverage    80.38%   80.37%   -0.00%     
===========================================
  Files          529      529              
  Lines        36862    36871       +9     
===========================================
+ Hits         29627    29633       +6     
- Misses        7235     7238       +3

Flag	Coverage Δ
django	`74.88% <86.21%> (-<0.01%)`	⬇️
sqlalchemy	`73.78% <86.21%> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
aiida/engine/daemon/execmanager.py	`66.94% <0.00%> (ø)`
aiida/manage/manager.py	`80.88% <75.00%> (-0.58%)`	⬇️
aiida/manage/configuration/profile.py	`94.51% <83.34%> (-0.46%)`	⬇️
aiida/engine/processes/calcjobs/manager.py	`86.73% <100.00%> (ø)`
aiida/orm/nodes/node.py	`96.29% <100.00%> (ø)`
aiida/tools/graph/graph_traversers.py	`88.60% <100.00%> (ø)`
aiida/tools/groups/paths.py	`91.62% <100.00%> (ø)`
aiida/tools/importexport/archive/migrators.py	`86.40% <100.00%> (ø)`
aiida/tools/importexport/common/config.py	`100.00% <100.00%> (ø)`
...ida/tools/importexport/dbimport/backends/common.py	`97.34% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ead3f39...e05ef2d. Read the comment docs.

unkcpz · 2021-07-22T07:14:46Z

Hi @sphuber, since I am here to look at mypy errors, I can try to assistant to complete this PR. Do you already start working on it, or can I directly add more commits into your branch?

sphuber · 2021-07-22T15:42:35Z

@unkcpz I have updated the branch with all that I have currently. It would be great if you could address the mypy errors. Feel free to add commits straight to my branch. Let me know if you don't have rights to do so. Thanks!

unkcpz · 2021-07-26T08:15:49Z

Hi @sphuber, I make some commits to settle the mypy errors.

Except for the following types of mypy complaints are not resolved:

`error: Module has no attribute "Node" [attr-defined]`

And lots of similar no attribute errors which I guess are related to the mypy bug as post at python/mypy#10826. mypy was not able to deduce the correct module from __all__.

incompatible with supertype errors

Similar errors of repository_metadata, _updatable_attributes and _hash_ignored_attributes from many files.The signatures are the attributes of parent class and are being overridden as properties in child class. Referring to python/mypy#4125 for more details.

sphuber

Thanks @unkcpz . Reverted some changes to the Group logic and have some other questions.

sphuber · 2021-07-26T14:43:41Z

aiida/manage/configuration/profile.py

+        backend = self.get_backend()
+        return Repository(backend=backend)
+
+    def get_backend(self) -> 'DiskObjectStoreRepositoryBackend':


I wouldn't call this get_backend. It would be too confusing with the database backend. If we really need this method, a better name would be get_repository_backend. However, why can't we just call get_repository().backend? Is it necessary to have this intermediate method?

a better name would be get_repository_backend

sure!

However, why can't we just call get_repository().backend?

It is because for Profile, get_repository().backend return an AbstractRepositoryBackend which has no attributes such as container defined in it. However, for Profile class, it sticks to the DiskObjectStoreRepositoryBackend, access to rather than the abstract backend class make it possible to deduce the attributes from the backend.

I am not sure if there is any better way to resolve this incompatible mypy error.

sphuber · 2021-07-26T14:46:57Z

aiida/manage/manager.py

+        if self._backend is None:
+            raise ValueError('Could not load the backend.')


Would be better to put this check directly after the logic that is supposed to set it, i.e. before the configure_logging call. Second, I don't think a ValueError is the correct exception for this. That is reserved for validation arguments of a method or function. What seems to be happening in this case, the only reason self._backend can be None is because the backend_type is neither BACKEND_DJANGO nor BACKEND_SQLA. So really we shouldn't have this check but add the else clause and raise an AssertionError because our internal logic is supposed to set these when loading a profile and that is what we are checking here: internal consistency of program logic.

Good point!

sphuber · 2021-07-26T14:47:33Z

aiida/manage/manager.py

@@ -206,7 +209,7 @@ def get_backend(self) -> 'Backend':

        """
        if self._backend is None:
-            self._load_backend()
+            return self._load_backend()


Why is this change necessary?

In order to resolve aiida/manage/manager.py:214: error: Incompatible return value type (got "Optional[Backend]", expected "Backend") [return-value].
I assume although the _load_backend() method actually set the self._backend and will never leave it to None, get_backend does not know it and assert the return value self._backend still can be None. mypy can't make an assumption about what the called function is doing.

Okay, I guess this is the place where the type assertion properly takes effect. We have:

return cast('Backend', self._backend)

In that case, I propose _load_backend method should have no return value.

sphuber · 2021-07-26T14:49:31Z

aiida/tools/importexport/dbimport/backends/django.py

@@ -38,7 +38,7 @@

 def import_data_dj(
    in_path: str,
-    group: Optional[Group] = None,
+    group: Optional[ImportGroup] = None,


I reverted this change as it is backwards incompatible. We are currently allowing users to use a plain Group to store nodes in upon importing. We cannot narrow this restriction down.

sphuber · 2021-07-26T14:50:12Z

aiida/tools/importexport/dbimport/backends/common.py

@@ -66,7 +66,7 @@ def collect_hashkeys(objects, hashkeys):
        container_export.export(set(hashkeys), container_profile, compress=True, callback=callback)


-def _make_import_group(*, group: Optional[ImportGroup], node_pks: List[int]) -> ImportGroup:
+def _make_import_group(*, group: Optional[ImportGroup], node_pks: List[int]) -> Optional[ImportGroup]:


To fix the mypy error related to inconsistent types, we should rather change this to Optional[Group]. I have already done this and reverted your other change. Reason why is explained in another comment

unkcpz

@sphuber thanks for review!

I reverted this change as it is backwards incompatible. We are currently allowing users to use a plain Group to store nodes in upon importing. We cannot narrow this restriction down.

Yes, should rollback. After changing that I found it failed some migrate tests.

unkcpz · 2021-07-27T01:42:48Z

aiida/manage/configuration/profile.py

+        backend = self.get_backend()
+        return Repository(backend=backend)
+
+    def get_backend(self) -> 'DiskObjectStoreRepositoryBackend':


a better name would be get_repository_backend

sure!

However, why can't we just call get_repository().backend?

It is because for Profile, get_repository().backend return an AbstractRepositoryBackend which has no attributes such as container defined in it. However, for Profile class, it sticks to the DiskObjectStoreRepositoryBackend, access to rather than the abstract backend class make it possible to deduce the attributes from the backend.

I am not sure if there is any better way to resolve this incompatible mypy error.

unkcpz · 2021-07-27T02:55:14Z

aiida/manage/manager.py

@@ -206,7 +209,7 @@ def get_backend(self) -> 'Backend':

        """
        if self._backend is None:
-            self._load_backend()
+            return self._load_backend()


In order to resolve aiida/manage/manager.py:214: error: Incompatible return value type (got "Optional[Backend]", expected "Backend") [return-value].
I assume although the _load_backend() method actually set the self._backend and will never leave it to None, get_backend does not know it and assert the return value self._backend still can be None. mypy can't make an assumption about what the called function is doing.

unkcpz · 2021-07-27T03:00:37Z

aiida/manage/manager.py

+        if self._backend is None:
+            raise ValueError('Could not load the backend.')


Good point!

unkcpz · 2021-07-27T06:51:59Z

aiida/manage/configuration/profile.py

        from aiida.repository import Repository
+
+        backend = self.get_backend()


Looking at failed CI, found here is a mistake. In case of disturbing you with too many push to your branch, I will fix this later with more valid commits.

Suggested change

backend = self.get_backend()

backend = self.get_repository_backend()

The libraries `yapf` and `mypy` support the `pyproject.toml` file for their configuration starting from v0.31 and v0.9, respectively. Co-authored-by: Jason Eu <morty.yu@yahoo.com>

chrisjsewell · 2021-07-28T03:36:54Z

And lots of similar no attribute errors which I guess are related to the mypy bug as post at python/mypy#10826. mypy was not able to deduce the correct module from __all__.

Well in fairness to mypy, these __all__ + import * constructs are basically a bit of a nightmare for static analysers.

We have to use pylint: disable=wildcard-import in all these files, which would suggest it is maybe not the best practice, and it also makes it quite difficult to see what APIs are being exposed at each level (#4511)

Additionally, it currently does not work with new default VS Code Python Language Server (Pylance) and so you get no auto-completions, etc.
Although here, following the issues: microsoft/pylance-release#289 -> microsoft/pyright#1877 -> https://github.com/microsoft/pyright/blob/main/docs/typed-libraries.md#library-interface, there is a work-around; to restructure the __all__ construction like:

__all__ = ()
__all__ += calculation.__all__
__all__ += process.__all__ 
__all__ += workflow.__all__

this does not help with mypy though

sphuber · 2021-07-28T10:47:44Z

Ok, well if this is a known problem of mypy, is there not some way to ignore these errors in a simple manner, i.e., without having to add inline disable comments on each respective line?

chrisjsewell · 2021-08-11T00:13:09Z

#5061 updates mypy to v0.901 😄

chrisjsewell · 2021-08-11T15:43:26Z

right lets see where we are at with this

chrisjsewell · 2021-08-11T16:03:36Z

Ok well the Module has no attribute errors are now gone, but yeh for the rest; I would suggest the code changes should go in a separate PR and this one should just be about moving the configuration

sphuber · 2021-08-11T16:15:18Z

I would suggest the code changes should go in a separate PR

What code changes? As far as I know, any code changes applied were simply to address the new errors that were now thrown by the updated version of mypy, so they would be necessary. And there are still a lot of errors left over.

chrisjsewell · 2021-08-11T16:24:14Z

aiida/manage/configuration/profile.py

+        backend = self.get_repository_backend()
+        return Repository(backend=backend)
+
+    def get_repository_backend(self) -> 'DiskObjectStoreRepositoryBackend':


what code changes

well you are introducing completely new methods 😬

chrisjsewell · 2021-08-11T16:26:53Z

any code changes applied were simply to address the new errors that were now thrown by the updated version of mypy

yeh but after #5061, I think these may be the cause of the errors, rather than fixes; this PR no longer actually updates the mypy version, so in principle moving the configuration should result in zero changes to the mypy outcome.

I'm happy to give it a try when I have time

chrisjsewell · 2021-08-11T17:13:31Z

moving the configuration should result in zero changes to the mypy outcome

😅 #5066

sphuber requested a review from chrisjsewell June 25, 2021 08:06

chrisjsewell self-assigned this Jul 21, 2021

unkcpz mentioned this pull request Jul 22, 2021

mypy keeps complaining error: unused "type: ignore" comment in pre-commit #5026

Closed

sphuber force-pushed the fix/yapf-mypy-config branch from 736e8d8 to 0bfd323 Compare July 22, 2021 15:41

unkcpz force-pushed the fix/yapf-mypy-config branch from f300011 to 9a1e2a9 Compare July 26, 2021 09:15

sphuber force-pushed the fix/yapf-mypy-config branch from 13d1996 to 7556343 Compare July 26, 2021 14:41

sphuber commented Jul 26, 2021

View reviewed changes

sphuber force-pushed the fix/yapf-mypy-config branch from 7556343 to 0fe96c8 Compare July 26, 2021 14:57

unkcpz force-pushed the fix/yapf-mypy-config branch from 0fe96c8 to d59452c Compare July 27, 2021 03:17

unkcpz reviewed Jul 27, 2021

View reviewed changes

sphuber force-pushed the fix/yapf-mypy-config branch from d59452c to 53ff345 Compare July 27, 2021 08:41

Pre-commit: move yapf and mypy config to pyproject.toml

c342428

The libraries `yapf` and `mypy` support the `pyproject.toml` file for their configuration starting from v0.31 and v0.9, respectively. Co-authored-by: Jason Eu <morty.yu@yahoo.com>

sphuber force-pushed the fix/yapf-mypy-config branch from 53ff345 to c342428 Compare July 27, 2021 08:46

chrisjsewell mentioned this pull request Aug 10, 2021

♻️ REFACTOR: Make __all__ imports explicit (via pre-commit) #5061

Merged

chrisjsewell added pr/blocked PR is blocked by another PR that should be merged first and removed pr/blocked PR is blocked by another PR that should be merged first labels Aug 11, 2021

Merge branch 'develop' into fix/yapf-mypy-config

e05ef2d

chrisjsewell reviewed Aug 11, 2021

View reviewed changes

chrisjsewell closed this Aug 11, 2021

sphuber deleted the fix/yapf-mypy-config branch August 11, 2021 17:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pre-commit: move `yapf` and `mypy` config to `pyproject.toml` #4996

Pre-commit: move `yapf` and `mypy` config to `pyproject.toml` #4996

sphuber commented Jun 25, 2021

sphuber commented Jun 25, 2021

codecov bot commented Jun 25, 2021 •

edited

Loading

unkcpz commented Jul 22, 2021

sphuber commented Jul 22, 2021

unkcpz commented Jul 26, 2021

sphuber left a comment

sphuber Jul 26, 2021

unkcpz Jul 27, 2021

sphuber Jul 26, 2021

unkcpz Jul 27, 2021

sphuber Jul 26, 2021

unkcpz Jul 27, 2021 •

edited

Loading

unkcpz Jul 27, 2021

sphuber Jul 26, 2021

sphuber Jul 26, 2021

unkcpz left a comment •

edited

Loading

unkcpz Jul 27, 2021

unkcpz Jul 27, 2021 •

edited

Loading

unkcpz Jul 27, 2021

unkcpz Jul 27, 2021

chrisjsewell commented Jul 28, 2021

sphuber commented Jul 28, 2021

chrisjsewell commented Aug 11, 2021

chrisjsewell commented Aug 11, 2021

chrisjsewell commented Aug 11, 2021

sphuber commented Aug 11, 2021

chrisjsewell Aug 11, 2021 •

edited

Loading

chrisjsewell commented Aug 11, 2021 •

edited

Loading

chrisjsewell commented Aug 11, 2021

		if self._backend is None:
		raise ValueError('Could not load the backend.')

		from aiida.repository import Repository

		backend = self.get_backend()

	backend = self.get_backend()
	backend = self.get_repository_backend()

Pre-commit: move yapf and mypy config to pyproject.toml #4996

Pre-commit: move yapf and mypy config to pyproject.toml #4996

Conversation

sphuber commented Jun 25, 2021

sphuber commented Jun 25, 2021

codecov bot commented Jun 25, 2021 • edited Loading

Codecov Report

unkcpz commented Jul 22, 2021

sphuber commented Jul 22, 2021

unkcpz commented Jul 26, 2021

error: Module has no attribute "Node" [attr-defined]

incompatible with supertype errors

sphuber left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

unkcpz Jul 27, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

unkcpz left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

unkcpz Jul 27, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chrisjsewell commented Jul 28, 2021

sphuber commented Jul 28, 2021

chrisjsewell commented Aug 11, 2021

chrisjsewell commented Aug 11, 2021

chrisjsewell commented Aug 11, 2021

sphuber commented Aug 11, 2021

chrisjsewell Aug 11, 2021 • edited Loading

Choose a reason for hiding this comment

chrisjsewell commented Aug 11, 2021 • edited Loading

chrisjsewell commented Aug 11, 2021

Pre-commit: move `yapf` and `mypy` config to `pyproject.toml` #4996

Pre-commit: move `yapf` and `mypy` config to `pyproject.toml` #4996

codecov bot commented Jun 25, 2021 •

edited

Loading

`error: Module has no attribute "Node" [attr-defined]`

unkcpz Jul 27, 2021 •

edited

Loading

unkcpz left a comment •

edited

Loading

unkcpz Jul 27, 2021 •

edited

Loading

chrisjsewell Aug 11, 2021 •

edited

Loading

chrisjsewell commented Aug 11, 2021 •

edited

Loading