Use dmypy instead of mypy #58

fgblomqvist · 2020-02-22T19:15:39Z

First time contributor checklist

I have read how to contribute to this project
I have read the code of conduct to this project

Contributor checklist

I am using the provided codeStyleConfig.xml
I have tested my contribution on these versions of PyCharm/IDEA:

IntelliJ IDEA 2019.3.1

My contribution is fully baked and ready to be merged as is
I ensure that all the open issues my contribution fixes are mentioned in the commit message of my first commit
using the Fixes #1234 syntax

Description

This PR intends to either switch this plugin over to use dmypy completely, or at least provide an option to do so. Using dmypy is much faster and efficient than using mypy, it will allow sub-second scans compared to the 5-15 seconds it could take for mypy to do the same task.

There is still some work left to do. E.g. deciding whether we should just always use dmypy or whether it should be configurable (and how that should work). A fix/solution is also needed to compensate for the fact that dmypy does not support --follow-imports silent. I'm proposing that the function that parses the mypy output can filter out any errors produced by --follow-imports error and that we default to that option instead.

Lastly, it probably just needs a tad more testing (e.g. in PyCharm, and perhaps some other projects).

Nonetheless, the code in this PR serves as a proof-of-concept for now, and I intend to follow through with getting it into a merge-able state as long as some decisions on the above are made.

Type of Changes

	Type
✓	✨ New feature

Related Issue

Closes #57
Closes #43

leinardi · 2020-02-23T14:31:18Z

Unfortunately I can't find the time to work on this right now, I'll try to do something the next weekend.

fgblomqvist · 2020-02-23T18:27:19Z

No worries, don't expect you to finish this PR. I'm thinking the best approach is to completely switch the plugin to use dmypy since that makes it simpler for the users and also makes the most sense in pretty much every single way. However, I did do some more research on the follow imports stuff and realized that follow imports error is not the way to go. Ideally we still want normal, which is not supported by the daemon yet. There does seem to be a workaround tho:
python/mypy#5870 (comment)
Utilizing that workaround might just have to become a dependency I suppose. Idk.

Nonetheless, feel free to provide your thoughts when you find the time. I might test the above out next week.

stale · 2020-04-25T05:44:57Z

This issue has been automatically marked as stale because it has not had activity in the last 60 days.

Bunkerbewohner · 2020-07-17T10:48:14Z

Update: by now dmypy supports the default option normal for follow-imports. So hypothetically it can be used instead of mypy. However, dmypy still is sort of experimental: Its output might vary slightly (in my case it complained about missing types where mypy wasn't complaining even when using the same config file). Also it sometimes can get stuck, e.g. when switching branches while the daemon is running. Therefore the use of dmypy should probably be an option rather than the default.

Bunkerbewohner · 2020-07-17T10:50:09Z

P.S. However I'd still be very keen on getting this dmypy option since for regular use cases the performance gain is immense. In one project I'm working on mypy runtimes decreased from 70+ seconds to 200ms for incremental changes.

leinardi · 2020-07-17T10:50:18Z

@Bunkerbewohner thanks for the update.

Bunkerbewohner

Thanks for the PR @fgblomqvist ! I'd definitely would like to see this feature, too!

src/main/java/com/leinardi/pycharm/mypy/mpapi/MypyRunner.java

fgblomqvist · 2020-07-19T15:34:27Z

Thanks for the update @Bunkerbewohner. I'm atm not currently actively developing in Python, but I am willing to try to put some more effort into this to get it merged. I made the changes you suggested, happy that they added the "normal" option. However, I'm in a new environment and I can't get this plugin to build anymore.

This is the error I'm getting:

/.../mypy-pycharm/src/main/java/com/leinardi/pycharm/mypy/toolwindow/TogglableTreeNode.java:49: error: incompatible types: inference variable T has incompatible bounds
        return Collections.unmodifiableList(children);
                                           ^
    equality constraints: TogglableTreeNode
    lower bounds: TreeNode
  where T is a type-variable:
    T extends Object declared in method <T>unmodifiableList(List<? extends T>)

I'm using Java 9 which I'm not sure is right (I'm no java dev). Maybe @leinardi can shed some light on this. Once I (or someone else) can get this to build I can verify that the changes I made work.

fgblomqvist · 2020-07-19T15:38:20Z

I rebased on top of the latest master.

leinardi · 2020-07-19T18:00:58Z

For me your branch builds fine, but I'm using Java 8:

$ java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-8u252-b09-1ubuntu1-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

However, when I use the plugin built from this branch, it seems to be completely broken for me and I keep getting this error inside the Event log:

19.51	Unexpected Exception Caught
				The scan failed due to an exception: Error while running Mypy: mypy: can't read file 'run': No such file or directory
				com.leinardi.pycharm.mypy.exception.MypyToolException: Error while running Mypy: mypy: can't read file 'run': No such file or directory
				at com.leinardi.pycharm.mypy.mpapi.MypyRunner.runMypy(MypyRunner.java:307)
				at com.leinardi.pycharm.mypy.mpapi.MypyRunner.scan(MypyRunner.java:261)
				at com.leinardi.pycharm.mypy.checker.ScanFiles.scan(ScanFiles.java:109)
				at com.leinardi.pycharm.mypy.checker.ScanFiles.checkFiles(ScanFiles.java:100)
				at com.leinardi.pycharm.mypy.checker.ScanFiles.call(ScanFiles.java:74)
				at com.leinardi.pycharm.mypy.MypyInspection.inspectFile(MypyInspection.java:88)
				at com.leinardi.pycharm.mypy.MypyInspection.lambda$checkFile$0(MypyInspection.java:65)
				at com.intellij.openapi.application.impl.ApplicationImpl$1.call(ApplicationImpl.ja... (show balloon)

This does not happen using the latest master so it seems to be related to this branch.

To reproduce the issue you can checkout the tag v0.7.10 of pyxel, import the project in PyCharm and then open the app.py (https://github.com/kitao/pyxel/blob/v0.7.10/pyxel/app.py):

This is how my environment looks:

Bunkerbewohner · 2020-07-19T18:37:45Z

src/main/java/com/leinardi/pycharm/mypy/mpapi/MypyRunner.java

@@ -272,19 +268,15 @@ public static String detectSystemMypyPath() {
 if (filesToScan.isEmpty()) {
 return Collections.emptyList();
 }
- boolean daemon = false;

 GeneralCommandLine cmd = new GeneralCommandLine(mypyPath);


The daemon has its own executable called "dmypy" which is next to "mypy" in the same folder. If useDaemon is true we would need to change that here.

Bunkerbewohner · 2020-07-19T18:40:37Z

@fgblomqvist thanks a lot!

@leinardi @fgblomqvist the problem is that dmypy has its own executable. The equivalent of "mypy --config-file mypy.ini file.py" using the daemon looks like this: "dmypy run -- --config-file mypy.ini file.py". I commented where it would need to be adjusted. I wonder if it's ok to simply take the path of mypy and prepend the "d" to the filename. If that file exists daemon mode is supported and should work, otherwise not.

fgblomqvist · 2020-07-19T19:05:32Z

@Bunkerbewohner ah that's new (they didn't use to have a separate executable, I'm pretty sure). Easy to fix. And yeah I think that sounds good enough.

@leinardi Hmm okay let me try with Java 8 then.

fgblomqvist · 2020-07-19T19:43:20Z

I was able to compile with Java 8 🎉

I pushed changes that does a very simple (and stupid) path-replacement to use dmypy instead of mypy is daemon mode is enabled. It gets the job done I think. However, when I run it on a local Python repo I got the daemon crashes with a KeyError sys. Googled a bit and it sounds like it might be a dmypy bug. Lmk if it works for you guys.

Regardless, the behavior can probably be improved a bit.

leinardi · 2020-07-19T20:21:53Z

I tried the latest branch and I found the following issues:

The "Use Daemon" option is not persisted: if I check it and press OK, if I re-open the settings is again unchecked:
If I run the plugin on the gwe module the plugin finds 279 errors instead using this branch but 33 if I build master. Probably some changes in this branch broke the mypy.ini recognition?

This branch:

master:

To reproduce you can run the plugin on this module: https://gitlab.com/leinardi/gwe/-/blob/master/gwe/__main__.py

fgblomqvist · 2020-07-19T21:21:04Z

Fixed the persistence problem. I sometimes now get around 50 errors in that project and sometimes no problems at all. Can't figure out how to get debug logging in IDEA to work so don't really know how to debug this further. Regardless, the problem you're seeing with 200+ errors is not something I can replicate.

Bunkerbewohner · 2020-07-25T12:56:09Z

Thanks @fgblomqvist! I have tested the current version on a project of my own and still have some issues. I get the following error when starting "Check Current File" using the plugin:

2020-07-25 14:45:56,754 [1525935]   INFO - .pycharm.mypy.mpapi.MypyRunner - Command Line string: /Users/mathias/Library/Java/JavaVirtualMachines/adopt-openj9-1.8.0_262/Contents/Home /Users/mathias/.pyenv/versions/3.8.1/bin/mypy -V 
2020-07-25 14:45:56,755 [1525936]   WARN - .pycharm.mypy.mpapi.MypyRunner - Error while checking Mypy path 
com.intellij.execution.process.ProcessNotCreatedException: Cannot run program "/Users/mathias/Library/Java/JavaVirtualMachines/adopt-openj9-1.8.0_262/Contents/Home": error=13, Permission denied
	at com.intellij.execution.configurations.GeneralCommandLine.createProcess(GeneralCommandLine.java:418)
	at com.leinardi.pycharm.mypy.mpapi.MypyRunner.isMypyPathValid(MypyRunner.java:96)
	at com.leinardi.pycharm.mypy.mpapi.MypyRunner.checkMypyAvailable(MypyRunner.java:179)
	at com.leinardi.pycharm.mypy.mpapi.MypyRunner.checkMypyAvailable(MypyRunner.java:152)
	at com.leinardi.pycharm.mypy.MypyInspection.inspectFile(MypyInspection.java:76)
	at com.leinardi.pycharm.mypy.MypyInspection.lambda$checkFile$0(MypyInspection.java:65)
	at com.intellij.openapi.application.impl.ApplicationImpl$1.call(ApplicationImpl.java:255)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.io.IOException: Cannot run program "/Users/mathias/Library/Java/JavaVirtualMachines/adopt-openj9-1.8.0_262/Contents/Home": error=13, Permission denied
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
	at com.intellij.execution.configurations.GeneralCommandLine.startProcess(GeneralCommandLine.java:449)
	at com.intellij.execution.configurations.GeneralCommandLine.createProcess(GeneralCommandLine.java:414)
	... 10 more
Caused by: java.io.IOException: error=13, Permission denied
	at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
	at java.base/java.lang.ProcessImpl.<init>(ProcessImpl.java:340)
	at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:271)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
	... 13 more

However, if I manually start the daemon on the command line the plugin works as expected and errors are checked super quickly. Also I don't get any unexpected mypy errors listed in my project which is quite sizeable (a clean mypy run takes more than 70s).

My settings look like this:

The "Test" button successfully finds the mypy executable.
Obviously the path in the error log above doesn't make much sense.
I will investigate further and see if I can get this to run properly on my machine (macOS).

Bunkerbewohner · 2020-07-25T12:59:10Z

src/main/java/com/leinardi/pycharm/mypy/MypyConfigService.java


 public MypyConfigService() {
 customMypyPath = "";
 mypyArguments = "";
 mypyConfigFilePath = "";
+ useDaemon = true;


dmypy seems to be still somewhat experimental. In productive use we definitely encountered some unexpected errors here and there, as well as apparant infinite loops where dmypy would never finish. Therefore I would strongly suggest to disable this option by default.

Bunkerbewohner · 2020-07-25T13:10:28Z

P.S. I changed the error logging a little and now the error I see is this:

The scan failed due to an exception: Error while running Mypy command /Users/mathias/.virtualenvs/kialo/bin/dmypy run -- --show-column-numbers --config-file /Users/mathias/kialo/backend/mypy.ini /Users/mathias/kialo/backend/kialo/entities/base_unit_test.py:
			[Errno 61] Connection refused

So the way that dmypy is invoked is correct including the path to the config file. No idea why launching the process doesn't work though. I mean in the earlier error it said "java.io.IOException: error=13, Permission denied" but why...

fgblomqvist · 2020-07-25T14:02:21Z

I had similar odd java errors when using adopt. Switched to standard openjdk which worked a lot better. So yeah beyond the weird java error which I don't think has to do with the plug-in, it seems like the only thing left to do is to disable the daemon by default. I can do that this coming week. Thanks for reporting back and lmk if you solve the java issue!

fgblomqvist · 2020-08-02T16:46:21Z

Got around to changing it, lmk if anything else should be done. When you have time @leinardi, please give it another go to see if you still see weird behavior.

leinardi · 2020-08-08T09:06:26Z

Hi @fgblomqvist, sorry for the late reply, it was a busy week at work.

I tested your branch again and the daemon checkbox works correctly. But now, when I enable it, mypy doesn't find any violation at all. The Event log shows no errors.

I made a short video to show the issue (unfortunately to be able to attach it I had to zip it): 2020-08-08 10-58-04.zip

fgblomqvist · 2020-08-08T13:03:46Z

I see. Yeah I had similar results sometimes. Idk if it's because the daemon is finicky or if there is something wrong with how I run it (but I don't think so) 🤔 .
Appreciate the response at least.

DetachHead · 2021-07-10T13:46:20Z

i believe i've fixed some of these issues https://github.com/fgblomqvist/mypy-pycharm/pull/1

…me reason it reports no problems when the file is located outside of the project

fix dmypy issues

DetachHead · 2021-07-19T10:03:33Z

some further outstanding issues i've identified

the daemon will sometimes report false positive when run on individual files that don't appear when running on the whole project (ie. with dmypy run -- mainn.py vs dmypy run -- .
the daemon will crash if there are lots of files in the project and you attempt to scan the whole project

looking at the mypy vscode plugin which uses the daemon and seems to be much more reliable, by default it only ever runs on the whole project and not individual files:

this isn't optimal though as it would need to run on ignored files and filter out those results, but i think it may result in more stability when using the daemon.

jesse-viola · 2022-04-14T17:14:45Z

Any updates on this PR? Would be a great feature.

nateschickler-puzzle · 2022-10-05T20:47:54Z

I updated the code to scan the whole project as per the above comment on my own branch, rebased, and tried running it on a project but am now running into this mypy bug: python/mypy#10709
Just wanted to report in case others are using ignore_missing_imports = True in their projects, dmypy may not run until that issue is resolved. I was not successful with the workaround mentioned by some commenters in the linked issue :(

isuro · 2023-07-20T17:57:16Z

Hey @fgblomqvist the issue linked by @nateschickler-puzzle seems to have been resolved. Any chance of getting this going again?

fgblomqvist · 2023-08-15T16:55:30Z

Hi @isuro, unfortunately I don't really have the time or motivation to continue this at the moment (haven't worked with Python in the last couple of years at least), but I welcome any attempts to pick it up! 🙂

I would start with @nateschickler-puzzle's work here: https://github.com/nateschickler-puzzle/mypy-pycharm
And then try to rebase it again and then see if it runs now that the bug is fixed. Projects like this tend to require quite a bit of testing/tinkering so it's easier to work on it if you actually use it daily.

stale bot added the Status: Stale label Apr 25, 2020

leinardi force-pushed the master branch from b37d3b2 to 6c5b7d6 Compare April 25, 2020 10:04

Bunkerbewohner suggested changes Jul 17, 2020

View reviewed changes

src/main/java/com/leinardi/pycharm/mypy/mpapi/MypyRunner.java Outdated Show resolved Hide resolved

src/main/java/com/leinardi/pycharm/mypy/mpapi/MypyRunner.java Outdated Show resolved Hide resolved

fgblomqvist added 6 commits July 19, 2020 11:37

Update status check for daemon

d515590

Fix command typo

3a29a15

Add valid follow imports for daemon

11cfc72

Remove unused import

dd32b9b

Remove follow-imports flag

dc72758

Add "Use Daemon" config option

438f4ee

Bunkerbewohner reviewed Jul 19, 2020

View reviewed changes

Use dmypy executable if daemon mode is enabled

ef635dd

Fix use daemon setting not persisting

66b1312

Bunkerbewohner reviewed Jul 25, 2020

View reviewed changes

Default to daemon off

1931413

retry on "The connection is busy" daemon errors

c1217c4

DetachHead and others added 2 commits July 10, 2021 23:47

use idea directory for temp dir when using the daemon, because for so…

6e00823

…me reason it reports no problems when the file is located outside of the project

Merge pull request #1 from DetachHead/master

534bdac

fix dmypy issues

fgblomqvist closed this by deleting the head repository Oct 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use dmypy instead of mypy #58

Use dmypy instead of mypy #58

fgblomqvist commented Feb 22, 2020 •

edited

Loading

leinardi commented Feb 23, 2020

fgblomqvist commented Feb 23, 2020 •

edited

Loading

stale bot commented Apr 25, 2020

Bunkerbewohner commented Jul 17, 2020

Bunkerbewohner commented Jul 17, 2020

leinardi commented Jul 17, 2020

Bunkerbewohner left a comment

fgblomqvist commented Jul 19, 2020

fgblomqvist commented Jul 19, 2020

leinardi commented Jul 19, 2020 •

edited

Loading

Bunkerbewohner Jul 19, 2020

Bunkerbewohner commented Jul 19, 2020

fgblomqvist commented Jul 19, 2020

fgblomqvist commented Jul 19, 2020

leinardi commented Jul 19, 2020

fgblomqvist commented Jul 19, 2020

Bunkerbewohner commented Jul 25, 2020

Bunkerbewohner Jul 25, 2020

Bunkerbewohner commented Jul 25, 2020

fgblomqvist commented Jul 25, 2020

fgblomqvist commented Aug 2, 2020

leinardi commented Aug 8, 2020 •

edited

Loading

fgblomqvist commented Aug 8, 2020

DetachHead commented Jul 10, 2021

DetachHead commented Jul 19, 2021

jesse-viola commented Apr 14, 2022

nateschickler-puzzle commented Oct 5, 2022

isuro commented Jul 20, 2023

fgblomqvist commented Aug 15, 2023

Use dmypy instead of mypy #58

Use dmypy instead of mypy #58

Conversation

fgblomqvist commented Feb 22, 2020 • edited Loading

First time contributor checklist

Contributor checklist

Description

Type of Changes

Related Issue

leinardi commented Feb 23, 2020

fgblomqvist commented Feb 23, 2020 • edited Loading

stale bot commented Apr 25, 2020

Bunkerbewohner commented Jul 17, 2020

Bunkerbewohner commented Jul 17, 2020

leinardi commented Jul 17, 2020

Bunkerbewohner left a comment

Choose a reason for hiding this comment

fgblomqvist commented Jul 19, 2020

fgblomqvist commented Jul 19, 2020

leinardi commented Jul 19, 2020 • edited Loading

Bunkerbewohner Jul 19, 2020

Choose a reason for hiding this comment

Bunkerbewohner commented Jul 19, 2020

fgblomqvist commented Jul 19, 2020

fgblomqvist commented Jul 19, 2020

leinardi commented Jul 19, 2020

fgblomqvist commented Jul 19, 2020

Bunkerbewohner commented Jul 25, 2020

Bunkerbewohner Jul 25, 2020

Choose a reason for hiding this comment

Bunkerbewohner commented Jul 25, 2020

fgblomqvist commented Jul 25, 2020

fgblomqvist commented Aug 2, 2020

leinardi commented Aug 8, 2020 • edited Loading

fgblomqvist commented Aug 8, 2020

DetachHead commented Jul 10, 2021

DetachHead commented Jul 19, 2021

jesse-viola commented Apr 14, 2022

nateschickler-puzzle commented Oct 5, 2022

isuro commented Jul 20, 2023

fgblomqvist commented Aug 15, 2023

fgblomqvist commented Feb 22, 2020 •

edited

Loading

fgblomqvist commented Feb 23, 2020 •

edited

Loading

leinardi commented Jul 19, 2020 •

edited

Loading

leinardi commented Aug 8, 2020 •

edited

Loading