Remove runtests.py #5274

elazarg · 2018-06-25T06:54:21Z

Replace runtests.py with a ~~gentle reminder to use pytest~~ script that executes pytest+lint+self check.
Remove waiter.py
Remove selfcheck unit test
Run self check from travis using python3 -m mypy --config-file mypy_self_check.ini -p mypy
Run linter explicitly from travis using flake8 -j0
Update docs

This is work-in-progress, here for review and testing. I tried to implement what I understood from the comments in #5270, but I'm not sure I got it right.

TODO: fix test run in typeshed.

Concludes #1673.

elazarg · 2018-06-26T07:01:57Z

I'm not completely sure what is the ideal workflow. I've separated the tests from the self check, but it is one of the tests that one should run before opening a PR.

pytest && flake8 && python3 -m mypy --config-file mypy_self_check.ini -p mypy is a lengthy command to run. Maybe leave runtests.py as a shorthand?

(Or maybe you meant something entirely different? @gvanrossum @ilevkivskyi)

emmatyping · 2018-06-26T11:08:50Z

I personally would prefer to leave the self check under pytest, as it is a bit of a lengthy command to type out the entire command, and not very memorable. However I defer to the judgement of others, I can always use tox :)

JukkaL · 2018-06-26T13:22:21Z

I don't have a strong preference about whether to run self check under pytest, but if it's not being run under pytest, we should definitely have a simple, consistent way of running self check locally instead of running a long command line. It would be good to have an easy way of self checking using dmypy as well.

ilevkivskyi · 2018-06-26T13:30:10Z

I would rather keep all three separate: tests, self-check, and flake8 (but this is also not a strong preference). For self-check I am mostly using Ctrk + Shift + M :-) If anyone else uses PyCharm, I can share my config for mypy plugin:

Mypy command: dmypy start -- --follow-imports=error --config-file=mypy_self_check.ini ; dmypy check mypy
PATH suffix: path/to/venv/bin

JelleZijlstra · 2018-06-26T14:30:38Z

I would actually prefer to have everything (including self-check and lint) run under pytest, so that it's easy to run all tests locally without having to remember a lot of commands.

elazarg · 2018-06-26T15:43:42Z

Looks like the simplest solution is to keep runtests as a simple scripts that executes pytest, self check and lint.

JukkaL · 2018-06-26T16:40:39Z

Looks like the simplest solution is to keep runtests as a simple scripts that executes pytest, self check and lint.

This is still worse that the legacy script since it will be harder to run just lint or self check. Whatever solution we use to replace runtests should make all of these easy (i.e. no need to remember complex command lines):

Run everything, including tests, lint and self check (matches what the travis build does).
Run only self check.
Run only lint.
Run some subset of tests (this already works).

Nice to haves, for extra credit:

It would be nice if the self check would run an incremental build by default.
Being able to run all tests without running self check or lint.
Provide also a way of running a dmypy self check, since these are often much faster than non-dmypy builds.

elazarg · 2018-06-26T17:23:50Z

Ok. Here's another suggestion:

$ ./runtests.py (runs everything)

$ ./runtests.py [lint] [self] [pytest] [args...] any subset of the three. Arguments are passed to pytest only without any preprocessing (if pytest is not mentioned, reject additional arguments). Arguments are trailing only.

Passing arguments to self check will be performed manually.

Possibly use self-dmypy or something similar instead of self for dmypy self check.

gvanrossum · 2018-06-26T18:06:27Z

Maybe even simpler, only use `runtests.py [lint] [self] [pytest]` -- for test selection it's best to directly call `pytest -k ...`.

elazarg · 2018-06-27T02:03:42Z

Done. The commands are ~~evaluated eagerly~~ asserted to find typos.

emmatyping

Just a few minor points. Overall looks great!

emmatyping · 2018-07-01T15:53:39Z

runtests.py

-import os
-from os.path import join, isdir
+from sys import exit
+from os import system
 import sys


Nit: maybe fold this and the sys.exit import into one import?

emmatyping · 2018-07-01T15:59:34Z

runtests.py

+for arg in args:
+    cmd = cmds[arg]
+    print('$', cmd)
+    res = (system(cmd) & 0x7F00) >> 8


Why the bit twiddling here? Wouldn't it be better to use subrocess.call?

I want to have the full command copy-pastable in the source. To clarify that this is a shorthand script, and that nothing more is going on than a mere execution of a command.

emmatyping · 2018-07-01T16:00:19Z

.travis.yml

-  - if [[ $TRAVIS_PYTHON_VERSION == '3.6' ]]; then flake8; fi
-  - if [[ $TRAVIS_PYTHON_VERSION == '3.5.1' ]]; then python runtests.py package; fi
+  - pytest -n12
+  - if [[ $TRAVIS_PYTHON_VERSION == '3.6' ]]; then flake8 -j0; fi


Perhaps we should have j12 here too?

I have no idea.

Done, because why not.

emmatyping · 2018-07-01T16:02:16Z

tox.ini


 [testenv:type]
 description = type check ourselves
 basepython = python3.6
-commands = python runtests.py self-check -p '-n0' -p '-v'
+commands = python3 -m mypy -p mypy


For consistency, we should point to the self-check config here.

elazarg · 2018-07-08T18:44:44Z

@ethanhs do you approve now?

emmatyping · 2018-07-08T18:46:25Z

You have a merge conflict, but otherwise I approve.

elazarg · 2018-07-09T05:03:39Z

I have no idea what's this conflict is about. It does not show up in my local merge, and the "resolve conflicts" key is disabled.

elazarg · 2018-07-09T05:04:48Z

Moreover, the conflict said to be in a file that should be removed completely.

elazarg · 2018-07-09T05:08:06Z

emmatyping · 2018-07-09T05:52:54Z

It is listed when I merge master as CONFLICT (modify/delete): waiter.py deleted in HEAD.... I believe you should be able to git rm waiter.py then commit the merge.

elazarg · 2018-07-09T06:10:28Z

But it does not exist.

~/workspace/mypy$ git status
On branch pytest-complete
Your branch is up to date with 'origin/pytest-complete'.

nothing to commit, working tree clean
~/workspace/mypy$ git rm waiter.py
fatal: pathspec 'waiter.py' did not match any files

emmatyping · 2018-07-09T06:23:23Z

Ah, if it doesn't exist on disk you need git rm --cached waiter.py. This is probably one of the more illogical git commands...

JelleZijlstra · 2018-07-09T14:35:00Z

The conflict is presumably that somebody else modified waiter.py on master, and you deleted it. The way to fix it would be to run git merge origin/master (I think, I don't do manual merges that often), and then when there's a conflict, git rm waiter.py as Ethan says.

elazarg · 2018-07-09T14:59:43Z

~/workspace/mypy$ git rm --cached waiter.py
fatal: pathspec 'waiter.py' did not match any files

elazarg · 2018-07-09T15:01:34Z

Sorry guys I'm so amazingly stupid. I've merged upstream/master but forgot git fetch upstream.

Michael0x2a · 2018-07-18T20:14:32Z

@ethanhs -- just to check, we can merge this now, right?

emmatyping · 2018-07-18T21:58:50Z

@Michael0x2a yes, this should be good to go.

ilevkivskyi · 2018-07-18T23:39:15Z

Thanks for all the work on tests @elazarg!

gvanrossum · 2018-07-18T23:57:13Z

Yes, thanks Elazar!

On Wed, Jul 18, 2018 at 4:39 PM Ivan Levkivskyi ***@***.***> wrote: Thanks for all the work on tests @elazarg <https://github.com/elazarg>! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5274 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACwrMpljctPPpxV0MIyRwAbRFzr_PTveks5uH8clgaJpZM4U1mgo> .

-- --Guido (mobile)

JukkaL · 2018-07-19T09:37:39Z

This is great! Thanks Elazar!

elazarg force-pushed the pytest-complete branch 5 times, most recently from 8722b57 to 1b03518 Compare June 26, 2018 06:39

elazarg changed the title ~~[WIP] Remove runtests.py~~ Remove runtests.py Jun 26, 2018

elazarg force-pushed the pytest-complete branch 2 times, most recently from 502b55e to aafab7a Compare June 27, 2018 02:02

elazarg force-pushed the pytest-complete branch from aafab7a to 31fe744 Compare June 27, 2018 02:35

make runtests.py a shorthand script; run selfcheck and lint from travis

effc9df

elazarg force-pushed the pytest-complete branch from 31fe744 to effc9df Compare June 27, 2018 02:37

emmatyping requested changes Jul 1, 2018

View reviewed changes

elazarg added 2 commits July 1, 2018 23:03

fix tox and imports

03960e7

flake -j12 for travis

2f65bc0

emmatyping approved these changes Jul 8, 2018

View reviewed changes

Merge remote-tracking branch 'upstream/master' into pytest-complete

2d9f5e2

merge upstream/master

b2c59e6

elazarg force-pushed the pytest-complete branch from 587c09a to b2c59e6 Compare July 9, 2018 15:03

ilevkivskyi merged commit 3e53036 into python:master Jul 18, 2018

ilevkivskyi mentioned this pull request Jul 18, 2018

Switch to pytest from our homegrown test runner #1673

Closed

Remove runtests.py #5274

Remove runtests.py #5274

Conversation

elazarg commented Jun 25, 2018 • edited Loading

elazarg commented Jun 26, 2018

emmatyping commented Jun 26, 2018 • edited Loading

JukkaL commented Jun 26, 2018

ilevkivskyi commented Jun 26, 2018

JelleZijlstra commented Jun 26, 2018

elazarg commented Jun 26, 2018

JukkaL commented Jun 26, 2018

elazarg commented Jun 26, 2018

gvanrossum commented Jun 26, 2018 via email

elazarg commented Jun 27, 2018 • edited Loading

emmatyping left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elazarg commented Jul 8, 2018

emmatyping commented Jul 8, 2018

elazarg commented Jul 9, 2018

elazarg commented Jul 9, 2018

elazarg commented Jul 9, 2018

emmatyping commented Jul 9, 2018

elazarg commented Jul 9, 2018

emmatyping commented Jul 9, 2018

JelleZijlstra commented Jul 9, 2018

elazarg commented Jul 9, 2018

elazarg commented Jul 9, 2018 • edited Loading

Michael0x2a commented Jul 18, 2018

emmatyping commented Jul 18, 2018

ilevkivskyi commented Jul 18, 2018

gvanrossum commented Jul 18, 2018 via email

JukkaL commented Jul 19, 2018

elazarg commented Jun 25, 2018 •

edited

Loading

emmatyping commented Jun 26, 2018 •

edited

Loading

elazarg commented Jun 27, 2018 •

edited

Loading

elazarg commented Jul 9, 2018 •

edited

Loading