Python3

[oliver-sanders]

Python 2.6 -> 3.7

• For the most part a minimal effort to get Cylc running under Py3
• Dependency base is now quite pleasing 🍾.
• All but two remote tests should pass 🍾
  • batch schedulers not tested
  • bumping this to future work
  • tests/authentication/10-remote-suite-passphrase-cache.t - awaiting conclusion of comms: future of [communication]method #2975
  • tests/cylc-message/00-ssh.t - awaiting conclusion of comms: future of [communication]method #2975
• All local tests should pass 🍾.
• All unittests should pass 🍾.

Related Issues

closes #2858
closes #2647
closes #2858
closes #2864
??? #2785

GUI issues which become obsolete:
closes #164
closes #1620
closes #1671
closes #1996
closes #2466
closes #2498
closes #2632
closes #2898
closes #2633
closes #2956

GUI issues which we may want to keep around for the moment:
addresses #2570
addresses #2580
addresses #2284

TODO:

• Scan suites in parallel
• Fix remaining test failures
• Docs
• pep8 (will do as last commit to make reviewing easier)
• Work out what we should do with uuid (removed from client-server interface at the moment).

Highlights:

• Remove non-py3 compatible components
  • GTK related code
  • cylc review (awaiting tornado re-write)
  • cherrypy
• Upgrade isodatetime
• Upgrade to Python 3.7 (unittests are now run under 3.6 as well, functional tests should pass under 3.6, untested)
• Replace cherrypy with minimal zmq network layer
  • The aim here was to put in the absolute minimum, we will want to improve on this later
  • Anon auth has been removed awaiting a superior mechanism, tests have been disabled
• Replace cylc graph --reference
• Refactor cylc scan
  • Aggressive refactor with interface changes, I can "tone it down" a little if we think this is likely to cause problems
  • See commit notes for details

Further Work

• ZMQ interface improvements: zmq: network layer improvements #2978
• CLI improvements: CLI refactor #2972
• Convert main loop to asyncio Replace scheduler.py main loop by asyncio loop #2866
• Cylc unicode support unicode #2979, cylc reg "[\n]\0 \/\/ exit" #2281
• Fully replace __cmp__ logic

Take a look through the commits, there have been a few changes beyond Python3.

[sadielbartholomew]

Very happy to see this go up as a PR! 🏆 Also the line count has taken quite a satisfying ✂️ (even accounting for the fact that cylc review will return) which cannot be a bad thing; goodbye GTK & CherryPy!

[oliver-sanders]

@kinow If you get a chance could you take a look at the two unittest failures (they are both Mock problems where the internals have changed so the tests have broken).

[hjoliver]

* Work out what we should do with uuid (removed from client-server interface at the moment).

From a quick look, I guess you mean client UUID, not server UUID? (the latter is needed to uniquely identify a suite run, and the new BOM infrastructure relies on this).

[hjoliver]

* Scan suites in parallel

Or use asyncio?

[hjoliver]

[cylc scan] Aggressive refactor with interface changes, I can "tone it down" a little if we think this is likely to cause problems

Looks fine to me! (from reading your commit log).

[kinow]

Am looking at the issue with unit tests now, but wanted to give a big shout out to @oliver-sanders for his organization with commits. I know given the number of changes here, most comments will be probably regarding things that changed and maybe a few that broke (or maybe none! 🤞). But if anyone else is curious, have a look at how well organized the changes are. I can pretty much follow his actions from readings his commit messages and looking at the grouped changes. 👏 👏 👏

Now on the broken unit tests. I got two broken tests:

# platform linux -- Python 3.7.0, pytest-3.8.0, py-1.6.0, pluggy-0.7.1
...
lib/cylc/tests/test_loggingutil.py::TestLoggingutil::test_value_error_raises_system_exit FAILED                                                                                                             [ 22%]
...
lib/cylc/tests/test_scheduler.py::TestScheduler::test_ioerror_is_ignored FAILED                                                                                                                             [ 31%]
...
================================================================================ 2 failed, 296 passed, 4 warnings in 18.36 seconds ================================================================================

First test_scheduler.py::TestScheduler::test_ioerror_is_ignored. This one was broken because Python 3 does a better job a type checking.

# This bugged line works in Python2
python -c 'import sys, os, mock; sys.path.append(os.path.join(mock.MagicMock(), "123"))'
# Same line fails in Python 3 (thankfully)
python3 -c 'import sys, os; from unittest import mock; sys.path.append(os.path.join(mock.MagicMock(), "123"))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.6/posixpath.py", line 80, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not MagicMock

FWIW, the code in CPython Python that checks it seems to be this one.

And test_loggingutil.py::TestLoggingutil::test_value_error_raises_system_exit. This one also had something that now is illegal in Python 3, comparing random objects with integers (again, related to more consistency with types, which is good).

# We had this in py2 before working
python -c 'import mock; mock.MagicMock() > 1'
# And now this one fails
python3 -c 'from unittest import mock; mock.MagicMock() > 1'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: '>' not supported between instances of 'MagicMock' and 'int'

@oliver-sanders sending a pull request (oliver-sanders#6) showing what needs to be changed to that all unit tests pass. That doesn't need to be merged. You can pick up the changes and put them in a nice commit (or edit previous commit, etc) :-) this way I shouldn't mess with your current nice organisation.

kinow@kinow-VirtualBox:~/Development/python/workspace/cylc$ pytest -x 
=============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.7.0, pytest-3.8.0, py-1.6.0, pluggy-0.7.1 -- /home/kinow/Development/python/anaconda3/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/home/kinow/Development/python/workspace/cylc/.hypothesis/examples')
rootdir: /home/kinow/Development/python/workspace/cylc, inifile: pytest.ini
plugins: remotedata-0.3.0, openfiles-0.3.0, env-0.6.2, doctestplus-0.1.3, cov-2.6.0, arraydiff-0.2, hypothesis-3.83.1
collected 298 items                                                                                                                                                                                               

lib/cylc/conditional_simplifier.py::cylc.conditional_simplifier.ConditionalSimplifier.listify PASSED                                                                                                        [  0%]
...
===================================================================================== 298 passed, 4 warnings in 16.32 seconds =====================================================================================

Cheers
Bruno

[oliver-sanders]

• Scan suites in parallel

Or use asyncio?

@hjoliver Yes, I was looking at doing this yesterday, probably the best way forward.

Something that occurred to me is that unless we are requesting suite [meta] items or state totals we don't actually need to bother the suite to get the required information as it is stored in the contact file. Would this be a good idea?

• Scan results would appear faster
• Avoids unnecessary requests to suites
• Means that we don't actually check suites are running but just report the state of the filesystem

[oliver-sanders]

From a quick look, I guess you mean client UUID, not server UUID?

Yes, sorry I mean the client UUID. The ZMQ client-server I've implemented does not register clients or assign them a UUID.

[wxtim]

Good Work! There's loads of changes, and very few issues that I can see.
I'm only partway through the diff, and since you are still working on it I'm not going to finalize my review.

The main one that seems to recur a good deal is 2to3 messing up by adding more brackets than are needed. :(

• Read code
  • bin/ diff
  • other diff
  • lib/python/test_diffr.py
  • lib/python/diffr.py
• Tests
  • Working locally
  • Travis
• Grep for leftover python2 idioms
  • cmp
• Assured myself that any automated things (Codacy) that have failed are ok - I'm waiting for you to respond to @kinow 's comments on lib/cylc/scheduler.py.

Suggest the following
https://github.com/oliver-sanders/cylc/pull/9/files

[wxtim]

Probably ought to have a test to ensure that sensible error is raised if int, str, float or any other non iterable is fed to it.

[matthewrmshin]

All minor comments.

[matthewrmshin]

Codacy has a few legit unused imports, unused statements, etc. You may want to take note.

[kinow]

I believe before or after this issue is merged, we can close #2647, as cylc_xdot.py is no more.

[matthewrmshin]

I have added some replies to the unresolved conversations. Address as necessary. This is good to go otherwise.

[oliver-sanders]

• Addressed feedback on unrelated code.
• Increased codacy happyness.
• Removed [communications] section.
• Removed HTTP(s) references from the documentation.
• Kept SSH tunnelling references in documentation awaiting comms: future of [communication]method #2975 .

[matthewrmshin]

@hjoliver 2 approvals. Do you want to take a look of this as well? Or are you happy for this to be merged?

(Note: subject to Travis CI returning a good status.)

[hjoliver]

I'm happy to merge. I'm (sadly) still not sufficiently au-fait with Python 3 differences to contribute much, and we don't need to fine-tune at this stage (cylc-8 is still almost a year away...). ... Thanks all, brilliant 💥

[hjoliver]

Travis CI passed this 😁

[hjoliver]

Codacy issues are not important, can be addressed later.

[dwsutherland]

Nice! 💥
No going back now!