WIP package cylc as a Python module (PyPI, Anaconda, RPM's, etc)

[kinow]

Hi,

While was playing with Cylc + Python 3 these past days, I realized that having the dependencies with the code gets a bit in the way when keeping track of code compatibility issues.

Besides appearing in the reports in my IDE and also in the command line, I checked Jinja2 and MarkupSafe, and both provide versions that appear to be compatible with Python 2.7 and 3.x. But if we have problems with our current version, updating it may cause some issues.

Also remembered that some time ago there was a request or some discussion about distributing Cylc via Anaconda. I have never released any conda package, but for PyPI I used the distutils and a simple setup.py. Looks like the documentation for conda packages mentions setup.py files too, though I suspect it's not a requirement.

This pull request contains a work in progress.

I understand the idea of having dependencies such as cherrypy, jinja, etc, with the package is to include the minimum required for running Cylc in the same repository (i.e. "batteries included").

However, I wonder if we could perhaps have Cylc as a single Python module, with the Cylc core, parsec (though maybe it could be an external module later...), and the jinja2 filters.

The minimum required dependencies would be listed in the setup.py file, under install_requires. Other optional dependencies like openssl, gtk, wouldn't be a blocker for installing Cylc, and could be listed under extras_require.

Dependencies that are not in PyPI, can also be referenced directly in setup.py via their GitHub repository (I think we would need this for isodatetime).

Besides helping to package Cylc for PyPI and Conda, I haven't tried it, but distutils includes some documentation for generating RPM files as well.

Finally, to maintain the current status of batteries included, we could also investigate customizing the packaging of Cylc, to instead of producing a binary file with Cylc code, to include also the libraries listed unr install_requires.

Here's an article that details how someone else did the same thing.

That way, we could keep the source code in Git, with the metadata to produce the binary files, and either manually or via Travis-CI, generate the desired binary packages (e.g. tar.gz, zip, rpm, xar, etc) and upload here to GitHub as a release.

The benefits I see for this change are:

• other Python devs would be familiar with the project structure (see Pandas, request, jinja, nasa/podaacpy)
• the code in the repository would be reduced to Cylc only
• producing code metrics would be simpler
• users would still be able to download Cylc from GitHub, but instead of cloning the repository, they would simply download a release in some binary format
• it would be easier for us to update dependencies (for compatibility between Python versions, new features, performance, or security), which would help with Port cylc to Python 3 #1874 IMO
• Cylc could be used as a dependency in other projects! We could even have the Web GUI as a separate project (though I wouldn't push for it now). But anybody would be able to install Cylc with pip install cylc or conda install cylc and then simply import the cylc modules and create new tools, libraries, etc.
• Same as above, but just to note that that would make it easier to import cylc and use it in Jupyter Notebooks too

Sorry for the wordy ticket! Hoping to discuss it in person with @hjoliver soon. But given that I still don't know much about the project nor the infrastructure used by users to run it, there could be cases for keeping the current structure. If so, I'd be OK with that too, and would just start trying to update the dependencies manually in my branch to match Python 2.7 + 3.x 😃

[hjoliver]

@kinow - thanks for looking into this. I'll discuss with the UK team tomorrow (Monday) but I think we'd all like to package cylc in a more standard way, and with no bundling, if possible. (To ease Python-3 porting too, as you've pointed out).

As I recall the following issues (might?) present something of a barrier:

• (py)GTK typically comes with system Python but is otherwise a very nasty dependency (e.g. for anaconda)
  • do you have any thoughts on this one?
• Jinja2 is bundled because some versions had loop scoping that afflicted our uses badly
  • this may be OK now, with recent versions of Jinja2
• Cherrypy was evolving too fast and chaotically
  • @matthewrmshin can probably comment on this one.

[hjoliver]

(I don't think "batteries included" is a particularly strong motivation - it's more about battery compatibility... but package dependency version requirements might solve that).

[matthewrmshin]

For information, the bundled CherryPy has been customized and the latest version has evolved into something completely different in the last 2 years. Hence I'd like to move on to another web framework as soon as possible.

[matthewrmshin]

PyGTK has been the biggest barrier for this work in the past. Hopefully this will be solved when we migrate away.

[matthewrmshin]

Batteries compatibility is important but should be achievable using configuration/package manager just as well these days.

We have now moved on from the era of reliance on the mercy of system admin to the era of container tech, so the batteries included philosophy is much less important.

[kinow]

Hi @hjoliver

(py)GTK typically comes with system Python but is otherwise a very nasty dependency (e.g. for anaconda)

do you have any thoughts on this one?

I have an idea that could work... I found the following info from the pygobject docs for porting.

"pygtk supported GTK+ 2.0 and Python 2 only. PyGObject supports GTK+ >=3.0 and Python 2/3. If you port away from pygtk you also have to move to GTK+ 3.0 at the same time. pygtkcompat described below can help you with that transition."

So far the UI seems to be the main showstopper for Python3. But I think we could:

• confirm that we won't have to do any major changes to the current UI code
• use a try/catch where first we try to import pygtk as right now, and in case of an error we try to import from pygobject classes

If that worked... then we could have this temporary GTK GUI solution to help users with the transitioning to the new Web GUI. But only if it doesn't mean having to re-write parts of the current UI.

[kinow]

@matthewrmshin

For information, the bundled CherryPy has been customized and the latest version has evolved into something completely different in the last 2 years. Hence I'd like to move on to another web framework as soon as possible.

I had somewhere in the Cylc code or user guide perhaps about how we distribute code of some libraries with a few customizations. But I wasn't aware of where we had those customizations. Hopefully the version that we have of CherryPy works with Python 2 (will check it later and comment in the Python 3 ticket). Thanks!

[kinow]

And happy that the batteries included feature of Cylc can now be changed a bit. Having the dependencies managed will help a lot for both the new Web GUI and for Python 3 🎉 🎉 🎉

[hjoliver]

@kinow - we have considered going to pygobject as an intermediate step but were concerned that it might take a lot of effort away from the web GUI. E.g. quote from this page:

http://dumbmatter.com/2012/02/some-notes-on-porting-from-pygtk-to-pygobject/

Is porting worth the effort? In 2012, it would probably have been more efficient to put this time towards porting my software to a web app.

(Mind you, 2012 was a long time ago now...)

[hjoliver]

Also on that page (in comments):

Compared to commercial solutions, PyGObject is a joke. The API specification is continuously in flux, and not stable long enough for anyone to seriously consider adopting it. But hey, who cares? at least the devs are having loads of fun!

(again, maybe things have improved since then ... but if PyGTK -> PyGObject is not easy we could waste a lot of time)

[matthewrmshin]

Discussed in meet...

If it is easy to port to PyGObject, (e.g. less than a day of work), then perhaps. Otherwise, the sequence of event should probably be:

remove-old-GUI & Python-3 => packaging & web-UI

To be done under a long-live development branch.

[hjoliver]

@kinow - unless PyGObject is super-easy (and reliable etc.) - which seems unlikely - I guess we should close this PR and redo it on the new dev branch for the web UI etc. (which we're not quite ready to make yet - I'll update you on this later). Do you agree?

[kinow]

I had a quick look at PyGObject, and having to learn what changed and not, plus the number of files to modify and confirm nothing is broken... just these two are enough to convince me it's not an easy task :-) so +1. Closing it.

Thanks!

[matthewrmshin]

(Marked as wontfix ... yet, but hopefully we'll be able to do this soon by removing PyGTK dependency.)