-
Notifications
You must be signed in to change notification settings - Fork 94
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP Make cylc a module #2834
WIP Make cylc a module #2834
Conversation
17393f9
to
280a230
Compare
This is still a work-in-progress branch, mainly waiting for:
|
I started this more because I wanted to play with Cylc's API in Jupyter Notebooks (which I think maybe would interest @jonnyhtw too?). Here's one example which I maintain using the branch in this pull request (though one of the notebooks dates a few years ago I think): |
I suppose we can create a new isodatetime release without metomi/isodatetime#101? In which case I'll try and do that early next week, so we don't end up blocking progress on other stuffs. |
Did not want to put any pressure on that @matthewrmshin, sorry. I will have another look at that pull request during next week, but my hopes are not too high for finding a solution that doesn't break anything else. I was thinking about discussing the Just raised this pull request to provide some example for the recent issue about packaging cylc. So feel free to focus on the Cylc release if you prefer, and leave the |
Hi there Definitely interested in Jupyter integration. I'm a big fan! Cheers J |
280a230
to
28ee2f3
Compare
Included a few more commits (will squash/edit later) to
Still a few more loose ends. Rebased onto master. |
Note: Conda docs mention a way to upload via Had a go at uploading to Conda via PYPI and then realized it wouldn't resolve dependencies via python setup.py sdist
anaconda upload dist/*.tar.gz
# quite easy to remove it from your channel later
anaconda remove kinow/pccora/0.2 Next week will finish experimenting with Conda, and hopefully have a |
Good primer on Python packaging: A tour on Python Packaging, Nick Mavrakis |
Finished the tests on Conda, using that small Python utility as test experiment. Notes:
conda skeleton pypi construct
anaconda upload /home/kinow/Development/python/anaconda3/conda-bld/linux-64/construct-2.5.1-py37_0.tar.bz2
...
Upload complete
conda package located at:
https://anaconda.org/kinow/construct After doing that, I had to include my own channel in the list of default channels, so that
Building the library and uploading itAfter that, it finally worked to build and upload it (it does both by default, by I disabled that in my kinow@kinow-VirtualBox:~$ conda install -c kinow pccora
Solving environment: done
## Package Plan ##
environment location: /home/kinow/Development/python/anaconda3
added / updated specs:
- pccora
The following packages will be downloaded:
package | build
---------------------------|-----------------
construct-2.5.1 | py37_0 110 KB kinow
pccora-0.3 | py37_0 18.2 MB kinow
------------------------------------------------------------
Total: 18.3 MB
The following NEW packages will be INSTALLED:
construct: 2.5.1-py37_0 kinow
pccora: 0.3-py37_0 kinow
Proceed ([y]/n)? y
Downloading and Extracting Packages
construct-2.5.1 | 110 KB | ######################################################################################################################################################################## | 100%
pccora-0.3 | 18.2 MB | ######################################################################################################################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
kinow@kinow-VirtualBox:~$ python
Python 3.7.0 (default, Jun 28 2018, 13:15:42)
[GCC 7.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pccora
>>> p = pccora.PCCORAParser()
>>> p
<pccora.pccora.PCCORAParser object at 0x7f5e6a9a3a58>
>>> ConclusionI think it is doable to generate a |
The only thing holding me from removing the |
Test coverage reporting is now in 🍾 so ... (as per your previous comment) ... this is almost ready to go? |
(I guess we need to document clearly that package installation does not address the PyGTK dependency; for that users might still need to rely on the system Python 2). |
ab40c23
to
bf1fff3
Compare
+1 And also need to figure out what to do about the release. That's, I think, the last pending issue for me.
Furthermore, This complete package, would be Cylc with all its dependencies included (and the Rebased the work, removing what had been implemented in the coverage reports pull request and was redundant here. |
d27cb11
to
3c8ceac
Compare
Rebased. Pending now Travis build. I believe this is ready to go, but not in a 7.x release. This makes Cylc a module, and allows users to install it via pip. However, it also removes dependencies, and users would have to change their installation process to comply with this change. So it might be better to leave the WIP for now, and merge it when going Python 3, IMO. |
Building with
I will package If that works, will start trying |
Testing the RPM:
To install with dependencies, we can just fake-use [root@4c921bbd54db rpms]# yum -y --nogpgcheck localinstall cylc*noarch.rpm
Loaded plugins: fastestmirror, ovl
Examining cylc-7.8.0_20_g1f592_dirty-1.noarch.rpm: cylc-7.8.0_20_g1f592_dirty-1.noarch
Marking cylc-7.8.0_20_g1f592_dirty-1.noarch.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package cylc.noarch 0:7.8.0_20_g1f592_dirty-1 will be installed
--> Processing Dependency: less for package: cylc-7.8.0_20_g1f592_dirty-1.noarch
Loading mirror speeds from cached hostfile
* base: mirror.ventraip.net.au
* extras: mirror.optus.net
* updates: ftp.swin.edu.au
--> Processing Dependency: python-jinja2 for package: cylc-7.8.0_20_g1f592_dirty-1.noarch
--> Processing Dependency: python-markupsafe for package: cylc-7.8.0_20_g1f592_dirty-1.noarch
--> Running transaction check
---> Package less.x86_64 0:458-9.el7 will be installed
---> Package python-jinja2.noarch 0:2.7.2-2.el7 will be installed
---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
===================================================================================================================================================================================================================
Package Arch Version Repository Size
===================================================================================================================================================================================================================
Installing:
cylc noarch 7.8.0_20_g1f592_dirty-1 /cylc-7.8.0_20_g1f592_dirty-1.noarch 9.7 M
Installing for dependencies:
less x86_64 458-9.el7 base 120 k
python-jinja2 noarch 2.7.2-2.el7 base 515 k
python-markupsafe x86_64 0.11-10.el7 base 25 k
Transaction Summary
===================================================================================================================================================================================================================
Install 1 Package (+3 Dependent packages)
Total size: 10 M
Total download size: 660 k
Installed size: 13 M
Downloading packages:
(1/3): less-458-9.el7.x86_64.rpm | 120 kB 00:00:00
(2/3): python-markupsafe-0.11-10.el7.x86_64.rpm | 25 kB 00:00:00
(3/3): python-jinja2-2.7.2-2.el7.noarch.rpm | 515 kB 00:00:00
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 906 kB/s | 660 kB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : python-markupsafe-0.11-10.el7.x86_64 1/4
Installing : python-jinja2-2.7.2-2.el7.noarch 2/4
Installing : less-458-9.el7.x86_64 3/4
Installing : cylc-7.8.0_20_g1f592_dirty-1.noarch 4/4
Verifying : cylc-7.8.0_20_g1f592_dirty-1.noarch 1/4
Verifying : python-jinja2-2.7.2-2.el7.noarch 2/4
Verifying : less-458-9.el7.x86_64 3/4
Verifying : python-markupsafe-0.11-10.el7.x86_64 4/4
Installed:
cylc.noarch 0:7.8.0_20_g1f592_dirty-1
Dependency Installed:
less.x86_64 0:458-9.el7 python-jinja2.noarch 0:2.7.2-2.el7 python-markupsafe.x86_64 0:0.11-10.el7
Complete! Interestingly... if you do not install [root@ead3bc7d5ffe /]# cylc check-software
Checking your software...
Individual results:
==========================================================================================
Package (version requirements) Outcome (version found)
==========================================================================================
*REQUIRED SOFTWARE*
Python (2.6+, <3).................................FOUND & min. version MET (2.7.5.final.0)
*OPTIONAL SOFTWARE for the GUI & dependency graph visualisation*
Traceback (most recent call last):
File "/usr/bin/cylc-check-software", line 335, in <module>
main()
File "/usr/bin/cylc-check-software", line 317, in main
functionality_print(tag)
File "/usr/bin/cylc-check-software", line 255, in functionality_print
opt_result[module] = check_py_module_ver(module, ver_req)
File "/usr/bin/cylc-check-software", line 168, in check_py_module_ver
import gtk
File "/usr/lib64/python2.7/site-packages/gtk-2.0/gtk/__init__.py", line 64, in <module>
_init()
File "/usr/lib64/python2.7/site-packages/gtk-2.0/gtk/__init__.py", line 52, in _init
_gtk.init_check()
RuntimeError: could not open display And to fix that in Docker. [root@ead3bc7d5ffe /]# yum install xorg-x11-server-Xvfb
[root@ead3bc7d5ffe /]# Xvfb :1 -screen 0 1024x768x16 &> xvfb.log &
[1] 771
[root@ead3bc7d5ffe /]# ps aux | grep X
root 771 0.3 0.2 99432 12508 pts/0 S 03:22 0:00 Xvfb :1 -screen 0 1024x768x16
root 774 0.0 0.0 9100 816 pts/0 S+ 03:22 0:00 grep --color=auto X
[root@ead3bc7d5ffe /]# DISPLAY=:1.0
[root@ead3bc7d5ffe /]# export DISPLAY
[root@ead3bc7d5ffe /]# cylc check-software
.... Now [root@ead3bc7d5ffe /]# cylc gui
Failed to open file “/usr/lib/python2.7/images/icon.png”: No such file or directory We are getting closer to a working RPM, that installs Cylc and all dependencies, built from sources. But the |
Blocking work on the modules, until #2918 is sorted out. Need to include the images in the package too... unless we want to go without building pygtk/pygraphviz, but instead ditching all GUI, and working only on the new module. If that's the case, #2918 can be closed, but this will be still blocked until Python3 work starts 👍 |
I'll create a "blocked" label for PRs like this... |
Good idea, thanks @hjoliver ! |
@kinow - did you mean to close this? |
I created this branch some months ago, while in parallel started bothering the maintainers of
isodatetime
to include thesetup.py
file over there 😬 Showed it to @hjoliver some days ago, but it was a huge mess of commits (you know, the ol' break here, fix over there kind of commits).After #1337, I thought it would be useful to open at least an WIP pull request so others could have a look at an example of a working
setup.py
for Cylc.At the moment it is producing an egg file for Cylc with
setuptools
+distutils
+ some Python witchcraft.