feat: Package as Python library #29

matthewfeickert · 2020-09-06T00:04:02Z

Partially addresses make pandamonium this pip-installable #26
Resolves Change default branch from python2 to master #28

This PR packages pandamonium as a modern Python library using a src/ directory structure that is dependent only on the required panda-client library on which it builds. It adds the pandamonium scripts as console scripts with the same CLI API and behavior within the scope of the virtual environment that the library is installed into. The old style of keeping things as global level scripts that can be installed and depend on existing global panda-client at the expense of making the library modules executable.

Requested squash and merge commit log

* Add BSD3 LICENSE
* Package as Python library with src/ directory structure
   - Uses setup.py, setup.cfg, and pyproject.toml
* Preserve global script style of installation through symlinking module files to top level of repository
* Add linting and packaging check GitHub Actions worfklows
* Add version updating through bumpversion
* Update installation and use instructions

It does not use Click to prove a CLI API for the time being as there is some discussion with @dguest on the desired level of dependencies. It is better to have the core functionality exist, so that after Issue #27 (providing a tag for the old style of global scope scripts) discussions can be had on this point.

A BSD3 license is also added which attributes @dguest as the author and copyright holder.

Additionally, GitHub Action workflows that lint the project with Black and pyflakes are added, in addition to a workflow that ensures that the library can be built using pep517.

As an example of how this looks at the dev stage on ATLAS Connect (that provides the required authentication certs that I setup below with emi and are the focus of Issue #24):

[feickert@login ~]$ hostname
login.usatlas.org
# I've already cloned my fork and am on the branch for this PR
[feickert@login ~]$ cd workarea/pandamonium/
[feickert@login pandamonium]$ virtualenv pandamonium-dev
[feickert@login pandamonium]$ source pandamonium-dev/bin/activate
(pandamonium-dev) [feickert@login pandamonium]$ which python
~/workarea/pandamonium/pandamonium-dev/bin/python
(pandamonium-dev) [feickert@login pandamonium]$ python --version
Python 2.7.5
(pandamonium-dev) [feickert@login pandamonium]$ pip --version
pip 20.2.2 from /home/feickert/workarea/pandamonium/pandamonium-dev/lib/python2.7/site-packages/pip (python 2.7)
(pandamonium-dev) [feickert@login pandamonium]$ pip install -e .
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Obtaining file:///home/feickert/workarea/pandamonium
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
Processing /home/feickert/.cache/pip/wheels/57/d0/3a/9922f4b7f7b07bd3f258322fadf315b061780f7b49c2c3f60a/panda_client-1.4.15-py2-none-any.whl
Installing collected packages: panda-client, pandamonium
  Running setup.py develop for pandamonium
Successfully installed panda-client-1.4.15 pandamonium
(pandamonium-dev) [feickert@login pandamonium]$ pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package      Version    Location
------------ ---------- ---------------------------------------
panda-client 1.4.15
pandamonium  0.1.dev116 /home/feickert/workarea/pandamonium/src
pip          20.2.2
setuptools   44.1.1
wheel        0.35.1
(pandamonium-dev) [feickert@login pandamonium]$ lsetup emi
************************************************************************
Requested:  emi ... 
 Setting up emi 4.0.2-1_200423.fix1 ... 
  Skipping: grid middleware already setup (from UI)
>>>>>>>>>>>>>>>>>>>>>>>>> Information for user <<<<<<<<<<<<<<<<<<<<<<<<<
 emi:
   Your proxy has 11h:59m:50s remaining
************************************************************************
(pandamonium-dev) [feickert@login pandamonium]$ pandamon | tail -n 5
done        12321     100%   user.feickert.364709.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ9WithSW.e7142_s3126_r9364_p4128._2020-09-03_09-35/ 
done        12320     100%   user.feickert.364706.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ6WithSW.e7142_s3126_r9364_p4128._2020-09-03_09-35/ 
done        12319     100%   user.feickert.364702.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ2WithSW.e7142_s3126_r9364_p4128._2020-09-03_09-35/ 
done        12318     100%   user.feickert.364700.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ0WithSW.e7142_s3126_r10201_p4128._2020-09-03_09-35/ 
done        12317     100%   user.feickert.364706.Pythia8EvtGen_A14NNPDF23LO_jetjet_JZ6WithSW.e7142_s3126_r10201_p4128._2020-09-03_09-35/

Keeping the old style means once can still do this

(base) [feickert@login ~]$ cd bin/
(base) [feickert@login bin]$ ls
virtualenv
(base) [feickert@login bin]$ git clone git@github.com:dguest/pandamonium.git
(base) [feickert@login bin]$ cd
(base) [feickert@login ~]$ tail -n 4 ~/.bashrc_user 
if [ -d "/home/${USER}/bin/pandamonium" ]; then
    PATH="/home/${USER}/bin/pandamonium${PATH:+:${PATH}}"; export PATH;
fi
(base) [feickert@login ~]$ source ~/.bashrc_user 
(base) [feickert@login ~]$ which pandamon
~/bin/pandamonium/pandamon

.bumpversion.cfg

matthewfeickert · 2020-09-06T00:07:11Z

~~Following the other required issues, the install docs need to be updated.~~ Done.

matthewfeickert · 2020-09-07T13:26:37Z

By the way, I highly advise that this become a squash and merge PR to make it a bit easier to manage. Otherwise I'll have to go in and do a bunch of rebasing and squashing to make it a much more manageable number of commits that are all meaningful and not a bunch of dev and testing.

These exists now as independnet scripts when installed in the virtual envrionment

matthewfeickert · 2020-09-08T18:36:34Z

@dguest This is ready for review now. I think you'll have some questions and comments, but in advance of all that I'd like to request that when this PR gets merged in that it is squashed and merged with the following commit log (instead of just the full list of all the commits in this PR):

Requested squash and merge commit log

* Add BSD3 LICENSE
* Package as Python library with src/ directory structure
   - Uses setup.py, setup.cfg, and pyproject.toml
* Preserve global script style of installation through symlinking module files to top level of repository
* Add linting and packaging check GitHub Actions worfklows
* Add version updating through bumpversion
* Update installation and use instructions

matthewfeickert · 2020-09-08T18:59:01Z

.github/workflows/publish-package.yml

+    - master
+    tags:
+    - v*
+  pull_request:
+    branches:
+    - master


@dguest Note that as these are set to master this workflow won't run until Issue #28 is resolved.

dguest · 2020-09-09T08:10:51Z

Neat, renaming the branches closed this MR. @matthewfeickert I don't know if you can retraget to master, that should be the same branch you'd initially intended to merge to.

matthewfeickert · 2020-09-09T08:17:41Z

Thanks for reopening this so that it properly targets master @dguest. This is now ready for review. 👍

dguest · 2020-09-16T06:53:27Z

@matthewfeickert, sorry this MR is hanging right now. I'm tempted to wait until we have a conclusion for #24. I guess we can discuss there: if it's going to take ages for CERN to fix something I suppose we should just merge this (or come up with another way to solve the error, i.e. by having instructions to self-sign the CERN root cert).

matthewfeickert · 2020-09-16T07:08:23Z

I'm tempted to wait until we have a conclusion for #24. I guess we can discuss there: if it's going to take ages for CERN to fix something I suppose we should just merge this (or come up with another way to solve the error, i.e. by having instructions to self-sign the CERN root cert).

Yeah I'm of like mind. Given that @kratsg and I pinged the JIRA ticket we might see some more activity soon. If we don't have a resolution in the next week then we can discuss trying to move forward with some other temporary solution.

matthewfeickert · 2020-09-21T18:24:53Z

@dguest As Issue #24 has been resolved can you please review this PR again?

README.md

dguest · 2020-09-21T18:40:26Z

README.md

@@ -1,6 +1,7 @@
-Pandamonium
-===========
+# Pandamonium


why you always gotta hate on my underline style? I like the thing that looks more like pretty in raw text form.

You're limited in the level of headings that you can use with this RST-like format. Additionally, it looks very much like the document is mixed RST and Makrdown which is pretty strange. (In all seriousness, have you tried writing in RST before? I'm not sure if you would like it more or less than Markdown).

I'll change this back though.

@dguest Is it possible to go to the equivalent of ### heading with this RST-like syntax? I didn't rewrite all of the README with this as it isn't clear how I'd do so given this limitation. It seems that this only goes to h2 headers given https://www.markdownguide.org/basic-syntax/#headings

yeah that's true. I always thought it was fine because ### it emphasizes these points less than a big fat underline. But I shouldn't hold up a MR over this obviously!

dguest · 2020-09-21T18:43:11Z

README.md

+### Notes if working on a remote server
+
+If you are working from a remote server where you do not have control over your Python runtimes (e.g. LXPLUS, ALTAS Connect login nodes) it is recommended that you bootstrap `virtualenv` and a default Python virtual environment by adding the following to your `.profile` or `.bash_profile`
+
+```
+# Ensure local virtualenv setup
+if [ ! -f "${HOME}/opt/venv/bin/virtualenv" ]; then
+    curl -sL --location --output /tmp/virtualenv.pyz https://bootstrap.pypa.io/virtualenv.pyz
+    python /tmp/virtualenv.pyz ~/opt/venv # Change this to python3 if available
+    ~/opt/venv/bin/pip install --upgrade pip
+    ~/opt/venv/bin/pip install virtualenv
+    mkdir -p ~/bin  # Ensure exists if new machine
+    ln -s ~/opt/venv/bin/virtualenv ~/bin/virtualenv
+fi
+
+# default venv from `virtualenv "${HOME}/.venvs/base"`
+if [ -d "${HOME}/.venvs/base" ]; then
+	  source "${HOME}/.venvs/base/bin/activate"
+fi
+```
+
+After that source your `.profile` or `.bash_profile` and then if you want to create a default Python virtual environment run
+
+```
+virtualenv "${HOME}/.venvs/base"
+```
+
+You will now be dropped into a virtual environment named `base` each time you login.
+The virtual environment is not special in anyway, so you should treat it as you would any other.
+


I would put this after the initial instructions. It's a lot of detail that people might already know. It's good to have it earlier but I'd rather keep the executive summary up front.

Moved this around now. So that it comes last under the Installation section.

dguest · 2020-09-21T18:46:03Z

README.md

+### Install from PyPI
+
+You can install [`pandamonium` from PyPI][pandamonium_PyPI] into any Python virtual environment by simply running
+
+```
+python -m pip install pandamonium
+```
+


this for example would be perfect higher up.

But I have an issue with your line wrapping. The idea here was that you should be able to run head -n100 README.md and get the basic instructions. If you're reading in a terminal it's sort of nice to have things wrap at 80.

Fixed this. This is good info to add in a CONTRIBUTING.md.

dguest · 2020-09-21T18:49:19Z

README.md

+
+#### Deprecation Warning
+
+You can currently just clone the repository and have `master` work the same way as `v0.1` on LXPLUS or ATLAS Connect, but this will be deprecated in the future in favor of installing `pandamonium` as a Python library.
+The motivation for this is that `pandamonium` does have hard requirements on other libraries, and it is better to fully contain them through the installation of the library through PyPI.
+
+[pandamonium_PyPI]: https://pypi.org/project/pandamonium/
+


would put after "use" section

README.md

dguest · 2020-09-21T18:58:40Z

In fact I think it might make sense to introduce another section on "job control" or something that talks about the stuff you can do once you've set up EMI: the resub / kill scripts won't work without that. It's probably also useful to make the distinction so people don't think they have to use EMI to run pandamon.

But that doesn't have to go into this MR either.

matthewfeickert · 2020-09-21T21:44:31Z

@dguest I think I've fixedup the README now with your feedback (thanks for it!). The only thing that I think we might still need to discuss once you review it again is if you're comfertable using # style of Markdown heading syntax to allow for h3 headings and lower as it seems that the RST-like syntax only allows for h1 and h2 headings.

I've additionally made Issues #31 and #32 to get addressed in separate PRs that can be focused on documentation, as this PR is focused on migration to a packaged repo structure.

dguest

Minor comments, but I'll address them after this review.

dguest · 2020-09-22T07:15:18Z

README.md

+If you want to install the original version of `pandamonium` before it became a
+library and was a set of global level Python scripts you can still do that.
+
+1. Clone the repository at tag [`v0.1`][tag_v0.1]


I think this is still out of date. It will work with master as well, won't it?

It is true that it will work with master but as that could possibly be deprecated in the future I wanted to add instructions that will for sure work.

dguest · 2020-09-22T07:17:55Z

README.md

+### Notes if working on a remote server
+
+If you are working from a remote server where you do not have control over your
+Python runtimes (e.g. LXPLUS, ALTAS Connect login nodes) it is recommended that
+you bootstrap `virtualenv` and a default Python virtual environment by adding
+the following to your `.bashrc` or `.bashrc_user`
+
+```
+# Ensure local virtualenv setup
+if [ ! -f "${HOME}/opt/venv/bin/virtualenv" ]; then
+    curl -sL --location --output /tmp/virtualenv.pyz https://bootstrap.pypa.io/virtualenv.pyz
+    python /tmp/virtualenv.pyz ~/opt/venv # Change this to python3 if available
+    ~/opt/venv/bin/pip install --upgrade pip
+    ~/opt/venv/bin/pip install virtualenv
+    mkdir -p ~/bin  # Ensure exists if new machine
+    ln -s ~/opt/venv/bin/virtualenv ~/bin/virtualenv
+fi
+
+# default venv from `virtualenv "${HOME}/.venvs/base"`
+if [ -d "${HOME}/.venvs/base" ]; then
+    source "${HOME}/.venvs/base/bin/activate"
+fi
+```
+
+After that source your `.profile` or `.bash_profile` and then if you want to
+create a default Python virtual environment run
+
+```
+virtualenv "${HOME}/.venvs/base"
+```
+
+You will now be dropped into a virtual environment named `base` each time you login.
+The virtual environment is not special in anyway, so you should treat it as you
+would any other.


I would put this after "use", and I think it's more appropriately called "working with virtualenv" or something like that. If someone has never used a venv this is a lot to dump on them, and if they have I guess they don't need this information so early on.

matthewfeickert added 3 commits September 5, 2020 01:27

Add src directory structure

e86a103

Add setup.py and setup.cfg

bed92d1

Add BSD3 LICENSE

844abf8

matthewfeickert commented Sep 6, 2020

View reviewed changes

.bumpversion.cfg Outdated Show resolved Hide resolved

matthewfeickert added 24 commits September 8, 2020 12:25

Add build system and check-manifest ignores

ccb5f3e

add MANIFEST.in

3e635f1

Add bumpversion config

42f7982

Add Github badge to README

cd26ab7

Require click

96f5aae

Add version

125af1c

Add cli basics

17d8ce3

Add lint CI

06d1235

Add publish-package CI

895c431

Add name correction

b5ef0b7

Fix name

47896dc

Rename to Python files and make not-executable

3bb6c05

Black everything in pre-commit

470949c

Placate most of pyflakes

977829c

Update gitignore

9b68df8

Add panda-clinet as required library

c309e7b

Add pandamonium scripts as console scripts

56e9c7d

These exists now as independnet scripts when installed in the virtual envrionment

Remove files not yet made from ignorelist

789186f

Update install instructions

fce1fc3

Use typical Markdown formatting

583594f

Add PyPI link

3445de4

Note that it is fully Python 3 compliant. :)

c0911c3

Note is a library

b9eaae0

Remove redundant check-manifest

09fc626

matthewfeickert force-pushed the feat/package-as-library branch from 78a46be to 92297c7 Compare September 8, 2020 18:25

matthewfeickert marked this pull request as ready for review September 8, 2020 18:35

matthewfeickert commented Sep 8, 2020

View reviewed changes

dguest closed this Sep 9, 2020

dguest reopened this Sep 9, 2020

dguest changed the base branch from python2 to master September 9, 2020 08:15

matthewfeickert mentioned this pull request Sep 21, 2020

SSL: CERTIFICATE_VERIFY_FAILED Error #24

Closed

matthewfeickert commented Sep 21, 2020

View reviewed changes

README.md Outdated Show resolved Hide resolved

dguest reviewed Sep 21, 2020

View reviewed changes

matthewfeickert added 5 commits September 21, 2020 16:20

Use Dan Markdown formatting

588776b

Reorder install instructions

1ba0c7a

Wrap text at 80 chars

532809a

Remove emi instructions and move deprecation warning

e0eeb1d

Add link to v0.1 release

d0c515d

This was referenced Sep 21, 2020

Add CONTRIBUTING.md #31

Open

Add "job control" section to README #32

Open

matthewfeickert requested a review from dguest September 21, 2020 21:42

dguest approved these changes Sep 22, 2020

View reviewed changes

dguest merged commit db20a61 into dguest:master Sep 22, 2020

matthewfeickert deleted the feat/package-as-library branch September 22, 2020 07:25

This was referenced Sep 22, 2020

make pandamonium this pip-installable #26

Closed

chore: Migrate from pep517 to python-build #33

Merged

feat: Package as Python library #29

feat: Package as Python library #29

Uh oh!

Conversation

matthewfeickert commented Sep 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

matthewfeickert commented Sep 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

matthewfeickert commented Sep 7, 2020

Uh oh!

matthewfeickert commented Sep 8, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dguest commented Sep 9, 2020

Uh oh!

matthewfeickert commented Sep 9, 2020

Uh oh!

dguest commented Sep 16, 2020

Uh oh!

matthewfeickert commented Sep 16, 2020

Uh oh!

matthewfeickert commented Sep 21, 2020

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dguest commented Sep 21, 2020

Uh oh!

matthewfeickert commented Sep 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dguest left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

matthewfeickert commented Sep 6, 2020 •

edited

Loading

matthewfeickert commented Sep 6, 2020 •

edited

Loading

matthewfeickert commented Sep 8, 2020 •

edited

Loading

matthewfeickert commented Sep 21, 2020 •

edited

Loading