4364 install prov flask #4461

pdurbin · 2018-02-08T18:18:39Z

connects to #4364 As a Dataverse Installation Administrator, I want to set up the Provenance system to run alongside Dataverse so that researchers using my installation can track and view provenance

Formatting is bad, will be fixed

See also ProvTools/prov-cpl#5

new file: scripts/vagrant/install-provenance.sh

coveralls · 2018-02-08T18:23:34Z

Coverage increased (+0.0009%) to 12.856% when pulling 01bfa41 on 4364-install-prov-flask into 2baa606 on develop.

pameyer

Short version:

cleaner separation between production build CPL RPM / production installation might be helpful (or existing separation clarified?). Possibly one script to setup the build VM/container, one to build the RPM, and one that installs and configures it?
preferable to use apache or nginx over the flask development server in production

pameyer · 2018-02-08T19:33:13Z

doc/sphinx-guides/source/developers/dev-environment.rst

+
+If you're feeling adventurous, you can attempt to install CPL directly on your Mac but this is not recommended. Rather, below CPL runs as a REST service within Vagrant.
+
+First, install Vagrant and VirtualBox as described in the :doc:`tools` section.


It seems like it would be less complex to move these two files (possibly renamed if they're too DV specific) to the prov-cpl repo.

If not, it would probably be more consistent if these were moved to conf/ (rather than doc/); or we should move the docker-related stuff to a different directory with the idea of having all dev VM/containerization/etc living in one place.

I put files in doc to make them easily downloadable from Sphinx. If anything I would move more conf files under Sphinx when they are used as examples. "Download this file, edit it, etc."

We can certainly see if @jacksonokuhn would like to have Vagrant files in the prov-cpl repo. I didn't want to inhibit progress by making that a dependency. My goal was to unblock Dataverse developers who wanted to get started on prov related work and needed a prov system running on their laptops. Incremental progress. We can make improvements in the future.

pameyer · 2018-02-08T19:36:00Z

doc/sphinx-guides/source/developers/dev-environment.rst

+
+curl -X PUT -d 'http://localhost:7777' http://localhost:8080/api/admin/settings/:ProvServiceUrl
+
+Installing CPL on Ubuntu


It would be good to mention which versions of ubuntu are expected to work. On version is mentioned below, but it might be clearer to move it to earlier in this section.

As someone who is relatively familiar with Vagrant, I would disagree with documenting the version of Ubuntu because what matters is the "base box" in the Vagrantfile. This is currently config.vm.box = "ubuntu/xenial64" which means 16.04 (the latest LTS as of this writing). Basically a comment here has the potential to get out of date. That said, I'm fine adding a comment if it's truly helpful.

pameyer · 2018-02-08T19:52:30Z

doc/sphinx-guides/source/developers/provenance.rst

+
+Build the RPM:
+
+``rpmbuild -ba dataverse-provenance.spec``


For consistency, it might be better to put these steps into a script that can be run from within the Vagrantfile. Manually editing the existing Vagrantfile might be simpler, but would also slow down automating things.

Maybe but I think @landreev and I are the only ones on the team who have ever built an RPM. I purposefully documented the steps in prose. I could see adding more automation once we've gotten more familiar with the process. Unfortunately, we can't use the HMDC Jenkins server to automate the RPM building because it's running el6 and there appear to be incompatibilities between el6 and el7. I left a comment about this at #4364 (comment) . For now the plan is the build this RPM in Vagrant and I don't want to be the only one who knows how. Note also that the version of the RPM matters. At this stage in the game, a human needs to make decisions about when to bump the version and when to get in touch with the upstream author about cutting a release, etc. It's complicated.

Ok - I was reminded of the discussions we had in #4369 about documentation / scripts for "surgery" on dependencies, which was why I mentioned it.

pameyer · 2018-02-08T19:56:43Z

doc/sphinx-guides/source/installation/provenance.rst

+
+Download :download:`dataverse-provenance.service  <../_static/installation/files/etc/systemd/system/dataverse-provenance.service>` and place it at ``/etc/systemd/system/dataverse-provenance.service``. Here are the default contents of that file but see below for adjustsments you might want to make:
+
+.. literalinclude:: ../_static/installation/files/etc/systemd/system/dataverse-provenance.service


Using the flask development server in production is probably something we should avoid.

Huh. Sure enough http://flask.pocoo.org/docs/0.12/deploying/#deployment says, "While lightweight and easy to use, Flask’s built-in server is not suitable for production as it doesn’t scale well and by default serves only one request at a time." Do you have any suggestions on how to deploy a Flask app in a better way? Heads up to @jacksonokuhn about this.

Apache / wsgi would be the way that I'd go; but there are other approaches.

pameyer · 2018-02-08T20:01:34Z

scripts/vagrant/install-provenance.sh

@@ -0,0 +1,79 @@
+#!/bin/bash


I'd suggest splitting these into build-provenance.sh and install-provenance.sh?

If anything, I might delete this script. It doesn't quite work, especially the part around postgresql-setup-conf.sql.

That sounds like a way to reduce the confusion.

pameyer · 2018-02-08T20:09:02Z

doc/sphinx-guides/source/_static/developers/prov/install/vagrant.sh

+sudo pip install flask
+cd /prov-cpl/bindings/python/RestAPI
+REST_SERVICE_USER=postgres # FIXME: create a "cplrest" user?
+su $REST_SERVICE_USER -s /bin/sh -c "python cpl-rest.py --host=0.0.0.0"


Not returning when vagrant up was a little surprising to me. I'd agree about creating a cplrest user; even though it shouldn't matter inside a dev VM.

I also ran into the vagrant halt doesn't halt problem with this file; but that may be more related to my system than this Vagrantfile.

Interesting. I haven't run into problems with vagrant halt (or vagrant destroy). I know it's weird to run the service as the postgres user but like you said, this is a dev VM and doesn't really matter. For production I'm using the name @jacksonokuhn and I talked about (cplservice) but I'm open to changing it.

cplservice for production sounds reasonable to me; but I'll defer to @jacksonokuhn . postgres for dev works (even if it's non-standard).

pameyer · 2018-02-08T20:28:40Z

doc/sphinx-guides/source/installation/provenance.rst

+Install ``dataverse-provenance`` RPM
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Download this :download:`dataverse-provenance RPM <../_static/installation/files/home/rpmbuild/rpmbuild/RPMS/x86_64/dataverse-provenance-0.1-1.x86_64.rpm>` and install it with:


guides.dataverse.org isn't currently available over https; distributing rpms over http is something that it's probably worth avoiding.

That's a fair point but no one has complained (that I'm aware of) about downloading the rapache RPM over plain HTTP: http://guides.dataverse.org/en/4.8.5/installation/r-rapache-tworavens.html#c-rapache

We did have a very security-minded individual take a look at our dev guide a couple years ago. You might enjoy reading through this comment, @pameyer: #2863 (comment)

If I installed rapache, it would be an issue I'd raise.

pameyer · 2018-02-08T20:36:37Z

scripts/vagrant/install-provenance.sh

+echo "Starting CPL REST service..."
+sudo bash -c "su -l $CPL_SERVICE_USER -c 'LD_LIBRARY_PATH=/usr/local/lib python cpl-rest.py --host=0.0.0.0 &'"
+echo "Checking version of CPL REST service..."
+curl http://localhost:5000/provapi/version


These checks failed when I ran them vagrant up; cpl-rest.py wasn't running; initial guess was cwd mismatch. Initial guess was incorrect; CPL module not installed in default python of cplsystem user.

As I mentioned above, the line about postgresql-setup-conf.sql isn't quite working. That's probably why the checks fail. The database user probably hasn't been created.

pameyer · 2018-02-08T20:37:49Z

scripts/vagrant/install-provenance.sh

+echo "Install dependencies for CPL REST service..."
+sudo yum install -y python-flask
+echo "Downloading CPL REST service script..."
+sudo bash -c "su -l $CPL_SERVICE_USER -c 'wget https://raw.githubusercontent.com/ProvTools/prov-cpl/8150ee315abc21712b49da2bf4cfdbf308eef1d7/bindings/python/RestAPI/cpl-rest.py'"


I'm confused why this script would download a file from a repository that has already been cloned.

The postgres user doesn't have access to /home/vagrant which is where the git repo has been cloned.

pdurbin · 2018-04-10T23:18:59Z

Closing. As explained at #4364 (comment) the RPM will be built using a Vagrantfile in the prov-cpl repo. The RPM will likely be hosted as a binary uploaded to a release of that repo. Installation instructions will reside in the prov-cpl repo.

pdurbin and others added 18 commits January 30, 2018 16:47

stub out install of prov flask in ubuntu in vagrant #4364

3f34669

for devs, provide Vagrant env to run CPL (Prov) as REST service #4364

8401132

add database setting :ProvServiceUrl #4364

9c19057

Added initial cpl ubuntu issues fixes #4364

fd21c12

Formatting is bad, will be fixed

restart postgres after config change (already running) #4364

264b5fc

better docs for installing prov for devs #4364

1fb7b50

note workaround for exposing prov system beyond localhost #4364

a4c850c

do initial sudo make install of prov-cpl on CentOS 7 #4364

d3eefcc

add RPM spec file and stub out install docs for RHEL/CentOS #4364

2f131b6

use new --host argument in cpl-rest.py #4364

c43a40d

See also ProvTools/prov-cpl#5

prov: clean up ubuntu dev instructions #4364

3dcb183

attempt to build Python bindings to CPL on CentOS 7 #4364

6b930af

add "install prov on centos" script (has FIXME) #4364

f31bf08

new file: scripts/vagrant/install-provenance.sh

build Python bindings for CPL using newer Swig #4364

de51414

install prov on centos in vagrant #4364

ea3fa5c

prov: add better RPM so sysadmins don't need make #4364

9a3261a

document how to install prov on CentOS 7 #4364

3dc2094

Merge branch 'develop' into 4364-install-prov-flask #4364

01bfa41

pdurbin mentioned this pull request Feb 8, 2018

As a Dataverse Installation Administrator, I want to set up the Provenance system to run alongside Dataverse so that researchers using my installation can track and view provenance #4364

Closed

pdurbin requested review from pameyer and landreev February 8, 2018 18:22

pdurbin requested a review from donsizemore February 8, 2018 20:05

pameyer reviewed Feb 8, 2018

View reviewed changes

pdurbin mentioned this pull request Feb 16, 2018

As a researcher, I want my provenance to be made available to other researchers at publish time so that they can make better use of my data #4381

Closed

pdurbin closed this Apr 10, 2018

matthew-a-dunlap deleted the 4364-install-prov-flask branch January 16, 2019 21:13

matthew-a-dunlap restored the 4364-install-prov-flask branch January 16, 2019 21:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

4364 install prov flask #4461

4364 install prov flask #4461

pdurbin commented Feb 8, 2018

coveralls commented Feb 8, 2018

pameyer left a comment

pameyer Feb 8, 2018

pdurbin Feb 12, 2018

pameyer Feb 8, 2018

pdurbin Feb 12, 2018

pameyer Feb 8, 2018

pdurbin Feb 12, 2018

pameyer Feb 12, 2018

pameyer Feb 8, 2018

pdurbin Feb 12, 2018

pameyer Feb 12, 2018

pameyer Feb 8, 2018

pdurbin Feb 12, 2018

pameyer Feb 12, 2018

pameyer Feb 8, 2018

pdurbin Feb 12, 2018

pameyer Feb 12, 2018

pameyer Feb 8, 2018

pdurbin Feb 12, 2018

pameyer Feb 12, 2018

pameyer Feb 8, 2018

pdurbin Feb 12, 2018

pameyer Feb 8, 2018

pdurbin Feb 12, 2018

pdurbin commented Apr 10, 2018


		If you're feeling adventurous, you can attempt to install CPL directly on your Mac but this is not recommended. Rather, below CPL runs as a REST service within Vagrant.

		First, install Vagrant and VirtualBox as described in the :doc:`tools` section.


		curl -X PUT -d 'http://localhost:7777' http://localhost:8080/api/admin/settings/:ProvServiceUrl

		Installing CPL on Ubuntu


		Download :download:`dataverse-provenance.service <../_static/installation/files/etc/systemd/system/dataverse-provenance.service>` and place it at ``/etc/systemd/system/dataverse-provenance.service``. Here are the default contents of that file but see below for adjustsments you might want to make:

		.. literalinclude:: ../_static/installation/files/etc/systemd/system/dataverse-provenance.service

4364 install prov flask #4461

4364 install prov flask #4461

Conversation

pdurbin commented Feb 8, 2018

coveralls commented Feb 8, 2018

pameyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pdurbin commented Apr 10, 2018