Proofscape Manage is intended for use by anyone interested in developing for the Proofscape projects. Its purpose is to help you set up and manage a development environment. It is designed for Unix-based systems, such as Linux or macOS. If anyone with expertise would like to help develop a Windows version, this would be most welcome.
You must have git and docker already installed. You need the ability to install
a Python 3.8 virtual environment. If you want to develop client-side code, you
need npm.
You will also need a good internet connection, as various steps involve pulling git repos, libraries from package repositories, and large Docker images from the web.
Choose a place in your filesystem for Proofscape to live. We recommend
~/proofscape, but you can choose any directory you want. For the remainder
of this guide, we will refer to this directory as PFSC_ROOT.
When making your choice, bear in mind that all of your Proofscape content
repos (if you plan on developing content) will live under PFSC_ROOT/lib,
while all the source repos will live under PFSC_ROOT/src. So choose a
convenient location.
Make the root directory if it doesn't already exist, cd into it, and clone this
repo there.
$ mkdir PFSC_ROOT
$ cd PFSC_ROOT
$ git clone https://github.com/proofscape/pfsc-manage.git
This project is designed for Python 3.8. If this is not available on
your system, we recommend using pyenv to
make it available.
Here and later in this guide, we will write a pyenv line to remind you to
use the right version of Python. If you choose to use an alternative way of
setting up a Python virtual environment at the desired version, you can ignore
these lines.
Set up the virtual environment as follows:
$ cd pfsc-manage
$ pyenv shell 3.8.12
$ python -m venv venv
$ . venv/bin/activate
(venv) $ pip install -U pip
(venv) $ pip install -e .
The last command performs a local installation of the pfsc-manage package
defined in this project. (See setup.py and manage.py.) This provides you
with a command line tool, pfsc, which you can use to perform all actions
necessary to set up the development environment, build docker images, run
container networks, etc.
Make a copy of the sample config file, at the top level of pfsc-manage:
$ cp sample_conf.py conf.py
The copy must be called conf.py.
You can use conf.py to make various settings, which should help you tailor
the tools provided by this repo to your use cases.
Note that conf.py is listed in .gitignore. It is not under
version control, so you should make backup copies as you see fit.
Alternatively, you may make as many copies of sample_conf.py as
you want, and store them in the conf_dir directory under any names you
choose. *.py files in this directory are ignored by git, so you should
again make your own backup copies.
Under this design, the conf.py at the top level of the project should not
be a file, but a symlink to one of the modules you define in conf_dir.
A working Proofscape system needs several subdirectories under PFSC_ROOT.
To make these, simply use:
(venv) $ pfsc makestruct
Note 1: Whenever you want to use the pfsc tool, the virtual environment
always needs to be active. This is indicated by the (venv) $ prompt.
Note 2: If you followed the recommended practice and put this repo (pfsc-manage)
directly under PFSC_ROOT, then the pfsc makestruct command will automatically
find the root dir, and figure out where to make all the subdirs. If you did it
differently, then you'll have to run pfsc makestruct --help for help on making
it work.
After running the pfsc makestruct command you should now have the following
directory structure:
PFSC_ROOT
|-- build
|-- deploy
| |-- .ssl
|-- graphdb
|-- lib
|-- PDFLibrary
|-- pfsc-manage
|-- src
|-- tmp
The purpose of the PFSC_ROOT/src dir is to be the home of any and all cloned
source repos on which you are working, or which are needed for building Docker
images. You're going to need at least the
pfsc-server repo, so go ahead and run
(venv) $ pfsc repo clone server
Afterward you might want to quickly cd into PFSC_ROOT/src and check that you
did indeed get a cloned copy of the pfsc-server Git repo.
If you are also interested in doing client-side development then you should also
(venv) $ pfsc repo clone ise
(venv) $ pfsc repo clone moose
in order to clone the pfsc-ise and pfsc-moose repos.
Note that these commands aren't doing anything too fancy. They are just cd-ing
into PFSC_ROOT/src and then running git clone with the approrpiate URLs. If
you wanted to, you could do this manually instead.
Install the dependencies for any repos you have cloned.
Like pfsc-manage, pfsc-server is a Python project. We therefore recommend
opening a new terminal tab, so that you can have a different virtual environment
active in each one. If you don't want to do that, you can always
(venv) $ deactivate
before switching between projects.
Now enter the pfsc-server project directory, and install the dependencies:
$ cd PFSC_ROOT/src/pfsc-server
$ pyenv shell 3.8.12
$ python -m venv venv
$ . venv/bin/activate
(venv) $ pip install -U pip
(venv) $ pip install -r req/dev.txt
(venv) $ pip install -r req/run.txt
For the JavaScript projects, you need npm.
$ cd PFSC_ROOT/src/pfsc-ise
$ npm install
$ cd PFSC_ROOT/src/pfsc-moose
$ npm install
For most development tasks you're going to need at least the pfsc-server
Docker image. The first time you build this can be quite slow, since we need
to begin by pulling base images.
First make sure the DOCKER_CMD setting in your conf.py is what it needs
to be (see comments in conf.py).
You're encouraged to look at
(venv) $ pfsc build server --help
and read about options that may eventually become relevant for you, but in order to get started quickly just do
(venv) $ pfsc build server latest
Again, this may take a while.
One of the main services provided by the pfsc-manage project is to automatically
generate deployment code. This includes docker-compose YAMLs, Nginx confs, .env
files, and more.
The command that generates all these files for you is pfsc deploy generate, and
you can learn a lot from
(venv) $ pfsc deploy generate --help
The expectation is that you will use pfsc deploy generate many times. Each time
you use it, it generates a new directory under PFSC_ROOT/deploy. You can either
set the name of the generated directory using the --dirname option, or else let
its name be automatically generated, using random words and a timestamp.
We will refer to each such directory as a deployment directory. You can feel free to delete these directories any time you want, but do not rename them, since their names are built into the code they contain, and they will not work if renamed.
In order to generate your first deployment directory, run
(venv) $ pfsc deploy generate
and accept the default setting for each interactive prompt that comes up (just hit
enter/return on each one). Now cd to PFSC_ROOT/deploy and you should see the new
directory. Its name will consist of random words and a timestamp.
For the remainder of this guide we will refer to the deployment directory you just
generated as FIRST_DEPLOY_DIR.
Before we move on, you should cd into FIRST_DEPLOY_DIR and take a look around.
Examine the contents of each of the generated files, and understand (or at least guess)
what they will do for you.
Note that among the generated files is a copy of conf.py as it stood
at the time that you ran pfsc deploy generate. This should help you to recreate the
same deployment later, if need be.
THIS SECTION IS OPTIONAL
Depending on how you will set up your graph database (gdb), you may want to define indexes to accelerate queries. Now that we have a deployment directory (which defines the location of the gdb, among other things), we can do this as follows.
NOTE: If you accepted the default settings when you ran pfsc deploy generate,
then you are not using Neo4j, and should skip this section.
If you are going to use Neo4j as your graph database engine, and want to define indexes to accelerate queries, you can proceed as follows.
First, we need a Neo4j container to be running. Therefore do the following:
(venv) $ cd PFSC_ROOT/deploy/FIRST_DEPLOY_DIR/layers
(venv) $ ./dc 100_db.yml up
(venv) $ pfsc gdb setup
Here you are using the automatically generated dc script to bring up just the
database layer of containers, before asking pfsc gdb setup to make the desired
indexes in the graph database.
Before taking down the Neo4j container, you might want to navigate to
localhost:7474 in a web browser (substitute your chosen port number if you changed
this setting in your conf.py) and run
CALL db.indexes()
in the Neo4j interactive browser, in order to see the indexes you just made.
In any case, when you're ready, take the database layer down with
(venv) $ ./dc 100_db.yml down
Support for setting up other graph databases...
You will sometimes be running the Proofscape server locally (like when running
its unit tests) and sometimes in a Docker container (like when testing through
a web browser). Therefore each deployment dir contains two corresponding config
files, local.env and docker.env, as you can see in FIRST_DEPLOY_DIR.
You don't have to worry about using docker.env. It is volume-mounted into the
appropriate container(s) by the docker-compose code in FIRST_DEPLOY_DIR, so
this is taken care of automatically.
You do however have to be conscious about the local configuration. Each time you
run pfsc deploy generate, the local.env for your new deployment dir is symlinked
into PFSC_ROOT/src/pfsc-server/instance/.env (unless you use the -L
or --no-local flag). This means that in general the most recently generated
deployment dir is the one whose local.env will configure the Proofscape server
during unit tests.
If you want to reactivate the local.env from an older deployment dir, you can
always do this using the pfsc deploy local command. For example, suppose you
have
PFSC_ROOT
|-- deploy
|-- foo
|-- bar
and bar's local.env is currently active. (Again, this simply means that the
symlink at PFSC_ROOT/src/pfsc-server/instance/.env currently points to it.)
If you wanted to instead reactivate foo's local.env, you could do this with
(venv) $ pfsc deploy local foo
In fact even
(venv) $ pfsc deploy local f
would work, since, rather than spelling out the entire name of the desired deployment dir, any prefix that uniquely determines the directory will work.
More sections to this guide will be coming soon. In the meantime we invite you to use the Discussions tab for this repo if you need help.