# Proofscape Manage

Proofscape Manage is intended for use by anyone interested in developing for the
Proofscape projects. Its purpose is to help you set up and manage a development
environment. It is designed for Unix-based systems, such as Linux or macOS.
If anyone with expertise would like to help develop a Windows version, this would
be most welcome.

You must have `git` and `docker` already installed. You need the ability to install
a Python 3.8 virtual environment. If you want to develop client-side code, you
need `npm`.

You will also need a good internet connection, as various steps involve pulling
git repos, libraries from package repositories, and large Docker images from the
web.


## Root Directory

Choose a place in your filesystem for Proofscape to live. We recommend
`~/proofscape`, but you can choose any directory you want. For the remainder
of this guide, we will refer to this directory as `PFSC_ROOT`.

When making your choice, bear in mind that all of your Proofscape _content_
repos (if you plan on developing content) will live under `PFSC_ROOT/lib`,
while all the source repos will live under `PFSC_ROOT/src`. So choose a
convenient location.


## Clone this repo, under the root dir

Make the root directory if it doesn't already exist, `cd` into it, and clone this
repo there.

    $ mkdir PFSC_ROOT
    $ cd PFSC_ROOT
    $ git clone https://github.com/proofscape/pfsc-manage.git


## Establish and activate a virtual environment

This project is designed for Python 3.8. If this is not available on
your system, we recommend using [`pyenv`](https://github.com/pyenv/pyenv) to
make it available.

Here and later in this guide, we will write a `pyenv` line to remind you to
use the right version of Python. If you choose to use an alternative way of
setting up a Python virtual environment at the desired version, you can ignore
these lines.

Set up the virtual environment as follows:

    $ cd pfsc-manage
    $ pyenv shell 3.8.12
    $ python -m venv venv
    $ . venv/bin/activate
    (venv) $ pip install -U pip
    (venv) $ pip install -e .

The last command performs a local installation of the `pfsc-manage` package
defined in this project. (See `setup.py` and `manage.py`.) This provides you
with a command line tool, `pfsc`, which you can use to perform all actions
necessary to set up the development environment, build docker images, run
container networks, etc.


## Make a `conf.py`

Make a copy of the sample config file, at the top level of `pfsc-manage`:

    $ cp sample_conf.py conf.py

The copy _must_ be called `conf.py`.

You can use `conf.py` to make various settings, which should help you tailor
the tools provided by this repo to your use cases.

Note that `conf.py` is listed in `.gitignore`. It is not under
version control, so you should make backup copies as you see fit.

### Multiple configurations

Alternatively, you may make as many copies of `sample_conf.py` as
you want, and store them in the `conf_dir` directory under any names you
choose. `*.py` files in this directory are ignored by `git`, so you should
again make your own backup copies.

Under this design, the `conf.py` at the top level of the project should not
be a file, but a symlink to one of the modules you define in `conf_dir`.


## Build the directory structure

A working Proofscape system needs several subdirectories under `PFSC_ROOT`.
To make these, simply use:

    (venv) $ pfsc makestruct

Note 1: Whenever you want to use the `pfsc` tool, the virtual environment
always needs to be active. This is indicated by the `(venv) $` prompt.

Note 2: If you followed the recommended practice and put this repo (`pfsc-manage`)
directly under `PFSC_ROOT`, then the `pfsc makestruct` command will automatically
find the root dir, and figure out where to make all the subdirs. If you did it
differently, then you'll have to run `pfsc makestruct --help` for help on making
it work.

After running the `pfsc makestruct` command you should now have the following
directory structure:

    PFSC_ROOT
     |-- build
     |-- deploy
     |    |-- .ssl
     |-- graphdb
     |-- lib
     |-- PDFLibrary
     |-- pfsc-manage
     |-- src
          |-- tmp


## Clone source repos

The purpose of the `PFSC_ROOT/src` dir is to be the home of any and all cloned
source repos on which you are working, or which are needed for building Docker
images. You're going to need at least the
`pfsc-server` repo, so go ahead and run

    (venv) $ pfsc repo clone server

Afterward you might want to quickly `cd` into `PFSC_ROOT/src` and check that you
did indeed get a cloned copy of the `pfsc-server` Git repo.

If you are also interested in doing client-side development then you should also

    (venv) $ pfsc repo clone ise
    (venv) $ pfsc repo clone moose

in order to clone the `pfsc-ise` and `pfsc-moose` repos.

Note that these commands aren't doing anything too fancy. They are just `cd`-ing
into `PFSC_ROOT/src` and then running `git clone` with the approrpiate `URLs`. If
you wanted to, you could do this manually instead.


## Install Dependencies

Install the dependencies for any repos you have cloned.

### `pfsc-server`

Like `pfsc-manage`, `pfsc-server` is a Python project. We therefore recommend
opening a new terminal tab, so that you can have a different virtual environment
active in each one. If you don't want to do that, you can always

    (venv) $ deactivate

before switching between projects.

Now enter the `pfsc-server` project directory, and install the dependencies:

    $ cd PFSC_ROOT/src/pfsc-server
    $ pyenv shell 3.8.12
    $ python -m venv venv
    $ . venv/bin/activate
    (venv) $ pip install -U pip
    (venv) $ pip install -r req/dev.txt
    (venv) $ pip install -r req/run.txt

### `pfsc-ise`

For the JavaScript projects, you need `npm`.

    $ cd PFSC_ROOT/src/pfsc-ise
    $ npm install

### `pfsc-moose`

    $ cd PFSC_ROOT/src/pfsc-moose
    $ npm install


## Build Docker images

For most development tasks you're going to need at least the `pfsc-server`
Docker image. The first time you build this can be quite slow, since we need
to begin by pulling base images.

First make sure the `DOCKER_CMD` setting in your `conf.py` is what it needs
to be (see comments in `conf.py`).

You're encouraged to look at

    (venv) $ pfsc build server --help

and read about options that may eventually become relevant for you, but in
order to get started quickly just do

    (venv) $ pfsc build server latest

Again, this may take a while.


## Make a first deployment directory

One of the main services provided by the `pfsc-manage` project is to automatically
generate deployment code. This includes docker-compose YAMLs, Nginx confs, `.env`
files, and more.

The command that generates all these files for you is `pfsc deploy generate`, and
you can learn a lot from

    (venv) $ pfsc deploy generate --help

The expectation is that you will use `pfsc deploy generate` many times. Each time
you use it, it generates a new directory under `PFSC_ROOT/deploy`. You can either
set the name of the generated directory using the `--dirname` option, or else let
its name be automatically generated, using random words and a timestamp.

We will refer to each such directory as a _deployment directory_.
You can feel free to delete these directories any time you want, but do not rename
them, since their names are built into the code they contain, and they will not
work if renamed.

In order to generate your first deployment directory, run

    (venv) $ pfsc deploy generate

and accept the default setting for each interactive prompt that comes up (just hit
enter/return on each one). Now `cd` to `PFSC_ROOT/deploy` and you should see the new
directory. Its name will consist of random words and a timestamp.

For the remainder of this guide we will refer to the deployment directory you just
generated as `FIRST_DEPLOY_DIR`.

Before we move on, you should `cd` into `FIRST_DEPLOY_DIR` and take a look around.
Examine the contents of each of the generated files, and understand (or at least guess)
what they will do for you.

Note that among the generated files is a copy of `conf.py` as it stood
at the time that you ran `pfsc deploy generate`. This should help you to recreate the
same deployment later, if need be.


## Set up databases

THIS SECTION IS OPTIONAL

Depending on how you will set up your graph database (gdb), you may want to
define indexes to accelerate queries.
Now that we have a deployment directory (which defines the location of the
gdb, among  other things), we can do this as follows.

### Defining indexes if using Neo4j

NOTE: If you accepted the default settings when you ran `pfsc deploy generate`,
then you are _not_ using Neo4j, and should skip this section.

If you are going to use Neo4j as your graph database engine, and want to define
indexes to accelerate queries, you can proceed as follows.

First, we need a Neo4j container to be running.
Therefore do the following:

    (venv) $ cd PFSC_ROOT/deploy/FIRST_DEPLOY_DIR/layers
    (venv) $ ./dc 100_db.yml up
    (venv) $ pfsc gdb setup

Here you are using the automatically generated `dc` script to bring up just the
database layer of containers, before asking `pfsc gdb setup` to make the desired
indexes in the graph database.

Before taking down the Neo4j container, you might want to navigate to
`localhost:7474` in a web browser (substitute your chosen port number if you changed
this setting in your `conf.py`) and run

    CALL db.indexes()

in the Neo4j interactive browser, in order to see the indexes you just made.

In any case, when you're ready, take the database layer down with

    (venv) $ ./dc 100_db.yml down

### Coming soon...

Support for setting up other graph databases...



## Local config

You will sometimes be running the Proofscape server locally (like when running
its unit tests) and sometimes in a Docker container (like when testing through
a web browser). Therefore each deployment dir contains two corresponding config
files, `local.env` and `docker.env`, as you can see in `FIRST_DEPLOY_DIR`.

You don't have to worry about using `docker.env`. It is volume-mounted into the
appropriate container(s) by the `docker-compose` code in `FIRST_DEPLOY_DIR`, so
this is taken care of automatically.

You do however have to be conscious about the local configuration. Each time you
run `pfsc deploy generate`, the `local.env` for your new deployment dir is symlinked
into `PFSC_ROOT/src/pfsc-server/instance/.env` (unless you use the `-L`
or `--no-local` flag). This means that in general _the most recently generated
deployment dir is the one whose `local.env` will configure the Proofscape server
during unit tests_.

If you want to reactivate the `local.env` from an older deployment dir, you can
always do this using the `pfsc deploy local` command. For example, suppose you
have

    PFSC_ROOT
     |-- deploy
          |-- foo
          |-- bar

and `bar`'s `local.env` is currently active. (Again, this simply means that the
symlink at `PFSC_ROOT/src/pfsc-server/instance/.env` currently points to it.)
If you wanted to instead reactivate `foo`'s `local.env`, you could do this with

    (venv) $ pfsc deploy local foo

In fact even

    (venv) $ pfsc deploy local f

would work, since, rather than spelling out the entire name of the desired
deployment dir, any prefix that uniquely determines the directory will work.


## Under construction

More sections to this guide will be coming soon. In the meantime we invite you
to use [the Discussions tab for this repo](https://github.com/proofscape/pfsc-manage/discussions)
if you need help.