This project is no longer publicly maintained

As of Nov. 1, 2022, Release 3.0.0 is the last public release of this project.

The TolaData team and Mercy Corps offer no warranty or support for this code. Use at your own risk.

Mercy Corps team members should direct questions to the internal TolaData team.

---

Tola Activity

The build status of the dev branch is:

TolaActivity is a software tool that is used to track program performance of development work across the globe. It includes the ability to set periodic targets for each program indicator, track those targets over time, and generate filterable IPTT reports on demand. It also includes a flexible Results Framework builder to enable hierarchical organization of indicators.

Creating a local TolaActivity instance

Running a local instance of TolaActivity makes development much faster. These instructions should get you up and running with a minimum of fuss if you have macOS or one of the many Ubunten. If they do not, we accept pull requests updating it. :)

Through all of these instructions, it is very important that a plain-text editor is used to edit the text files. For instance, TextEdit or Notepad are fine, MS Word is emphatically not. Even some plain-text editors default to a rich text format, so make sure you are saving plain text.

Install software dependencies

At this itme, TolaActivity requires Python 3. It has been thoroughly tested with versions 3.6, and lightly tested with version 3.8. It is suggested that Developers use Python 3.8 or greater. These instructions assume MySQL is being used as Django's datastore. Version 5.7 has been thoroughly tested, but version 8 should work as well.

macOS

On macOS, you can use Homebrew to install much of the software needed for TolaActivity. To see if you have Homebrew installed, run which brew at the command line. If you don't get a file path, it is not installed and you should run the following command from the command line.

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Before starting the install process, you may also want to request a copy of the settings.secret.yml file and a dump of an existing TolaActivity database.

To see if you have Python 3 or MySQL installed, run the following commands:

$ python3 --version
$ mysql --version

At the Terminal command line, run the following commands to install:

$ brew install python@3.8
$ brew install mysql@5.7

Add these lines to your shell profile ~/.bash_profile or ~/.zshrc (you may need to create it)

export PATH="/usr/local/opt/openssl/bin:$PATH"
export LIBRARY_PATH="/usr/local/opt/openssl/lib/:$LIBRARY_PATH"
export PATH="/usr/local/opt/mysql@5.7/bin:$PATH"

export LDFLAGS="-L/usr/local/opt/openssl/lib"
export CPPFLAGS="-I/usr/local/opt/openssl/include"
export LDFLAGS="-L/usr/local/opt/mysql@5.7/lib"
export CPPFLAGS="-I/usr/local/opt/mysql@5.7/include"

For macOS with ARM chip add this instead:

export PATH="/opt/homebrew/opt/openssl@1.1/bin:$PATH"
export LIBRARY_PATH="/opt/homebrew/opt/openssl@1.1/lib/:$LIBRARY_PATH"
export PATH="/opt/homebrew/opt/mysql@5.7/bin:$PATH"
export PATH="/opt/homebrew/opt/mysql-client@5.7/bin:$PATH"
export PKG_CONFIG_PATH="/opt/homebrew/opt/mysql-client@5.7/lib/pkgconfig"

export LDFLAGS="-L/opt/homebrew/opt/openssl@1.1/lib"
export CPPFLAGS="-I/opt/homebrew/opt/openssl@1.1/include"
export LDFLAGS="-L/opt/homebrew/opt/mysql@5.7/lib"
export CPPFLAGS="-I/opt/homebrew/opt/mysql@5.7/include"

Back at the command line:

$ source ~/.bash_profile # if using bash, e.g on MacOS 10.14 or older
$ source ~/.zshrc # if using zsh, e.g. on MacOS 10.15 or newer
$ brew install mysql-client
$ brew install py2cairo pango
$ pip3 install virtualenv

For macOS ARM chip add this instead you might have to specify the mysql_client version:

$ brew install mysql-client@5.7

You should now also start the mysql server:

brew services start mysql@5.7

Now that you've installed the necessary macOS software dependencies, you can move down to the section Install and configure the TolaActivity source files.

Windows

For Windows installations, install the Windows Subsystem for Linux using a Ubuntu distribution as the base. Once this has successfully been installed, launch a Powershell window and proceed with the Ubuntu instructions below.

Ubuntu

You will need Python3 installed. TolaData has been tested on Python 3.6 but should work on later versions as well.

$ python3 --version

If one of these doesn't yield a Python installation, install it.

$ sudo apt update
$ sudo apt install python3

Now install some other linux packages that will be needed later.

$ sudo apt install mysql-server libmysqlclient-dev mysql-client
$ sudo apt install libsasl2-dev libssl-dev
$ sudo apt-get install python3-dev libffi-dev
$ sudo apt-get install libxml2-dev libxmlsec1-dev libxmlsec1-openssl pkg-config
$ sudo apt-get install libcairo2-dev libpango1.0-dev
$ sudo apt install virtualenv

Note that some users have had to manually build the libxml2 library, as mentioned in the python-xmlsec documentation.

wget http://xmlsoft.org/sources/libxml2-2.9.1.tar.gz
tar -xvf libxml2-2.9.1.tar.gz
cd libxml2-2.9.1
./configure && make && make install

You should now also start the mysql server:

sudo service mysql start

Now you're ready to install and configure the TolaData source files.

Install and configure the TolaActivity source files

Install the source files and the python libraries

All operating systems should now be ready to install TolaActivity source files and do some OS-independent installations.

$ git clone https://github.com/mercycorps/toladata.git
$ cd toladata
$ virtualenv -p python3.8 venv
$ source venv/bin/activate  # you should see '(venv)' appear on the left of your command prompt

Install pip-tools and use the pip-sync command to install, upgrade, and uninstall your virtual environment with the necessary dependencies to match the requirements.txt and dev-requirements.txt contents.

$ pip install pip-tools
$ pip-sync requirements.txt dev-requirements.txt

You may need to perform additional installations if you receive an error installing the requirements.txt. For example:

• Failed building wheel for xmlsec. Run command brew install libxmlsec1

Modify the config file

If you have a copy of the settings.secret.yml file, place it in the TolaActivity/config directory. In Windows, you will need to copy it from where the file is stored on your hard drive. You may want to modify the file per the instructions below (not with an MS Office product!) before moving it to the TolaActivity folder. To move the file, if the file is in your Downloads directory, you could use a command that looks something like this:

$ cp /mnt/c/Users/<your_username>/Downloads/settings.secret.yml config

If you don't have a copy of the settings.secret.py file, then copy the sample file thusly:

$ cp config/sample-settings.secret.yml config/settings.secret.yml

Open the config/settings.secret.yml file with a plain-text editor.

1. Find the node named, "DATABASES" and set the database NAME, USER, and PASSWORD as appropriate. If you have a dump of an existing TolaActivity database, set the NAME to "tola_activty". The result should resemble the following:

   DATABASES:
     default:
       ENGINE: "django.db.backends.mysql"
       NAME: "<db_name>" # "myDatabaseName"
       USER: "<db_username>" # "myUsername"
       PASSWORD: "<password>" # "SooperSekritWord"
       HOST: "localhost"
       PORT: ""

   The rest of the instructions uses the values commented above.

2. Create the log directory

   $ mkdir /User/<username>/logs

   Modify the LOGFILE entry so it points to the file in the logs directory you just created. For example:

   LOGFILE: '/home/<username>/logs/django_error.log'

Set up Django's MySQL backing store

Log into mysql and create the database, create the user, and grant permissions with the following commands using the same database Name, User, and Password used in the settings.secret.yml file. The commands below uses the example values commented in the previous section.

$ mysql -u root  # Ubuntu users will need to use sudo for this line
mysql> CREATE DATABASE myDatabaseName;
mysql> CREATE USER 'myUsername'@'localhost' IDENTIFIED BY 'SooperSekritWord';
mysql> GRANT ALL ON myDatabaseName.* TO 'myUsername'@'localhost';
mysql> exit

Set up Django

If you have a copy of an existing Tola database, you can load through the mysql command using the path to the .sql file. When prompted, you should provide the database password.

$ mysql -u tola -p tola_activity < /path/to/file.sql

Run the database migrations, even if you just uploaded the .sql file.

$ python manage.py migrate

If you get an error during migration

During migration, you might see an error like the following:

Applying social_django.0005_auto_20160727_2333
  django.db.utils.OperationalError: (1071, 'Specified key was too long; max key length is 1000 bytes')

The social_django app creates a unique_together relationship between two rows that concatenate to a value too long for the destination row. To fix this, manually change the following two fields:

• social_auth_association.server_url to varchar(100)
• social_auth_association.handle to varchar(100)

You can address this through the MySQL CLI:

$ mysql -u tola -p tola_activity  # you will be prompted for the database password
mysql> ALTER TABLE social_auth_association MODIFY server_url varchar(100) NOT NULL;
mysql> ALTER TABLE social_auth_association MODIFY handle varchar(100) NOT NULL;
mysql> exit

...then re-run the migration as normal:

$ python manage.py migrate

Start the server:

Start the Django development server with the runserver command.

$ python manage.py runserver
Performing system checks...

System check identified 1 issue (0 silenced).
March 20, 2018 - 11:51:55
Django version 1.11.2, using settings 'tola.settings.local'
Starting development server at http://0.0.0.0:8000/
Quit the server with CONTROL-C.

Try launching Tola in your browser

In your browser, navigate to localhost:8000. You should see a TolaActivity login screen. You should also be able login through the Google+ link.

Creating a local user

If you are not logging in through the Google+ link, you will need to create a local user so you can log in. You can use the Python Shell or MySQL to create a local superuser and add the additional data that is required.

Using Python Shell

$ ./manage.py shell
>>> from django.contrib.auth.models import User
>>> from workflow.models impot TolaUser
>>> me = User.objects.create(username="<your_email>", email="<your_email">, is_superuser=True, is_staff=True)
>>> me.set_password(<password>)
>>> me.save()
>>> TolaUser.objects.create(user=me, name="<first_last>")
>>> exit()

Using MySQL

Use the createsuperuser command and enter in a username and password to be used to login. When it asks for an email, you can just hit the Enter key to skip the question (email not needed to login).

$ python manage.py createsuperuser
Username: <my_username>
Email address:
Password: <my_password>
Password (again): <my_password>
Superuser created successfully.

Log into MySQL and get the id of the record you just added from the auth.user table using the following query:

$ mysql -u tola -p tola_activity
mysql> SELECT id, username FROM auth_user ORDER BY id DESC LIMIT 5;
+----+----------+
| id | username |
+----+----------+
|  1 | myname   |
+----+----------+

Note the value for id to use in the next step.

Insert the id value from the auth_user table into the user_id field of the workflow_tolauser table (Name can be first and/or last name, username, or anything else):

mysql> INSERT INTO workflow_tolauser (name, privacy_disclaimer_accepted, user_id, language) VALUES ("<My Name>", 1, "<id>", "en");
mysql> exit

Restart the Tola Activity server

$ python manage.py runserver
Performing system checks...

System check identified no issues (0 silenced).
March 26, 2018 - 23:38:10
Django version 1.11.2, using settings 'tola.settings.local'
Starting development server at http://127.0.0.1:8000/
Quit the server with CONTROL-C.

You should now be able to login using the username and password fields of the login screen. Once you have logged in, you will be taken to the home page.

For Developers

Front-end development setup and dev server

Tola uses Webpack and npm installed packages in node_modules to build javascript bundles. During development, you will need to run the webpack development server to have the latest JS bundles available, and to re-generate the bundles if you modify any JS handled by Webpack.

Directions for installing npm can be found below. It can also be installed via homebrew on macOS.

$ brew install npm (sudo apt install npm )

Our current front-end build harness uses node-sass v4.12* which requires node v12 (or possibly older). If you are working with more than one project, you may want to install nvm (Node Version Manager). To install nvm:

$ curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash

After installing nvm, add version 12:

$ nvm install 12

Install all node_module package dependencies using npm

(optional) Switch to Node 12:

$ nvm use 12

To install:

$ cd /path/to/tola_data
$ npm run install:dev

Note: You made need to periodiclly run this after doing a git pull if package.json has been updated with new dependencies. This is similar to running pip install -r requements.txt if the requirements.txt has been updated.

Start the webpack development server

$ npm run watch

This should be done along side ./manage.py runserver

Run JS unit tests

$ npm test

It's also possible to run the tests in "watch" mode

$ npm test -- --watch

or

$ npm run test:watch

Build bundles for production

When you are ready to deploy to an external server, you will need to build and check-in the production ready bundles. These are generated with:

$ npm run build:prod

or use the alias

$ npm run build

Installing and running the front-end harness

See also the front-end architecture roadmap.

This is optional if you are not doing significant front-end development. You can bypass the frontend build by dropping selectors into /path/to/project/tola/static/css/app.css.

Installation & Startup:

1. Globally install npm. Here are general instructions to do so. On my Mac I prefer to install it via Homebrew
2. Install local dependencies:

   $ cd /path/to/project/
   $ npm install

   This will install all necessary node modules into node_modules at the project root.
3. Start a watch task that will copy necessary libraries and compile static files while you work:

   $ cd /path/to/project/
   $ npm start

   You can also configure PyCharm to run this task in the background:
   1. Select Run → Edit Configurations...
   2. Add new configuration (⌘-n or click the + button)
   3. Choose npm in the wee popup
   4. PyCharm should automagically select the correct package.json: /path/to/project/package.json
   5. Choose the Command start

npm will compile a single global css file at /path/to/project/tola/static/css/tola.css. This file includes the entire Bootstrap library (previously in bootstrap.min.css and bootstrap-multiselect.min.css), our custom selectors (previously in app.css), and overrides to Bootstrap (previously in bootstrap_overrides.css)

Other tips:

1. Never edit the compiled css (tola.css) directly. Any manual changes to compiled css files will be overwritten the next time the css is regenerated. They are theoretically retrievable via Git but see #3, below. Remember: you can always bypass the harness by dropping css selectors directly into app.css.

2. But seriously, you should just put your css into a .scss file & compile it properly. Valid css is also valid scss. If you’re not sure where to write a selector, append it to the end of the master scss file: /path/to/project/scss/tola.scss.

3. Please commit your compiled css to GitHub, preferably in the same commit as your edits to our scss files.

4. There is no need to resolve merge conflicts in compiled css. Resolve them in the scss files first, then regenerate your css and accept all changes from the right (HEAD) side.

5. Suggestions for Frontend coding practices are forthcoming.

Updating Python libraries

Tola uses pip-tools to manage Python dependencies. The requirements.in file is where changes to Python libraries should be made. pip-compile can then be used to generate the requirements.txt file. This allows us to list only those libraries we are explicitly adding in requirements.in, with a description of what the library is being used for. pip-tools is then used to handle adding dependencies, installation, and upgrades. See pip-tools documentation for details of use.