Selene -- Mycroft's Server Backend

Selene provides the services used by Mycroft Core to manage devices, skills and settings. It consists of two repositories. This one contains Python and SQL representing the database definition, data access layer, APIs and scripts. The second repository, Selene UI, contains Angular web applications that use the APIs defined in this repository.

There are four APIs defined in this repository, account management, single sign on, skill marketplace and device. The first three support account.mycroft.ai (aka home.mycroft.ai), sso.mycroft.ai, and market.mycroft.ai, respectively. The device API is how devices running Mycroft Core communicate with the server. Also included in this repository is a package containing batch scripts for maintenance and the definition of the database schema.

Each API is designed to run independently of the others. Code common to each of the APIs, such as the Data Access Layer, can be found in the "shared" directory. The shared code is an independent Python package required by each of the APIs. Each API has its own Pipfile so that it can be run in its own virtual environment.

Installation

The Python code utilizes features introduced in Python 3.7, such as data classes. Pipenv is used for virtual environment and package management. If you prefer to use pip and pyenv (or virtualenv), you can find the required libraries in the files named "Pipfile". These instructions will use pipenv commands.

If the Selene applications will be servicing a large number of devices (enterprise usage, for example), it is recommended that each of the applications run on their own server or virtual machine. This configuration makes it easier to scale and monitor each application independently. However, all applications can be run on a single server. This configuration could be more practical for a household running a handful of devices.

These instructions will assume a multi-server setup for several thousand devices. To run on a single server servicing a small number of devices, the recommended system requirements are 4 CPU, 8GB RAM and 100GB of disk. There are a lot of manual steps in this section that will eventually be replaced with an installation script.

All Selene applications are time zone agnostic. It is recommended that the time zone on any server running Selene be UTC.

It is recommended to create an application specific user. In these instructions this user will be mycroft.

Postgres DB

• Recommended server configuration: Ubuntu 18.04 LTS (server install), 2 CPU, 4GB RAM, 50GB disk.
• Use the package management system to install Python 3.7, Python 3 pip and PostgreSQL 10

sudo apt-get install postgresql python3.7 python python3-pip

• Set Postgres to start on boot

sudo systemctl enable postgresql

• Clone the selene-backend and documentation repositories

sudo mkdir -p /opt/selene
sudo chown -R mycroft:users /opt/selene
cd /opt/selene
git clone https://github.com/MycroftAI/selene-backend.git

• Create the virtual environment for the database code

sudo python3.7 -m pip install pipenv
cd /opt/selene/selene-backend/db
pipenv install

• Download files from geonames.org used to populate the geography schema tables

mkdir -p /opt/selene/data
cd /opt/selene/data
wget http://download.geonames.org/export/dump/countryInfo.txt
wget http://download.geonames.org/export/dump/timeZones.txt
wget http://download.geonames.org/export/dump/admin1CodesASCII.txt
wget http://download.geonames.org/export/dump/cities500.zip

• Add environment variables containing these passwords for the bootstrap script

export DB_PASSWORD=<selene user password>
export POSTGRES_PASSWORD=<postgres user password>

• Generate secure passwords for the postgres user and selene user on the database

sudo -u postgres psql -c "ALTER USER postgres PASSWORD '$POSTGRES_PASSWORD'"
sudo -u postgres psql -c "CREATE ROLE selene WITH LOGIN ENCRYPTED PASSWORD '$DB_PASSWORD'"

• Run the bootstrap script

cd /opt/selene/selene-backend/db/scripts
pipenv run python bootstrap_mycroft_db.py

• Note: if you get an authentication error you can temporarily edit /etc/postgresql/<version>/main/pg_hba.conf replacing the following lines:

# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust

• By default, Postgres only listens on localhost. This will not do for a multi-server setup. Change the listen_addresses value in the posgresql.conf file to the private IP of the database server. This file is owned by the postgres user so use the following command to edit it (substituting vi for your favorite editor)

sudo -u postgres vi /etc/postgres/10/main/postgresql.conf

• By default, Postgres only allows connections from localhost. This will not do for a multi-server setup either. Add an entry to the pg_hba.conf file for each server that needs to access this database. This file is also owned by the postgres user so use the following command to edit it (substituting vi for your favorite editor)

sudo -u postgres vi /etc/postgres/10/main/pg_hba.conf

• Instructions on how to update the pg_hba.conf file can be found in Postgres' documentation. Below is an example for reference.

# IPv4 Selene connections
host    mycroft         selene          <private IP address>/32          md5

• Restart Postgres for the postgres.conf and pg_hba.conf changes to take effect.

sudo systemctl restart postgresql

Redis DB

• Recommended server configuration: Ubuntu 18.04 LTS, 1 CPU, 1GB RAM, 5GB disk. So as to not reinvent the wheel, here are some easy-to-follow instructions for installing Redis on Ubuntu 18.04.
• By default, Redis only listens on local host. For multi-server setups, one additional step is to change the "bind" variable in /etc/redis/redis.conf to be the private IP of the Redis host.

APIs

The majority of the setup for each API is the same. This section defines the steps common to all APIs. Steps specific to each API will be defined in their respective sections.

• Add an application user to the VM. Either give this user sudo privileges or execute the sudo commands below as a user with sudo privileges. These instructions will assume a user name of "mycroft"
• Use the package management system to install Python 3.7, Python 3 pip and Python 3.7 Developer Tools

sudo apt install python3.7 python3-pip python3.7-dev
sudo python3.7 -m pip install pipenv

• Setup the Backend Application Directory

sudo mkdir -p /opt/selene
sudo chown -R mycroft:users /opt/selene

• Setup the Log Directory

sudo mkdir -p /var/log/mycroft
sudo chown -R mycroft:users /var/log/mycroft

• Clone the Selene Backend Repository

cd /opt/selene
git clone https://github.com/MycroftAI/selene-backend.git

• If running in a test environment, be sure to checkout the "test" branch of the repository

Single Sign On API

Recommended server configuration: Ubuntu 18.04 LTS, 1 CPU, 1GB RAM, 5GB disk

• Create the virtual environment and install the requirements for the application

cd /opt/selene/selene-backend/api/sso
pipenv install

Account API

• Recommended server configuration: Ubuntu 18.04 LTS, 1 CPU, 1GB RAM, 5GB disk
• Create the virtual environment and install the requirements for the application

cd /opt/selene/selene-backend/api/account
pipenv install

Marketplace API

• Recommended server configuration: Ubuntu 18.04 LTS, 1 CPU, 1GB RAM, 10GB disk
• Create the virtual environment and install the requirements for the application

cd /opt/selene/selene-backend/api/market
pipenv install

Device API

• Recommended server configuration: Ubuntu 18.04 LTS, 2 CPU, 2GB RAM, 50GB disk
• Create the virtual environment and install the requirements for the application

cd /opt/selene/selene-backend/api/public
pipenv install

Precise API

• Recommended server configuration: Ubuntu 18.04 LTS, 1 CPU, 1GB RAM, 5GB disk
• Create the virtual environment and install the requirements for the application

cd /opt/selene/selene-backend/api/precise
pipenv install

Running the APIs

Each API is configured to run on port 5000. This is not a problem if each is running in its own VM but will be an issue if all APIs are running on the same server, or if port 5000 is already in use. To address these scenarios, change the port numbering in the uwsgi.ini file for each API.

Single Sign On API

• The SSO application uses three JWTs for authentication. First is an access key, which is required to authenticate a user for API calls. Second is a refresh key that automatically refreshes the access key when it expires. Third is a reset key, which is used in a password reset scenario. Generate a secret key for each JWT.
• Any data that can identify a user is encrypted. Generate a salt that will be used with the encryption algorithm.
• Access to the Github API is required to support logging in with your Github account. Details can be found here.
• The password reset functionality sends an email to the user with a link to reset their password. Selene uses SendGrid to send these emails so a SendGrid account and API key are required.
• Define a systemd service to run the API. The service defines environment variables that use the secret and API keys generated in previous steps.

sudo vim /etc/systemd/system/sso_api.service

[Unit]
Description=Mycroft Single Sign On Api
After=network.target

[Service]
User=mycroft
Group=www-data
Restart=always
Type=simple
WorkingDirectory=/opt/selene/selene-backend/api/sso
ExecStart=/usr/local/bin/pipenv run uwsgi --ini uwsgi.ini
Environment=DB_HOST=<IP address or name of database host>
Environment=DB_NAME=mycroft
Environment=DB_PASSWORD=<selene database user password>
Environment=DB_PORT=5432
Environment=DB_USER=selene
Environment=GITHUB_CLIENT_ID=<github client id>
Environment=GITHUB_CLIENT_SECRET=<github client secret>
Environment=JWT_ACCESS_SECRET=<access secret>
Environment=JWT_REFRESH_SECRET=<refresh secret>
Environment=JWT_RESET_SECRET=<reset secret>
Environment=SALT=<salt value>
Environment=SELENE_ENVIRONMENT=<test/prod>
Environment=SENDGRID_API_KEY=<sendgrid API key>
Environment=SSO_BASE_URL=<base url for single sign on application>

[Install]
WantedBy=multi-user.target

• Start the sso_api service and set it to start on boot

sudo systemctl start sso_api.service
sudo systemctl enable sso_api.service

Account API

• The account API uses the same authentication mechanism as the single sign on API. The JWT_ACCESS_SECRET, JWT_REFRESH_SECRET and SALT environment variables must be the same values as those on the single sign on API.
• This application uses the Redis database so the service needs to know where it resides.
• Define a systemd service to run the API. The service defines environment variables that use the secret and API keys generated in previous steps.

sudo vim /etc/systemd/system/account_api.service

[Unit]
Description=Mycroft Account API
After=network.target

[Service]
User=mycroft
Group=www-data
Restart=always
Type=simple
WorkingDirectory=/opt/selene/selene-backend/api/account
ExecStart=/usr/local/bin/pipenv run uwsgi --ini uwsgi.ini
Environment=DB_HOST=<db host IP address or name>
Environment=DB_NAME=mycroft
Environment=DB_PASSWORD=<selene user database password>
Environment=DB_PORT=5432
Environment=DB_USER=selene
Environment=JWT_ACCESS_SECRET=<same as value for single sign on>
Environment=JWT_REFRESH_SECRET=<same as value for single sign on>
Environment=OAUTH_BASE_URL=<url for oauth service>
Environment=REDIS_HOST=<IP address or name of redis host>
Environment=REDIS_PORT=6379
Environment=SELENE_ENVIRONMENT=<test/prod>
Environment=SALT=<same as value for single sign on>

[Install]
WantedBy=multi-user.target

• Start the account_api service and set it to start on boot

sudo systemctl start account_api.service
sudo systemctl enable account_api.service

Marketplace API

• The marketplace API uses the same authentication mechanism as the single sign on API. The JWT_ACCESS_SECRET, JWT_REFRESH_SECRET and SALT environment variables must be the same values as those on the single sign on API.
• This application uses the Redis database so the service needs to know where it resides.
• Define a systemd service to run the API. The service defines environment variables that use the secret and API keys generated in previous steps.

sudo vim /etc/systemd/system/market_api.service

[Unit]
Description=Mycroft Marketplace API
After=network.target

[Service]
User=mycroft
Group=www-data
Restart=always
Type=simple
WorkingDirectory=/opt/selene/selene-backend/api/market
ExecStart=/usr/local/bin/pipenv run uwsgi --ini uwsgi.ini
Environment=DB_HOST=<db host IP address or name>
Environment=DB_NAME=mycroft
Environment=DB_PASSWORD=<selene user database password>
Environment=DB_PORT=5432
Environment=DB_USER=selene
Environment=JWT_ACCESS_SECRET=<same as value for single sign on>
Environment=JWT_REFRESH_SECRET=<same as value for single sign on>
Environment=OAUTH_BASE_URL=<url for oauth service>
Environment=REDIS_HOST=<IP address or name of redis host>
Environment=REDIS_PORT=6379
Environment=SELENE_ENVIRONMENT=<test/prod>
Environment=SALT=<same as value for single sign on>

[Install]
WantedBy=multi-user.target

• Start the market_api service and set it to start on boot

sudo systemctl start market_api.service
sudo systemctl enable market_api.service

• The marketplace API assumes that the skills it supplies to the web application are in the Postgres database. To get them there, a script needs to be run to download them from Github. The script requires the GITHUB_USER, GITHUB_PASSWORD, DB_HOST, DB_NAME, DB_USER and DB_PASSWORD environment variables to run. Use the same values as those in the service definition files.

cd /opt/selene/selene-backend/batch
pipenv install
pipenv run python load_skill_display_data.py --core-version <specify core version, e.g. 19.02>

Device API

• The device API uses the same authentication mechanism as the single sign on API. The JWT_ACCESS_SECRET, JWT_REFRESH_SECRET and SALT environment variables must be the same values as those on the single sign on API.
• This application uses the Redis database so the service needs to know where it resides.
• The weather skill requires a key to the Open Weather Map API
• The speech to text engine requires a key to Google's STT API.
• The Wolfram Alpha skill requires an API key to the Wolfram Alpha API
• Define a systemd service to run the API. The service defines environment variables that use the secret and API keys generated in previous steps.

sudo vim /etc/systemd/system/public_api.service

[Unit]
Description=Mycroft Public API
After=network.target

[Service]
User=mycroft
Group=www-data
Restart=always
Type=simple
WorkingDirectory=/opt/selene/selene-backend/api/public
ExecStart=/usr/local/bin/pipenv run uwsgi --ini uwsgi.ini
Environment=DB_HOST=<db host IP address or name>
Environment=DB_NAME=mycroft
Environment=DB_PASSWORD=<selene user database password>
Environment=DB_PORT=5432
Environment=DB_USER=selene
Environment=EMAIL_SERVICE_HOST=<email host>
Environment=EMAIL_SERVICE_PORT=<email port>
Environment=EMAIL_SERVICE_USER=<email user>
Environment=EMAIL_SERVICE_PASSWORD=<email password>
Environment=GOOGLE_STT_KEY=<Google STT API key>
Environment=JWT_ACCESS_SECRET=<same as value for single sign on>
Environment=JWT_REFRESH_SECRET=<same as value for single sign on>
Environment=OAUTH_BASE_URL=<url for oauth service>
Environment=OWM_KEY=<Open Weather Map API Key>
Environment=OWM_URL=https://api.openweathermap.org/data/2.5
Environment=REDIS_HOST=<IP address or name of redis host>
Environment=REDIS_PORT=6379
Environment=SELENE_ENVIRONMENT=<test/prod>
Environment=SALT=<same as value for single sign on>
Environment=WOLFRAM_ALPHA_KEY=<Wolfram Alpha API Key
Environment=WOLFRAM_ALPHA_URL=https://api.wolframalpha.com

[Install]
WantedBy=multi-user.target

• Start the public_api service and set it to start on boot

sudo systemctl start public_api.service
sudo systemctl enable public_api.service

Testing the endpoints

Before we continue, let's make sure that your endpoints are operational - for this we'll use the public_api endpoint as an example.

1. As we do not yet have a http router configured, we must change the uwsgi configuration for the endpoint we want to test. This is contained in: /opt/selene/selene-backend/api/public/uwsgi.ini. Here we want to replace

   socket = :$PORT
   

   with

   http = :$PORT
   

   then restart the service:

   sudo systemctl restart public_api.service
   

2. Check the status of the systemd service:

   systemctl status public_api.service
   

   Should report the service as "active (running)"

3. Send a GET request from a remote device:

   curl -v http://$IP_ADDRESS:$PORT/code?state=this-is-a-test
   

   You can also monitor this from the service logs by running:

   journalctl -u public_api.service -f
   

Other Considerations

DNS

There are multiple ways to setup DNS. This document will not dictate how to do so for Selene. However, here is an example, based on how DNS is setup at Mycroft AI...

Each application runs on its own sub-domain. Assuming a top level domain of "mycroft.ai" the subdomains are:

• account.mycroft.ai
• api.mycroft.ai
• market.mycroft.ai
• sso.mycroft.ai

The APIs that support the web applications are directories within the sub-domain (e.g. account.mycroft.ai/api). Since the device API is externally facing, it is versioned. It's subdirectory must be "v1".

Reverse Proxy

There are multiple tools available for setting up a reverse proxy that will point your DNS entries to your APIs. As such, the decision on how to set this up will be left to the user.

SSL

It is recommended that Selene applications be run using HTTPS. To do this an SSL certificate is necessary.

Let's Encrypt is a great way to easily set up SSL certificates for free.

What About the GUI???

Once the database and API setup is complete, the next step is to setup the GUI, The README file for the Selene UI repository contains the instructions for setting up the web applications.

Getting Involved

This is an open source project and we would love your help. We have prepared a contributing guide to help you get started.

If this is your first PR or you're not sure where to get started, say hi in Mycroft Chat and a team member would be happy to guide you. Join the Mycroft Forum for questions and answers.