Skip to content

Latest commit

 

History

History
273 lines (181 loc) · 10.1 KB

DEVELOPMENT.md

File metadata and controls

273 lines (181 loc) · 10.1 KB
Raw

Setting up a local development environment

Fork and clone the repository onto your computer. You can either run the project using a Docker container or by setting it up Manually.

The project is organized into the following directories:

  • common: Features that are shared between the frontend and backend of Common Voice, such as Typescript types.
  • locales: Information about activated languages on the site, automatically generated by yarn import-locales
  • maintenance: Static code for maintenance mode on Common Voice
  • scripts: Some scripts for managing data
  • server: The server-side app logic, written in TypeScript.
  • web: The Common Voice website files, written in TypeScript. We use React to build the website.
  • bundler: Service that is creating the dataset release bundles for Common Voice, written in TypeScript.

Docker

We provide a docker-compose setup to orchestrate the local development environment configuration using Docker. This is our recommended way of setting up your local development environment, as it will bypass the need to install each component separately.

Requirements

Configuration

You can find configurable options, like the port Common Voice is running on, in /server/src/config-helper.ts. If you want to modify any of these values, you will need to create a configuration file.

If you're using Docker, you should save this file as .env-local-docker (see .env-local-docker.example) in the root directory of the project, and it will be formatted like unix env values, with each key having a CV_ prefix. For example:

CV_DB_ROOT_PASS="root"
CV_MYSQLHOST="db"
CV_IMPORT_SENTENCES="true"

Copy the example with:

> cd common-voice
> cp .env-local-docker.example .env-local-docker

This will instruct your application to import the sentences located in server/data/* on start up. This step is IMPORTANT to be able to contribute to specific languages.

Setup steps

Run the following commands:

> cd common-voice
> docker-compose up

This is going to:

  • Launch a mysql instance configured for common-voice
  • Launch a fake GCP Cloud Storage instance to store files locally and avoid going through setting up GCP Cloud Storage
  • Mount the project using a Docker volume to allow reflecting changes to the codebase directly to the container
  • Import sentences from server/data/*
  • Launch common-voice server
  • Launch bundler service

Once you've have imported the sentences for all locales (or just the ones that are of interest to you) open a new terminal and flush the redis cache:

> docker exec -it redis redis-cli FLUSHALL

This will ensure that on the next restart the languages, that we just imported sentences for, will be available for contribution.

Restart the server and you should be able to visit the website at http://localhost:9000.

Notes:

The bundler service is not strictly needed to run the common voice website. Run the following commands to just run the minimal setup:

> cd common-voice
> docker-compose up web

Docker can be a very memory-intensive process. If you notice intermittent failures, or if features like auto-rebuilding crash, try increasing Docker's available memory from within Docker's Preferences > Resources settings.**

Apple M1 Silicon

There is an outstanding issue where MySQL doesn't work in Docker on Apple Silicon (as of 17th January 2022). You can instead replace it with MariaDB in a docker-compose.override.yml file.

services:
  db:
    image: mariadb:10.4
    command: mysqld --sql_mode="STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION"

Troubleshooting

If you get an error like the following when running native Docker (not Docker for Desktop),

ERROR: Couldn't connect to Docker daemon at http+docker://localhost - is it running?

You may need to build a new image. You can do that by issuing the following commands:

> cd docker/
> docker build .

Then after this you can:

> cd ..
> docker-compose up

You may have to run these commands as root/superuser.

Manual installation

You may want to set up each component manually. In which case you will need:

Requirements

Configuration

You can find configurable options, like the port Common Voice is running on, in /server/src/config-helper.ts. If you want to modify any of these values, you will need to create a configuration file.

If you installed the app manually, create a /config.json with the config you want to override in JSON format. The keys will not have a CV_ prefix. For example:

{
  "IMPORT_SENTENCES": false,
  "MYSQLDBNAME": "voice",
  "MYSQLHOST": "127.0.0.1",
}

Database preparation

Once the required components are installed, you need to prepare your database.

You can either create a MySQL superuser that that uses the default DB_ROOT_USER and DB_ROOT_PASS values from /server/src/config-helper.ts or create your own config as described above.

Cloud Storage configuration

The Common Voice project uses Google Cloud Storage for voice clip storage. This will be provided for you if you use the Docker installation, but if you are doing local development you will need to set up your own Cloud Storage instance. For detailed although outdated instructions on how to do that, see HOWTO_S3.md. The steps to setup Buckets on GCP should be similar.

Setup steps

Make sure your MySQL server is running. Then run the following commands:

> yarn
> yarn start

This will:

  1. Install all JavaScript dependencies.
  2. Build and serve files located in the web folder on localhost.
  3. Save uploaded voice clips onto Google's Cloud Storage.
  4. Lint and rebuild all js files on every change.

You can then access the website at http://localhost:9000.

Linting

We're using ESLint (with Typescript) and Prettier to lint the project.

Install ESlint and Prettier extentions into your code editor or you can run this for all files with:

yarn lint

For now this is not automatically checked on Pull Requests or blocking while we fix existing issues.

Authentication

If you want to work with login-related features (Profile, Dashboard, Goals, ...) you'll need to set up authentication:

  1. Create an Auth0 account.
  2. Click "Applications" from the dashboard. Create a new one, or use the default application.
  3. On "Applications" still, next to your application, click the "three dots" icon, then Settings.
  4. Add http://localhost:9000/callback to the "Allowed Callback URLs" list.
  5. Copy the Auth0 application settings into your configuration file. These are found in the same Settings tab as the "Allowed Callback URLs".

For Docker, in .env-local-docker:

CV_AUTH0_DOMAIN="<domain_here>"
CV_AUTH0_CLIENT_ID="<client_id_here>"
CV_AUTH0_CLIENT_SECRET="<client_secret_here>"

For local development, in config.json:

"AUTH0": {
 "DOMAIN": "<domain_here>",
 "CLIENT_ID": "<client_id_here>",
 "CLIENT_SECRET": "<client_secret_here>"
}
  1. You can add more login options to your app from the "Connections" tab
  2. Restart your local Common Voice instance
  3. You will now be able to create a new user by clicking on "Log in" on your Common Voice instance and then switching over to the "Sign Up" tab on the login dialog.

Database migrations

We use db-migrate for running database migrations.

To add a migration run: yarn migrate:local create <MIGRATION_NAME>.

At the moment you manually have to change the migration file extension to .ts. A migration has to expose the following API:

export const up = async function (db: any): Promise<any> {
  return null
}

export const down = async function (): Promise<any> {
  return null
}

Migrations are always run when the server is started.

To migrate up or down manually run: yarn migrate:local [up|down].

This will execute the most recently added migration in the migrations folder.

Localization

We're using Fluent to localize strings. You can find examples all over the frontend code. Strings that appear in the english message files, can then be translated on Pontoon. Some things to note regarding string changes are documented on MDN.

Updating languages

To update the list of locales run:

> yarn import-locales

Running End to End Tests

Some end to end tests require you to create a test user for login purposes. To do so you will need to create a cypress.env.json file with the login details for that user and your auth0 domain. This file should be located in the root of the web directory. Your JSON file should look like this:

{
  "auth0_domain": "auth0_domain",
  "test_user_email": "test user email",
  "test_user_password": "test user password"
}

Submitting an Issue

Did you notice a bug? Do you have a feature request? Please file an issue here on GitHub.

Something Else?

Want to talk about something but can't find a home for it here? Head to our Discourse Category to discuss everything from feedback and ideas to questions and random musings.