workbench-server

The ecoacoustics workbench server. Manages the structure and metadata for environmental audio data. Provides an API for client access.

(Previously known as baw-server)

Dependencies

This project's dev environment is managed by Docker. Please ensure the latest version of Docker Desktop is installed on your machine

Audio processing and other long-running tasks are performed using baw-workers.

Contributing

We're working on that.

Development requirements

Docker and docker-compose are required to run this application.

Windows

Install:

• Optionally: At least Windows 10 Version 2004

• Docker desktop

  • a recent version (>3.4) with Docker Compose V2!

• WSL Version 2 (and add a default Ubuntu distro): https://docs.microsoft.com/en-us/windows/wsl/install-win10

• Enable the metadata flag for DrvFS mounts: https://docs.microsoft.com/en-us/windows/wsl/wsl-config#configure-per-distro-launch-settings-with-wslconf

• If you have issue with bind mount permissions, boot the container as the root user and chown -R 1000 . to grant access to files

• Clear container logs with:

  • sudo find /mnt/wsl/docker-desktop-data/version-pack-data/community/docker/containers -name *.log -exec truncate -s 0 {} \;
  • warning: this can corrupt the docker logs, use with caution

MacOS

Install Docker Desktop

Linux

Install docker as normal.

Environment Setup

Clone this repo, then change directory to your cloned directory and on your host machine run

$ docker-compose up

This will prepare a complete development environment. To see what is involved in the setup, look at the Dockerfile and bin/setup files.

You can stop the running containers using ctrl+c which is equivalent to docker compose stop.

• docker compose stop will stop the containers
• docker compose down will stop containers, remove containers, and delete networks
  • images will not be deleted
  • the primary application state (on the postgres volume) will not be removed

NOTE: changes in the Dockerfile will not be reflected in docker compose images or containers unless the compose project is destroyed or the containers are rebuilt.

Destroy or rebuild the docker environment

By default docker volume state is persisted between restarts of docker compose. This means you can return to your previous development session easily. If, however, you want to start from scratch you can remove state by doing one of the following.

To start from scratch by removing all containers, images, and volumes:

$ docker-compose down --remove-orphans --volumes --rmi local

To rebuild the web service / baw-server image (e.g. to update dependencies) but keep state from our volume:

$ docker-compose build

IDE

We recommend using VSCode and VSCode's remote container development mode, with the included workspace. When you open the baw-server.code-workspace with vscode it should ask you to open the workspace dev container with a prompt similar to this:

Development

If you're using VSCode's remote container development features you'll need to start the web server. See the command below.

Otherwise, Start by running, on your host machine:

$ docker-compose up

Web server

For the following commands:

• if you're using VSCode's remote container feature you can run these commands in the built in terminal
• if you're entering these commands from your host
  • cd to this directory
  • prefix commands with docker compose web exec to run the commands once off
  • if the service isn't started, use docker compose web start to start it
  • use docker compose web run xxx to run the web service and then run command xxx
  • docker compose exec web bash to enter an interactive session

Common tasks that you may need:

• bundle install to install a new gem
• rails db:xxx to manipulate database
  • rails db:setup - does all of the below
  • rails db:create - create database
  • rails db:migrate - run migrations
  • rails db:seed - seed the database with default data
• rails console to use the rails console
• rails start to run a web server
• passenger-config restart-app / to restart passenger and hot-reload code
• touch tmp/restart.txt to trigger a passenger reload before every request

Note: the binstubs in the bin folder automatically load bundler. You should not need to use bundle exec.

Docker commands:

• docker compose stop will stop the containers
• docker compose stop web stop web container so you can do something else
• docker compose exec bundle exec passenger start - the default action for docker compose up
• use docker compose restart workers workers_test scheduler scheduler_test to restart rails apps in headless services
  • this is useful for installing a new gem in associated containers after updating the Gemfile (entrypoint.sh does the bundle install)

When running the server in development or test modes, these configuration files will be used:

• config/settings/development.yml
• config/settings/test.yml

They are based on config/settings/default.yml.

Workers

• run commands on the worker container
  • e.g. docker compose workers exec ...
• list worker commands: baw-workers -T or baw-workers (using the manually maintained binstub)
• run a new worker: baw-workers baw:worker:run
• run a scheduler to process time-delayed tasks: baw-workers baw:worker:run_scheduler

Debugging:

Insert a debugger statement where you would like the debugger to stop. Alternatively you can enter a break point after the debugger starts.

(host) > docker compose exec web bash
(container) > RAILS_ENV='test' rdbg --command -- /home/baw_web/baw-server/bin/rspec /home/baw_web/baw-server/spec/lib/modules/filter/query_spec.rb -e association

We've also defined a helper script to run the debugger:

(container) > debug rspec /home/baw_web/baw-server/spec/lib/modules/filter/query_spec.rb -e association

More information here: https://github.com/ruby/debug

Tests

The tests are run using rspec:

bin/rspec

Tests can also be run with a specified seed using rspec:

rspec --seed <number>

There are some slower tests that do not always need to be run. These are marked with the slow metadata and can be skipped with:

rspec --tag ~slow

Style

Use this style guide as a reference: https://github.com/rubocop-hq/ruby-style-guide.

Documentation

Generate API documentation with:

bin/generate_docs.sh

Other commands

These commands should be executed automatically but are listed because they are helpful to know.

• Create the test database: rails db:create RAILS_ENV=test
• Then migrate and seed the test database: rails db:migrate db:seed RAILS_ENV=test
• Prepare the local development database: rails db:setup RAILS_ENV=development
• Run rspec tests: rspec
• Run rspec tests and generate a HTML report: rspec --format html --out rspec_results.html
• Generate API documentation: bin/generate_docs.sh

Production setup and deploying

Create production settings file config/settings/production.yml based on config/settings/default.yml. Create staging settings file config/settings/staging.yml based on config/settings/default.yml.

We deploy using Ansible.

If you want to use background workers, you'll need to set up Redis. A basic redis setup is included with the docker-compose file.

Creating a release

1. git switch master && git pull
2. Set the GitHub auth token for docs generation:
   • export CHANGELOG_GITHUB_TOKEN=xxx
3. Determine the next version name. Check the ./VERSION file for the current version.
4. Run the docker_build_and_push.ps1 script with the next version as the first argument:
   • e.g. ./docker_build_and_push.ps1 6.1.0

Architecture

• Two main parts
  • web server (Rails)
  • job workers (under lib/gems/baw-workers)
• docker compose is used to run
  • postgresql
  • redis
  • a single worker (for test and dev environments)
  • a single scheduler (for test and dev environments)
  • an instance of sftpgo
  • an instance of PBS (for test and dev environments)

Credits

This project was originally created and maintained by @cofiem - all the amazing things it does are a credit to them.

Other major credits go to:

• @atruskie
• @peichins
• @Allcharles
• @JessieLOliver for design and user experience feedback

License

Apache License, Version 2.0