Skip to content
This repository has been archived by the owner on Nov 14, 2022. It is now read-only.

Installation for production

Jump to bottom
Elliot Oram edited this page Jul 23, 2018 · 6 revisions

Autoreduction Installation

This a guide on how to install the Autoreduction service for production.

Overview

Autoreduction runs best if it's split into at least three machines:

The advantage of this is you can have as many Queue Processing machines as required, these are the machines that will do the actual reduction using Mantid or any other software used for data reduction.

Utilities Machine

MySQL

To start with, download the latest version of MySQL 32Bit for Windows from here. Before starting to install it, you should also download and install the C++ redistributable package 2013 from here as MySQL server is dependent on it.

Once you have installed the C++ package, go through the MySQL installation process and make sure to install the "Developer Default" option. There will be also be a "Check Requirements" page where you will have to click "Execute" to install missing dependencies. You will also have to set the root password at this stage.

Once installation has completed, you should be able to use the included MySQL workbench to connect to the local database. Once you're happy everything is working as intended, you can move onto installing ActiveMQ.

ActiveMQ

Firstly, you should install Java on the utilities machine. You can find the latest version of the JRE here. After installing the JRE, you can download the ActiveMQ zip from here. After extracting the zip to a sensible location, you will need to configure your Windows ports so that external processes can access the ActiveMQ on port 61616. You can find instructions on how to do this on Windows here.

After configuring the firewall, you should set up ActiveMQ as a service so that it will automatically restart if the machine is rebooted. To do this, navigate to C:\ActiveMQ\apache-activemq-5.15.0\bin\win32 and run the InstallService.bat file as an administrator. This will install ActiveMQ as a service.

To optionally set a password on the ActiveMQ, you can modify the activemq.xml by adding the following section to the broker section:

<plugins>
    <simpleAuthenticationPlugin>
        <users>
            <authenticationUser username="user" password="password" groups="users,admins"/>
        </users>
    </simpleAuthenticationPlugin>
</plugins>

EndOfRunMonitor

Copy the code for the EndOfRunMonitor across to the utilities machine. To install this, you are going to need to install Python and some extra dependencies. Here is the list:

  • Download and install Python 32 Bit from here
  • Download and run the get-pip script from here.
  • Download and install Pywin32 from here
  • Download and install WinRar here
  • Download and install Stomp.py here
  • Download and install suds-jurko from here
  • Finally, download and install Python-ICAT from here

Archive Monitor

This is the back up service that is used to catch any runs that are missed by the EndOfRunMonitor.

  • From a windows cmd admin console run python isis_archive_monitor_win_service.py install
  • Open the windows Services application (you can find this from the start menu) in admin mode and you should now see a list of all windows services.
  • Locate the AutoReduce Archive Monitorright click the service and click Start from the context menu

Once all of the dependencies have been successfully installed, you need to navigate to your EndOfRunMonitor folder. Once in there, edit the settings.py file to ensure you are connecting to the correct locations. Once the settings have been updated, run python ISIS_monitor_win_service to install the EndOfRunMonitor as a Windows service. You will then need to manually update the service in your service explorer and change the user that it runs as such that it has the correct permissions to work properly.

WebApp Machine

Install the WebApp

Copy across the files for the WebApp and then edit the settings.py and apache/apache_django_wsgi such that they have the correct configuration. If you are using the UOWS login page then you will probably need to add a certficate so that Apache and Django trust the site. To do this, get the certificate from here and place it on the WebApp machine. Then, add its directory to the settings.py file. This will ensure that the connection is trusted.

Install Apache

Download Apache from here and run the MSI to install it. You will then need the Mod_WSGI package for Windows to be able to link the Django app up to the Apache server. This can be downloaded here. Get the 32 bit version of the mod_wsgi and place it in Apache2.2/modules.

You will now need to edit your httpd.conf to include the mod_wsgi library. To do this, add the following line in the 'LoadModule' section: LoadModule wsgi_module modules/mod_wsgi-py27-VC9.so. At the end of the file, you will need to add a line to include the wsgi.conf from the Django WebApp.

Run the generate_apache_wsgi.py file supplying a command line argument of the root project directory e.g. python Webapp/autoreduce_webapp/apache/generate_apache_wsgi.py C:\autoreduction

You should then be able to start Apache as either a service or just through the command line by running httpd.exe found in Apache2.2/bin/httpd.exe.

Queue Processing Machine

  • Install external project dependencies:

    • yum install python-pip

  • Create the a local clone of the git repository with sparse checkout enabled:

$ mkdir queue_processors
$ cd queue_processors
$ git init
$ git remote add origin -f https://github.com/ISISScientificComputing/autoreduce.git
$ git config core.sparsecheckout true
$ echo -e "QueueProcessors/*\nsetup.py\nrequirements.txt\nutils/*" > .git/info/sparse-checkout
$ git pull origin master

  • Navigate to the parent directory of the git repository and use pip install -e <name of git directory> to install the project to the PYTHONPATH.

  • Edit all settings.py files to ensure they have the correct credentials to connecting to external services (Queue service and database).

  • Create /autoreducetmp folder - mkdir /autoreduce. Change owner and group to user that will be used to run queueProcessor (to store temporary created reduction files) using chmod.

  • Start the processors manually with python queue_processor_daemon.py start in each of the queue processors or ./start.sh to start both simultaneously .

Mounting the ISIS Archive

The ISIS archive must also be mounted on the autoreduce machine. This will also require you to make a credentials file with the credentials you will use to access the archive. To do this and to make sure the mount persists through reboots, use the following:

vim /archive.cred

Add the following lines to the file (but replace the fake details with your real credentials) Leave isis as the domain:

username=user
password=pass
domain=isis

After this, use the following commands to create a folder to mount the drive into:

mkdir /isis
vim /etc/fstab

Add the following line to the fstab file:

//isis.cclrc.ac.uk/inst$/       /isis      cifs   _netdev,rw,credentials=/archive.creds,iocharset=utf8,soft, 0 0

Then, as root, use the following command to update from fstab and add the ISIS mount to the autoreduce machine:

mount -a

Home

Development

Workflow and best practices

Incident mitigation workflows

ISIS Specific

Post-mortems

Outdated pages

Clone this wiki locally