Skip to content

Applications deployment on EEE

Jump to bottom
bartoszbosak edited this page Dec 7, 2016 · 7 revisions

The following section presents a proposed way of MUSCLE2 applications deployment on computing resources. This instruction assumes that MUSCLE2 was installed accordingly with the Deployment on EEE document. In particular the instruction should be a cornerstone for the deployments of ComPat applications on the EEE resources.

Location

In order to enable group work with applications, the applications should be deployed into the shared / group directory. For the EEE it is recomended to install the applications in domain-based subdirectories of that directory, e.g. $COMPAT_SHARED/Fusion, $COMPAT_SHARED/Bio.

Modules

Since the applications often have multiple dependencies and they need to utilise specific environment settings, the handy solution is to employ Environment Modules to configure the runtime environment of particular applications.

All ComPat modules should be created in the Modules directory being the 1st-level subdirectory of the ComPat shared directory ($COMPAT_SHARED). For the QCG purposes, the same directory - so the same set of modules - should be linked to the hidden .qcg-modules subdirectory as well.

For the application modules there should be a special directory/file structure preserved. In principle the applications' modules should go into the $COMPAT_SHARED/Modules/compat/apps/APPLICATION_DOMAIN directory.

Example - SimpleExample app from the MUSCLE2 distribution

The following script presents very simplistic module for the example MUSCLE2 application. In this script we load only one module that configure the MUSCLE2 settings. In real applications developers will need to load more modules (e.g. MPI libs, solvers) or make other environment configuration (e.g. set environment variables).

#%Module1.0#####################################################################
##
## SimpleExample app Modulefile
##
proc ModulesHelp { } {
        global version

        puts stderr "\tThis module sets environment for SimpleExample application"
        puts stderr "\tThe module details:"
        puts stderr "\t* target resource: Inula"
}

module-whatis   "Sets environment for the SimpleExample application"

# loading required modules / setting required common variables
module load compat/common/muscle2

set curMod [module-info name]

if { [ module-info mode load ] } {
        puts stderr "$curMod load complete."
}

if { [ module-info mode remove ] } {
        puts stderr "$curMod unload complete."
}

set     version      "default"

The module should be placed somewhere into $COMPAT_SHARED/Modules/compat/apps directory. For the above module and the application compiled with the settings specific for the Eagle cluster it may be: $COMPAT_SHARED/Modules/compat/apps/examples/simpleexample/eagle/default

Such a module could be loaded then with the following command:

module load compat/apps/examples/simpleexample/inula/default

Submission with QCG

In order to provide flexibility for development of applications, MUSCLE2 is exposed in QCG as a separate application-like item. The users may use it as any other QCG application that takes input file as a parameter - in case of MUSCLE it would be a *.cxa.rb file. The special settings for the domain applications can be loaded during the preprocessing directly from the job description. In particular the preprocessing phase may include loading a specific module. Thanks to such a solution, the users may freely play with their own applications.

The following QCG-Simple description presents an example script that may be used to execute SimpleExample application on the inula cluster.

#QCG note=SimpleExample
#QCG name=SE

#QCG grant=qcgdevtests3
#QCG host=inula
#QCG walltime=PT5M
#QCG queue=plgrid-testing
#QCG nodes=1:2
#QCG persistent

#############################################
## Start job using the muscle2 application ##
#############################################
#QCG application=muscle2

######################################################################
## load default module for the selected app in the preprocess phase ## 
######################################################################
#QCG preprocess=module load compat/apps/examples/simpleexample 

#QCG argument=SimpleExample.cxa.rb
#QCG stage-in-file=SimpleExample.cxa.rb
#QCG stage-out-dir=. -> results

#QCG output=output
#QCG error=error

The QCG job may be submitted with QCG-SimpleClient. If stored as SimpleExample.qcg the command will look as follows:

qcg-sub SimpleExample.qcg

Submission with JobProfile (XML) description (the old way)

The QCG job may be also defined using a XML description called JobProfile. The following XML allows to define the same job as above in a different manner.

<qcgJob appId="SimpleExample" xmlns:jxb="http://java.sun.com/xml/ns/jaxb" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" project="qcgdevtests3">
  <task persistent="true" taskId="task">
    <requirements>
      <topology>
        <processes processesId="" masterGroup="true">
          <processesMap slotsPerNode="2">
            <processesPerNode>2</processesPerNode>
          </processesMap>
          <candidateHosts>
            <hostName>inula</hostName>
          </candidateHosts>
          <reservation type="LOCAL">NO_RESERVATION</reservation>
        </processes>
      </topology>
    </requirements>
    <execution type="mapper">
      <executable>
        <application name="muscle2"/>
      </executable>
      <arguments>
        <value>SimpleExample.cxa.rb</value>
      </arguments>
      <stdout>
        <directory>
          <location type="URL">gsiftp://qcg.man.poznan.pl/~/COMPAT/MuscleTests/SimpleExample/output-${JOB_ID}-${TASK_ID}</location>
        </directory>
      </stdout>
      <stderr>
        <directory>
          <location type="URL">gsiftp://qcg.man.poznan.pl/~/COMPAT/MuscleTests/SimpleExample/error-${JOB_ID}-${TASK_ID}</location>
        </directory>
      </stderr>
      <stageInOut>
        <file name="SimpleExample.cxa.rb" type="in">
          <location type="URL">gsiftp://qcg.man.poznan.pl/~/COMPAT/MuscleTests/SimpleExample/SimpleExample.cxa.rb</location>
        </file>
        <directory name="." type="out">
          <location type="URL">gsiftp://qcg.man.poznan.pl/~/COMPAT/MuscleTests/SimpleExample/results-${JOB_ID}-${TASK_ID}</location>
        </directory>
      </stageInOut>
      <environment>
        <variable name="QCG_MODULES_LIST">compat/apps/examples/simpleexample</variable>
      </environment>
    </execution>
    <executionTime>
      <executionDuration>P0Y0M0DT0H05M</executionDuration>
    </executionTime>
  </task>
</qcgJob>

The submission of JobProfile description with QCG-Client requires -X switch:

qcg-sub -X SimpleExample.xml
Clone this wiki locally