Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Suddenly stopped working with slurm #8

Open
pgcudahy opened this issue Mar 12, 2022 · 10 comments
Open

Suddenly stopped working with slurm #8

pgcudahy opened this issue Mar 12, 2022 · 10 comments

Comments

@pgcudahy
Copy link

pgcudahy commented Mar 12, 2022 •
edited
Loading

I've been using sos a lot in the past week on my university cluster but today it has suddenly stopped working with a strange environment issue. None of my steps which worked previously work anymore. For an example, here's my step to run gubbins on a multi-fasta alignment.

%sosrun gubbins -q yale_hpc_task_spooler \
-r yale_hpc_task_spooler mem="4GB" cores=2 walltime="04:00:00" nodes=1

[gubbins]
input: output_from("generate_ska_alignment")["ska_alignment"], group_by="all"
output: tree = f'/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/\
Taiwan_MKansasii.final_tree.tre',
filtered_multifasta = f'/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/\
Taiwan_MKansasii.filtered_polymorphic_sites.fasta',
recombination_gff = f'/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/\
Taiwan_MKansasii.recombination_predictions.gff'
task: walltime='06:00:00', mem="24GB", cores=8, nodes=1,
    workdir='/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/'

run: expand = True
    module load miniconda
    conda activate gubbins
    cd {_output:d}
    run_gubbins.py \
        -p Taiwan_MKansasii \
        --threads 8 \
        {_input}

But it fails with this error

t025f4ebfe0a4e5d4	failed

Created 19 min ago
Started 19 min ago
Ran for 0s
TASK:
=====
run(fr"""module load miniconda
conda activate gubbins
cd {_output:d}
run_gubbins.py \
    -p Taiwan_MKansasii \
    --threads 4 \
    {_input}

""")

TAGS:
=====
gubbins pgc29 w49339814024d0f8f

ENVIRONMENT:
============
__signature_vars__    {'_input', '_output', 'run'}
_depends              []
_index                0
_input                [file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.aln')]
_output               [file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.final_tree.tre'), file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.filtered_polymorphic_sites.fasta'), file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.recombination_predictions.gff')]
_runtime              {'cores': 8,
 'localhost': {'paths': {'home': '/home/pgc29/',
                         'scratch': '/home/pgc29/scratch60/'}},
 'max_cores': 200,
 'max_mem': 1200000000000,
 'max_walltime': '72:00:00',
 'mem': 24000000000,
 'nodes': 1,
 'queue': 'yale_hpc_task_spooler',
 'run_mode': 'run',
 'sig_mode': 'default',
 'verbosity': 2,
 'walltime': '06:00:00',
 'workdir': '/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/'}
step_name             'gubbins'
workflow_id           'w49339814024d0f8f'

EXECUTION STATS:
================

execution script:
================
#!/bin/bash
#SBATCH --time=06:00:00
#SBATCH --nodes=1
#SBATCH --cpus-per-task=8
#SBATCH --mem-per-cpu=3G
#SBATCH --job-name=t025f4ebfe0a4e5d4
#SBATCH --output=/home/pgc29/.sos/tasks/t025f4ebfe0a4e5d4.out
#SBATCH --error=/home/pgc29/.sos/tasks/t025f4ebfe0a4e5d4.err
#SBATCH --requeue
#SBATCH --partition=general,scavenge
cd /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/
/home/pgc29/.local/bin/sos execute t025f4ebfe0a4e5d4 -v 2 -s default -m run


standard error:
================
INFO: t025f4ebfe0a4e5d4 started
module () {  eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
ml () {  eval $($LMOD_DIR/ml_cmd "$@")
}
# -*- shell-script -*-
#
########################################################################
# Start Lmod BASHENV
########################################################################

if [ -z "${LMOD_SH_DBG_ON+x}" ]; then
   case "$-" in
     *v*x*) __lmod_vx='vx' ;;
     *v*)   __lmod_vx='v'  ;;
     *x*)   __lmod_vx='x'  ;;
   esac;
fi

[ -n "${__lmod_vx:-}" ] && set +$__lmod_vx 
Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for this output (/usr/share/lmod/lmod/init/bash)
Shell debugging restarted

########################################################################
# End Lmod BASHENV
########################################################################
#
# Local Variables:
# mode: shell-script
# indent-tabs-mode: nil
# End:
module load miniconda
Lmod has detected the following error: The following module(s) are unknown:
"miniconda"

Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore_cache load "miniconda"

Also make sure that all modulefiles written in TCL start with the string
#%Module



false
Task t025f4ebfe0a4e5d4 exits with code 1

The important bit is Lmod has detected the following error: The following module(s) are unknown: "miniconda". But when I submit the job manually (using sbatch) the following script runs fine

#!/bin/bash
#SBATCH --time=06:00:00
#SBATCH --nodes=1
#SBATCH --cpus-per-task=8
#SBATCH --mem-per-cpu=3G
#SBATCH --job-name=gubbins
#SBATCH --output=/home/pgc29/.sos/tasks/t025f4ebfe0a4e5d4.out
#SBATCH --error=/home/pgc29/.sos/tasks/t025f4ebfe0a4e5d4.err
#SBATCH --requeue
#SBATCH --partition=general,scavenge
cd /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/
module load miniconda
conda activate gubbins
run_gubbins.py \
    -p Taiwan_MKansasii \
    --threads 8 \
    /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.aln

This way lmod finds the miniconda module just fine and runs. My sos step was working on the cluster a couple of days ago. Now all of my sos scripts are failing with lmod not finding any modules, so it doesn't seem to be a module specific problem, but lmod itself. What could cause this?
@BoPeng
Copy link
Contributor

BoPeng commented Mar 12, 2022

It is very difficult to test what is going on here. However, judging from the error message, it looks like the computing node does not have this module. Can you find the job id of the failed job, get the node on which the task is executed, ssh to the node, and check module load runs over there?

@pgcudahy
Copy link
Author

pgcudahy commented Mar 12, 2022 •
edited
Loading

Yes, when I ssh into the node, lmod loads miniconda without issue. I've tried other sos scripts that previously worked and use common modules like GATK, trimmomatic, BWA, etc. and in every case the job fails with lmod not finding the module.

[pgc29@c23n08 ~]$ sacct -j 23712498 --format=JobID,JobName,MaxRSS,Elapsed,NodeList
JobID           JobName     MaxRSS    Elapsed        NodeList 
------------ ---------- ---------- ---------- --------------- 
23712498     t025f4ebf+              00:00:04          c18n01 
23712498.ba+      batch      1488K   00:00:04          c18n01 
23712498.ex+     extern      1076K   00:00:04          c18n01 
[pgc29@c23n08 ~]$ ssh c18n01
Warning: Permanently added 'c18n01,10.178.118.1' (ECDSA) to the list of known hosts.
[pgc29@c18n01 ~]$ module load miniconda
[pgc29@c18n01 ~]$ conda
usage: conda [-h] [-V] command ...

conda is a tool for managing and deploying applications, environments and packages.

@BoPeng
Copy link
Contributor

BoPeng commented Mar 12, 2022 •
edited
Loading

ok, on the computing node can you run the script directly?

cd /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/
/home/pgc29/.local/bin/sos execute t025f4ebfe0a4e5d4 -v 2 -s default -m run

The module load command is supposed to be executed from here.

I need to run an errand now but the next thing you can try is to print the env inside the task script. Maybe slurm needs an option to pass all env into the script (which for PBS is -v so you may need the #SBATCH version of $PBS -v). Since it worked previously, so the admin might have changed some system settings.

Just my guess.

@pgcudahy
Copy link
Author 

[pgc29@c18n01 gubbins]$ /home/pgc29/.local/bin/sos execute t025f4ebfe0a4e5d4 -v 2 -s default -m run
INFO: t025f4ebfe0a4e5d4 started

It seems to be running fine when started this way. When I use another terminal to log onto the cluster I see

[pgc29@c23n08 ~]$ sos status t025f4ebfe0a4e5d4 -v4 
t025f4ebfe0a4e5d4	running

Created 1 hr 30 min ago
Started 43 sec ago
Ran for 43 sec
TASK:
=====
run(fr"""module load miniconda
conda activate gubbins
cd {_output:d}
run_gubbins.py \
    -p Taiwan_MKansasii \
    --threads 4 \
    {_input}

""")

TAGS:
=====
gubbins pgc29 w49339814024d0f8f

ENVIRONMENT:
============
__signature_vars__    {'_output', 'run', '_input'}
_depends              []
_index                0
_input                [file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.aln')]
_output               [file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.final_tree.tre'), file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.filtered_polymorphic_sites.fasta'), file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.recombination_predictions.gff')]
_runtime              {'cores': 8,
 'localhost': {'paths': {'home': '/home/pgc29/',
                         'scratch': '/home/pgc29/scratch60/'}},
 'max_cores': 200,
 'max_mem': 1200000000000,
 'max_walltime': '72:00:00',
 'mem': 24000000000,
 'nodes': 1,
 'queue': 'yale_hpc_task_spooler',
 'run_mode': 'run',
 'sig_mode': 'default',
 'verbosity': 2,
 'walltime': '06:00:00',
 'workdir': '/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/'}
step_name             'gubbins'
workflow_id           'w49339814024d0f8f'


t025f4ebfe0a4e5d4.soserr:
=========================
module () {  eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
ml () {  eval $($LMOD_DIR/ml_cmd "$@")
}
# -*- shell-script -*-
#
########################################################################
# Start Lmod BASHENV
########################################################################

if [ -z "${LMOD_SH_DBG_ON+x}" ]; then
   case "$-" in
     *v*x*) __lmod_vx='vx' ;;
     *v*)   __lmod_vx='v'  ;;
     *x*)   __lmod_vx='x'  ;;
   esac;
fi

[ -n "${__lmod_vx:-}" ] && set +$__lmod_vx 
Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for this output (/usr/share/lmod/lmod/init/bash)
Shell debugging restarted

########################################################################
# End Lmod BASHENV
########################################################################
#
# Local Variables:
# mode: shell-script
# indent-tabs-mode: nil
# End:
module load miniconda
EBDEVELMINICONDA=/ysm-gpfs/apps/software/miniconda/4.11.0/easybuild/miniconda-4.11.0-easybuild-devel; export EBDEVELMINICONDA; EBROOTMINICONDA=/ysm-gpfs/apps/software/miniconda/4.11.0; export EBROOTMINICONDA; EBVERSIONMINICONDA=4.11.0; export EBVERSIONMINICONDA; LOADEDMODULES=StdEnv:settarg:miniconda/4.11.0; export LOADEDMODULES; MANPATH=/ysm-gpfs/apps/software/miniconda/4.11.0/share/man:/ysm-gpfs/apps/software/miniconda/4.11.0/man:/usr/share/lmod/lmod/share/man::; export MANPATH; MODULEPATH=/gpfs/ysm/apps/modules/base:/gpfs/ysm/apps/modules/bio:/gpfs/ysm/apps/modules/cae:/gpfs/ysm/apps/modules/chem:/gpfs/ysm/apps/modules/compiler:/gpfs/ysm/apps/modules/data:/gpfs/ysm/apps/modules/debugger:/gpfs/ysm/apps/modules/devel:/gpfs/ysm/apps/modules/geo:/gpfs/ysm/apps/modules/ide:/gpfs/ysm/apps/modules/lang:/gpfs/ysm/apps/modules/lib:/gpfs/ysm/apps/modules/math:/gpfs/ysm/apps/modules/mpi:/gpfs/ysm/apps/modules/numlib:/gpfs/ysm/apps/modules/perf:/gpfs/ysm/apps/modules/phys:/gpfs/ysm/apps/modules/system:/gpfs/ysm/apps/modules/toolchain:/gpfs/ysm/apps/modules/tools:/gpfs/ysm/apps/modules/vis:/gpfs/ysm/apps/modules/restricted:/etc/modulefiles:/usr/share/modulefiles:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core; export MODULEPATH; __LMOD_REF_COUNT_PATH=/ysm-gpfs/apps/software/miniconda/4.11.0:1\;/ysm-gpfs/apps/software/miniconda/4.11.0/sbin:1\;/ysm-gpfs/apps/software/miniconda/4.11.0/bin:1\;/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins/bin:1\;OBJ/_x86_64_06_avx2:1\;/home/pgc29/byobu/bin:1\;/ysm-gpfs/apps/software/miniconda/4.8.3/condabin:1\;/ysm-gpfs/apps/software/miniconda/4.8.3:1\;/ysm-gpfs/apps/software/miniconda/4.8.3/bin:1\;/gpfs/ysm/apps/software/R/3.6.1-foss-2018b/bin:1\;/ysm-gpfs/apps/software/ImageMagick/7.0.8-11-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/LittleCMS/2.9-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/Ghostscript/9.23-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/GSL/2.5-GCC-7.3.0-2.30/bin:1\;/ysm-gpfs/apps/software/UDUNITS/2.2.26-foss-2018b/bin:1\;/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/sbin:1\;/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libsndfile/1.0.28-GCCcore-7.3.0/bin:1\;/gpfs/ysm/apps/software/GDAL/2.4.4-foss-2018b-Python-3.7.0/bin:1\;/ysm-gpfs/apps/software/libgeotiff/1.4.2-foss-2018b/bin:1\;/ysm-gpfs/apps/software/PROJ/5.0.0-foss-2018b/bin:1\;/ysm-gpfs/apps/software/JasPer/2.0.14-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/GEOS/3.6.2-foss-2018b-Python-3.7.0/bin:1\;/ysm-gpfs/apps/software/netCDF/4.6.1-foss-2018b/bin:1\;/ysm-gpfs/apps/software/HDF5/1.10.2-foss-2018b/bin:1\;/ysm-gpfs/apps/software/Python/3.7.0-foss-2018b/bin:1\;/ysm-gpfs/apps/software/cURL/7.60.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/Tk/8.6.8-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/Java/1.8.0_261:1\;/ysm-gpfs/apps/software/Java/1.8.0_261/bin:1\;/ysm-gpfs/apps/software/LibTIFF/4.0.9-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/NASM/2.13.03-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/SQLite/3.24.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/Tcl/8.6.8-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libreadline/7.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/cairo/1.14.12-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/GLib/2.54.3-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/PCRE/8.41-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/gettext/0.19.8.1-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libxml2/2.9.8-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/XZ/5.2.4-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/LLVM/6.0.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/nettle/3.4-foss-2018b/bin:1\;/ysm-gpfs/apps/software/X11/20180604-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/fontconfig/2.13.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/sbin:1\;/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/ncurses/6.1-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/freetype/2.9.1-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libpng/1.6.34-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/expat/2.2.5-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/bzip2/1.0.6-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/FFTW/3.3.8-gompi-2018b/bin:1\;/ysm-gpfs/apps/software/OpenBLAS/0.3.1-GCC-7.3.0-2.30/bin:1\;/ysm-gpfs/apps/software/OpenMPI/3.1.1-GCC-7.3.0-2.30/bin:1\;/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/sbin:1\;/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/numactl/2.0.11-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/binutils/2.30-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/GCCcore/7.3.0/bin:1\;/gpfs/ysm/bin:1\;/opt/TurboVNC/bin:1\;/usr/lib64/qt-3.3/bin:1\;/usr/local/bin:1\;/usr/bin:1\;/usr/local/sbin:1\;/usr/sbin:1\;/home/pgc29/.local/bin:2\;/home/pgc29/bin:2; export __LMOD_REF_COUNT_PATH; PATH=/ysm-gpfs/apps/software/miniconda/4.11.0:/ysm-gpfs/apps/software/miniconda/4.11.0/sbin:/ysm-gpfs/apps/software/miniconda/4.11.0/bin:/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins/bin:OBJ/_x86_64_06_avx2:/home/pgc29/byobu/bin:/ysm-gpfs/apps/software/miniconda/4.8.3/condabin:/ysm-gpfs/apps/software/miniconda/4.8.3:/ysm-gpfs/apps/software/miniconda/4.8.3/bin:/gpfs/ysm/apps/software/R/3.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/ImageMagick/7.0.8-11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LittleCMS/2.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Ghostscript/9.23-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GSL/2.5-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/UDUNITS/2.2.26-foss-2018b/bin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libsndfile/1.0.28-GCCcore-7.3.0/bin:/gpfs/ysm/apps/software/GDAL/2.4.4-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/libgeotiff/1.4.2-foss-2018b/bin:/ysm-gpfs/apps/software/PROJ/5.0.0-foss-2018b/bin:/ysm-gpfs/apps/software/JasPer/2.0.14-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GEOS/3.6.2-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/netCDF/4.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/HDF5/1.10.2-foss-2018b/bin:/ysm-gpfs/apps/software/Python/3.7.0-foss-2018b/bin:/ysm-gpfs/apps/software/cURL/7.60.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tk/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Java/1.8.0_261:/ysm-gpfs/apps/software/Java/1.8.0_261/bin:/ysm-gpfs/apps/software/LibTIFF/4.0.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/NASM/2.13.03-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/SQLite/3.24.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tcl/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libreadline/7.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/cairo/1.14.12-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GLib/2.54.3-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/PCRE/8.41-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/gettext/0.19.8.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libxml2/2.9.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/XZ/5.2.4-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LLVM/6.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/nettle/3.4-foss-2018b/bin:/ysm-gpfs/apps/software/X11/20180604-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/fontconfig/2.13.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/ncurses/6.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/freetype/2.9.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libpng/1.6.34-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/expat/2.2.5-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/bzip2/1.0.6-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/FFTW/3.3.8-gompi-2018b/bin:/ysm-gpfs/apps/software/OpenBLAS/0.3.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/OpenMPI/3.1.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/numactl/2.0.11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/binutils/2.30-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GCCcore/7.3.0/bin:/gpfs/ysm/bin:/opt/TurboVNC/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/pgc29/.local/bin:/home/pgc29/bin; export PATH; PKG_CONFIG_PATH=/ysm-gpfs/apps/software/miniconda/4.11.0/lib/pkgconfig; export PKG_CONFIG_PATH; PYTHONNOUSERSITE=True; export PYTHONNOUSERSITE; _LMFILES_=/etc/modulefiles/StdEnv.lua:/usr/share/lmod/lmod/modulefiles/Core/settarg.lua:/gpfs/ysm/apps/modules/tools/miniconda/4.11.0.lua; export _LMFILES_; _ModuleTable001_=X01vZHVsZVRhYmxlXz17WyJNVHZlcnNpb24iXT0zLFsiY19yZWJ1aWxkVGltZSJdPWZhbHNlLFsiY19zaG9ydFRpbWUiXT0wLjAwMTExNjAzNzM2ODc3NDQsZGVwdGhUPXt9LGZhbWlseT17fSxtVD17U3RkRW52PXtbImZuIl09Ii9ldGMvbW9kdWxlZmlsZXMvU3RkRW52Lmx1YSIsWyJmdWxsTmFtZSJdPSJTdGRFbnYiLFsibG9hZE9yZGVyIl09MSxwcm9wVD17bG1vZD17WyJzdGlja3kiXT0xLH0sfSxbInN0YWNrRGVwdGgiXT0wLFsic3RhdHVzIl09ImFjdGl2ZSIsWyJ1c2VyTmFtZSJdPSJTdGRFbnYiLFsid1YiXT0iTS4qemZpbmFsIix9LG1pbmljb25kYT17WyJmbiJdPSIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL3Rvb2xzL21pbmljb25kYS80LjExLjAubHVhIixbImZ1bGxO; export _ModuleTable001_; _ModuleTable002_=YW1lIl09Im1pbmljb25kYS80LjExLjAiLFsibG9hZE9yZGVyIl09Myxwcm9wVD17fSxbInN0YWNrRGVwdGgiXT0wLFsic3RhdHVzIl09ImFjdGl2ZSIsWyJ1c2VyTmFtZSJdPSJtaW5pY29uZGEiLFsid1YiXT0iMDAwMDAwMDA0LjAwMDAwMDAxMS4qemZpbmFsIix9LHNldHRhcmc9e1siZm4iXT0iL3Vzci9zaGFyZS9sbW9kL2xtb2QvbW9kdWxlZmlsZXMvQ29yZS9zZXR0YXJnLmx1YSIsWyJmdWxsTmFtZSJdPSJzZXR0YXJnIixbImxvYWRPcmRlciJdPTIscHJvcFQ9e30sWyJzdGFja0RlcHRoIl09MCxbInN0YXR1cyJdPSJhY3RpdmUiLFsidXNlck5hbWUiXT0ic2V0dGFyZyIsWyJ3ViJdPSJNLip6ZmluYWwiLH0sfSxtcGF0aEE9eyIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL2Jh; export _ModuleTable002_; _ModuleTable003_=c2UiLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL2JpbyIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvY2FlIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9jaGVtIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9jb21waWxlciIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvZGF0YSIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvZGVidWdnZXIiLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL2RldmVsIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9nZW8iLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL2lkZSIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvbGFuZyIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvbGliIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9tYXRoIiwiL2dwZnMveXNtL2FwcHMv; export _ModuleTable003_; _ModuleTable004_=bW9kdWxlcy9tcGkiLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL251bWxpYiIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvcGVyZiIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvcGh5cyIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvc3lzdGVtIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy90b29sY2hhaW4iLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL3Rvb2xzIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy92aXMiLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL3Jlc3RyaWN0ZWQiLCIvZXRjL21vZHVsZWZpbGVzIiwiL3Vzci9zaGFyZS9tb2R1bGVmaWxlcyIsIi91c3Ivc2hhcmUvbW9kdWxlZmlsZXMvTGludXgiLCIvdXNyL3NoYXJlL21vZHVsZWZpbGVzL0NvcmUiLCIvdXNyL3NoYXJl; export _ModuleTable004_; _ModuleTable005_=L2xtb2QvbG1vZC9tb2R1bGVmaWxlcy9Db3JlIix9LFsic3lzdGVtQmFzZU1QQVRIIl09Ii9ldGMvbW9kdWxlZmlsZXM6L3Vzci9zaGFyZS9tb2R1bGVmaWxlczovdXNyL3NoYXJlL21vZHVsZWZpbGVzL0xpbnV4Oi91c3Ivc2hhcmUvbW9kdWxlZmlsZXMvQ29yZTovdXNyL3NoYXJlL2xtb2QvbG1vZC9tb2R1bGVmaWxlcy9Db3JlIix9; export _ModuleTable005_; _ModuleTable_Sz_=5; export _ModuleTable_Sz_; conda deactivate; unset CONDA_EXE; unset _CE_CONDA; unset _CE_M; unset -f __conda_activate; unset -f __conda_reactivate; unset -f __conda_hashr; unset CONDA_SHLVL; unset _CONDA_EXE; unset _CONDA_ROOT; unset -f conda; source /ysm-gpfs/apps/software/miniconda/4.11.0/etc/profile.d/conda.sh;

CommandNotFoundError: Your shell has not been properly configured to use 'conda deactivate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.


export CONDA_EXE='/ysm-gpfs/apps/software/miniconda/4.11.0/bin/conda'
export _CE_M=''
export _CE_CONDA=''
export CONDA_PYTHON_EXE='/ysm-gpfs/apps/software/miniconda/4.11.0/bin/python'

# Copyright (C) 2012 Anaconda, Inc
# SPDX-License-Identifier: BSD-3-Clause

__add_sys_prefix_to_path() {
    # In dev-mode CONDA_EXE is python.exe and on Windows
    # it is in a different relative location to condabin.
    if [ -n "${_CE_CONDA}" ] && [ -n "${WINDIR+x}" ]; then
        SYSP=$(\dirname "${CONDA_EXE}")
    else
        SYSP=$(\dirname "${CONDA_EXE}")
        SYSP=$(\dirname "${SYSP}")
    fi

    if [ -n "${WINDIR+x}" ]; then
        PATH="${SYSP}/bin:${PATH}"
        PATH="${SYSP}/Scripts:${PATH}"
        PATH="${SYSP}/Library/bin:${PATH}"
        PATH="${SYSP}/Library/usr/bin:${PATH}"
        PATH="${SYSP}/Library/mingw-w64/bin:${PATH}"
        PATH="${SYSP}:${PATH}"
    else
        PATH="${SYSP}/bin:${PATH}"
    fi
    \export PATH
}

__conda_exe() (
    __add_sys_prefix_to_path
    "$CONDA_EXE" $_CE_M $_CE_CONDA "$@"
)

__conda_hashr() {
    if [ -n "${ZSH_VERSION:+x}" ]; then
        \rehash
    elif [ -n "${POSH_VERSION:+x}" ]; then
        :  # pass
    else
        \hash -r
    fi
}

__conda_activate() {
    if [ -n "${CONDA_PS1_BACKUP:+x}" ]; then
        # Handle transition from shell activated with conda <= 4.3 to a subsequent activation
        # after conda updated to >= 4.4. See issue #6173.
        PS1="$CONDA_PS1_BACKUP"
        \unset CONDA_PS1_BACKUP
    fi
    \local ask_conda
    ask_conda="$(PS1="${PS1:-}" __conda_exe shell.posix "$@")" || \return
    \eval "$ask_conda"
    __conda_hashr
}

__conda_reactivate() {
    \local ask_conda
    ask_conda="$(PS1="${PS1:-}" __conda_exe shell.posix reactivate)" || \return
    \eval "$ask_conda"
    __conda_hashr
}

conda() {
    \local cmd="${1-__missing__}"
    case "$cmd" in
        activate|deactivate)
            __conda_activate "$@"
            ;;
        install|update|upgrade|remove|uninstall)
            __conda_exe "$@" || \return
            __conda_reactivate
            ;;
        *)
            __conda_exe "$@"
            ;;
    esac
}

if [ -z "${CONDA_SHLVL+x}" ]; then
    \export CONDA_SHLVL=0
    # In dev-mode CONDA_EXE is python.exe and on Windows
    # it is in a different relative location to condabin.
    if [ -n "${_CE_CONDA:+x}" ] && [ -n "${WINDIR+x}" ]; then
        PATH="$(\dirname "$CONDA_EXE")/condabin${PATH:+":${PATH}"}"
    else
        PATH="$(\dirname "$(\dirname "$CONDA_EXE")")/condabin${PATH:+":${PATH}"}"
    fi
    \export PATH

    # We're not allowing PS1 to be unbound. It must at least be set.
    # However, we're not exporting it, which can cause problems when starting a second shell
    # via a first shell (i.e. starting zsh from bash).
    if [ -z "${PS1+x}" ]; then
        PS1=
    fi
fi
PATH='/ysm-gpfs/apps/software/miniconda/4.11.0/condabin:/ysm-gpfs/apps/software/miniconda/4.11.0:/ysm-gpfs/apps/software/miniconda/4.11.0/sbin:/ysm-gpfs/apps/software/miniconda/4.11.0/bin:/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins/bin:OBJ/_x86_64_06_avx2:/home/pgc29/byobu/bin:/ysm-gpfs/apps/software/miniconda/4.8.3/condabin:/ysm-gpfs/apps/software/miniconda/4.8.3:/ysm-gpfs/apps/software/miniconda/4.8.3/bin:/gpfs/ysm/apps/software/R/3.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/ImageMagick/7.0.8-11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LittleCMS/2.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Ghostscript/9.23-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GSL/2.5-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/UDUNITS/2.2.26-foss-2018b/bin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libsndfile/1.0.28-GCCcore-7.3.0/bin:/gpfs/ysm/apps/software/GDAL/2.4.4-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/libgeotiff/1.4.2-foss-2018b/bin:/ysm-gpfs/apps/software/PROJ/5.0.0-foss-2018b/bin:/ysm-gpfs/apps/software/JasPer/2.0.14-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GEOS/3.6.2-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/netCDF/4.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/HDF5/1.10.2-foss-2018b/bin:/ysm-gpfs/apps/software/Python/3.7.0-foss-2018b/bin:/ysm-gpfs/apps/software/cURL/7.60.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tk/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Java/1.8.0_261:/ysm-gpfs/apps/software/Java/1.8.0_261/bin:/ysm-gpfs/apps/software/LibTIFF/4.0.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/NASM/2.13.03-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/SQLite/3.24.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tcl/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libreadline/7.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/cairo/1.14.12-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GLib/2.54.3-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/PCRE/8.41-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/gettext/0.19.8.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libxml2/2.9.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/XZ/5.2.4-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LLVM/6.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/nettle/3.4-foss-2018b/bin:/ysm-gpfs/apps/software/X11/20180604-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/fontconfig/2.13.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/ncurses/6.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/freetype/2.9.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libpng/1.6.34-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/expat/2.2.5-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/bzip2/1.0.6-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/FFTW/3.3.8-gompi-2018b/bin:/ysm-gpfs/apps/software/OpenBLAS/0.3.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/OpenMPI/3.1.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/numactl/2.0.11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/binutils/2.30-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GCCcore/7.3.0/bin:/gpfs/ysm/bin:/opt/TurboVNC/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/pgc29/.local/bin:/home/pgc29/bin'; export PATH; TARG='OBJ/_x86_64_06_avx2'; export TARG; unset 'TARG_BUILD_SCENARIO'; unset 'TARG_COMPILER_FAMILY'; unset 'TARG_EXTRA'; TARG_HOST='farnam'; export TARG_HOST; TARG_MACH='x86_64_06_avx2'; export TARG_MACH; TARG_MACH_DESCRIPT='Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz'; export TARG_MACH_DESCRIPT; unset 'TARG_MPI_FAMILY'; TARG_OS='Linux-3.10.0-1160.42.2.el7.x86_64'; export TARG_OS; TARG_OS_FAMILY='Linux'; export TARG_OS_FAMILY; TARG_SUMMARY='x86_64_06_avx2'; export TARG_SUMMARY; unset 'TARG_TITLE_BAR'; TARG_TITLE_BAR_PAREN=' '; export TARG_TITLE_BAR_PAREN;
conda activate gubbins
PS1='(gubbins)'
export PATH='/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins/bin:/ysm-gpfs/apps/software/miniconda/4.11.0/condabin:/ysm-gpfs/apps/software/miniconda/4.11.0:/ysm-gpfs/apps/software/miniconda/4.11.0/sbin:/ysm-gpfs/apps/software/miniconda/4.11.0/bin:/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins/bin:OBJ/_x86_64_06_avx2:/home/pgc29/byobu/bin:/ysm-gpfs/apps/software/miniconda/4.8.3/condabin:/ysm-gpfs/apps/software/miniconda/4.8.3:/ysm-gpfs/apps/software/miniconda/4.8.3/bin:/gpfs/ysm/apps/software/R/3.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/ImageMagick/7.0.8-11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LittleCMS/2.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Ghostscript/9.23-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GSL/2.5-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/UDUNITS/2.2.26-foss-2018b/bin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libsndfile/1.0.28-GCCcore-7.3.0/bin:/gpfs/ysm/apps/software/GDAL/2.4.4-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/libgeotiff/1.4.2-foss-2018b/bin:/ysm-gpfs/apps/software/PROJ/5.0.0-foss-2018b/bin:/ysm-gpfs/apps/software/JasPer/2.0.14-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GEOS/3.6.2-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/netCDF/4.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/HDF5/1.10.2-foss-2018b/bin:/ysm-gpfs/apps/software/Python/3.7.0-foss-2018b/bin:/ysm-gpfs/apps/software/cURL/7.60.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tk/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Java/1.8.0_261:/ysm-gpfs/apps/software/Java/1.8.0_261/bin:/ysm-gpfs/apps/software/LibTIFF/4.0.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/NASM/2.13.03-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/SQLite/3.24.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tcl/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libreadline/7.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/cairo/1.14.12-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GLib/2.54.3-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/PCRE/8.41-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/gettext/0.19.8.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libxml2/2.9.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/XZ/5.2.4-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LLVM/6.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/nettle/3.4-foss-2018b/bin:/ysm-gpfs/apps/software/X11/20180604-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/fontconfig/2.13.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/ncurses/6.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/freetype/2.9.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libpng/1.6.34-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/expat/2.2.5-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/bzip2/1.0.6-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/FFTW/3.3.8-gompi-2018b/bin:/ysm-gpfs/apps/software/OpenBLAS/0.3.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/OpenMPI/3.1.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/numactl/2.0.11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/binutils/2.30-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GCCcore/7.3.0/bin:/gpfs/ysm/bin:/opt/TurboVNC/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/pgc29/.local/bin:/home/pgc29/bin'
export CONDA_PREFIX='/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins'
export CONDA_SHLVL='1'
export CONDA_DEFAULT_ENV='/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins'
export CONDA_PROMPT_MODIFIER='(gubbins)'
export CONDA_EXE='/ysm-gpfs/apps/software/miniconda/4.11.0/bin/conda'
export _CE_M=''
export _CE_CONDA=''
export CONDA_PYTHON_EXE='/ysm-gpfs/apps/software/miniconda/4.11.0/bin/python'
cd /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins
run_gubbins.py \
    -p Taiwan_MKansasii \
    --threads 4 \
    /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.aln

And I see output files starting to appear where they should.

@BoPeng
Copy link
Contributor

BoPeng commented Mar 12, 2022

So this means some env was not passed to the script correctly. The next step would be creating a shell script with sos execute, use sbatch to submit the job. If the job fails, maybe add an option --export=ALL.

--export=ALLDefault mode if --export is not specified. All of the user's environment will be loaded (either from the caller's environment or from a clean environment if --get-user-env is specified).

@pgcudahy
Copy link
Author

That works too!?
I used the script

#!/bin/bash
#SBATCH --time=06:00:00
#SBATCH --nodes=1
#SBATCH --cpus-per-task=8
#SBATCH --mem-per-cpu=3G
#SBATCH --job-name=gubbins
#SBATCH --output=/home/pgc29/gubbins.out
#SBATCH --error=/home/pgc29/gubbins.err
#SBATCH --requeue
#SBATCH --partition=general,scavenge
#SBATCH --export=ALL
cd /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/
/home/pgc29/.local/bin/sos execute t54475dda19ec2b2d -v 2 -s default -m run

And after sbatch it's happily working away. Unfortunately I just got a text from my ISP that my service is going down for the next few hours. I'm in a remote area on a wireless microwave connection so this happens. It's also late my time so I won't be able to respond again until tomorrow. Thanks again for you help with my esoteric issues. I really appreciate it.

@pgcudahy
Copy link
Author

In case this helps

[pgc29@c16n03 ~]$ sos status t54475dda19ec2b2d -v4
t54475dda19ec2b2d	running

Created 36 min ago
Started 5 min ago
Ran for 5 min
TASK:
=====
run(fr"""module load miniconda
conda activate gubbins
cd {_output:d}
run_gubbins.py \
    -p Taiwan_MKansasii \
    --threads 8 \
    {_input}

""")

TAGS:
=====
gubbins pgc29 w9142b9731f79a5e8

ENVIRONMENT:
============
__signature_vars__    {'_output', 'run', '_input'}
_depends              []
_index                0
_input                [file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.aln')]
_output               [file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.final_tree.tre'), file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.filtered_polymorphic_sites.fasta'), file_target('/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.recombination_predictions.gff')]
_runtime              {'cores': 8,
 'localhost': {'paths': {'home': '/home/pgc29/',
                         'scratch': '/home/pgc29/scratch60/'}},
 'max_cores': 200,
 'max_mem': 1200000000000,
 'max_walltime': '72:00:00',
 'mem': 24000000000,
 'nodes': 1,
 'queue': 'yale_hpc_task_spooler',
 'run_mode': 'run',
 'sig_mode': 'default',
 'verbosity': 2,
 'walltime': '06:00:00',
 'workdir': '/home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/'}
step_name             'gubbins'
workflow_id           'w9142b9731f79a5e8'


t54475dda19ec2b2d.soserr:
=========================
module () {  eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
ml () {  eval $($LMOD_DIR/ml_cmd "$@")
}
# -*- shell-script -*-
#
########################################################################
# Start Lmod BASHENV
########################################################################

if [ -z "${LMOD_SH_DBG_ON+x}" ]; then
   case "$-" in
     *v*x*) __lmod_vx='vx' ;;
     *v*)   __lmod_vx='v'  ;;
     *x*)   __lmod_vx='x'  ;;
   esac;
fi

[ -n "${__lmod_vx:-}" ] && set +$__lmod_vx 
Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for this output (/usr/share/lmod/lmod/init/bash)
Shell debugging restarted

########################################################################
# End Lmod BASHENV
########################################################################
#
# Local Variables:
# mode: shell-script
# indent-tabs-mode: nil
# End:
module load miniconda
EBDEVELMINICONDA=/ysm-gpfs/apps/software/miniconda/4.11.0/easybuild/miniconda-4.11.0-easybuild-devel; export EBDEVELMINICONDA; EBROOTMINICONDA=/ysm-gpfs/apps/software/miniconda/4.11.0; export EBROOTMINICONDA; EBVERSIONMINICONDA=4.11.0; export EBVERSIONMINICONDA; LOADEDMODULES=StdEnv:settarg:miniconda/4.11.0; export LOADEDMODULES; MANPATH=/ysm-gpfs/apps/software/miniconda/4.11.0/share/man:/ysm-gpfs/apps/software/miniconda/4.11.0/man:/usr/share/lmod/lmod/share/man::; export MANPATH; MODULEPATH=/gpfs/ysm/apps/modules/base:/gpfs/ysm/apps/modules/bio:/gpfs/ysm/apps/modules/cae:/gpfs/ysm/apps/modules/chem:/gpfs/ysm/apps/modules/compiler:/gpfs/ysm/apps/modules/data:/gpfs/ysm/apps/modules/debugger:/gpfs/ysm/apps/modules/devel:/gpfs/ysm/apps/modules/geo:/gpfs/ysm/apps/modules/ide:/gpfs/ysm/apps/modules/lang:/gpfs/ysm/apps/modules/lib:/gpfs/ysm/apps/modules/math:/gpfs/ysm/apps/modules/mpi:/gpfs/ysm/apps/modules/numlib:/gpfs/ysm/apps/modules/perf:/gpfs/ysm/apps/modules/phys:/gpfs/ysm/apps/modules/system:/gpfs/ysm/apps/modules/toolchain:/gpfs/ysm/apps/modules/tools:/gpfs/ysm/apps/modules/vis:/gpfs/ysm/apps/modules/restricted:/etc/modulefiles:/usr/share/modulefiles:/usr/share/modulefiles/Linux:/usr/share/modulefiles/Core:/usr/share/lmod/lmod/modulefiles/Core; export MODULEPATH; __LMOD_REF_COUNT_PATH=/ysm-gpfs/apps/software/miniconda/4.11.0:1\;/ysm-gpfs/apps/software/miniconda/4.11.0/sbin:1\;/ysm-gpfs/apps/software/miniconda/4.11.0/bin:1\;/home/pgc29/byobu/bin:1\;/ysm-gpfs/apps/software/miniconda/4.8.3/condabin:1\;/ysm-gpfs/apps/software/miniconda/4.8.3:1\;/ysm-gpfs/apps/software/miniconda/4.8.3/bin:1\;/gpfs/ysm/apps/software/R/3.6.1-foss-2018b/bin:1\;/ysm-gpfs/apps/software/ImageMagick/7.0.8-11-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/LittleCMS/2.9-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/Ghostscript/9.23-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/GSL/2.5-GCC-7.3.0-2.30/bin:1\;/ysm-gpfs/apps/software/UDUNITS/2.2.26-foss-2018b/bin:1\;/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/sbin:1\;/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libsndfile/1.0.28-GCCcore-7.3.0/bin:1\;/gpfs/ysm/apps/software/GDAL/2.4.4-foss-2018b-Python-3.7.0/bin:1\;/ysm-gpfs/apps/software/libgeotiff/1.4.2-foss-2018b/bin:1\;/ysm-gpfs/apps/software/PROJ/5.0.0-foss-2018b/bin:1\;/ysm-gpfs/apps/software/JasPer/2.0.14-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/GEOS/3.6.2-foss-2018b-Python-3.7.0/bin:1\;/ysm-gpfs/apps/software/netCDF/4.6.1-foss-2018b/bin:1\;/ysm-gpfs/apps/software/HDF5/1.10.2-foss-2018b/bin:1\;/ysm-gpfs/apps/software/Python/3.7.0-foss-2018b/bin:1\;/ysm-gpfs/apps/software/cURL/7.60.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/Tk/8.6.8-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/Java/1.8.0_261:1\;/ysm-gpfs/apps/software/Java/1.8.0_261/bin:1\;/ysm-gpfs/apps/software/LibTIFF/4.0.9-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/NASM/2.13.03-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/SQLite/3.24.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/Tcl/8.6.8-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libreadline/7.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/cairo/1.14.12-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/GLib/2.54.3-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/PCRE/8.41-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/gettext/0.19.8.1-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libxml2/2.9.8-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/XZ/5.2.4-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/LLVM/6.0.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/nettle/3.4-foss-2018b/bin:1\;/ysm-gpfs/apps/software/X11/20180604-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/fontconfig/2.13.0-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/sbin:1\;/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/ncurses/6.1-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/freetype/2.9.1-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/libpng/1.6.34-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/expat/2.2.5-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/bzip2/1.0.6-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/FFTW/3.3.8-gompi-2018b/bin:1\;/ysm-gpfs/apps/software/OpenBLAS/0.3.1-GCC-7.3.0-2.30/bin:1\;/ysm-gpfs/apps/software/OpenMPI/3.1.1-GCC-7.3.0-2.30/bin:1\;/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/sbin:1\;/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/numactl/2.0.11-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/binutils/2.30-GCCcore-7.3.0/bin:1\;/ysm-gpfs/apps/software/GCCcore/7.3.0/bin:1\;/gpfs/ysm/bin:1\;/opt/TurboVNC/bin:1\;/usr/lib64/qt-3.3/bin:1\;/usr/local/bin:1\;/usr/bin:1\;/usr/local/sbin:1\;/usr/sbin:1\;/home/pgc29/.local/bin:1\;/home/pgc29/bin:1; export __LMOD_REF_COUNT_PATH; PATH=/ysm-gpfs/apps/software/miniconda/4.11.0:/ysm-gpfs/apps/software/miniconda/4.11.0/sbin:/ysm-gpfs/apps/software/miniconda/4.11.0/bin:/home/pgc29/byobu/bin:/ysm-gpfs/apps/software/miniconda/4.8.3/condabin:/ysm-gpfs/apps/software/miniconda/4.8.3:/ysm-gpfs/apps/software/miniconda/4.8.3/bin:/gpfs/ysm/apps/software/R/3.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/ImageMagick/7.0.8-11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LittleCMS/2.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Ghostscript/9.23-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GSL/2.5-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/UDUNITS/2.2.26-foss-2018b/bin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libsndfile/1.0.28-GCCcore-7.3.0/bin:/gpfs/ysm/apps/software/GDAL/2.4.4-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/libgeotiff/1.4.2-foss-2018b/bin:/ysm-gpfs/apps/software/PROJ/5.0.0-foss-2018b/bin:/ysm-gpfs/apps/software/JasPer/2.0.14-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GEOS/3.6.2-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/netCDF/4.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/HDF5/1.10.2-foss-2018b/bin:/ysm-gpfs/apps/software/Python/3.7.0-foss-2018b/bin:/ysm-gpfs/apps/software/cURL/7.60.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tk/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Java/1.8.0_261:/ysm-gpfs/apps/software/Java/1.8.0_261/bin:/ysm-gpfs/apps/software/LibTIFF/4.0.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/NASM/2.13.03-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/SQLite/3.24.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tcl/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libreadline/7.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/cairo/1.14.12-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GLib/2.54.3-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/PCRE/8.41-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/gettext/0.19.8.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libxml2/2.9.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/XZ/5.2.4-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LLVM/6.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/nettle/3.4-foss-2018b/bin:/ysm-gpfs/apps/software/X11/20180604-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/fontconfig/2.13.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/ncurses/6.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/freetype/2.9.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libpng/1.6.34-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/expat/2.2.5-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/bzip2/1.0.6-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/FFTW/3.3.8-gompi-2018b/bin:/ysm-gpfs/apps/software/OpenBLAS/0.3.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/OpenMPI/3.1.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/numactl/2.0.11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/binutils/2.30-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GCCcore/7.3.0/bin:/gpfs/ysm/bin:/opt/TurboVNC/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/pgc29/.local/bin:/home/pgc29/bin; export PATH; PKG_CONFIG_PATH=/ysm-gpfs/apps/software/miniconda/4.11.0/lib/pkgconfig; export PKG_CONFIG_PATH; PYTHONNOUSERSITE=True; export PYTHONNOUSERSITE; _LMFILES_=/etc/modulefiles/StdEnv.lua:/usr/share/lmod/lmod/modulefiles/Core/settarg.lua:/gpfs/ysm/apps/modules/tools/miniconda/4.11.0.lua; export _LMFILES_; _ModuleTable001_=X01vZHVsZVRhYmxlXz17WyJNVHZlcnNpb24iXT0zLFsiY19yZWJ1aWxkVGltZSJdPTcyMDAsWyJjX3Nob3J0VGltZSJdPTAuMDAxMTE2MDM3MzY4Nzc0NCxkZXB0aFQ9e30sZmFtaWx5PXt9LG1UPXtTdGRFbnY9e1siZm4iXT0iL2V0Yy9tb2R1bGVmaWxlcy9TdGRFbnYubHVhIixbImZ1bGxOYW1lIl09IlN0ZEVudiIsWyJsb2FkT3JkZXIiXT0xLHByb3BUPXtsbW9kPXtbInN0aWNreSJdPTEsfSx9LFsic3RhY2tEZXB0aCJdPTAsWyJzdGF0dXMiXT0iYWN0aXZlIixbInVzZXJOYW1lIl09IlN0ZEVudiIsWyJ3ViJdPSJNLip6ZmluYWwiLH0sbWluaWNvbmRhPXtbImZuIl09Ii9ncGZzL3lzbS9hcHBzL21vZHVsZXMvdG9vbHMvbWluaWNvbmRhLzQuMTEuMC5sdWEiLFsiZnVsbE5h; export _ModuleTable001_; _ModuleTable002_=bWUiXT0ibWluaWNvbmRhLzQuMTEuMCIsWyJsb2FkT3JkZXIiXT0zLHByb3BUPXt9LFsic3RhY2tEZXB0aCJdPTAsWyJzdGF0dXMiXT0iYWN0aXZlIixbInVzZXJOYW1lIl09Im1pbmljb25kYSIsWyJ3ViJdPSIwMDAwMDAwMDQuMDAwMDAwMDExLip6ZmluYWwiLH0sc2V0dGFyZz17WyJmbiJdPSIvdXNyL3NoYXJlL2xtb2QvbG1vZC9tb2R1bGVmaWxlcy9Db3JlL3NldHRhcmcubHVhIixbImZ1bGxOYW1lIl09InNldHRhcmciLFsibG9hZE9yZGVyIl09Mixwcm9wVD17fSxbInN0YWNrRGVwdGgiXT0wLFsic3RhdHVzIl09ImFjdGl2ZSIsWyJ1c2VyTmFtZSJdPSJzZXR0YXJnIixbIndWIl09Ik0uKnpmaW5hbCIsfSx9LG1wYXRoQT17Ii9ncGZzL3lzbS9hcHBzL21vZHVsZXMvYmFz; export _ModuleTable002_; _ModuleTable003_=ZSIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvYmlvIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9jYWUiLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL2NoZW0iLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL2NvbXBpbGVyIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9kYXRhIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9kZWJ1Z2dlciIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvZGV2ZWwiLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL2dlbyIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvaWRlIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9sYW5nIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9saWIiLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL21hdGgiLCIvZ3Bmcy95c20vYXBwcy9t; export _ModuleTable003_; _ModuleTable004_=b2R1bGVzL21waSIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvbnVtbGliIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9wZXJmIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9waHlzIiwiL2dwZnMveXNtL2FwcHMvbW9kdWxlcy9zeXN0ZW0iLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL3Rvb2xjaGFpbiIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvdG9vbHMiLCIvZ3Bmcy95c20vYXBwcy9tb2R1bGVzL3ZpcyIsIi9ncGZzL3lzbS9hcHBzL21vZHVsZXMvcmVzdHJpY3RlZCIsIi9ldGMvbW9kdWxlZmlsZXMiLCIvdXNyL3NoYXJlL21vZHVsZWZpbGVzIiwiL3Vzci9zaGFyZS9tb2R1bGVmaWxlcy9MaW51eCIsIi91c3Ivc2hhcmUvbW9kdWxlZmlsZXMvQ29yZSIsIi91c3Ivc2hhcmUv; export _ModuleTable004_; _ModuleTable005_=bG1vZC9sbW9kL21vZHVsZWZpbGVzL0NvcmUiLH0sWyJzeXN0ZW1CYXNlTVBBVEgiXT0iL2V0Yy9tb2R1bGVmaWxlczovdXNyL3NoYXJlL21vZHVsZWZpbGVzOi91c3Ivc2hhcmUvbW9kdWxlZmlsZXMvTGludXg6L3Vzci9zaGFyZS9tb2R1bGVmaWxlcy9Db3JlOi91c3Ivc2hhcmUvbG1vZC9sbW9kL21vZHVsZWZpbGVzL0NvcmUiLH0=; export _ModuleTable005_; _ModuleTable_Sz_=5; export _ModuleTable_Sz_; conda deactivate; unset CONDA_EXE; unset _CE_CONDA; unset _CE_M; unset -f __conda_activate; unset -f __conda_reactivate; unset -f __conda_hashr; unset CONDA_SHLVL; unset _CONDA_EXE; unset _CONDA_ROOT; unset -f conda; source /ysm-gpfs/apps/software/miniconda/4.11.0/etc/profile.d/conda.sh;

CommandNotFoundError: Your shell has not been properly configured to use 'conda deactivate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.


export CONDA_EXE='/ysm-gpfs/apps/software/miniconda/4.11.0/bin/conda'
export _CE_M=''
export _CE_CONDA=''
export CONDA_PYTHON_EXE='/ysm-gpfs/apps/software/miniconda/4.11.0/bin/python'

# Copyright (C) 2012 Anaconda, Inc
# SPDX-License-Identifier: BSD-3-Clause

__add_sys_prefix_to_path() {
    # In dev-mode CONDA_EXE is python.exe and on Windows
    # it is in a different relative location to condabin.
    if [ -n "${_CE_CONDA}" ] && [ -n "${WINDIR+x}" ]; then
        SYSP=$(\dirname "${CONDA_EXE}")
    else
        SYSP=$(\dirname "${CONDA_EXE}")
        SYSP=$(\dirname "${SYSP}")
    fi

    if [ -n "${WINDIR+x}" ]; then
        PATH="${SYSP}/bin:${PATH}"
        PATH="${SYSP}/Scripts:${PATH}"
        PATH="${SYSP}/Library/bin:${PATH}"
        PATH="${SYSP}/Library/usr/bin:${PATH}"
        PATH="${SYSP}/Library/mingw-w64/bin:${PATH}"
        PATH="${SYSP}:${PATH}"
    else
        PATH="${SYSP}/bin:${PATH}"
    fi
    \export PATH
}

__conda_exe() (
    __add_sys_prefix_to_path
    "$CONDA_EXE" $_CE_M $_CE_CONDA "$@"
)

__conda_hashr() {
    if [ -n "${ZSH_VERSION:+x}" ]; then
        \rehash
    elif [ -n "${POSH_VERSION:+x}" ]; then
        :  # pass
    else
        \hash -r
    fi
}

__conda_activate() {
    if [ -n "${CONDA_PS1_BACKUP:+x}" ]; then
        # Handle transition from shell activated with conda <= 4.3 to a subsequent activation
        # after conda updated to >= 4.4. See issue #6173.
        PS1="$CONDA_PS1_BACKUP"
        \unset CONDA_PS1_BACKUP
    fi
    \local ask_conda
    ask_conda="$(PS1="${PS1:-}" __conda_exe shell.posix "$@")" || \return
    \eval "$ask_conda"
    __conda_hashr
}

__conda_reactivate() {
    \local ask_conda
    ask_conda="$(PS1="${PS1:-}" __conda_exe shell.posix reactivate)" || \return
    \eval "$ask_conda"
    __conda_hashr
}

conda() {
    \local cmd="${1-__missing__}"
    case "$cmd" in
        activate|deactivate)
            __conda_activate "$@"
            ;;
        install|update|upgrade|remove|uninstall)
            __conda_exe "$@" || \return
            __conda_reactivate
            ;;
        *)
            __conda_exe "$@"
            ;;
    esac
}

if [ -z "${CONDA_SHLVL+x}" ]; then
    \export CONDA_SHLVL=0
    # In dev-mode CONDA_EXE is python.exe and on Windows
    # it is in a different relative location to condabin.
    if [ -n "${_CE_CONDA:+x}" ] && [ -n "${WINDIR+x}" ]; then
        PATH="$(\dirname "$CONDA_EXE")/condabin${PATH:+":${PATH}"}"
    else
        PATH="$(\dirname "$(\dirname "$CONDA_EXE")")/condabin${PATH:+":${PATH}"}"
    fi
    \export PATH

    # We're not allowing PS1 to be unbound. It must at least be set.
    # However, we're not exporting it, which can cause problems when starting a second shell
    # via a first shell (i.e. starting zsh from bash).
    if [ -z "${PS1+x}" ]; then
        PS1=
    fi
fi
PATH='OBJ/_x86_64_06_avx2:/ysm-gpfs/apps/software/miniconda/4.11.0/condabin:/ysm-gpfs/apps/software/miniconda/4.11.0:/ysm-gpfs/apps/software/miniconda/4.11.0/sbin:/ysm-gpfs/apps/software/miniconda/4.11.0/bin:/home/pgc29/byobu/bin:/ysm-gpfs/apps/software/miniconda/4.8.3/condabin:/ysm-gpfs/apps/software/miniconda/4.8.3:/ysm-gpfs/apps/software/miniconda/4.8.3/bin:/gpfs/ysm/apps/software/R/3.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/ImageMagick/7.0.8-11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LittleCMS/2.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Ghostscript/9.23-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GSL/2.5-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/UDUNITS/2.2.26-foss-2018b/bin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libsndfile/1.0.28-GCCcore-7.3.0/bin:/gpfs/ysm/apps/software/GDAL/2.4.4-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/libgeotiff/1.4.2-foss-2018b/bin:/ysm-gpfs/apps/software/PROJ/5.0.0-foss-2018b/bin:/ysm-gpfs/apps/software/JasPer/2.0.14-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GEOS/3.6.2-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/netCDF/4.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/HDF5/1.10.2-foss-2018b/bin:/ysm-gpfs/apps/software/Python/3.7.0-foss-2018b/bin:/ysm-gpfs/apps/software/cURL/7.60.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tk/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Java/1.8.0_261:/ysm-gpfs/apps/software/Java/1.8.0_261/bin:/ysm-gpfs/apps/software/LibTIFF/4.0.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/NASM/2.13.03-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/SQLite/3.24.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tcl/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libreadline/7.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/cairo/1.14.12-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GLib/2.54.3-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/PCRE/8.41-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/gettext/0.19.8.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libxml2/2.9.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/XZ/5.2.4-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LLVM/6.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/nettle/3.4-foss-2018b/bin:/ysm-gpfs/apps/software/X11/20180604-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/fontconfig/2.13.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/ncurses/6.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/freetype/2.9.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libpng/1.6.34-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/expat/2.2.5-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/bzip2/1.0.6-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/FFTW/3.3.8-gompi-2018b/bin:/ysm-gpfs/apps/software/OpenBLAS/0.3.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/OpenMPI/3.1.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/numactl/2.0.11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/binutils/2.30-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GCCcore/7.3.0/bin:/gpfs/ysm/bin:/opt/TurboVNC/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/pgc29/.local/bin:/home/pgc29/bin'; export PATH; TARG='OBJ/_x86_64_06_avx2'; export TARG; unset 'TARG_BUILD_SCENARIO'; unset 'TARG_COMPILER_FAMILY'; unset 'TARG_EXTRA'; TARG_HOST='farnam'; export TARG_HOST; TARG_MACH='x86_64_06_avx2'; export TARG_MACH; TARG_MACH_DESCRIPT='Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz'; export TARG_MACH_DESCRIPT; unset 'TARG_MPI_FAMILY'; TARG_OS='Linux-3.10.0-1160.42.2.el7.x86_64'; export TARG_OS; TARG_OS_FAMILY='Linux'; export TARG_OS_FAMILY; TARG_SUMMARY='x86_64_06_avx2'; export TARG_SUMMARY; unset 'TARG_TITLE_BAR'; TARG_TITLE_BAR_PAREN=' '; export TARG_TITLE_BAR_PAREN;
conda activate gubbins
PS1='(gubbins)'
export PATH='/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins/bin:OBJ/_x86_64_06_avx2:/ysm-gpfs/apps/software/miniconda/4.11.0/condabin:/ysm-gpfs/apps/software/miniconda/4.11.0:/ysm-gpfs/apps/software/miniconda/4.11.0/sbin:/ysm-gpfs/apps/software/miniconda/4.11.0/bin:/home/pgc29/byobu/bin:/ysm-gpfs/apps/software/miniconda/4.8.3/condabin:/ysm-gpfs/apps/software/miniconda/4.8.3:/ysm-gpfs/apps/software/miniconda/4.8.3/bin:/gpfs/ysm/apps/software/R/3.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/ImageMagick/7.0.8-11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LittleCMS/2.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Ghostscript/9.23-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GSL/2.5-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/UDUNITS/2.2.26-foss-2018b/bin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/ICU/61.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libsndfile/1.0.28-GCCcore-7.3.0/bin:/gpfs/ysm/apps/software/GDAL/2.4.4-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/libgeotiff/1.4.2-foss-2018b/bin:/ysm-gpfs/apps/software/PROJ/5.0.0-foss-2018b/bin:/ysm-gpfs/apps/software/JasPer/2.0.14-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GEOS/3.6.2-foss-2018b-Python-3.7.0/bin:/ysm-gpfs/apps/software/netCDF/4.6.1-foss-2018b/bin:/ysm-gpfs/apps/software/HDF5/1.10.2-foss-2018b/bin:/ysm-gpfs/apps/software/Python/3.7.0-foss-2018b/bin:/ysm-gpfs/apps/software/cURL/7.60.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tk/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Java/1.8.0_261:/ysm-gpfs/apps/software/Java/1.8.0_261/bin:/ysm-gpfs/apps/software/LibTIFF/4.0.9-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/NASM/2.13.03-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/SQLite/3.24.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/Tcl/8.6.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libreadline/7.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/cairo/1.14.12-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GLib/2.54.3-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/PCRE/8.41-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/gettext/0.19.8.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libxml2/2.9.8-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/XZ/5.2.4-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/LLVM/6.0.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/nettle/3.4-foss-2018b/bin:/ysm-gpfs/apps/software/X11/20180604-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/fontconfig/2.13.0-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/util-linux/2.32-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/ncurses/6.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/freetype/2.9.1-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/libpng/1.6.34-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/expat/2.2.5-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/bzip2/1.0.6-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/FFTW/3.3.8-gompi-2018b/bin:/ysm-gpfs/apps/software/OpenBLAS/0.3.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/OpenMPI/3.1.1-GCC-7.3.0-2.30/bin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/sbin:/ysm-gpfs/apps/software/hwloc/1.11.10-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/numactl/2.0.11-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/binutils/2.30-GCCcore-7.3.0/bin:/ysm-gpfs/apps/software/GCCcore/7.3.0/bin:/gpfs/ysm/bin:/opt/TurboVNC/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/pgc29/.local/bin:/home/pgc29/bin'
export CONDA_PREFIX='/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins'
export CONDA_SHLVL='1'
export CONDA_DEFAULT_ENV='/gpfs/ysm/project/cohen_theodore/pgc29/conda_envs/gubbins'
export CONDA_PROMPT_MODIFIER='(gubbins)'
export CONDA_EXE='/ysm-gpfs/apps/software/miniconda/4.11.0/bin/conda'
export _CE_M=''
export _CE_CONDA=''
export CONDA_PYTHON_EXE='/ysm-gpfs/apps/software/miniconda/4.11.0/bin/python'
cd /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins
run_gubbins.py \
    -p Taiwan_MKansasii \
    --threads 8 \
    /home/pgc29/scratch60/Taiwan_MKansasii/data/gubbins/Taiwan_MKansasii.aln

@pgcudahy
Copy link
Author

I added #SBATCH --export=ALL to my hosts.yml config, but still get the original error when submitting from a notebook.

@BoPeng
Copy link
Contributor

BoPeng commented Mar 12, 2022

The notebook simply runs sbatch to submit jobs so you will need to check if you can manually submit jobs with that module load command.

@pgcudahy
Copy link
Author

To try and further debug this I saw that the original error message said Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for this output (/usr/share/lmod/lmod/init/bash)

So I added #SBATCH --export=LMOD_SH_DBG_ON=1 to my hosts.yml config, and now it has started working again. I have no idea why this changed anything, but it works. Probably an idiosyncrasy of my cluster and nothing to do with SOS so sorry to take up your time.

Best,
Patrick

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pgcudahy @BoPeng