Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/logging #86

Merged
merged 6 commits into from
Sep 8, 2017
Merged

Feature/logging #86

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4d237b4
templates for submitting batch jobs can log job ID now
Sep 2, 2017
c3e11be
added comments about whereabouts of a log file and also displaying in…
Sep 2, 2017
ba84476
added logging submission time
Sep 2, 2017
b9b6c5b
added logging stderr from batch system command
Sep 3, 2017
58ccaf9
added exit traps for deleteing temporary files
Sep 7, 2017
252526d
added comments for clearer code understanding
Sep 8, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 28 additions & 1 deletion mcpartools/scheduler/data/submit_slurm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,29 @@
#!/bin/bash
sbatch {options_args:s} --array=1-{jobs_no:d} --output="{log_dir:s}/output_%j_%a.log" --error="{log_dir:s}/error_%j_%a.log" {script_path:s}

# Log file submit.log will be created in the same directory submit.sh is located
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a clear description what is the difference between submit.log and "{log_dir:s}/output_%j_%a.log" files. First one will contain stderr and stdout from sbatch command (including job id and probably job execution status). Second one contains stdout from the MC engine executable (doing our simulation).

# submit.log is for storing stdout and stderr of sbatch command, for log info from individual jobs see {log_dir:s} directory
LOGFILE="$(cd $(dirname $0) && pwd)/submit.log"
echo -n "" > "$LOGFILE"

OUT=`mktemp`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a bit of comment what is happening here. Code does much better job now, but starts to be much harder to read.

Sth like: "we create now two temporary files, which will store stderr and stdout of sbatch command. Their content will be used for furter analysis, i.e. extraction of jobid. Files will be removed on script exit (see trap command)"

ERR=`mktemp`
sbatch {options_args:s} --array=1-{jobs_no:d} --output="{log_dir:s}/output_%j_%a.log" --error="{log_dir:s}/error_%j_%a.log" --parsable {script_path:s} > $OUT 2>$ERR

echo "Saving logs to $LOGFILE"

# if sbatch command ended with a success log following info
if [ $? -eq 0 ] ; then
echo "Job ID: `cat $OUT | cut -d ";" -f 1`" > "$LOGFILE"
echo "Submission time: `date +"%Y-%m-%d %H:%M:%S"`" >> "$LOGFILE"
fi

# if output from stderr isn't an empty string then log it as well to submit.log
if [ "`cat $ERR`" != "" ] ; then
echo "---------------------" >> "$LOGFILE"
echo "ERROR MESSAGE" >>"$LOGFILE"
echo "---------------------" >> "$LOGFILE"
cat $ERR >> "$LOGFILE"
fi

rm $OUT
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if script is killed or sbatch crashes ?
Consider making trap like this one: https://stackoverflow.com/questions/687014/removing-created-temp-files-in-unexpected-bash-exit

rm $ERR
28 changes: 27 additions & 1 deletion mcpartools/scheduler/data/submit_torque.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,29 @@
#!/usr/bin/env bash

qsub {options_args:s} -t 1-{jobs_no:d} -o {log_dir:s} -e {log_dir:s} {script_path:s}
# Log file submit.log will be created in the same directory submit.sh is located
# submit.log is for storing stdout and stderr of qsub command, for log info from individual jobs see {log_dir:s} directory
LOGFILE="$(cd $(dirname $0) && pwd)/submit.log"
echo -n "" > "$LOGFILE"

OUT=`mktemp`
ERR=`mktemp`
qsub {options_args:s} -t 1-{jobs_no:d} -o {log_dir:s} -e {log_dir:s} -terse {script_path:s} > $OUT 2> $ERR

echo "Saving logs to $LOGFILE"

# if qsub command ended with a success log following info
if [ $? -eq 0 ] ; then
echo "Job ID: `cat $OUT | cut -d ";" -f 1`" > "$LOGFILE"
echo "Submission time: `date +"%Y-%m-%d %H:%M:%S"`" >> "$LOGFILE"
fi

# if output from stderr isn't an empty string then log it as well to submit.log
if [ "`cat $ERR`" != "" ] ; then
echo "---------------------" >> "$LOGFILE"
echo "ERROR MESSAGE" >>"$LOGFILE"
echo "---------------------" >> "$LOGFILE"
cat $ERR >> "$LOGFILE"
fi

rm $OUT
rm $ERR