case.submit does not display system error messages

[worleyph]

Example, if two jobs are submitted to the debug queue on Titan, the second submittsion will fail. An informative error message used to be output (CIME 5.1 and earlier), but this is now missing:

 Submitting job script qsub    -q debug -l walltime=00:15:00 -A cli115 case.run 
 ERROR: Command: 'qsub    -q debug -l walltime=00:15:00 -A cli115 case.run ' failed with error ''

This is not the only useful error mesage that is now missing, but I don't have any other examples at the moment.

I vaguely remember already describing this problem somewhere, but can not find an existing open issue.

[jgfouca]

@worleyph , you are correct and @rljacob already noted this on the ESMCI side and it's already been fixed there.

[worleyph]

Thanks - that is probably what I remember - Rob volunteering to create this issue on the ESMCI side, based on a Slack conversation.

[worleyph]

@rljacob and @jgfouca , is the fix from the ESMCI side in CIME5.3 (i.e., in next)?

[jgfouca]

@worleyph yes, and it's on master now too.

[worleyph]

Not working correctly. Please reopen. I'll try to document this evening.

…

Sent from my iPhone

On Apr 19, 2017, at 12:08, James Foucar ***@***.***> wrote: @worleyph yes, and it's on master now too. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[worleyph]

@jgfouca , with current master, with one job already submitted to the debug queue on Titan, and another job submitted to the debug queue has the following error message:

 ./case.submit 
 ...
 Submitting job script qsub    -q debug -l walltime=00:45:00 -A cli115 case.run 
 ERROR: Command: 'qsub    -q debug -l walltime=00:45:00 -A cli115 case.run ' failed with error 'None'

If I instead submit a non-ACME job (using qsub directly), e.g. an interactive job request, I get

 >qsub -I -q debug -A cli115 -l walltime=30:00 nodes=1 (y|n|e|a)? yes
 
  Job not submitted 
  
  You currently have a job in the debug queue. Each 
   user is allowed to have only one job at a time in the       
   debug queue.  Please wait until job 3347570 completes  
   before submitting another job to the debug queue.     

Previously this error message was passed back when ./case.submit errored out.

I see that something cahnged; what was ' ', is now 'None', but not sure that this is a meaningful change :-).

[jgfouca]

@worleyph I'm seeing this too. Don't know what happened as this was working and tested on ESMCI. I'll look at it.

[jgfouca]

@worleyph apparently there was a bad interaction between two PRs on the ESMCI side. The first fixed this issue, then a subsequent PR (also by me) messed with combining output for shell commands and re-broke it. It's now once again fixed on the ESMCI side. It should make it back to ACME fairly soon or I can duplicate the fix (risking conflicts) on the ACME side if it's urgent.

[worleyph]

@jgfouca , I labelled this as 'minor', but I am only one person. I can't really make the determination as to whether this can be held back until the next ESMCI sync-up. I don't know how often failures occur, and what different failure modes there are (and how important feedback is in resolving these).

[philipwjones]

I've encountered this too, but have been able to diagnose failures from other log output, so guess not a showstopper. As long as the next update is relatively soon, can probably wait.

[worleyph]

@jgfouca , as I indicated in my e-mail to you and Peter Caldwell, I could use this fix now if you are willing to share. I'll just drop it into my working version.

[jgfouca]

@worleyph , apologies, must have deleted the email from github and I didn't notice your mention.

The fix is this:
ESMCI/cime#1390