bugfix cluster_queue in SGE with queue='' as new default

[sebastian-luna-valero]

After refactoring the default value for queue in the default pipeline.ini in #390 the SGE submission did not work properly.

This pull request solves that issue.

[IanSudbery]

This commit does not set queue='' as default.
The default queue is hardcoded in Parameters.py and is set to "all.q"

We needed a way to override this, so I invented "NONE" as a special keyword to specify that no queue should be provided. With this commit, there is no way to override the hardcoded default so as to specify no queue.

[sebastian-luna-valero]

Although there is a hard-coded default queue="all.q" in Parameters.py, that's overwritten when loading the default values in the pipeline.ini in the configuration folder:
https://github.com/CGATOxford/CGATPipelines/blob/master/CGATPipelines/configuration/pipeline.ini#L80

That was changed from queue=all.q to queue= in the PR mentioned above (#390)

The printconfig command confirms that in runtime:

cgatflow testing printconfig | grep queue

The best way to override this is with a $HOME/.cgat file, having:

[cluster]
queue=your-value-here

There is also a --cluster-queue=CLUSTER_QUEUE option for the cgatflow command.

[IanSudbery]

So at the moment we all have ~/.cgat files that say

[cluster]
queue=NONE

Because we need to force no queue specification for submission to our cluster to work.

Are you saying that if we set

[cluster]
queue=

That this will overwrite the all.q default with ''?

[sebastian-luna-valero]

Yes. In fact, if your ~/.cgat file only contains:

[cluster]
queue=

I think you can get rid of it after pulling these two PRs.

[IanSudbery]

As far as I can see, this is being set by adding something to the default ini file. Thus, as far as I can tell, it looks like it would only apply to either the standard pipelines, or to new pipelines developed from the new default pipeline.ini, but not to old pipelines.

Also, will every one at CGAT now need a ~/.cgat file, specifying "all.q"?

[sebastian-luna-valero]

Yes, it is defined in the default ini file. Production pipelines will use that configuration, I am not sure about obsolete ones.

I have reconfigured the job scheduler in CGAT so all.q is the default queue, even if queue='' so no need to have a ~/.cgat file in our systems right now.

[IanSudbery]

Perhaps it might be worth taking the hard coded "all.q" out of Parameters.py?

[sebastian-luna-valero]

Indeed, changing the hard-coded value from 'cluster_queue': 'all.q' to 'cluster_queue': '' in Parameters.py would be the next step. However, I wonder whether it is just better to wait for this and other changes to happen after we start working on the new CGAT core repository.