Add support for configurable job uniqueness. #265
Conversation
This is needed for proper job array support see also pull requests in the xdmod and xdmod-supremm and xdmod-xsede code base.
|
xdmod: ubccr/xdmod#1723 |
config/config.json
Outdated
| @@ -40,6 +40,7 @@ | |||
| "hostname_mode": "hostname", | |||
| "pcp_log_dir": "/data/pcp-logs/my_cluster_name", | |||
| "script_dir": "/data/jobscripts/my_cluster_name", | |||
| "job_uniq_mode": "local_job_id", | |||
There was a problem hiding this comment.
Doesn't there need to be a corresponding update to configure_resource() in supremm_setup.py for this config parameter?
There was a problem hiding this comment.
Originally I planned for this to be just to support ACCESS XDMoD (this code has been running on the XSEDE/ACCESS version for many years via a custom patch). Have to think about this...........
There was a problem hiding this comment.
Ok. Plan is to autodetect the job unique identifier - no need for a configuration setting.
src/supremm/xdmodaccount.py
Outdated
| WHEN jf.`local_job_id_raw` IS NULL THEN IF(jf.`local_job_array_index` = -1, jf.`local_jobid`, CONCAT(jf.`local_jobid`, '_', jf.`local_job_array_index`)) | ||
| WHEN sj.`source_format` = 'slurm' THEN jf.`local_job_id_raw` | ||
| ELSE IF(jf.`local_job_array_index` = -1, jf.`local_jobid`, CONCAT(jf.`local_jobid`, '_', jf.`local_job_array_index`)) | ||
| END AS job_uniq_id, |
There was a problem hiding this comment.
Does job_uniq_id need to be `job_uniq_id` (with backticks)? Or is that a bit pedantic? Just thinking if it should match the syntax for the other aliases.
src/supremm/xdmodaccount.py
Outdated
| WHEN jf.`local_job_id_raw` IS NULL THEN IF(jf.`local_job_array_index` = -1, jf.`local_jobid`, CONCAT(jf.`local_jobid`, '_', jf.`local_job_array_index`)) | ||
| WHEN sj.`source_format` = 'slurm' THEN jf.`local_job_id_raw` | ||
| ELSE IF(jf.`local_job_array_index` = -1, jf.`local_jobid`, CONCAT(jf.`local_jobid`, '_', jf.`local_job_array_index`)) | ||
| END AS job_uniq_index, |
There was a problem hiding this comment.
Same comment here as line 31.
Cherry pick #265 into v2 branch
This is needed for proper job array support see also pull requests in the xdmod and xdmod-supremm and xdmod-xsede code base.