Add support for 'paths' in configuration to allow one to search for scheduler binary in non-standard location #1757

shahzebsiddiqui · 2024-04-18T19:03:19Z

This PR will attempt to add the following keys in configuration file where one can specify an alternate location where to search for scheduler binaries that can be useful if they are not in $PATH

    paths:
      lsf: "/usr/local/bin"
      slurm: "/usr/local/bin"
      pbs: "/usr/local/bin"
      cobalt: "/usr/local/bin"
      torque: "/usr/local/bin"

some of the keys that it can be declared are 'pbs', 'torque', 'lsf', 'slurm', 'cobalt' for specifying directory paths where to search for binaries.

…hich will be a list of colon separated list of directory paths where to search for binaries

…nary

binary paths as a dictionary. An optional argument custom_dirs can be used to search for binary in additional paths that will be appended to PATH

needed. We are doing this check later in SiteConfiguration class

method from the class Change Torque class to be subclass of Scheduler instead of PBS and make use of custom_dirs to properly detect the binary paths. We have changed logic on how to determine a scheduler is active

correct binary paths

cobalt binaries

in the if condition since we still want to poll them and they will be cancelled once maxpendtime is reached

For sleep_cancel job we will hold the job so it cancels

for more information, see https://pre-commit.ci

job also forgot to call method retrieve_jobdata in gather method for SlurmExecutor class which is needed to get the job data used for detecting the exitcode

remove unnecessary return statement that caused issue with cobalt job to not cancel job since return value was None

for more information, see https://pre-commit.ci

…nstall we needed to add option to wget --no-check-certificate to get the script to install. We updated the example buildspecs for pbs tests

…g each keyword

change logic for when to retrieve get_queues method for PBS and Torque only if check method pass

shahzebsiddiqui added 11 commits April 17, 2024 16:51

add property name 'paths' that is an object which can specify paths.

407400f

some of the keys that it can be declared are 'pbs', 'torque', 'lsf', 'slurm', 'cobalt' for specifying directory paths where to search for binaries.

modify implementation of Scheduler class to pass custom_dirs method w…

2672eb7

…hich will be a list of colon separated list of directory paths where to search for binaries

update shutil.which call to specify additional paths to search for bi…

1075911

…nary

add method check_binaries that will be used for returning a list of

64ae60b

binary paths as a dictionary. An optional argument custom_dirs can be used to search for binary in additional paths that will be appended to PATH

remove scheduler check from BuildTestSystem class since its not really

30b6f37

needed. We are doing this check later in SiteConfiguration class

fix import issues since class were moved to another file

f1ad769

change logic of detection class. We have removed the check_binaries

2839095

method from the class Change Torque class to be subclass of Scheduler instead of PBS and make use of custom_dirs to properly detect the binary paths. We have changed logic on how to determine a scheduler is active

change logic for PBSExecutor and PBSJob to ensure it picks up the

2aa5b53

correct binary paths

change logic for SlurmExecutor and SlurmJob to ensure it picks up the

30ddc76

correct binary paths

change logic for LSFExecutor and LSFJob to ensure it picks up the

a084624

correct binary paths

change logic for CobaltExecutor and CobaltJob to ensure it picks up

0938efb

cobalt binaries

pull-request-size bot added the size/L label Apr 18, 2024

shahzebsiddiqui and others added 13 commits April 18, 2024 19:42

suspending jobs were dropped from table of polled jobs so we added them

555b638

in the if condition since we still want to poll them and they will be cancelled once maxpendtime is reached

change executor names for example tests.

06a74cb

For sleep_cancel job we will hold the job so it cancels

[pre-commit.ci] auto fixes from pre-commit.com hooks

0716224

for more information, see https://pre-commit.ci

fix regular expression pattern for hostname

e0036a5

add log statement for showing returncode for slurm job when it finishes

7869240

job also forgot to call method retrieve_jobdata in gather method for SlurmExecutor class which is needed to get the job data used for detecting the exitcode

update hostname for jlse

f624a2b

need to have regular expression in single quotes.

fdf2d7a

remove unnecessary return statement that caused issue with cobalt job to not cancel job since return value was None

[pre-commit.ci] auto fixes from pre-commit.com hooks

a35e7d6

for more information, see https://pre-commit.ci

update pbs setup script for installing miniconda since it failed to i…

a42f66c

…nstall we needed to add option to wget --no-check-certificate to get the script to install. We updated the example buildspecs for pbs tests

add paths keyword in main configuration file with comments documentin…

0fc2155

…g each keyword

add paths to example configuration files for schema test validation

de457c8

add documentation for 'paths' in configuring buildtest page

0a98bde

add method active for Torque class

1976fa2

change logic for when to retrieve get_queues method for PBS and Torque only if check method pass

shahzebsiddiqui merged commit 50deac0 into devel Apr 19, 2024
36 of 38 checks passed

shahzebsiddiqui deleted the paths_for_sched_libs_in_configuration branch April 19, 2024 18:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for 'paths' in configuration to allow one to search for scheduler binary in non-standard location #1757

Add support for 'paths' in configuration to allow one to search for scheduler binary in non-standard location #1757

shahzebsiddiqui commented Apr 18, 2024

Add support for 'paths' in configuration to allow one to search for scheduler binary in non-standard location #1757

Add support for 'paths' in configuration to allow one to search for scheduler binary in non-standard location #1757

Conversation

shahzebsiddiqui commented Apr 18, 2024