Biobb_remote is a package to allow biobb's to be executed on remote sites through ssh
Provides SSHCredentials to manage the generation, and installation of ssh credentials
credentials = SSHCredentials(host='', userid='', generate_key=False, look_for_keys=True)
- host (str): remote host name
- userid (str): remote user name
- generate_key (bool): Generates new public/private keys pair
- look_for_keys (bool): Look user's .ssh keys if keys not set
(void) credentials.save(output_path, public_key_path=None, private_key_path=None, passwd=None)
Stores SSHCredentials object in a external file
- output_path (str): Path to file
- public_key_path (str): Path to a standard public key file
- private_key_path (str): Path to a standard private key file
- passwd (str): Password to encrypt private key (optional)
(void) credentials.load_from_file(credentials_path, passwd=None)
Recovers SSHCredentials object from disk file
- credentials_path (str): Path to packed credentials file.
- passwd (str): Passwd to decrypt private key (optional)
(void) credentials.load_from_private_key_file(private_path, passwd=None)
Recovers SSHCredentials object from disk file
- private_path (str): Path to private key file.
- passwd (str): Passwd to decrypt private key (optional)
(void) credentials.generate_key(nbits=2048)
Generates RSA keys pair
- nbits (int): number of bits the generated key
(str) credentials.get_public_key(suffix='@biobb')
Returns a readable publik key suitable to addto authorized keys
- suffix (str): Added to the key for identify it.
(str) credentials.get_private_key(passwd=None)
Returns a readable possibly encrypted private key
- passwd (str): Password to encrypt private key (optional)
(bool) credentials.check_host_auth()
Checks for public_key in remote .ssh/authorized_keys file. Requires users' SSH access to host
(void) credentials.install_host_auth(file_bck='bck')
Installs public_key on remote .ssh/authorized_keys file. Requires users' SSH access to host
- file_bck (str): Generates an authorized.file_bck file with original authorized keys
(void) credentials.remove_host_auth(file_bck='biobb')
Removes public_key on remote .ssh/authorized_keys file. Requires users' SSH access to host
- file_bck (str): Generates an authorized.file_bck file with original authorized keys
Class wrapping ssh operations
ssh_session = SSHSession(ssh_data=None, credentials_path=None, private_path=None, passwd=None)
- ssh_data (SSHCredentials) : SSHCredentials object
- credentials_path (str) : Path to packed credentials file to use
- private_path (str): Path to private key file
- passwd (str): Password to decrypt credentials (optional)
(str) ssh_session.run_command(command)
Runs command on remote. Returns stdout + stderr
- command (str): Command line to execute
(bool | file_handle) ssh_session.run_sftp(oper, input_file_path, output_file_path='')
Runs SFTP session on remote
- oper (str): Operation to perform, one of * get (gets a single file from input_file_path (remote) to output_file_path (local) ) * put (puts a single file from input_file_path (local) to output_file_path (remote) * create (creates a file in output_file_path (remote) from input_file_path string- * file (opens a remote file in input_file_path for read). Returns a file handle. * listdir (returns a list of files in remote input_file_path
- input_file_path (str): Input file path or input string
- output_file_path (str): Output file path
DataBundle Class to manage bundles of input/output files
data_bundle = DataBundle(bundle_id)
- bundle_id (str): Id for the data bundle
data_bundle.add_file(file_path)
Adds a single file to the data bundle
- file_path (str): Path to the file to add
data_bundle.add_dir(dir_path)
Adds all files from a directory
- dir_path (str): Path to the directory to add
([str]) data_bundle.get_file_names()
Generates a list of names or included files
(str) data_bundle.to_json()
Generates a Json dump
Task Abstract module to handle remote tasks. Not for direct use, extend to include specific queueing systems
task.UNKNOWN = 0
task.SUBMITTED = 1
task.RUNNING = 2
task.CANCELLED = 3
task.FINISHED = 4
task.CLOSING = 5
task.JOB_STATUS (dict)
task=Task(host=None, userid=None, look_for_keys=True, debug_ssh=False)
Classe to handle task execution
- host (str): Remote host
- userid (str): Remote user id
- look_for_keys (bool): Look for keys available in user's .ssh directory debug_ssh (bool): Open SSH session with debug activated
(void) task.load_data_from_file(file_path, mode='json')
Loads accumulated task data from external file
- file_path (str): Path to file
- mode (str): Format. Json | Pickle
(void) task.save(save_file_path, mode='json', verbose=False)
Saves current task status in a external file. Can be used to recover session at a later time.
- save_file_path (str): Path to file
- mode (str): Format to use json|pickle.
- verbose (bool): Print additional information
(void) task.set_credentials(credentials, passwd=None):
Loads ssh credentials from SSHCredentials object or from a external file
- credentials (SSHCredentials | str): SSHCredentials object or a path to a file containing the data
- passwd (str): Password to decrypt private key when loaded from file (optional)
(void) task.set_private_key(private_path, passwd=None):
Inserts private key from external file
- private_path (str): Path to private key file
- passwd (str, optional): Password to decrypt private key
(void) task.load_host_config(host_config_path)
Loads a pre-defined host configuration file (json format)
- host_config_path (str): Path to the configuration file
(void) task.set_custom_settings(self, ref_setting='default', patch=None, clean=False)
Generates a custom queue setting based on existing one
- ref_setting (str): Base settings to modify
- patch (dict): Patch to apply
- clean (bool): Clean existing settings
(void) task.prep_auto_settings(total_cores=0, nodes=0, cpus_per_task=1, num_gpus=0)
Generates queue configuration settings for balancing MPI/OMP/GPU.
- total_cores (int): Aproximated number of cores to use
- nodes (int): Number of complete nodes to use (overrides total_cores)
- cpus_per_task (int): OMP processes per MPI task to allocate
- num_gpus (int): Num of GPUs per node to allocate
(void) task.set_local_data_bundle(local_data_path, add_files=True)
Builds local data bundle from a local directory
- local_data_path (str): Path to local data directory
- add_files (bool): On create, add all files in the directory.
(void) task.prep_remote_workdir(remote_base_path)
Creates empty remote working dir (remote_base_path/task.id)
- remote_base_path (str): Path to remote base directory, task folders created within
(void) task.send_input_data(remote_base_path, create_dir=True, overwrite=True, new_only=True)
Uploads data bundle files to remote working dir
- remote_base_path (str): Remote base path for all task activites. Each task will create a unique working dir (re-usable).
- create_dir(str): Create remote working directory before sending data.
- overwrite (bool): Upload files even if they already exists in the remote working dir.
- new_only (bool): Overwrite only with newer files
(str) task.get_remote_py_script(python_import, files, command, properties='')
Generates 1 line python code to be executed in the queue script using python -c
- python_import (str): Python import line(s) to include (; separated)
- files (dict): File names to associate to biobb required path parameters
- command (str): biobb class to launch
- properties (dict | str): Either a dict, path to a json or yaml config file or a 1-line Json with the required biobb parameters
(str) task.get_remote_comm_line(command, files, use_biobb=False, properties='', cmd_settings=''):
Generates a command line for queue script. Can be used to launch a biobb module or any command line remotely.
- job_name (str): Job name to display (optional, used to identify queue jobs, and stdout/stderr logs)
- command (str): Command to execute
- files (dict): Input/output files. "--" prefix added if only a parameter name is provided
- use_biobb (bool): Set to prepend biobb path on host
- properties (dict): BioBB properties
- cmd_settings (dict): Additional settings to add to the command line, pre-set bundles can be configured in host config data.
(void) task.submit(job_name=None, queue_settings='default', modules=None, local_run_script='', conda_env='', save_file_path=None, poll_time=0)
Submits task to remote. Optionally waits until completion.
- job_name (str): Job name to display in the queuing system. Stdout/stderr logs are named as job.name.(out|err). Optional, defaults to queue default behaviour.
- queue_settings (str): Label for set of queue settings (defined in host configuration). Use 'custom' for user defined settings (see set_custom_settings)
- modules (str): modules to activate (defined in host configuration)
- conda_env (str): Conda environment to activate
- local_run_script (str): Path to local script to run or a string with the script itself (identified by leading '#' tag)
- save_file_path (str): Path to local task log file to update after submit (Default None),
- poll_time (int): if set, polls periodically for job completion (seconds)
(void) task.cancel(remove_data=False)
Cancels running task
- remove_data (bool): Removes remoted workign directory.
(str) task.check_queue()
Check queue status. Returns output of the remote appropriate command
(void) check_job(update=True, poll_time=0, save_file_path=None)
Prints job status to stdout
- update (bool): update status before printing it
- poll_time (int): poll until job finished. Poll interval in seconds.
- save_file_path (str): Path to local task log file to update status (Default None),
(void) task.get_remote_file(file):
Gets file from remote working dir
- file (str): File name
([stdout, stderr]) task.get_logs()
Get queue logs
(void) task.get_output_data(local_data_path='', files_only=None, overwrite=True, new_only=True)
Downloads remote working dir contents to local path
- local_data_path (str): Path to local directory
- files_only ([str]): Only download files in list, if empty download all files
- overwrite (bool): Overwrite local files if they exist
- new_only (bool): Overwrite only with newer files
(void) task.clean_remote()
Remove remote working dir
Task Class extended to include specific settings for Slurm queueing system
Host configuration files
Generates kay pairs to be consumed by other utilities
credentials [-h] [--user USERID] [--host HOSTNAME]
[--pubkey_path PUBKEY_PATH] [--nbits NBITS] --keys_path
KEYS_PATH [--privkey_path PRIVKEY_PATH]
{create,get_pubkey,get_private}
Simple sftp service
scp_service [-h] --keys_path KEYS_PATH [-i INPUT_FILE_PATH]
[-o OUTPUT_FILE_PATH]
{get,put,create,file,listdir}
Simple remote ssh command
ssh_command [-h] --keys_path KEYS_PATH [command [command ...]]
Complete set of functions to manage slurm submissions remotely
slurm_test [-h] --keys_path KEYS_PATH [--script SCRIPT_PATH]
[--local_data LOCAL_DATA_PATH] [--remote REMOTE_PATH]
[--queue_settings Q_SETTINGS] [--module MODULE]
[--task_data TASK_FILE_PATH]
{submit,queue,cancel,status,get_data,put_data}
v1.2.2 November 2021
This software has been developed in the MMB group (http://mmb.irbbarcelona.org) at the BSC (http://www.bsc.es/) & IRB (https://www.irbbarcelona.org/) for the European BioExcel (http://bioexcel.eu/), funded by the European Commission (EU H2020 675728).
- (c) 2015-2021 Barcelona Supercomputing Center
- (c) 2015-2021 Institute for Research in Biomedicine
Licensed under the GNU Lesser General Public License v2.1, see the file LICENSE for details.
