hpc parameter+ #53
Open
hpc parameter+ #53
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added `--cluster` argument (ilifu or galahad, but could be expanded to run as a standard on other slurm based HPC).
Read in default / limiting values from a default config which specifies which HPC facility has which limitations in their nodes etc. This should eventually allow for easy addition of further slurm based HPCs.
…around default calls.
Bug fixes and
Passing command line arguments from initial call to allow for other configurations / calls to be used.
Maintain cli in recursive processMeerKAT.py calls.
Added call parameters to avoid que pair (QP) errors.
Saving selected HPC name in config during build to allow for future use without setting hpc explicitly in CLI. i.e. for running.
Importing `processMeerKAT.py` and `config_parser.py` from the parent directory has caused many issues. Using this fix is not ideal, but it should be a consistent fix across systems.
Using hpc option saved in built config to remove global dependency by reading in known hpc config.
|
+1 to this change! I've just opened a PR on the PR with some minor fixes I found. One outlying issue here is that now the help screen is pretty bare, e.g. I think the new arg parsing messes with the help output. A similar issue was talked about here, but I'm not sure if that solution would really help here. |
Minor parsing fixes + new HPC + CLI config convention bug fix.
|
Ping @mb010 @Jordatious @Kitchi, I've got a big fix for this PR in mb010#2. It'd be great if this could be reviewed an merged, if approved. |
Fixing args and merge casa6 branch
Config fix for hpc parameter priority.
Pipeline V2.0 release
Changed casa6 --> master to reflect the merge for V2.0
|
Any chance this can be merged? I don't know who owns / holds responsibility for this repo, but if its functionally dead, this feature has been tested and will help others to use it in the future. :) Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I aim to write code that can generalise to other slurm facilities within ~30min of someone cloning the code.
With a few minor changes / improvements / bug fixes the primary change is the
known_hpc.cfgfile which contains all global variables previously inprocessMeerKAT.pyand a few parameters to help build system specific sbatch files. This should allow users to add the limits of their hpc facility and have their submissions checked against those values, instead of the ilifu limits (node, mem, cores etc.).