Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change Cluster-based run_script/submit_workflow script path location #162

Closed
bch0w opened this issue Mar 2, 2023 · 1 comment
Closed

Comments

@bch0w
Copy link
Member

bch0w commented Mar 2, 2023

System modules derived from the Cluster class use an external run script to execute tasks on compute nodes. This run script is essentially a python script that loads functions from Pickle files and executes them. This allows tasks to be partitioned from a master job to each compute node.

run_call = " ".join([
f"{self.run_call_header}",
f"{os.path.join(ROOT_DIR, 'system', 'runscripts', 'run')}",
f"--funcs {funcs_fid}",
f"--kwargs {kwargs_fid}",
f"--environment SEISFLOWS_TASKID={{task_id}},{self.environs}"
])

Currently the run calls (located in the run() function) points to a symlink to a Python script, which was used to make things clearer. However when SeisFlows is installed via Pip, these symlinks are not carried through.

I see two ways to address this:

  1. add a MANIFEST.in file to keep the symlinks
  2. change the paths from 'run' to 'run_funcs.py' (which is carried through on pip installation)

If 2 is chosen, it might be beneficial to define the path to this script as a class variable once (in Cluster) rather than having it floating around in lots of different places. The same should be done for the submit workflow function:

submit_call = " ".join([
f"{self.submit_call_header}",
f"{os.path.join(ROOT_DIR, 'system', 'runscripts', 'submit')}",
f"--workdir {workdir}",
f"--parameter_file {parameter_file}",
])

bch0w added a commit that referenced this issue Mar 8, 2023
@bch0w
Copy link
Member Author

bch0w commented Mar 8, 2023

This is addressed in 1163f46 with a Manifest.in file

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

1 participant