Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhance use of batch scripts #133

Merged
merged 80 commits into from
Dec 12, 2024
Merged

Conversation

paulf81
Copy link
Collaborator

@paulf81 paulf81 commented Dec 5, 2024

This draft pull requests starts the effort to use more draft scripts to automate certain start-up processes and avoid issues related to bad conda set ups and helics_port selections.

Proposed changes

  • A series of start up scripts are added to a high-level folder scripts/
  • The run scripts are modified to use these scripts
  • A first script find_and_kill_helics.sh looks to find and kill any open helics jobs
  • activate_conda.sh sets up the CONDA environment within bash
  • get_helics_port.sh looks for an open set of 10 consecutive ports, and assigns the first to HELICS_PORT

When we feel good about this

  • Need to modify hercules_runscript_CLcontrol and similar to accept HELICS_PORT as a command line argument and not something specified in input files

Before completing

  • Update all examples to new standards
  • Confirm all examples run
  • Make sure pointed back at develop

@paulf81 paulf81 closed this Dec 5, 2024
@paulf81 paulf81 reopened this Dec 5, 2024
@paulf81 paulf81 changed the base branch from feature/standard_example to develop December 5, 2024 21:53
@paulf81
Copy link
Collaborator Author

paulf81 commented Dec 7, 2024

@misi9170 and @genevievestarke , at this example 8 works with the method of looking for open ports and chaining them through. I did find still it can happen you need to force a new port, but the new first_port variable in the runscript provides a single location to force this change. Before I replicate these changes to the other examples, would you mind to have a look over example 8, see if it runs for you and let me know if don't like anything?

Copy link
Collaborator

@genevievestarke genevievestarke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@paulf81 I reviewed this pull request! The changes seemed split between Ex 08 and Ex 09, so I updated the input file for Ex 08 to make a hopefully comprehensive example. I also adjusted floris_standin.py and amr_wind_standin.py to hopefully act the same in the examples. Let me know if I broke something!

source "$CONDA_PATH"
conda activate hercules
# Locate the scripts folder
SCRIPTS_DIR="../../scripts"
Copy link
Collaborator

@misi9170 misi9170 Dec 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are the forward slashes going to create issues for Windows users? This PR is not the first to use forward slashes in shell scripts, but just wondering if it's something we need to be thinking about

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you run the script with bash though? I'm not sure,

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if a windows user is using wsl, or git bash should be fine, if not maybe we just need a windows version of these scripts? Is there a windows native equivalent of bash?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, perhaps we should just be encouraging Windows users to use WSL... presumably, users could try to run things in the Command Prompt, and I'm not sure it would work.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good to me

@paulf81
Copy link
Collaborator Author

paulf81 commented Dec 11, 2024

Thanks @genevievestarke , I think I've carried the changes through the rest of the examples now

temp = read_amr_wind_input(amr_input_file)

# Check that helics_port is an integer
# If helics_port provided update with value
if helics_port is not None:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need an else clause? Is the idea that helics_port can be provided by the user as an argument, or it can come from the input yaml?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as written if helics_port is not provided, it doesn't over write the value already implied by the input yaml, but if there is no value there we'd need to do something, not sure which way we wanted to go

@misi9170
Copy link
Collaborator

misi9170 commented Dec 11, 2024

How confident are we that it is only processes that have "helics" in the name that cause the usual "Helics broker could not connect" (or similar) message? I had thought that some of the python calls can also cause this hangup, because they have also received the helics_port and were trying to connect via Helics to the broker, but I may well be wrong about that. I can play around with it a bit and see if it seems to work for me

@paulf81
Copy link
Collaborator Author

paulf81 commented Dec 11, 2024

How confident are we that it is only processes that have "helics" in the name that cause the usual "Helics broker could not connect" (or similar) message? I had thought that some of the python calls can also cause this hangup, because they have also received the helics_port and were trying to connect via Helics to the broker, but I may well be wrong about that. I can play around with it a bit and see if it seems to work for me

Not too confident! I've seen there are still some exceptions (at least on my machine) that cause the later helics federates to go to a higher port number. A nice thing about the new format is that there is single place to set the port number, so it's pretty quick to just try a new starting port in the event of trouble

@paulf81 paulf81 marked this pull request as ready for review December 11, 2024 21:05
@paulf81 paulf81 merged commit 3062d78 into NREL:develop Dec 12, 2024
3 checks passed
@paulf81 paulf81 deleted the feature/batch_scripts branch December 12, 2024 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants