Skip to content

Commit

Permalink
Version/1.12.1 (#473)
Browse files Browse the repository at this point in the history
* fix default worker bug with all steps

* version bump and requirements fix

* Bugfix/filename-special-vars (#425)

* fix file naming bug

* fix filename bug with variable as study name

* add tests for the file name special vars changes

* modify changelog

* implement Luc's suggestions

* remove replace line

* Create dependabot-changelog-updater.yml

* testing outputs of modifying changelog

* delete dependabot-changelog-updater

* feature/pdf-docs (#427)

* first attempt at adding pdf

* fixing build error

* modify changelog to show docs changes

* fix errors Luc found in the build logs

* trying out removal of latex

* reverting latex changes back

* uncommenting the latex_elements settings

* adding epub to see if latex will build

* adding a latex engine variable to conf

* fix naming error with latex_engine

* attempting to add a logo to the pdf build

* testing an override to the searchtools file

* revert back to not using searchtools override

* update changelog

* bugfix/openfoam_singularity_issues (#426)

* fix openfoam_singularity issues

* update requirements and descriptions for openfoam examples

* bugfix/output-path-substitution (#430)

* fix bug with output_path and variable substitution

* add tests for cli substitutions

* bugfix/scheduler-permission-error (#436)

* Release/1.10.2 (#437)

* bump version to 1.10.2

* bump version in CHANGELOG

* resolve develop to main merge issues (#439)

* fix default worker bug with all steps

* version bump and requirements fix

* dependabot/certifi-requests-pygments (#441)

* Bump certifi from 2022.12.7 to 2023.7.22 in /docs

Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22.
- [Commits](certifi/python-certifi@2022.12.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* add all dependabot changes and update CHANGELOG

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* bugfix/server-pip-redis-conf (#443)

* add *.conf to the MANIFEST file so pip will grab the redis.conf file

* add note explaining how to fix a hanging merlin server start

* modify CHANGELOG

* add second export option to docs and fix typo

* bump to version 1.10.3 (#444)

* bugfix/sphinx-5.3.0-requirement (#446)

* Version/1.10.3 (#445)

* fix default worker bug with all steps

* version bump and requirements fix

* Bugfix/filename-special-vars (#425)

* fix file naming bug

* fix filename bug with variable as study name

* add tests for the file name special vars changes

* modify changelog

* implement Luc's suggestions

* remove replace line

* Create dependabot-changelog-updater.yml

* testing outputs of modifying changelog

* delete dependabot-changelog-updater

* feature/pdf-docs (#427)

* first attempt at adding pdf

* fixing build error

* modify changelog to show docs changes

* fix errors Luc found in the build logs

* trying out removal of latex

* reverting latex changes back

* uncommenting the latex_elements settings

* adding epub to see if latex will build

* adding a latex engine variable to conf

* fix naming error with latex_engine

* attempting to add a logo to the pdf build

* testing an override to the searchtools file

* revert back to not using searchtools override

* update changelog

* bugfix/openfoam_singularity_issues (#426)

* fix openfoam_singularity issues

* update requirements and descriptions for openfoam examples

* bugfix/output-path-substitution (#430)

* fix bug with output_path and variable substitution

* add tests for cli substitutions

* bugfix/scheduler-permission-error (#436)

* Release/1.10.2 (#437)

* bump version to 1.10.2

* bump version in CHANGELOG

* resolve develop to main merge issues (#439)

* fix default worker bug with all steps

* version bump and requirements fix

* dependabot/certifi-requests-pygments (#441)

* Bump certifi from 2022.12.7 to 2023.7.22 in /docs

Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22.
- [Commits](certifi/python-certifi@2022.12.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* add all dependabot changes and update CHANGELOG

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* bugfix/server-pip-redis-conf (#443)

* add *.conf to the MANIFEST file so pip will grab the redis.conf file

* add note explaining how to fix a hanging merlin server start

* modify CHANGELOG

* add second export option to docs and fix typo

* bump to version 1.10.3 (#444)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* change hardcoded sphinx requirement

* update CHANGELOG

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feature/vlauncher (#447)

* fix file naming error for iterative workflows

* fixed small bug with new filepath naming

* add VLAUNCHER functionality

* add docs for VLAUNCHER and modify changelog

* re-word docs and fix table format

* add a test for vlauncher

* run fix-style and add a test for vlauncher

* Add the find_vlaunch_var and setup_vlaunch functions.
The numeric value of the shell variables may not be defined until run
time, so replace with variable strings instead of values.
Consolidate the commands into one function.

* Add variable set for (t)csh.

* Run fix-style

* make step settings the defaults and ignore commented lines

* add some additional tests

* remove regex library import

---------

Co-authored-by: Joseph M. Koning <koning1@llnl.gov>

* release/1.11.0 (#448)

* bugfix/skewed-sample-hierarchy (#450)

* add patch for skewed sample hierarchy/additional samples

* update changelog

* catch narrower range of exceptions

* bugfix/lsf-gpu-typo (#453)

* fix typo in batch.py that causes a bug

* change print statements to log statements

* release/1.11.1 (#454)

* Add Pytest Fixtures to Test Suite (#456)

* begin work on integration refactor; create fixtures and initial tests

* update CHANGELOG and run fix-style

* add pytest fixtures and README explaining them

* add tests to demonstrate how to use the fixtures

* move/rename some files and modify integration's README

* add password change to redis.pass file

* fix lint issues

* modify redis pwd for test server to be constant for each test

* fix lint issue only caught on github ci

* Bugfix for WEAVE CI (#457)

* begin work on integration refactor; create fixtures and initial tests

* update CHANGELOG and run fix-style

* add pytest fixtures and README explaining them

* add tests to demonstrate how to use the fixtures

* move/rename some files and modify integration's README

* add password change to redis.pass file

* fix lint issues

* modify redis pwd for test server to be constant for each test

* fix lint issue only caught on github ci

* add fix for merlin server startup

* update CHANGELOG

* bugfix/monitor-shutdown (#452)

* add celery query to see if workers still processing tasks

* fix merlin status when using redis as broker

* fix consumer count bug and run fix-style

* fix linter issues

* update changelog

* update docs for monitor

* remove unused exception I previously added

* first attempt at using pytest fixtures for monitor tests

* (partially) fix launch_workers fixture so it can be used in multiple classes

* fix linter issues and typo on pytest decorator

* update black's python version and fix style issue

* remove print statements from celeryadapter.py

* workers manager is now allowed to be used as a context manager

* add one thing to changelog and remove print statement

* Add the missing restart keyword to the specification docs. (#459)

* docs/conversion-to-mkdocs (#460)

* remove a merge conflict statement that was missed

* add base requirements for mkdocs

* set up configuration for API docs

* start work on porting user guide to mkdocs

* add custom styling and contact page

* begin work on porting tutorial to mkdocs

* add new examples page

* move old sphinx docs to their own folder (*delete later*)

* modify some admonitions to be success

* modify hello examples page and port step 3 of tutorial to mkdocs

* fix typo in hello example

* finish porting step 4 of tutorial to mkdocs

* port part 5 of the tutorial to mkdocs

* copy faq and contributing from old docs

* port step 6 of tutorial to mkdocs

* remove unused prereq

* port step 7 of tutorial to mkdocs

* add more detailed instructions on contributing

* move venv page into installation and add spack instructions too

* add configuration docs

* add content to user guide landing page

* port celery page to mkdocs

* rearrange configuration pages to add in merlin server configuration instructions

* port command line page to mkdocs

* finish new landing page

* change size of merlin logo

* port variables page to mkdocs

* fix broken links to configuration page

* port FAQ to mkdocs

* fix incorrect requirement name

* update CHANGELOG

* attempt to get docs to build through readthedocs

* port docker page to mkdocs

* port contributing guide to mkdocs

* add new 'running studies' page

* add path changes to images

* add a page on how to interpret study output

* add page on the spec file

* remove old sphinx docs that are no longer needed

* added README to docs and updated CHANGELOG

* fix copyright and hello_samples tree

* rearrange images/stylesheets and statements that use them

* add suggestions from Luc and Joe

* add tcsh instructions for venv activation

* add Charle's suggestions for the landing page

* change tcsh mentions to csh

* openfoam tutorial modifications (#463)

* feature/revamped status (#464)

* feature/new-status (#442)

* add backend functionality for merlin status

* add frontend functionality for merlin status

* add tests for merlin status

* run fix-style and remove import of deprecated function

* update CHANGELOG

* add more logging statements, make better use of glob

* run fix-style

* clean up test files a bit

* fix test suite after step_name_map mod

* add avg/std dev run time calculations to status

* modify status tests to accommodate new avg/std dev calculations

* fix linter issues

* fix lint issue and add test for avg/std dev calc

* feature/detailed-status (#451)

* Version/1.11.0 (#449)

* fix default worker bug with all steps

* version bump and requirements fix

* Bugfix/filename-special-vars (#425)

* fix file naming bug

* fix filename bug with variable as study name

* add tests for the file name special vars changes

* modify changelog

* implement Luc's suggestions

* remove replace line

* Create dependabot-changelog-updater.yml

* testing outputs of modifying changelog

* delete dependabot-changelog-updater

* feature/pdf-docs (#427)

* first attempt at adding pdf

* fixing build error

* modify changelog to show docs changes

* fix errors Luc found in the build logs

* trying out removal of latex

* reverting latex changes back

* uncommenting the latex_elements settings

* adding epub to see if latex will build

* adding a latex engine variable to conf

* fix naming error with latex_engine

* attempting to add a logo to the pdf build

* testing an override to the searchtools file

* revert back to not using searchtools override

* update changelog

* bugfix/openfoam_singularity_issues (#426)

* fix openfoam_singularity issues

* update requirements and descriptions for openfoam examples

* bugfix/output-path-substitution (#430)

* fix bug with output_path and variable substitution

* add tests for cli substitutions

* bugfix/scheduler-permission-error (#436)

* Release/1.10.2 (#437)

* bump version to 1.10.2

* bump version in CHANGELOG

* resolve develop to main merge issues (#439)

* fix default worker bug with all steps

* version bump and requirements fix

* dependabot/certifi-requests-pygments (#441)

* Bump certifi from 2022.12.7 to 2023.7.22 in /docs

Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22.
- [Commits](certifi/python-certifi@2022.12.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* add all dependabot changes and update CHANGELOG

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* bugfix/server-pip-redis-conf (#443)

* add *.conf to the MANIFEST file so pip will grab the redis.conf file

* add note explaining how to fix a hanging merlin server start

* modify CHANGELOG

* add second export option to docs and fix typo

* bump to version 1.10.3 (#444)

* bugfix/sphinx-5.3.0-requirement (#446)

* Version/1.10.3 (#445)

* fix default worker bug with all steps

* version bump and requirements fix

* Bugfix/filename-special-vars (#425)

* fix file naming bug

* fix filename bug with variable as study name

* add tests for the file name special vars changes

* modify changelog

* implement Luc's suggestions

* remove replace line

* Create dependabot-changelog-updater.yml

* testing outputs of modifying changelog

* delete dependabot-changelog-updater

* feature/pdf-docs (#427)

* first attempt at adding pdf

* fixing build error

* modify changelog to show docs changes

* fix errors Luc found in the build logs

* trying out removal of latex

* reverting latex changes back

* uncommenting the latex_elements settings

* adding epub to see if latex will build

* adding a latex engine variable to conf

* fix naming error with latex_engine

* attempting to add a logo to the pdf build

* testing an override to the searchtools file

* revert back to not using searchtools override

* update changelog

* bugfix/openfoam_singularity_issues (#426)

* fix openfoam_singularity issues

* update requirements and descriptions for openfoam examples

* bugfix/output-path-substitution (#430)

* fix bug with output_path and variable substitution

* add tests for cli substitutions

* bugfix/scheduler-permission-error (#436)

* Release/1.10.2 (#437)

* bump version to 1.10.2

* bump version in CHANGELOG

* resolve develop to main merge issues (#439)

* fix default worker bug with all steps

* version bump and requirements fix

* dependabot/certifi-requests-pygments (#441)

* Bump certifi from 2022.12.7 to 2023.7.22 in /docs

Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22.
- [Commits](certifi/python-certifi@2022.12.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* add all dependabot changes and update CHANGELOG

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* bugfix/server-pip-redis-conf (#443)

* add *.conf to the MANIFEST file so pip will grab the redis.conf file

* add note explaining how to fix a hanging merlin server start

* modify CHANGELOG

* add second export option to docs and fix typo

* bump to version 1.10.3 (#444)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* change hardcoded sphinx requirement

* update CHANGELOG

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feature/vlauncher (#447)

* fix file naming error for iterative workflows

* fixed small bug with new filepath naming

* add VLAUNCHER functionality

* add docs for VLAUNCHER and modify changelog

* re-word docs and fix table format

* add a test for vlauncher

* run fix-style and add a test for vlauncher

* Add the find_vlaunch_var and setup_vlaunch functions.
The numeric value of the shell variables may not be defined until run
time, so replace with variable strings instead of values.
Consolidate the commands into one function.

* Add variable set for (t)csh.

* Run fix-style

* make step settings the defaults and ignore commented lines

* add some additional tests

* remove regex library import

---------

Co-authored-by: Joseph M. Koning <koning1@llnl.gov>

* release/1.11.0 (#448)

* bugfix/skewed-sample-hierarchy (#450)

* add patch for skewed sample hierarchy/additional samples

* update changelog

* catch narrower range of exceptions

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Joseph M. Koning <koning1@llnl.gov>

* add functionality for the detailed-status command

* add tests for detailed-status

* fix linter issues

* update changelog

* general cleanup and add log statements

* slightly modify two tests

* default status renderer now uses json status format

* remove inaccurate comment

* bugfix/lsf-gpu-typo (#453)

* fix typo in batch.py that causes a bug

* change print statements to log statements

* release/1.11.1 (#454)

* Add Pytest Fixtures to Test Suite (#456)

* begin work on integration refactor; create fixtures and initial tests

* update CHANGELOG and run fix-style

* add pytest fixtures and README explaining them

* add tests to demonstrate how to use the fixtures

* move/rename some files and modify integration's README

* add password change to redis.pass file

* fix lint issues

* modify redis pwd for test server to be constant for each test

* fix lint issue only caught on github ci

* Bugfix for WEAVE CI (#457)

* begin work on integration refactor; create fixtures and initial tests

* update CHANGELOG and run fix-style

* add pytest fixtures and README explaining them

* add tests to demonstrate how to use the fixtures

* move/rename some files and modify integration's README

* add password change to redis.pass file

* fix lint issues

* modify redis pwd for test server to be constant for each test

* fix lint issue only caught on github ci

* add fix for merlin server startup

* update CHANGELOG

* bugfix/monitor-shutdown (#452)

* add celery query to see if workers still processing tasks

* fix merlin status when using redis as broker

* fix consumer count bug and run fix-style

* fix linter issues

* update changelog

* update docs for monitor

* remove unused exception I previously added

* first attempt at using pytest fixtures for monitor tests

* (partially) fix launch_workers fixture so it can be used in multiple classes

* fix linter issues and typo on pytest decorator

* update black's python version and fix style issue

* remove print statements from celeryadapter.py

* workers manager is now allowed to be used as a context manager

* add one thing to changelog and remove print statement

* Add the missing restart keyword to the specification docs. (#459)

* add Jeremy's suggestion to change vars option to output-path

* remove unnecessary lines from CHANGELOG

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Joseph M. Koning <koning1@llnl.gov>
Co-authored-by: Joe Koning <koning@users.noreply.github.com>

* feature/queue info (#461)

* remove a merge conflict statement that was missed

* add queue-info functionality

* add tests for queue-info

* update CHANGELOG

* add try/except for forceful termination of test workers

* change github workflow to use py38 with black instead of py36

* run fix-style with py 3.12 and fix a typo in a test

* add filetype check for dump option

* add banner print statement

* docs/revamped status (#462)

* fix broken image link in README

* add new commands to the command line page

* add monitoring docs layout and complete status cmds page

* fix bug with dumping queue-info to files

* add docs for queue-info

* add documentation for 'query-workers'

* add reference to new query-workers docs and split a paragraph

* fix small bug with --steps option of monitor

* add documentation for monitor command

* update CHANGELOG

* fix dump-csv image for queue-info

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Joseph M. Koning <koning1@llnl.gov>
Co-authored-by: Joe Koning <koning@users.noreply.github.com>

* release/1.12.0 (#465)

* remove a merge conflict statement that was missed

* bump version to 1.12.0

* feature/retry_priority (#468)

* remove a merge conflict statement that was missed

* add a 'pip freeze' call in github workflow to view reqs versions

* add new retry priority as highest task priority

* update CHANGELOG

* add in MID priority

* change default priority to use priority map MID value

* docs/server-cross-node (#470)

* remove a merge conflict statement that was missed

* add a 'pip freeze' call in github workflow to view reqs versions

* rename the merlin server config page

* add instructions for running a cross-node workflow w/ containerized server

* update CHANGELOG

* bugfix/initial-status-issues (#471)

* remove a merge conflict statement that was missed

* add a 'pip freeze' call in github workflow to view reqs versions

* fix bug with dry run status

* set MANPAGER for detailed-status

* fix bug with 1 sample removing the status file

* add support for multiple workers on one step in status files

* update test suite to accommodate changes to workers in status files

* add catch and potential fix for JSONDecodeError

* fix docstring of a test

* update CHANGELOG.md

* run fix style and add Luc's suggestions

* run fix-style with python 3.12

* added additional check for status file while condensing

* add try/except to catch an error for dumping statuses

* release/1.12.1 (#472)

* remove a merge conflict statement that was missed

* add a 'pip freeze' call in github workflow to view reqs versions

* bump version to 1.12.1

* fix a lint issue that somehow slipped through the cracks

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Joseph M. Koning <koning1@llnl.gov>
Co-authored-by: Joe Koning <koning@users.noreply.github.com>
Co-authored-by: Jane Herriman <xorJane@users.noreply.github.com>
  • Loading branch information
5 people authored Apr 16, 2024
1 parent 60e46d7 commit b4321d0
Show file tree
Hide file tree
Showing 79 changed files with 1,416 additions and 430 deletions.
1 change: 1 addition & 0 deletions .github/workflows/push-pr_workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,7 @@ jobs:
python3 -m pip install --upgrade pip
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
pip3 install -r requirements/dev.txt
pip freeze
- name: Install singularity
run: |
Expand Down
18 changes: 18 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,24 @@ All notable changes to Merlin will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [1.12.1]
### Added
- New Priority.RETRY value for the Celery task priorities. This will be the new highest priority.
- Support for the status command to handle multiple workers on the same step
- Documentation on how to run cross-node workflows with a containerized server (`merlin server`)

### Changed
- Modified some tests in `test_status.py` and `test_detailed_status.py` to accommodate bugfixes for the status commands

### Fixed
- Bugfixes for the status commands:
- Fixed "DRY RUN" naming convention so that it outputs in the progress bar properly
- Fixed issue where a step that was run with one sample would delete the status file upon condensing
- Fixed issue where multiple workers processing the same step would break the status file and cause the workflow to crash
- Added a catch for the JSONDecodeError that would potentially crash a run
- Added a FileLock to the status write in `_update_status_file()` of `MerlinStepRecord` to avoid potential race conditions (potentially related to JSONDecodeError above)
- Added in `export MANPAGER="less -r"` call behind the scenes for `detailed-status` to fix ASCII error

## [1.12.0]
### Added
- A new command `merlin queue-info` that will print the status of your celery queues
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.0.
# This file is part of Merlin, Version: 1.12.1.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down
Loading

0 comments on commit b4321d0

Please sign in to comment.