Skip to content

Commit

Permalink
Version/1.12.2 (#492)
Browse files Browse the repository at this point in the history
* fix default worker bug with all steps

* version bump and requirements fix

* Bugfix/filename-special-vars (#425)

* fix file naming bug

* fix filename bug with variable as study name

* add tests for the file name special vars changes

* modify changelog

* implement Luc's suggestions

* remove replace line

* Create dependabot-changelog-updater.yml

* testing outputs of modifying changelog

* delete dependabot-changelog-updater

* feature/pdf-docs (#427)

* first attempt at adding pdf

* fixing build error

* modify changelog to show docs changes

* fix errors Luc found in the build logs

* trying out removal of latex

* reverting latex changes back

* uncommenting the latex_elements settings

* adding epub to see if latex will build

* adding a latex engine variable to conf

* fix naming error with latex_engine

* attempting to add a logo to the pdf build

* testing an override to the searchtools file

* revert back to not using searchtools override

* update changelog

* bugfix/openfoam_singularity_issues (#426)

* fix openfoam_singularity issues

* update requirements and descriptions for openfoam examples

* bugfix/output-path-substitution (#430)

* fix bug with output_path and variable substitution

* add tests for cli substitutions

* bugfix/scheduler-permission-error (#436)

* Release/1.10.2 (#437)

* bump version to 1.10.2

* bump version in CHANGELOG

* resolve develop to main merge issues (#439)

* fix default worker bug with all steps

* version bump and requirements fix

* dependabot/certifi-requests-pygments (#441)

* Bump certifi from 2022.12.7 to 2023.7.22 in /docs

Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22.
- [Commits](certifi/python-certifi@2022.12.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* add all dependabot changes and update CHANGELOG

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* bugfix/server-pip-redis-conf (#443)

* add *.conf to the MANIFEST file so pip will grab the redis.conf file

* add note explaining how to fix a hanging merlin server start

* modify CHANGELOG

* add second export option to docs and fix typo

* bump to version 1.10.3 (#444)

* bugfix/sphinx-5.3.0-requirement (#446)

* Version/1.10.3 (#445)

* fix default worker bug with all steps

* version bump and requirements fix

* Bugfix/filename-special-vars (#425)

* fix file naming bug

* fix filename bug with variable as study name

* add tests for the file name special vars changes

* modify changelog

* implement Luc's suggestions

* remove replace line

* Create dependabot-changelog-updater.yml

* testing outputs of modifying changelog

* delete dependabot-changelog-updater

* feature/pdf-docs (#427)

* first attempt at adding pdf

* fixing build error

* modify changelog to show docs changes

* fix errors Luc found in the build logs

* trying out removal of latex

* reverting latex changes back

* uncommenting the latex_elements settings

* adding epub to see if latex will build

* adding a latex engine variable to conf

* fix naming error with latex_engine

* attempting to add a logo to the pdf build

* testing an override to the searchtools file

* revert back to not using searchtools override

* update changelog

* bugfix/openfoam_singularity_issues (#426)

* fix openfoam_singularity issues

* update requirements and descriptions for openfoam examples

* bugfix/output-path-substitution (#430)

* fix bug with output_path and variable substitution

* add tests for cli substitutions

* bugfix/scheduler-permission-error (#436)

* Release/1.10.2 (#437)

* bump version to 1.10.2

* bump version in CHANGELOG

* resolve develop to main merge issues (#439)

* fix default worker bug with all steps

* version bump and requirements fix

* dependabot/certifi-requests-pygments (#441)

* Bump certifi from 2022.12.7 to 2023.7.22 in /docs

Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22.
- [Commits](certifi/python-certifi@2022.12.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* add all dependabot changes and update CHANGELOG

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* bugfix/server-pip-redis-conf (#443)

* add *.conf to the MANIFEST file so pip will grab the redis.conf file

* add note explaining how to fix a hanging merlin server start

* modify CHANGELOG

* add second export option to docs and fix typo

* bump to version 1.10.3 (#444)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* change hardcoded sphinx requirement

* update CHANGELOG

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* remove github text that was causing errors

* feature/vlauncher (#447)

* fix file naming error for iterative workflows

* fixed small bug with new filepath naming

* add VLAUNCHER functionality

* add docs for VLAUNCHER and modify changelog

* re-word docs and fix table format

* add a test for vlauncher

* run fix-style and add a test for vlauncher

* Add the find_vlaunch_var and setup_vlaunch functions.
The numeric value of the shell variables may not be defined until run
time, so replace with variable strings instead of values.
Consolidate the commands into one function.

* Add variable set for (t)csh.

* Run fix-style

* make step settings the defaults and ignore commented lines

* add some additional tests

* remove regex library import

---------

Co-authored-by: Joseph M. Koning <koning1@llnl.gov>

* release/1.11.0 (#448)

* bugfix/skewed-sample-hierarchy (#450)

* add patch for skewed sample hierarchy/additional samples

* update changelog

* catch narrower range of exceptions

* bugfix/lsf-gpu-typo (#453)

* fix typo in batch.py that causes a bug

* change print statements to log statements

* release/1.11.1 (#454)

* Add Pytest Fixtures to Test Suite (#456)

* begin work on integration refactor; create fixtures and initial tests

* update CHANGELOG and run fix-style

* add pytest fixtures and README explaining them

* add tests to demonstrate how to use the fixtures

* move/rename some files and modify integration's README

* add password change to redis.pass file

* fix lint issues

* modify redis pwd for test server to be constant for each test

* fix lint issue only caught on github ci

* Bugfix for WEAVE CI (#457)

* begin work on integration refactor; create fixtures and initial tests

* update CHANGELOG and run fix-style

* add pytest fixtures and README explaining them

* add tests to demonstrate how to use the fixtures

* move/rename some files and modify integration's README

* add password change to redis.pass file

* fix lint issues

* modify redis pwd for test server to be constant for each test

* fix lint issue only caught on github ci

* add fix for merlin server startup

* update CHANGELOG

* bugfix/monitor-shutdown (#452)

* add celery query to see if workers still processing tasks

* fix merlin status when using redis as broker

* fix consumer count bug and run fix-style

* fix linter issues

* update changelog

* update docs for monitor

* remove unused exception I previously added

* first attempt at using pytest fixtures for monitor tests

* (partially) fix launch_workers fixture so it can be used in multiple classes

* fix linter issues and typo on pytest decorator

* update black's python version and fix style issue

* remove print statements from celeryadapter.py

* workers manager is now allowed to be used as a context manager

* add one thing to changelog and remove print statement

* Add the missing restart keyword to the specification docs. (#459)

* docs/conversion-to-mkdocs (#460)

* remove a merge conflict statement that was missed

* add base requirements for mkdocs

* set up configuration for API docs

* start work on porting user guide to mkdocs

* add custom styling and contact page

* begin work on porting tutorial to mkdocs

* add new examples page

* move old sphinx docs to their own folder (*delete later*)

* modify some admonitions to be success

* modify hello examples page and port step 3 of tutorial to mkdocs

* fix typo in hello example

* finish porting step 4 of tutorial to mkdocs

* port part 5 of the tutorial to mkdocs

* copy faq and contributing from old docs

* port step 6 of tutorial to mkdocs

* remove unused prereq

* port step 7 of tutorial to mkdocs

* add more detailed instructions on contributing

* move venv page into installation and add spack instructions too

* add configuration docs

* add content to user guide landing page

* port celery page to mkdocs

* rearrange configuration pages to add in merlin server configuration instructions

* port command line page to mkdocs

* finish new landing page

* change size of merlin logo

* port variables page to mkdocs

* fix broken links to configuration page

* port FAQ to mkdocs

* fix incorrect requirement name

* update CHANGELOG

* attempt to get docs to build through readthedocs

* port docker page to mkdocs

* port contributing guide to mkdocs

* add new 'running studies' page

* add path changes to images

* add a page on how to interpret study output

* add page on the spec file

* remove old sphinx docs that are no longer needed

* added README to docs and updated CHANGELOG

* fix copyright and hello_samples tree

* rearrange images/stylesheets and statements that use them

* add suggestions from Luc and Joe

* add tcsh instructions for venv activation

* add Charle's suggestions for the landing page

* change tcsh mentions to csh

* openfoam tutorial modifications (#463)

* feature/revamped status (#464)

* feature/new-status (#442)

* add backend functionality for merlin status

* add frontend functionality for merlin status

* add tests for merlin status

* run fix-style and remove import of deprecated function

* update CHANGELOG

* add more logging statements, make better use of glob

* run fix-style

* clean up test files a bit

* fix test suite after step_name_map mod

* add avg/std dev run time calculations to status

* modify status tests to accommodate new avg/std dev calculations

* fix linter issues

* fix lint issue and add test for avg/std dev calc

* feature/detailed-status (#451)

* Version/1.11.0 (#449)

* fix default worker bug with all steps

* version bump and requirements fix

* Bugfix/filename-special-vars (#425)

* fix file naming bug

* fix filename bug with variable as study name

* add tests for the file name special vars changes

* modify changelog

* implement Luc's suggestions

* remove replace line

* Create dependabot-changelog-updater.yml

* testing outputs of modifying changelog

* delete dependabot-changelog-updater

* feature/pdf-docs (#427)

* first attempt at adding pdf

* fixing build error

* modify changelog to show docs changes

* fix errors Luc found in the build logs

* trying out removal of latex

* reverting latex changes back

* uncommenting the latex_elements settings

* adding epub to see if latex will build

* adding a latex engine variable to conf

* fix naming error with latex_engine

* attempting to add a logo to the pdf build

* testing an override to the searchtools file

* revert back to not using searchtools override

* update changelog

* bugfix/openfoam_singularity_issues (#426)

* fix openfoam_singularity issues

* update requirements and descriptions for openfoam examples

* bugfix/output-path-substitution (#430)

* fix bug with output_path and variable substitution

* add tests for cli substitutions

* bugfix/scheduler-permission-error (#436)

* Release/1.10.2 (#437)

* bump version to 1.10.2

* bump version in CHANGELOG

* resolve develop to main merge issues (#439)

* fix default worker bug with all steps

* version bump and requirements fix

* dependabot/certifi-requests-pygments (#441)

* Bump certifi from 2022.12.7 to 2023.7.22 in /docs

Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22.
- [Commits](certifi/python-certifi@2022.12.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* add all dependabot changes and update CHANGELOG

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* bugfix/server-pip-redis-conf (#443)

* add *.conf to the MANIFEST file so pip will grab the redis.conf file

* add note explaining how to fix a hanging merlin server start

* modify CHANGELOG

* add second export option to docs and fix typo

* bump to version 1.10.3 (#444)

* bugfix/sphinx-5.3.0-requirement (#446)

* Version/1.10.3 (#445)

* fix default worker bug with all steps

* version bump and requirements fix

* Bugfix/filename-special-vars (#425)

* fix file naming bug

* fix filename bug with variable as study name

* add tests for the file name special vars changes

* modify changelog

* implement Luc's suggestions

* remove replace line

* Create dependabot-changelog-updater.yml

* testing outputs of modifying changelog

* delete dependabot-changelog-updater

* feature/pdf-docs (#427)

* first attempt at adding pdf

* fixing build error

* modify changelog to show docs changes

* fix errors Luc found in the build logs

* trying out removal of latex

* reverting latex changes back

* uncommenting the latex_elements settings

* adding epub to see if latex will build

* adding a latex engine variable to conf

* fix naming error with latex_engine

* attempting to add a logo to the pdf build

* testing an override to the searchtools file

* revert back to not using searchtools override

* update changelog

* bugfix/openfoam_singularity_issues (#426)

* fix openfoam_singularity issues

* update requirements and descriptions for openfoam examples

* bugfix/output-path-substitution (#430)

* fix bug with output_path and variable substitution

* add tests for cli substitutions

* bugfix/scheduler-permission-error (#436)

* Release/1.10.2 (#437)

* bump version to 1.10.2

* bump version in CHANGELOG

* resolve develop to main merge issues (#439)

* fix default worker bug with all steps

* version bump and requirements fix

* dependabot/certifi-requests-pygments (#441)

* Bump certifi from 2022.12.7 to 2023.7.22 in /docs

Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22.
- [Commits](certifi/python-certifi@2022.12.07...2023.07.22)

---
updated-dependencies:
- dependency-name: certifi
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* add all dependabot changes and update CHANGELOG

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* bugfix/server-pip-redis-conf (#443)

* add *.conf to the MANIFEST file so pip will grab the redis.conf file

* add note explaining how to fix a hanging merlin server start

* modify CHANGELOG

* add second export option to docs and fix typo

* bump to version 1.10.3 (#444)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* change hardcoded sphinx requirement

* update CHANGELOG

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* feature/vlauncher (#447)

* fix file naming error for iterative workflows

* fixed small bug with new filepath naming

* add VLAUNCHER functionality

* add docs for VLAUNCHER and modify changelog

* re-word docs and fix table format

* add a test for vlauncher

* run fix-style and add a test for vlauncher

* Add the find_vlaunch_var and setup_vlaunch functions.
The numeric value of the shell variables may not be defined until run
time, so replace with variable strings instead of values.
Consolidate the commands into one function.

* Add variable set for (t)csh.

* Run fix-style

* make step settings the defaults and ignore commented lines

* add some additional tests

* remove regex library import

---------

Co-authored-by: Joseph M. Koning <koning1@llnl.gov>

* release/1.11.0 (#448)

* bugfix/skewed-sample-hierarchy (#450)

* add patch for skewed sample hierarchy/additional samples

* update changelog

* catch narrower range of exceptions

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Joseph M. Koning <koning1@llnl.gov>

* add functionality for the detailed-status command

* add tests for detailed-status

* fix linter issues

* update changelog

* general cleanup and add log statements

* slightly modify two tests

* default status renderer now uses json status format

* remove inaccurate comment

* bugfix/lsf-gpu-typo (#453)

* fix typo in batch.py that causes a bug

* change print statements to log statements

* release/1.11.1 (#454)

* Add Pytest Fixtures to Test Suite (#456)

* begin work on integration refactor; create fixtures and initial tests

* update CHANGELOG and run fix-style

* add pytest fixtures and README explaining them

* add tests to demonstrate how to use the fixtures

* move/rename some files and modify integration's README

* add password change to redis.pass file

* fix lint issues

* modify redis pwd for test server to be constant for each test

* fix lint issue only caught on github ci

* Bugfix for WEAVE CI (#457)

* begin work on integration refactor; create fixtures and initial tests

* update CHANGELOG and run fix-style

* add pytest fixtures and README explaining them

* add tests to demonstrate how to use the fixtures

* move/rename some files and modify integration's README

* add password change to redis.pass file

* fix lint issues

* modify redis pwd for test server to be constant for each test

* fix lint issue only caught on github ci

* add fix for merlin server startup

* update CHANGELOG

* bugfix/monitor-shutdown (#452)

* add celery query to see if workers still processing tasks

* fix merlin status when using redis as broker

* fix consumer count bug and run fix-style

* fix linter issues

* update changelog

* update docs for monitor

* remove unused exception I previously added

* first attempt at using pytest fixtures for monitor tests

* (partially) fix launch_workers fixture so it can be used in multiple classes

* fix linter issues and typo on pytest decorator

* update black's python version and fix style issue

* remove print statements from celeryadapter.py

* workers manager is now allowed to be used as a context manager

* add one thing to changelog and remove print statement

* Add the missing restart keyword to the specification docs. (#459)

* add Jeremy's suggestion to change vars option to output-path

* remove unnecessary lines from CHANGELOG

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Joseph M. Koning <koning1@llnl.gov>
Co-authored-by: Joe Koning <koning@users.noreply.github.com>

* feature/queue info (#461)

* remove a merge conflict statement that was missed

* add queue-info functionality

* add tests for queue-info

* update CHANGELOG

* add try/except for forceful termination of test workers

* change github workflow to use py38 with black instead of py36

* run fix-style with py 3.12 and fix a typo in a test

* add filetype check for dump option

* add banner print statement

* docs/revamped status (#462)

* fix broken image link in README

* add new commands to the command line page

* add monitoring docs layout and complete status cmds page

* fix bug with dumping queue-info to files

* add docs for queue-info

* add documentation for 'query-workers'

* add reference to new query-workers docs and split a paragraph

* fix small bug with --steps option of monitor

* add documentation for monitor command

* update CHANGELOG

* fix dump-csv image for queue-info

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Joseph M. Koning <koning1@llnl.gov>
Co-authored-by: Joe Koning <koning@users.noreply.github.com>

* release/1.12.0 (#465)

* remove a merge conflict statement that was missed

* bump version to 1.12.0

* feature/retry_priority (#468)

* remove a merge conflict statement that was missed

* add a 'pip freeze' call in github workflow to view reqs versions

* add new retry priority as highest task priority

* update CHANGELOG

* add in MID priority

* change default priority to use priority map MID value

* docs/server-cross-node (#470)

* remove a merge conflict statement that was missed

* add a 'pip freeze' call in github workflow to view reqs versions

* rename the merlin server config page

* add instructions for running a cross-node workflow w/ containerized server

* update CHANGELOG

* bugfix/initial-status-issues (#471)

* remove a merge conflict statement that was missed

* add a 'pip freeze' call in github workflow to view reqs versions

* fix bug with dry run status

* set MANPAGER for detailed-status

* fix bug with 1 sample removing the status file

* add support for multiple workers on one step in status files

* update test suite to accommodate changes to workers in status files

* add catch and potential fix for JSONDecodeError

* fix docstring of a test

* update CHANGELOG.md

* run fix style and add Luc's suggestions

* run fix-style with python 3.12

* added additional check for status file while condensing

* add try/except to catch an error for dumping statuses

* release/1.12.1 (#472)

* remove a merge conflict statement that was missed

* add a 'pip freeze' call in github workflow to view reqs versions

* bump version to 1.12.1

* fix a lint issue that somehow slipped through the cracks

* Fix filenames for OpenFoam tutorial (#475)

* bugfix/deep-merge-existing-keys (#476)

* remove a merge conflict statement that was missed

* add a 'pip freeze' call in github workflow to view reqs versions

* remove DeepMergeException and add conflict_handler to dict_deep_merge

* add conflict handler to dict_deep_merge

* fix broken tests for detailed-status

* use caplog fixture rather than IO stream

* add ability to define module-specific fixtures

* add tests for read/write status files and conlict handling

* add caplog explanation to docstrings

* update CHANGELOG

* run fix-style

* add pytest-mock as dependency for test suite

* clean up input check in dict_deep_merge

* Improved Info (#477)

* Add merlin version to banner

* Add python package info to and clean up 'merlin info'

* Add some unit tests

* Force GitHub runner checkout to grab the whole history, fixing CHANGELOG test bug

* Update CHANGELOG to show bugfix to CHANGELOG test

* Target is in source's history (#478)

* New github action test to make sure target has been merged into source

* Fix link to merlin banner image (#479)

* bugfix/status_nested_workspace (#480)

* remove a merge conflict statement that was missed

* have status ignore nested workspaces and modify merge rules

* update CHANGELOG

* fixed issue with escape sequences in ascii art

* apply Luc's suggestion

* add setuptools as a requirement since python 3.12 doesn't have it natively

* modify unit tests for status to use pytest rather than unittest

* update CHANGELOG

* add fixtures for status testing and add nested workflow test

* update CHANGELOG

* bugfix/celery-chord-error (#481)

* remove a merge conflict statement that was missed

* add celery results backend patch to stop ChordErrors

* add MERLIN_RAISE_ERROR return code

* add tests to ensure chord error isn't raised

* add RAISE_ERROR to docs

* update CHANGELOG

* fix lint issues

* up the sleep time on the chord error test

* add new steps to the chord err test spec

* add tree statement to the new test for debugging

* upping sleep time to see if that fixes github action for python 3.7

* change sleep time for new test based on python version

* run fix style

* remove specific sleep time for diff python versions

* release/1.12.2b1 (#482)

* remove a merge conflict statement that was missed

* bump version to 1.12.2b1

* bugfix/flux-nodes (#484)

* remove a merge conflict statement that was missed

* fix flux node allocation issue

* allow for vars to be used with nodes settings of workers/batch

* add tests for var usage with nodes

* update CHANGELOG

* run fix-style

* bugfix/flux-nodes-prior-versions (#487)

* add a version check for flux when getting node count

* update CHANGELOG

* add major version check for flux

* Change Task ID to directory path (#486)

* Modifying task id to include directory

* Adding Several New Unit Tests (#490)

* remove a merge conflict statement that was missed

* add pytest coverage library and add sample_index coverage

* run fix style and add module header

* add tests for encryption modules

* add unit tests for util_sampling

* run fix-style and fix typo

* create directory for context managers and fix issue with an encryption test

* add a context manager for spinning up/down the redis server

* fix issue with path in one test

* rework CONFIG functionality for testing

* refactor config fixture so it doesn't depend on redis server to be started

* split CONFIG fixtures into rabbit and redis configs, run fix-style

* add unit tests for broker.py

* add unit tests for the Config object

* update CHANGELOG

* make CONFIG fixtures more flexible for tests

* add tests for results_backend.py

* fix lint issues for most recent changes

* fix filename issue in setup.cfg and move celeryadapter tests to integration suite

* add ssl filepaths to mysql config object

* add unit tests for configfile.py

* add tests for the utils.py file in config/

* create utilities file and constants file

* move create_dir function to utils.py

* add tests for merlin/examples/generator.py

* run fix-style and update changelog

* add a 'pip freeze' call in github workflow to view reqs versions

* re-delete the old config test files

* fix tests/bugs introduced by merging in develop

* add a unit test file for the dumper module

* begin work on server tests and modular fixtures

* start work on tests for RedisConfig

* add tests for RedisConfig object

* add tests for RedisUsers class

* change server fixtures to use redis config files

* add tests for AppYaml class

* final cleanup of server_utils

* fix lint issues

* parametrize setup examples tests

* sort example output

* ensure directory is changed back on no outdir test

* sort the specs in examples output

* fix lint issues

* start writing tests for server config

* add pytest coverage library and add sample_index coverage

* run fix style and add module header

* add tests for encryption modules

* add unit tests for util_sampling

* run fix-style and fix typo

* create directory for context managers and fix issue with an encryption test

* add a context manager for spinning up/down the redis server

* fix issue with path in one test

* rework CONFIG functionality for testing

* refactor config fixture so it doesn't depend on redis server to be started

* split CONFIG fixtures into rabbit and redis configs, run fix-style

* add unit tests for broker.py

* add unit tests for the Config object

* update CHANGELOG

* make CONFIG fixtures more flexible for tests

* add tests for results_backend.py

* fix lint issues for most recent changes

* fix filename issue in setup.cfg and move celeryadapter tests to integration suite

* add ssl filepaths to mysql config object

* add unit tests for configfile.py

* add tests for the utils.py file in config/

* create utilities file and constants file

* move create_dir function to utils.py

* add tests for merlin/examples/generator.py

* run fix-style and update changelog

* fix tests/bugs introduced by merging in develop

* add a unit test file for the dumper module

* begin work on server tests and modular fixtures

* start work on tests for RedisConfig

* add tests for RedisConfig object

* add tests for RedisUsers class

* change server fixtures to use redis config files

* add tests for AppYaml class

* final cleanup of server_utils

* fix lint issues

* parametrize setup examples tests

* sort example output

* ensure directory is changed back on no outdir test

* sort the specs in examples output

* fix lint issues

* start writing tests for server config

* bake in LC_ALL env variable setting for server cmds

* add tests for parse_redis_output

* fix issue with scope of fixture after rebase

* run fix-style

* split up create_server_config and write tests for it

* add tests for config_merlin_server function

* add tests for pull_server_config

* add tests for pull_server_image

* finish writing tests for server_config.py

* add tests for server_commands.py

* run fix-style

* update README for testing directory

* update the temp_output_directory to include python version

* mock the open.write to try to fix github CI

* ensure config dir is created

* update CHANGELOG

* add print of exception to OSError catch in pull_server_image

* change name of config_file in test that's failing

* update CHANGELOG

* add Ryan and Joe's suggestions

* update tests to use newly named functions

* fix linter issue

* release/1.12.2 (#491)

* update version to 1.12.2

* fix style issue

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Joseph M. Koning <koning1@llnl.gov>
Co-authored-by: Joe Koning <koning@users.noreply.github.com>
Co-authored-by: Jane Herriman <xorJane@users.noreply.github.com>
Co-authored-by: Luc Peterson <peterson76@llnl.gov>
  • Loading branch information
6 people authored Oct 28, 2024
1 parent 9e27798 commit edbfe9b
Show file tree
Hide file tree
Showing 99 changed files with 7,232 additions and 673 deletions.
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,9 @@ flux.out
slurm*.out
docs/build/

# Tox files
# Test files
.tox/*
.coverage

# Jupyter
jupyter/.ipynb_checkpoints
Expand Down
27 changes: 23 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ All notable changes to Merlin will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [1.12.2b1]
## [1.12.2]
### Added
- Conflict handler option to the `dict_deep_merge` function in `utils.py`
- Ability to add module-specific pytest fixtures
Expand All @@ -19,6 +19,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- New step return code `$(MERLIN_RAISE_ERROR)` to force an error to be raised by a task (mainly for testing)
- Added description of this to docs
- New test to ensure a single failed task won't break a workflow
- Several new unit tests for the following subdirectories:
- `merlin/common/`
- `merlin/config/`
- `merlin/examples/`
- `merlin/server/`
- Context managers for the `conftest.py` file to ensure safe spin up and shutdown of fixtures
- `RedisServerManager`: context to help with starting/stopping a redis server for tests
- `CeleryWorkersManager`: context to help with starting/stopping workers for tests
- Ability to copy and print the `Config` object from `merlin/config/__init__.py`
- Equality method to the `ContainerFormatConfig` and `ContainerConfig` objects from `merlin/server/server_util.py`

### Changed
- `merlin info` is cleaner and gives python package info
Expand All @@ -28,6 +38,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Added fixtures for `merlin status` tests that copy the workspace to a temporary directory so you can see exactly what's run in a test
- Batch block and workers now allow for variables to be used in node settings
- Task id is now the path to the directory
- Split the `start_server` and `config_server` functions of `merlin/server/server_commands.py` into multiple functions to make testing easier
- Split the `create_server_config` function of `merlin/server/server_config.py` into two functions to make testing easier
- Combined `set_snapshot_seconds` and `set_snapshot_changes` methods of `RedisConfig` into one method `set_snapshot`

### Fixed
- Bugfix for output of `merlin example openfoam_wf_singularity`
Expand Down Expand Up @@ -95,8 +108,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- this required adding a decent amount of test files to help with the tests; these can be found under the tests/unit/study/status_test_files directory
- Pytest fixtures in the `conftest.py` file of the integration test suite
- NOTE: an export command `export LC_ALL='C'` had to be added to fix a bug in the WEAVE CI. This can be removed when we resolve this issue for the `merlin server` command
- Tests for the `celeryadapter.py` module
- New CeleryTestWorkersManager context to help with starting/stopping workers for tests
- Coverage to the test suite. This includes adding tests for:
- `merlin/common/`
- `merlin/config/`
- `merlin/examples/`
- `celeryadapter.py`
- Context managers for the `conftest.py` file to ensure safe spin up and shutdown of fixtures
- `RedisServerManager`: context to help with starting/stopping a redis server for tests
- `CeleryWorkersManager`: context to help with starting/stopping workers for tests
- Ability to copy and print the `Config` object from `merlin/config/__init__.py`

### Changed
- Reformatted the entire `merlin status` command
Expand Down Expand Up @@ -132,7 +152,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- The `merlin monitor` command will now keep an allocation up if the queues are empty and workers are still processing tasks
- Add the restart keyword to the specification docs
- Cyclical imports and config imports that could easily cause ci issues

## [1.11.1]
### Fixed
- Typo in `batch.py` that caused lsf launches to fail (`ALL_SGPUS` changed to `ALL_GPUS`)
Expand Down
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down
2 changes: 1 addition & 1 deletion config.mk
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ else
endif

VER?=1.0.0
VSTRING=[0-9]\+\.[0-9]\+\.[0-9]\+
VSTRING=[0-9]\+\.[0-9]\+\.[0-9]\+\(b[0-9]\+\)\?
YEAR=20[0-9][0-9]
NEW_YEAR?=2023
CHANGELOG_VSTRING="## \[$(VSTRING)\]"
Expand Down
4 changes: 2 additions & 2 deletions merlin/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down Expand Up @@ -38,7 +38,7 @@
import sys


__version__ = "1.12.2b1"
__version__ = "1.12.2"
VERSION = __version__
PATH_TO_PROJ = os.path.join(os.path.dirname(__file__), "")

Expand Down
2 changes: 1 addition & 1 deletion merlin/ascii_art.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down
2 changes: 1 addition & 1 deletion merlin/celery.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down
2 changes: 1 addition & 1 deletion merlin/common/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down
2 changes: 1 addition & 1 deletion merlin/common/abstracts/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down
2 changes: 1 addition & 1 deletion merlin/common/abstracts/enums/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down
2 changes: 1 addition & 1 deletion merlin/common/dumper.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1
# This file is part of Merlin, Version: 1.12.2
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down
2 changes: 1 addition & 1 deletion merlin/common/openfilelist.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down
2 changes: 1 addition & 1 deletion merlin/common/opennpylib.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down
4 changes: 2 additions & 2 deletions merlin/common/sample_index.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down Expand Up @@ -225,8 +225,8 @@ def __setitem__(self, full_address, sub_tree):

# Replace if we already have something at this address.
if delete_me is not None:
self.children.__delitem__(full_address)
SampleIndex.check_valid_addresses_for_insertion(full_address, sub_tree)
self.children.__delitem__(full_address)
self.children[full_address] = sub_tree
return
raise KeyError
Expand Down
2 changes: 1 addition & 1 deletion merlin/common/sample_index_factory.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down
2 changes: 1 addition & 1 deletion merlin/common/security/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down
11 changes: 5 additions & 6 deletions merlin/common/security/encrypt.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down Expand Up @@ -52,11 +52,10 @@ def _get_key_path():
except AttributeError:
key_filepath = "~/.merlin/encrypt_data_key"

try:
key_filepath = os.path.abspath(os.path.expanduser(key_filepath))
except KeyError as e:
raise ValueError("Error! No password provided for RabbitMQ") from e
return key_filepath
if key_filepath is None:
raise ValueError("Error! No password provided for RabbitMQ")

return os.path.abspath(os.path.expanduser(key_filepath))


def _gen_key(key_path):
Expand Down
2 changes: 1 addition & 1 deletion merlin/common/security/encrypt_backend_traffic.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down
2 changes: 1 addition & 1 deletion merlin/common/tasks.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down
3 changes: 2 additions & 1 deletion merlin/common/util_sampling.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down Expand Up @@ -35,6 +35,7 @@
import numpy as np


# TODO should we move this to merlin-spellbook?
def scale_samples(samples_norm, limits, limits_norm=(0, 1), do_log=False):
"""Scale samples to new limits, either log10 or linearly.
Expand Down
33 changes: 31 additions & 2 deletions merlin/config/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand All @@ -31,7 +31,7 @@
"""
Used to store the application configuration.
"""

from copy import copy
from types import SimpleNamespace
from typing import Dict, List, Optional

Expand All @@ -56,6 +56,35 @@ def __init__(self, app_dict):
self.results_backend: Optional[SimpleNamespace]
self.load_app_into_namespaces(app_dict)

def __copy__(self):
"""
A magic method to allow this class to be copied with copy(instance_of_Config).
"""
cls = self.__class__
result = cls.__new__(cls)
copied_attrs = {
"celery": copy(self.__dict__["celery"]),
"broker": copy(self.__dict__["broker"]),
"results_backend": copy(self.__dict__["results_backend"]),
}
result.__dict__.update(copied_attrs)
return result

def __str__(self):
"""
A magic method so we can print the CONFIG class.
"""
formatted_str = "config:"
attrs = {"celery": self.celery, "broker": self.broker, "results_backend": self.results_backend}
for name, attr in attrs.items():
if attr is not None:
items = (f" {k}: {v!r}" for k, v in attr.__dict__.items())
joined_items = "\n".join(items)
formatted_str += f"\n {name}:\n{joined_items}"
else:
formatted_str += f"\n {name}:\n None"
return formatted_str

def load_app_into_namespaces(self, app_dict: Dict) -> None:
"""
Makes the application dictionary into a namespace, sets the attributes of the Config from the namespace values.
Expand Down
16 changes: 5 additions & 11 deletions merlin/config/broker.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down Expand Up @@ -85,13 +85,13 @@ def get_rabbit_connection(include_password, conn="amqps"):
password_filepath = CONFIG.broker.password
LOG.debug(f"Broker: password filepath = {password_filepath}")
password_filepath = os.path.abspath(expanduser(password_filepath))
except KeyError as e: # pylint: disable=C0103
raise ValueError("Broker: No password provided for RabbitMQ") from e
except (AttributeError, KeyError) as exc:
raise ValueError("Broker: No password provided for RabbitMQ") from exc

try:
password = read_file(password_filepath)
except IOError as e: # pylint: disable=C0103
raise ValueError(f"Broker: RabbitMQ password file {password_filepath} does not exist") from e
except IOError as exc:
raise ValueError(f"Broker: RabbitMQ password file {password_filepath} does not exist") from exc

try:
port = CONFIG.broker.port
Expand Down Expand Up @@ -205,12 +205,6 @@ def get_connection_string(include_password=True):
except AttributeError:
broker = ""

try:
config_path = CONFIG.celery.certs
config_path = os.path.abspath(os.path.expanduser(config_path))
except AttributeError:
config_path = None

if broker not in BROKERS:
raise ValueError(f"Error: {broker} is not a supported broker.")
return _sort_valid_broker(broker, include_password)
Expand Down
2 changes: 1 addition & 1 deletion merlin/config/celeryconfig.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down
2 changes: 1 addition & 1 deletion merlin/config/configfile.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
#
# LLNL-CODE-797170
# All rights reserved.
# This file is part of Merlin, Version: 1.12.2b1.
# This file is part of Merlin, Version: 1.12.2.
#
# For details, see https://github.com/LLNL/merlin.
#
Expand Down
Loading

0 comments on commit edbfe9b

Please sign in to comment.