1.7.2 (#264)

* locked maestro version

* incremented to 1.5.3

* Update CHANGELOG.md

* Feature/koning/vhost no leading slash (#217)

* The broker name can now be amqps (with ssl) or amqp (without ssl).
The default rabbitmq vhost is now <user> instead of /<user>.

* Undo fix typos.

* Fix spacing.

* Add amqps conn for self doc.

* Add inital running workers check to monitor. This should eliminate any (#218)

race conditions.

* updated timeout (#223)

* updated default task timeout

* Added to CHANGELOG

* Feature/koning/create encrypt key init (#219)

* Add init_key to create the encryption key when merlin config is run.

* Move import to point of use for encryption.

* Feature/koning/monitor upgrade (#221)

* Fix sleep variable.

* Change another sleep variable.

* Change brackets.

* Fix clarity for jobs but no consumers condition.

* Fix comment.

* Update comment.

* Move check_merlin_status to router.

* Run fix-style.

* Bugfix/ben/expand name (#220)

* working on bugfix

* achieved desired behavior

* made expansion anf filewriting conditional

* fixed style

* updated CHANGELOG

* fixed samples and restart

* fixed style

* used variable

* added name expansion test

* temporarily commented out one test

* fixed style

* added debug block

* changes

* corrected bug

* updated example; scripts are now copied into merlin_info

* Feature/ben/expose visibility timeout (#224)

* exposed visibility timeout in config file

* added visibility timeouts to default config files

* fixed style

* added seconds to clarify name

* fixed attribute name

* locked celery version at 4.4.2

* updated CHANGELOG

* v1.5.4

* adjusted CHANGELOG

* adjusted CHANGELOG

* Feature/koning/info conn timeout (#226)

* Add a timeout check for the merlin info kombu connection test. The redis
server will not comply with the connection_timeout config.

* Update docs.

* Load default timeout if non in config file (#229)

* 1.6.1

* bugfix (#232)

* bugfix

* removed repetition in logic

* Log sample generation to merlin_info (#234)

* sample generation is now logged to merlin_info

* Added to CHANGELOG

* fixed style

* Maestro 1.1.8+ update (#233)

* working on adding maestro's schema validation to merlin

* updated schemas

* adding merlin_schema to spec logic

* added debug prints

* added print

* fixing vestigial Maestro calls

* adjusted merlin_schemas

* fixed json file

* validation progress

* added from_dict replacement logic

* adjusted maestrowf version req

* fixed string

* schema improvements

* merlinsection schema updates

* merlin section error parsing appears to work

* updated CHANGELOG

* fixed merge error

* Added walltime to json schema

* Add the bank and walltime keywords to the batch slurm launch, these (#236)

will not alter the lsf launch.

* Bugfix for new celery versions (#231)

* Change the expand_tasks_with_samples task to use an immutable signature,
no return code is neccessary.

* Change only expand_tasks_with_samples to si()

* Increase celery version requirement to the latest (4.4.5).

* Update CHANGELOG.

Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com>

* hidden test specs (#237)

* added tests to run_tests.py

* updated merlinsection schema

* added full_spec test

* added to CHANGELOG

* merlinspec change

* use regex to find variable tokens (#239)

* now using regex to find variable tokens

* added shell ref function

* added to CHANGELOG

* Revert maestro 1.1.8 support (#241)

* reverted support of Maestro 1.1.8

* removed tests that use maestro validation

* fixed style

* fixed cli tests, reactivated 3 old ones (#242)

* fixed cli tests, reactivated 3 old ones

* added to CHANGELOG

* More provenance specs + logic improvements (#240)

* added orig.yaml and expanded.yaml

* added partial provenance spec

* fixed style

* added to CHANGELOG

* removed comments

* made env replacement more precise

* fixed restart bug

* changed provenance file names

* logic update

* fixed style

* expansion now working

* added cli test logic to test all 3 provenance file types

* updated CHANGELOG

* expansion sans io (#243)

* reverted support of Maestro 1.1.8

* removed tests that use maestro validation

* fixed style

* added sections property to spec

* reducing number of file writes and reads

* provenance specs are correct

* fixed style

* removed unused imports

* cleaning up logic

* fixed style

* final logic improvements

* removed unused property

* updated CHANGELOG

* v1.6.2

* debugging study

* added provenance tests

* fixed path in test

* added check for workflow name expanding to invalid filename

* Docs/ben/level max dirs (#250)

* added to documentation

* added to CHANGELOG

* added default

* updated docs to reflect recent provenance spec update (#251)

* Feature/ben/pgen (#248)

* added basic pgen support

* added to CHANGELOG

* pgen, pargs, and env appear to work

* added to CHANGELOG

* fixed typo

* added pgen cli test

* added faq entry on pgen

* Update Dockerfile (#254)

* Update CHANGELOG.md

* pretty yaml (#252)

* starting

* building pretty_dump function

* updates

* fixed style

* tweaks

* valid yaml

* added yaml_sections

* added to CHANGELOG

* removed old yaml representer code

* added cli test for equality of provenance specs

* fixed pgen test

* disabled test while not working

* fixed yaml lists for source, paths, git

* tweaks / fixes

* env variable bugfix (#247)

* removed expansion of env variables in provenance specs

* added to CHANGELOG

* corrected unit tests

* added todos

* working on expand_env_var function

* fixed type bug

* removed comments, prints, fixed style

* added to CHANGELOG

* tweaks

* fixed output path

* allowed study name to use env vars

* fixed style

* added to CHANGELOG, added support for 'restart'

* Update CHANGELOG.md

* added docs strings

* added to docs

* updated comment

* fixed CHANGELOG

* removed debug print

* flux examples fix (#253)

* Update flux example workflows to use new kvs query methods.

* Update CHANGELOG with fix info.

* Add coker fix comment.

* Update CHANGELOG.md

* added gitlab ci file

* improved gitlab ci setup

* renamed file

* hotfix to allow distributed cli tests to run

* redis update (#255)

* Update redis to new version with TLS support.

* Add changelog.

* Bugfix for rediss cert_reqs keyword. (#256)

* expose celery (#245)

* added default app.yaml, still in progress...

* added celery defaults

* Update app.yaml

* config override appears to be working

* corrected value name

* shifted some properties to be optional defaults

* removed unneeded file

* improvements

* fixed style

* fixed style

* hid 'no override' message

* tweaked var names

* added 2 tests for celery app (#246)

* fixed test

* fixed workers bug

* fixed unit tests for running offline

* removed 2 unit tests

* fixed CHANGELOG

* added info print of all celery configurations being overridden

* 1.7.0

* 1.7.0

* copy samplesfile to merlin_info (#259)

* copy samplesfile to merlin_info

* added to CHANGELOG

* Bugfix/koning/process exceptions (#260)

* Add bugfix log.

* Add an exception catcher for the multiprocessing Process class.

* fixed style

* 1.7.1

* hotfix

* fix

* 1.7.2

Co-authored-by: Joe Koning <koning@users.noreply.github.com>
Co-authored-by: Luc Peterson <peterson76@llnl.gov>
Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com>

Makefile

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #
@@ -38,7 +38,7 @@
 import sys
 
 
-__version__ = "1.7.1"
+__version__ = "1.7.2"
 VERSION = __version__
 PATH_TO_PROJ = os.path.join(os.path.dirname(__file__), "")
 

merlin/ascii_art.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/celery.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/abstracts/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/abstracts/enums/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/openfilelist.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/opennpylib.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/sample_index.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/sample_index_factory.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/security/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/security/encrypt.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/security/encrypt_backend_traffic.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/tasks.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/common/util_sampling.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/config/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/config/broker.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/config/configfile.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/config/results_backend.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/data/celery/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/display.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/examples/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/examples/examples.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/examples/generator.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/exceptions/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/log_formatter.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/main.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/merlin_templates.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/router.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/spec/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/spec/all_keys.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/spec/defaults.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/spec/expansion.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #
@@ -155,7 +155,11 @@ def determine_user_variables(*user_var_dicts):
         {'TARGET': 'target_dir',
         'PATH': '$(SPECROOT)/target_dir'}
     """
-    all_var_dicts = dict(ChainMap(*user_var_dicts))
+    # TODO move this logic to specification.py?
+    try:
+        all_var_dicts = dict(ChainMap(*user_var_dicts))
+    except TypeError:
+        all_var_dicts = {}
     determined_results = {}
     for key, val in all_var_dicts.items():
         if key in RESERVED:

merlin/spec/specification.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/study/__init__.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/study/batch.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/study/celeryadapter.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/study/dag.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/study/script_adapter.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

merlin/study/step.py

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.7.1.
+# This file is part of Merlin, Version: 1.7.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #