Bugfix for new celery versions #231
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Develop
no return code is neccessary.
ben-bay
approved these changes
Jun 11, 2020
|
I think we will need to test this on some real work workflows before merging. |
|
What kind of workflows do you have in mind? I've run this successfully on feature_demo and null_spec. |
|
Real ones like the multi-physics simulation workflows. |
Ok, I don't have one set up so this may be a job for @lucpeterson |
|
looks good to me: was able to run hydra via flux on lassen with celery 4.4.5 both distributed and locally (with the result.get() error no longer there) |
lucpeterson
approved these changes
Jun 18, 2020
|
OK, good the specific issue that may arise is that this task would not propagate its result to the other tasks due to the immutable flag setting. But since this task creates other tasks that do have the result propagated , it is probably not an issue. |
|
so what's the right test? is this put to bed? is it to run like 1000 hello worlds? i tested only 10, which is below the threshold for the directory |
|
That is the question, what is the correct test? I think that if you were not seeing a chord completing correctly, some tasks being executed before others, not waiting for all tasks to complete before executing a task. |
ben-bay
added a commit
that referenced
this pull request
Jul 6, 2020
* locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov>
ben-bay
added a commit
that referenced
this pull request
Jul 23, 2020
* locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com>
ben-bay
added a commit
that referenced
this pull request
Aug 3, 2020
* locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com>
ben-bay
added a commit
that referenced
this pull request
Aug 4, 2020
* locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com>
ben-bay
added a commit
that referenced
this pull request
Aug 4, 2020
* locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 * Update CHANGELOG.md * Update CHANGELOG.md Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com>
ben-bay
added a commit
that referenced
this pull request
Aug 4, 2020
* locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 * Update CHANGELOG.md * Update CHANGELOG.md * 1.7.3 * 1.7.3 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com>
lucpeterson
added a commit
to lucpeterson/merlin
that referenced
this pull request
Aug 19, 2020
* 1.7.0 (LLNL#257) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (LLNL#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (LLNL#218) race conditions. * updated timeout (LLNL#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (LLNL#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (LLNL#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (LLNL#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (LLNL#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (LLNL#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (LLNL#229) * 1.6.1 * bugfix (LLNL#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (LLNL#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (LLNL#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (LLNL#236) will not alter the lsf launch. * Bugfix for new celery versions (LLNL#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (LLNL#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (LLNL#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (LLNL#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (LLNL#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (LLNL#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (LLNL#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (LLNL#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (LLNL#251) * Feature/ben/pgen (LLNL#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (LLNL#254) * Update CHANGELOG.md * pretty yaml (LLNL#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (LLNL#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (LLNL#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (LLNL#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (LLNL#256) * expose celery (LLNL#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (LLNL#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.1 hotfix (LLNL#261) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (LLNL#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (LLNL#218) race conditions. * updated timeout (LLNL#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (LLNL#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (LLNL#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (LLNL#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (LLNL#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (LLNL#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (LLNL#229) * 1.6.1 * bugfix (LLNL#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (LLNL#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (LLNL#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (LLNL#236) will not alter the lsf launch. * Bugfix for new celery versions (LLNL#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (LLNL#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (LLNL#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (LLNL#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (LLNL#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (LLNL#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (LLNL#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (LLNL#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (LLNL#251) * Feature/ben/pgen (LLNL#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (LLNL#254) * Update CHANGELOG.md * pretty yaml (LLNL#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (LLNL#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (LLNL#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (LLNL#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (LLNL#256) * expose celery (LLNL#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (LLNL#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (LLNL#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (LLNL#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.2 (LLNL#264) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (LLNL#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (LLNL#218) race conditions. * updated timeout (LLNL#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (LLNL#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (LLNL#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (LLNL#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (LLNL#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (LLNL#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (LLNL#229) * 1.6.1 * bugfix (LLNL#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (LLNL#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (LLNL#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (LLNL#236) will not alter the lsf launch. * Bugfix for new celery versions (LLNL#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (LLNL#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (LLNL#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (LLNL#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (LLNL#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (LLNL#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (LLNL#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (LLNL#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (LLNL#251) * Feature/ben/pgen (LLNL#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (LLNL#254) * Update CHANGELOG.md * pretty yaml (LLNL#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (LLNL#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (LLNL#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (LLNL#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (LLNL#256) * expose celery (LLNL#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (LLNL#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (LLNL#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (LLNL#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.2 (LLNL#265) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (LLNL#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (LLNL#218) race conditions. * updated timeout (LLNL#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (LLNL#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (LLNL#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (LLNL#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (LLNL#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (LLNL#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (LLNL#229) * 1.6.1 * bugfix (LLNL#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (LLNL#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (LLNL#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (LLNL#236) will not alter the lsf launch. * Bugfix for new celery versions (LLNL#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (LLNL#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (LLNL#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (LLNL#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (LLNL#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (LLNL#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (LLNL#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (LLNL#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (LLNL#251) * Feature/ben/pgen (LLNL#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (LLNL#254) * Update CHANGELOG.md * pretty yaml (LLNL#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (LLNL#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (LLNL#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (LLNL#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (LLNL#256) * expose celery (LLNL#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (LLNL#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (LLNL#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (LLNL#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 * Update CHANGELOG.md * Update CHANGELOG.md Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.3 (LLNL#266) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (LLNL#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (LLNL#218) race conditions. * updated timeout (LLNL#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (LLNL#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (LLNL#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (LLNL#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (LLNL#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (LLNL#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (LLNL#229) * 1.6.1 * bugfix (LLNL#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (LLNL#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (LLNL#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (LLNL#236) will not alter the lsf launch. * Bugfix for new celery versions (LLNL#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (LLNL#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (LLNL#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (LLNL#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (LLNL#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (LLNL#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (LLNL#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (LLNL#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (LLNL#251) * Feature/ben/pgen (LLNL#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (LLNL#254) * Update CHANGELOG.md * pretty yaml (LLNL#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (LLNL#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (LLNL#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (LLNL#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (LLNL#256) * expose celery (LLNL#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (LLNL#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (LLNL#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (LLNL#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 * Update CHANGELOG.md * Update CHANGELOG.md * 1.7.3 * 1.7.3 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com>
ben-bay
added a commit
that referenced
this pull request
Aug 26, 2020
* locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 * Update CHANGELOG.md * Update CHANGELOG.md * 1.7.3 * 1.7.3 * Refactor/ben/spec logic (#267) * improved lots of logic * added minimum format test * added to CHANGELOG * added to CHANGELOG * added 3 cli tests * added shortcut var * sorted imports * removed erroneous note from doc * null spec updates (for paper) (#270) * made samples generation actually null; now uses touch, once only * improving samples * updated job launcher * modifications * added echo * added echo * added times * fixed slurm command logic * script improvements to accomodate multiple node logs * read output script updates * read output script fixes * updated times * updated times * script updates * script changes * launcher script improvement: added run id to queue names * tweak * added account logic for quartz and pascal * path * fixed bug that mixed queues at w=32 and w=64 * bugfix * tweak * added new launcher specs * added 24 hour cap * fixed sample vals * script update * launcher changes * updated merlin monitor call * updates * fixed style * Feature/koning/maestro remove submit message (#269) * Add a MerlinStepRecord to remove Maestro message when tasks resubmit. * Add the dependent objects for MerlinStepRecord. * Add CHANGELOG entry for bugfix. * Revert quick fix changes. * fixed style * preparing run * script updates * updated launcher script: automated node list vals * removed predefined node lists * Update CHANGELOG.md * Update CHANGELOG.md * Feature/ben/null chain (#272) * added null chain scripts * spec improvements * removed sim spec, as null spec can now be customized to wait longer * added to merlin monitor * specified copied file name * added time to null_chain * max sample ramp * added to CHANGELOG * fixed style * 1.7.4 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com>
AlexanderWinterLLNL
added a commit
that referenced
this pull request
Jul 6, 2021
* Squash and merge develop to Main * Master (#2) * 1.7.0 (#257) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.1 hotfix (#261) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.2 (#264) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.2 (#265) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 * Update CHANGELOG.md * Update CHANGELOG.md Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.3 (#266) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 * Update CHANGELOG.md * Update CHANGELOG.md * 1.7.3 * 1.7.3 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * Revert "Master (#2)" This reverts commit 1d7495b. * [Flake8 src refactor] display.py and broker.py refactored. * [Standardize CI linting and Makefile] Updated the Makefile and CI to hold one standard for linting from setup.cfg. * Add ability to enter step walltime in HH:MM:SS format. With HH and MM optional. Previously, script adapters had assumed a format that depends on the batch type. Now it'll get converted internally to the right format. * Updated CHANGELOG and docs * Add testing for integers too * An explicit conversion test. Making mypy and black happy. * Move to 1.8.0 * Explictly wrap queue name string in quotes to make it more robust (#320) * Explictly wrap queue name string in quotes to make it more robust * Move CHANGELOG note to v1.8.0 * Update CHANGELOG.md * Update test_time_formats.py Utilized typing for all variables, utilizing PyTest parametrized marks to allow testing to flow regardless of individual fail states of test data. * Update test_time_formats.py Added test to verify some exception handling, compressed tests to not have as much indirection. * Typo in variable name change * Fix style before merging, update to Makefile so fix-style upgrades all dev packages before style check. (#321) Co-authored-by: Alexander Cameron Winter <winter27@quartz1148.llnl.gov> * Revert "Merge branch 'main' into develop" using -m option. This reverts commit a9276b8, reversing changes made to 63a094b. Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> Co-authored-by: Alexander Cameron Winter <winter27@quartz1148.llnl.gov> Co-authored-by: Alexander Winter <80929291+AlexanderWinterLLNL@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #203
Closes #225