Version 1.10.2 (#438)

* fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix
LLNL · Aug 7, 2023 · 261e035 · 261e035
1 parent ab739c6
commit 261e035
Show file tree

Hide file tree

Showing 72 changed files with 249 additions and 114 deletions.
diff --git a/.gitignore b/.gitignore
@@ -69,3 +69,4 @@ jupyter/testDistributedSamples.py
 dist/
 build/
 .DS_Store
+.vscode/
diff --git a/.readthedocs.yaml b/.readthedocs.yaml
@@ -11,3 +11,5 @@ sphinx:
 python:
   install:
     - requirements: docs/requirements.txt
+
+formats: [pdf]
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,25 @@ All notable changes to Merlin will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.10.2]
+### Fixed
+- A bug where the .orig, .partial, and .expanded file names were using the study name rather than the original file name
+- A bug where the openfoam_wf_singularity example was not being found
+- Some build warnings in the docs (unknown targets, duplicate targets, title underlines too short, etc.)
+- A bug where when the output path contained a variable that was overridden, the overridden variable was not changed in the output_path
+- A bug where permission denied errors happened when checking for system scheduler
+
+### Added
+- Tests for ensuring `$(MERLIN_SPEC_ORIGINAL_TEMPLATE)`, `$(MERLIN_SPEC_ARCHIVED_COPY)`, and `$(MERLIN_SPEC_EXECUTED_RUN)` are stored correctly
+- A pdf download format for the docs
+- Tests for cli substitutions
+
+### Changed
+- The ProvenanceYAMLFileHasRegex condition for integration tests now saves the study name and spec file name as attributes instead of just the study name
+  - This lead to minor changes in 3 tests ("local override feature demo", "local pgen feature demo", and "remote feature demo") with what we pass to this specific condition
+- Updated scikit-learn requirement for the openfoam_wf_singularity example
+- Uncommented Latex support in the docs configuration to get pdf builds working
+
 ## [1.10.1]
 ### Fixed
 - A bug where assigning a worker all steps also assigned steps to the default worker

diff --git a/Makefile b/Makefile
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -66,7 +66,7 @@
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
-language = None
+language = "en"
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
@@ -122,21 +122,24 @@
 
 # -- Options for LaTeX output ------------------------------------------------
 
+latex_engine = "pdflatex"
 latex_elements = {
     # The paper size ('letterpaper' or 'a4paper').
     #
-    # 'papersize': 'letterpaper',
+    'papersize': 'letterpaper',
     # The font size ('10pt', '11pt' or '12pt').
     #
-    # 'pointsize': '10pt',
+    'pointsize': '10pt',
     # Additional stuff for the LaTeX preamble.
     #
-    # 'preamble': '',
+    'preamble': '',
     # Latex figure (float) alignment
     #
-    # 'figure_align': 'htbp',
+    'figure_align': 'htbp',
 }
 
+latex_logo = "../images/merlin.png"
+
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).

diff --git a/docs/source/faq.rst b/docs/source/faq.rst
@@ -383,7 +383,7 @@ in the ``launch_args`` variable in the batch section.
 What is PBS?
 ~~~~~~~~~~~~
 Another job scheduler. See `Portable Batch System
-https://en.wikipedia.org/wiki/Portable_Batch_System`_
+<https://en.wikipedia.org/wiki/Portable_Batch_System>`_
 .
 This functionality is only available to launch a flux scheduler.
 

diff --git a/docs/source/merlin_developer.rst b/docs/source/merlin_developer.rst
@@ -164,8 +164,7 @@ properties labeled ``name`` and ``population`` that are both required, it would
     }
 
 Here, ``name`` can only be a string but ``population`` can be both a string and an integer. 
-For help with json schema formatting, check out the `step-by-step getting started guide 
-<https://json-schema.org/learn/getting-started/step-by-step.html>`_.
+For help with json schema formatting, check out the `step-by-step getting started guide`_.
 
 The next step is to enable this block in the schema validation process. To do this we need to:
 

diff --git a/docs/source/merlin_variables.rst b/docs/source/merlin_variables.rst
@@ -161,7 +161,7 @@ Reserved variables
             $(MERLIN_INFO)/*.expanded.yaml
 
 The ``LAUNCHER`` Variable
-+++++++++++++++++++++
++++++++++++++++++++++++++
 
 ``$(LAUNCHER)`` is a special case of a reserved variable since it's value *can* be changed.
 It serves as an abstraction to launch a job with parallel schedulers like :ref:`slurm<slurm>`,

diff --git a/docs/source/modules/installation/installation.rst b/docs/source/modules/installation/installation.rst
@@ -229,7 +229,7 @@ If everything is set up correctly, you should see:
 
 
 (OPTIONAL) Docker Advanced Installation
-----------------------------
+---------------------------------------
 
 RabbitMQ Server
 +++++++++++++++

diff --git a/merlin/__init__.py b/merlin/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #
@@ -38,7 +38,7 @@
 import sys
 
 
-__version__ = "1.10.1"
+__version__ = "1.10.2"
 VERSION = __version__
 PATH_TO_PROJ = os.path.join(os.path.dirname(__file__), "")
 

diff --git a/merlin/ascii_art.py b/merlin/ascii_art.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/celery.py b/merlin/celery.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/__init__.py b/merlin/common/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/abstracts/__init__.py b/merlin/common/abstracts/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/abstracts/enums/__init__.py b/merlin/common/abstracts/enums/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/openfilelist.py b/merlin/common/openfilelist.py
@@ -8,7 +8,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/opennpylib.py b/merlin/common/opennpylib.py
@@ -8,7 +8,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/sample_index.py b/merlin/common/sample_index.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/sample_index_factory.py b/merlin/common/sample_index_factory.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/security/__init__.py b/merlin/common/security/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/security/encrypt.py b/merlin/common/security/encrypt.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/security/encrypt_backend_traffic.py b/merlin/common/security/encrypt_backend_traffic.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/tasks.py b/merlin/common/tasks.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/common/util_sampling.py b/merlin/common/util_sampling.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/config/__init__.py b/merlin/config/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/config/broker.py b/merlin/config/broker.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/config/celeryconfig.py b/merlin/config/celeryconfig.py
@@ -10,7 +10,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/config/configfile.py b/merlin/config/configfile.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/config/results_backend.py b/merlin/config/results_backend.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/config/utils.py b/merlin/config/utils.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/data/celery/__init__.py b/merlin/data/celery/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/display.py b/merlin/display.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/examples/__init__.py b/merlin/examples/__init__.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/examples/examples.py b/merlin/examples/examples.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #

diff --git a/merlin/examples/generator.py b/merlin/examples/generator.py
@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.10.1.
+# This file is part of Merlin, Version: 1.10.2.
 #
 # For details, see https://github.com/LLNL/merlin.
 #
@@ -83,6 +83,8 @@ def list_examples():
         directory = os.path.join(os.path.join(EXAMPLES_DIR, example_dir), "")
         specs = glob.glob(directory + "*.yaml")
         for spec in specs:
+            if "template" in spec:
+                continue
             with open(spec) as f:  # pylint: disable=C0103
                 try:
                     spec_metadata = yaml.safe_load(f)["description"]

diff --git a/merlin/examples/workflows/openfoam_wf/openfoam_wf.yaml b/merlin/examples/workflows/openfoam_wf/openfoam_wf.yaml
@@ -3,6 +3,7 @@ description:
     description: |
       A parameter study that includes initializing, running,
       post-processing, collecting, learning and visualizing OpenFOAM runs
+      using docker.
 
 
 env:

diff --git a/merlin/examples/workflows/openfoam_wf/openfoam_wf_template.yaml b/merlin/examples/workflows/openfoam_wf/openfoam_wf_template.yaml
@@ -3,6 +3,7 @@ description:
     description: |
       A parameter study that includes initializing, running,
       post-processing, collecting, learning and visualizing OpenFOAM runs
+      using docker.
 
 
 env:

diff --git a/merlin/examples/workflows/openfoam_wf/requirements.txt b/merlin/examples/workflows/openfoam_wf/requirements.txt
@@ -1,3 +1,3 @@
 Ofpp==0.11
-scikit-learn==0.21.3
+scikit-learn>=1.0.2
 matplotlib==3.1.1
-Original file line number
+Diff line change
@@ Expand Up / @@ -69,3 +69,4 @@ jupyter/testDistributedSamples.py @@
     dist/
     build/
     .DS_Store
+    .vscode/