Skip to content

Commit

Permalink
Release of v1.1.4 (#193)
Browse files Browse the repository at this point in the history
* Addition of user enabled workspace hashing (#145)

* Addition of hashing to Study parameterization.

* Addition of the hashws option to argparse.

* Addition of a warning note for users who use labels in steps.

* Update setup.py to 1.1.4dev

* More generalized FluxScriptAdapter (#149)

* Addition of a more general flux ScriptAdapter.

* Addition of some casting from int to str

* Corrected "gpus" to "ngpus"

* Rework jobspec construction to make a valid jobspec.

* Check for empty value for cores per task.

* README tweak to update quickstart link. (#139)

* typos. fixes #141 (#154)

* Correction of flake8 style errors [new version of flake8].

* Update to setup.py to reflect dev version 1.0

* Correction to safe pathing for missed cases and make_safe_path enhancements. (#157)

* Made pickle and log path string safe for pathing.

* Tweaks to make_safe_path to include a base path.

* Updates to make_safe_path usage

* Correction to not modify the iterator copy.

* Correction to fix the format of output status time to avoid a comma that breaks printing. (#160)

* Addition of a utility function for formatting times to H:M:S

* _StepRecord time methods now call the new utility function.

* Tweaks to add days to the format to avoid 3 digit hours.

* Tweak to formatting.

* Made the day format more parsable.

* Removal of _stage_linear since it is now not needed. (#156)

* Removal of _stage_linear since it is now not needed.

* Addition of linear LULESH samples.

* Update the dev to 1.1.

* Addition of pargs for passing parameters to custom parameter generation (#152)

* Addition of a utility method to create a dictionary from a list of key-value pairs.

* Addition of the pargs interface for passing parameters to custom parameter generation.

* Addition of a Monte Carlo example that accepts pargs.

* Addition of pargs check for dependency on pgen.

* Addition of clearer error message for malformed parameters.

* Update setup.py

* do not overwrite log file. (#162)

Signed-off-by: Peter Robinson <robinson96@llnl.gov>

* Added confirmation message after launching a study (#163)

* Enhancements to store relative pathing in the metadata file. (#165)

* Changes to make workspaces reflect relative pathing based on step names.

* Addition of an alternative output format based on step combinations.

* Addition of tag to LULESH git dependency. (#169)

* Script Adapter Plugin (#167) (#170)

Fixes #167 

* added pytest to requirements

added Pipfile and pipenv settings

* Added property key to Abstract.ScriptAdapter (#167)
Also added impementation and tests to verify that existing functionality isn't changed

* updated factory to use key when registering adapters(#167)

* cleanedup linelength

* cleaned up imports to be specific to module (#167)

* added tests to verify exception for unknown adapter

* moved adapters tests to individual files

* added test to verify scriptadapter functionality (#167)

updated gitignore to have testing and pycharm ignores

testing existing adapters in factory (#167)

added test to verify factories.keys matches get_valid_adapters (#167)

added copyright to file

* updated __init__ modules to do dynamic includes

* removed unneeeded imports

* updated dependency versions

* fixed all flake8 errors

* updated to run flake8 and pytest when run locally

* updated tests to have documentation about purpose and function as requested in #170

* fixed line length

* Removal of nose from requirements.

* updated to remove nose from the requirements

* PyYAML vulnerability fix (#171)

* Locking the version of PyYAML to be above 2.1 because of an arbitrary code execution vulnerability.

* Addition of a version condition to pyyaml to patch a vulnerability.

* Update of Pipfile.lock to match Pipefile.

* Minor tweak to indentation for flake8 failure.

* fixed pyyaml to requirements (#172)

* Addition of a loader to the yaml load call. (#174)

Fixes #173 

* Addition of a loader to the yaml load call.

* Addition of a catch if the loader attribute is missing.

* Correction to install enum34 for Python versions < 3.4 (#176)

* Moved enum34 to condition dependent on Python<3.4.

* Addition of conditional enum34 install for requirements.txt.

* Correction of requirements.txt syntax for python version.

* Addition of a Dockerfile for tutorials and ease of trying out. (#178)

* Addition of a Dockerfile for quick tutorials.

* Tweaks for Docker and addition of git.

* Tweak to Docker file for caching.

* Addition of Docker documentation.

* Tweaks to Docker documentation.

* Removal of markdown ##

* Take out shebang from shell definition and add it when script is written. (#181)

* Take out shebang from shell definiton and at it when script is written.

* Include shebang in cmd and fix format of string written to file.

* Tweaks to fix malformed log statements. (#182)

* Correction to message when stating no to launch.

* Enhance shell batch setting to apply to scheduler scripts. (#183)

* Extension of shebang feature to allow users to specify shells.

* Addition of debug message to print kwargs.

* Addition of kwargs.

* Addition of basic batch settings to LULESH sample.

* Addition of kwargs to Flux adapters.

* Docstring tweaks.

* Docstring update.

* Fixes the addition of the shebang header for SLURM (#184)

* Docstring correction for LocalAdapter.

* Correction to addition of exec line at top of scripts.

* Correction to an accidental reassignment of cmd.

* Removal of an assignment of self._exec in SLURM adapter.

* Change to transition adapter returns to Record objects. (#177)

* Addition of a Record class for storing general data.

* Addition of SubmissionRecord type.

* Update to the order of for record parameters.

* Changes to StepRecord to expect SubmissionRecord returns.

* Updates to SLURM and local adapters to use SubmissionRecords.

* Slight tweak to LocalAdapter docstring.

* Tweak to have SubmissionRecord initialize its base.

* Addition of CancellationRecord class.

* Changes to CancellationRecord to map based on status.

* Additional interface additions and tweaks.

* Changes to have cancel use CancellationRecords.

* Update to ExecutionGraph to use records.

* Updates to SLURM and local adapters to use SubmissionRecords.

* Slight tweak to LocalAdapter docstring.

* Addition of CancellationRecord class.

* Additional interface additions and tweaks.

* Changes to have cancel use CancellationRecords.

* Cherry pick of execution commit.

* Removal of redundant "get" definiton.

* Addition of a SLURM enabled LULESH sample specification.

* Addition of output for stdout and stderr for Local adapter.

* Correction of file to open.

* Addition of 3.7 to testing stack.

* Added 3.7 to tox.ini.

* Removal of py37 in testing.

* Addition of py37 to travisCI (#187)

* Correction to 3.7 specification.

* Removed py37 from tox testing.

* Readded py37 to tox and removed duplicate from travis.

* Addition of build status badge.

* Update SLURM sample spec to add missing walltime.

* Addition of dumping the environment to a YAML file. (#190)

* Addition of documentation that covers the set up of a simple study (#168)

* Addition of simple Hello World spec.

* Addition of basics page to index.

* Addition of hello_world documentation.

* Additions to hello_world.

* More documentation in single step section.

* Continued edits to Hello World.

* Addition of parameter section.

* Addition of a note about %% token.

* Addition of directory structure.

* Continuation of parameter documentation.

* Removal of the depends key.

* Addition of the env section description.

* Addition of a link to Docker documentation for Dockerfiles.

* Addition of single parameter hello world.

* Correction of double colons.

* Correction of indentation.

* Addition of print out to verify output.

* Addition of sample specifications for multi and single params.

* Addition of more documentation for single param.

* Additional output to show parameter results.

* Correction to formatting.

* Addition of samples.

* Addition of simple Hello World spec.

* Addition of basics page to index.

* Addition of hello_world documentation.

* Additions to hello_world.

* More documentation in single step section.

* Continued edits to Hello World.

* Addition of parameter section.

* Addition of a note about %% token.

* Addition of directory structure.

* Continuation of parameter documentation.

* Removal of the depends key.

* Addition of the env section description.

* Addition of a link to Docker documentation for Dockerfiles.

* Addition of single parameter hello world.

* Correction of double colons.

* Correction of indentation.

* Addition of print out to verify output.

* Addition of sample specifications for multi and single params.

* Addition of more documentation for single param.

* Additional output to show parameter results.

* Correction to formatting.

* Updates to docstrings for data structures.

* Updates to clear Sphinx warnings.

* Removal of escape on the *args becuase of flake8 failure.

* Clean up of existing hello world specs.

* Addition of multistep example spec.

* Removal of * to fix sphinx errors.

* Correction to some docstrings.

* Tweaks to specs for consistent naming.

* Finished multi-step parameterized example.

* Tweaks to hello world docs.

* Addition of link to examples on GitHub.

* Correction of link to examples.

* Correction of link to examples (again).

* Removal of Pipfile.lock.

* Additions to gitignore for vscode and pipenv.

* Marking for v1.1.4 release.

* Corrected a missed merge for release v1.1.4
  • Loading branch information
Francesco Di Natale authored Jul 15, 2019
1 parent 085229d commit 51c62a3
Show file tree
Hide file tree
Showing 46 changed files with 2,076 additions and 235 deletions.
27 changes: 26 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
*.pyc
build/
dist/
*egg-info/
Expand All @@ -7,8 +6,34 @@ venv
.output/
sample_output/

# VSCode
.vscode

# Doxygen output
docs/html/

# Testing output
.tox/
testing
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*.cover
.hypothesis/
.pytest_cache/


*.py[cod]
__pycache__/
htmlcov/
wheelhouse/

pylint_*.txt
Pipfile.lock

#pycharm
.idea/
6 changes: 6 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,12 @@ python:
- "3.5"
- "3.6"

matrix:
include:
- python: 3.7
dist: xenial
sudo: true

install:
- pip install tox-travis
- pip install flake8
Expand Down
6 changes: 6 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
FROM ubuntu
LABEL maintainer="Francesco Di Natale dinatale3@llnl.gov"

RUN apt-get update && apt-get install -y python python-pip git
ADD . /maestrowf
RUN pip install -U /maestrowf
26 changes: 26 additions & 0 deletions Pipfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
"enum34" = "*"
filelock = "*"
six = "*"
tabulate = "*"
Fabric = "*"
PyYAML = ">= 4.2b1"
maestrowf = {path = "."}

[dev-packages]
"flake8" = "*"
pydocstyle = "*"
pylint = "*"
tox = "*"
coverage = "*"
sphinx_rtd_theme = "*"
Sphinx = "*"
pytest = "*"

[requires]
python_version = "3.6"
35 changes: 2 additions & 33 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# Maestro Workflow Conductor (MaestroWF)
[![Build Status](https://travis-ci.org/LLNL/maestrowf.svg?branch=develop)](https://travis-ci.org/LLNL/maestrowf)
[![PyPI](https://img.shields.io/pypi/v/maestrowf.svg)](https://pypi.python.org/pypi?name=maestrowf&version=1.0.0&:action=display)
[![Issues](https://img.shields.io/github/issues/LLNL/maestrowf.svg)](https://github.com/LLNL/maestrowf/issues)
[![Forks](https://img.shields.io/github/forks/LLNL/maestrowf.svg)](https://github.com/LLNL/maestrowf/network)
Expand Down Expand Up @@ -68,39 +69,7 @@ If you plan to develop on MaestroWF, install the repository directly using:

### Quickstart Example

MaestroWF comes packed with a basic example using LULESH, a proxy application provided
by LLNL. Information and source code for LULESH can be found [here](https://codesign.llnl.gov/lulesh.php).

The example performs the following workflow locally:
- Download LULESH from the webpage linked above and decompress it.
- Substitute all necessary variables with their serial compilers and make LULESH.
- Execute a small parameter sweep of varying size and iterations (a simple sensitivity study)

Two copies of the workflow are in the ```samples/lulesh``` directory for unix and macosx.
This is due to differences with ```sed```. In order to execute the sample study simply
execute from the root directory of the repository:

Unix:

$ maestro run ./samples/lulesh/lulesh_sample1_unix.yaml

MacOSX:

$ maestro run ./samples/lulesh/lulesh_sample1_macosx.yaml

When prompted, reply in the affirmative:

$ Would you like to launch the study?[yn] y

Maestro will create a timestamped directory in ```sample_output/lulesh```.

To monitor the study run:

$ maestro status sample_output/lulesh/<study_dir>

To cancel the study:

$ maestro cancel sample_output/lulesh/<study_dir>
MaestroWF comes packed with a basic example using LULESH, a proxy application provided by LLNL. You can find the Quick Start guide [here](https://maestrowf.readthedocs.io/en/latest/quick_start.html#).

----------------

Expand Down
21 changes: 19 additions & 2 deletions docs/source/getting_started.rst
Original file line number Diff line number Diff line change
@@ -1,10 +1,27 @@
Getting Started
================

Maestro Docker Container
************************

In order to set up the Docker container execute the following from the root of the Maestro repository::

$ docker --build -t maestrowf .

To launch the interactive shell of the Ubuntu image simply run::

$ docker run -it maestrowf

Once inside the Docker container, the following should bring up help::

$ maestro -h

For more information on using Dockerfiles, checkout Docker's `documentation <https://docs.docker.com/engine/reference/builder/>`.

Installing MaestroWF
*********************

MaestroWF can be installed via pip::
MaestroWF can be installed via pip outside of Docker with the following::

$ pip install maestrowf

Expand All @@ -13,7 +30,7 @@ MaestroWF can be installed via pip::
Once installed run::

$ maestro -h

usage: maestro [-h] [-l LOGPATH] [-d DEBUG_LVL] [-c] {cancel,run,status} ...

The Maestro Workflow Conductor for specifiying, launching, and managing general workflows.
Expand Down
Loading

0 comments on commit 51c62a3

Please sign in to comment.