Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue/5004 #5468

Merged
merged 73 commits into from
Jun 18, 2020
Merged

Issue/5004 #5468

merged 73 commits into from
Jun 18, 2020

Commits on Feb 3, 2020

  1. added docs and explicit data types

    @markjschreiber
    markjschreiber committed Feb 3, 2020
    Configuration menu
    Copy the full SHA
    5a550e9 View commit details
    Browse the repository at this point in the history

Commits on Feb 4, 2020

  1. added scala docs and explicit types

    @markjschreiber
    markjschreiber committed Feb 4, 2020
    Configuration menu
    Copy the full SHA
    cc3de61 View commit details
    Browse the repository at this point in the history

  2. added log statements for job definition

    @markjschreiber
    markjschreiber committed Feb 4, 2020
    Configuration menu
    Copy the full SHA
    bace95f View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2020

  1. refactoring to remove the need to create a job definition for every task

    @markjschreiber
    markjschreiber committed Feb 26, 2020
    Configuration menu
    Copy the full SHA
    62aaaa9 View commit details
    Browse the repository at this point in the history

Commits on Feb 29, 2020

  1. further work on job definitions

    @markjschreiber
    markjschreiber committed Feb 29, 2020
    Configuration menu
    Copy the full SHA
    c7d2f3f View commit details
    Browse the repository at this point in the history

  2. Some clean up

    @markjschreiber
    markjschreiber committed Feb 29, 2020
    Configuration menu
    Copy the full SHA
    732a772 View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2020

  1. Added docs and Some clean up

    @markjschreiber
    markjschreiber committed Mar 2, 2020
    Configuration menu
    Copy the full SHA
    7b6db73 View commit details
    Browse the repository at this point in the history

  2. Strategy to name and re-use job definitions

    @markjschreiber
    markjschreiber committed Mar 2, 2020
    Configuration menu
    Copy the full SHA
    96de1c2 View commit details
    Browse the repository at this point in the history

  3. Script bucket name validation and tests

    @markjschreiber
    markjschreiber committed Mar 2, 2020
    Configuration menu
    Copy the full SHA
    8a6bd16 View commit details
    Browse the repository at this point in the history

Commits on Mar 3, 2020

  1. Adjust regex to allow "/" in parts of s3 bucket name

    @markjschreiber
    markjschreiber committed Mar 3, 2020
    Configuration menu
    Copy the full SHA
    6463590 View commit details
    Browse the repository at this point in the history

  2. Include docker image name in hash

    @markjschreiber
    markjschreiber committed Mar 3, 2020
    Configuration menu
    Copy the full SHA
    40b13e8 View commit details
    Browse the repository at this point in the history

  3. use script not commandLine

    @markjschreiber
    markjschreiber committed Mar 3, 2020
    Configuration menu
    Copy the full SHA
    a2626bd View commit details
    Browse the repository at this point in the history

  4. debugging s3 script check and creation

    @markjschreiber
    markjschreiber committed Mar 3, 2020
    Configuration menu
    Copy the full SHA
    67646d7 View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2020

  1. Moved logic to check for existing job definition inside of the submit… 

    … call back
    @markjschreiber
    markjschreiber committed Mar 4, 2020
    Configuration menu
    Copy the full SHA
    4223aaa View commit details
    Browse the repository at this point in the history

  2. don't perform job definition creation if it has been found

    @markjschreiber
    markjschreiber committed Mar 4, 2020
    Configuration menu
    Copy the full SHA
    fa17501 View commit details
    Browse the repository at this point in the history

  3. Changing s3 script path handling and removing old s3 (v1) client

    @markjschreiber
    markjschreiber committed Mar 4, 2020
    Configuration menu
    Copy the full SHA
    61ffce0 View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2020

  1. fixed volumes and mounts and s3 interaction

    @markjschreiber
    markjschreiber committed Mar 5, 2020
    Configuration menu
    Copy the full SHA
    13c3690 View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2020

  1. removed all previous dependency on AWS SDK 1, fixed region issue

    @markjschreiber
    markjschreiber committed Mar 6, 2020
    Configuration menu
    Copy the full SHA
    329950e View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2020

  1. Changed job definition naming system. Some refactoring. Progress towa… 

    …rd removing proxy.
    @markjschreiber
    markjschreiber committed Mar 9, 2020
    Configuration menu
    Copy the full SHA
    39d45ba View commit details
    Browse the repository at this point in the history

Commits on Mar 10, 2020

  1. Refactored things that can be moved from job definition to batch job … 

    …to promote re-use of definitions.

Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Mar 10, 2020
    Configuration menu
    Copy the full SHA
    319f7f8 View commit details
    Browse the repository at this point in the history

Commits on Mar 12, 2020

  1. Removed proxy resources 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Mar 12, 2020
    Configuration menu
    Copy the full SHA
    f060449 View commit details
    Browse the repository at this point in the history

  2. Adjustments to environment, job definition and reconfigured script 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Mar 12, 2020
    Configuration menu
    Copy the full SHA
    4e623e6 View commit details
    Browse the repository at this point in the history

  3. changed log level 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Mar 12, 2020
    Configuration menu
    Copy the full SHA
    9935cf5 View commit details
    Browse the repository at this point in the history

  4. added type annotations and docs 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Mar 12, 2020
    Configuration menu
    Copy the full SHA
    fe6c4cd View commit details
    Browse the repository at this point in the history

  5. added type annotations and docs 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Mar 12, 2020
    Configuration menu
    Copy the full SHA
    6d95a5d View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2020

  1. Added toString for debugging 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Mar 13, 2020
    Configuration menu
    Copy the full SHA
    4552c5c View commit details
    Browse the repository at this point in the history

  2. Logging, code comments and docs 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Mar 13, 2020
    Configuration menu
    Copy the full SHA
    220db40 View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2020

  1. Generalized client building and removed duplicated code 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Mar 16, 2020
    Configuration menu
    Copy the full SHA
    94c680c View commit details
    Browse the repository at this point in the history

  2. added toString method 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Mar 16, 2020
    Configuration menu
    Copy the full SHA
    206d56c View commit details
    Browse the repository at this point in the history

  3. download input files from s3 to container 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Mar 16, 2020
    Configuration menu
    Copy the full SHA
    2a1fcff View commit details
    Browse the repository at this point in the history

Commits on Mar 17, 2020

  1. handle input and output files 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Mar 17, 2020
    Configuration menu
    Copy the full SHA
    3baf925 View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2020

  1. tee output streams 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Mar 19, 2020
    Configuration menu
    Copy the full SHA
    cda7b44 View commit details
    Browse the repository at this point in the history

  2. Added ability to determine the container, cluster and EC2 instance of… 

    … a job.

Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Mar 19, 2020
    Configuration menu
    Copy the full SHA
    c7d48a3 View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2020

  1. remove commented out code 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Mar 20, 2020
    Configuration menu
    Copy the full SHA
    64438a8 View commit details
    Browse the repository at this point in the history

  2. defend against nulls 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Mar 20, 2020
    Configuration menu
    Copy the full SHA
    5aed018 View commit details
    Browse the repository at this point in the history

Commits on Mar 30, 2020

  1. removed commented out code 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Mar 30, 2020
    Configuration menu
    Copy the full SHA
    f1c9c2a View commit details
    Browse the repository at this point in the history

Commits on Mar 31, 2020

  1. Updated README.md to reflect changes to the backend. 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Mar 31, 2020
    Configuration menu
    Copy the full SHA
    0d27996 View commit details
    Browse the repository at this point in the history

Commits on Apr 8, 2020

  1. Change to ensure working directory is "inside" the container and not … 

    …a mounted volume. Better trapping of errors and exit codes

Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 8, 2020
    Configuration menu
    Copy the full SHA
    c24dfb0 View commit details
    Browse the repository at this point in the history

  2. removed redundant / 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 8, 2020
    Configuration menu
    Copy the full SHA
    35c85c5 View commit details
    Browse the repository at this point in the history

  3. Tweaks suggested in code review 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 8, 2020
    Configuration menu
    Copy the full SHA
    8407b26 View commit details
    Browse the repository at this point in the history

  4. Corrected error handling and paths in script. Other tweaks from code … 

    …review.

Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 8, 2020
    Configuration menu
    Copy the full SHA
    a63a535 View commit details
    Browse the repository at this point in the history

  5. Corrected information about AMI requirements 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 8, 2020
    Configuration menu
    Copy the full SHA
    aa44d4c View commit details
    Browse the repository at this point in the history

  6. write rc.txt to S3 on failure 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 8, 2020
    Configuration menu
    Copy the full SHA
    b9fb3bd View commit details
    Browse the repository at this point in the history

  7. Merge branch 'develop' into issue/5004 

    # Conflicts:
#	supportedBackends/aws/src/main/scala/cromwell/backend/impl/aws/AwsBatchAsyncBackendJobExecutionActor.scala
#	supportedBackends/aws/src/main/scala/cromwell/backend/impl/aws/AwsBatchJobPaths.scala
    @markjschreiber
    markjschreiber committed Apr 8, 2020
    Configuration menu
    Copy the full SHA
    21c0232 View commit details
    Browse the repository at this point in the history

Commits on Apr 9, 2020

  1. Removed incorrect \n from expected result 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 9, 2020
    Configuration menu
    Copy the full SHA
    7db0133 View commit details
    Browse the repository at this point in the history

  2. removed commented out code 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 9, 2020
    Configuration menu
    Copy the full SHA
    ae27121 View commit details
    Browse the repository at this point in the history

  3. Tweaks suggested in Code Review 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 9, 2020
    Configuration menu
    Copy the full SHA
    e5e86c4 View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2020

  1. Updated bucket and queue for new backend conf 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 16, 2020
    Configuration menu
    Copy the full SHA
    78468b0 View commit details
    Browse the repository at this point in the history

  2. Oops should be bucket name not bucket url 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 16, 2020
    Configuration menu
    Copy the full SHA
    a771be9 View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2020

  1. better reconfigured script 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 21, 2020
    Configuration menu
    Copy the full SHA
    75d0cc7 View commit details
    Browse the repository at this point in the history

  2. better handling of list files and stdout/ stderr delocalization 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 21, 2020
    Configuration menu
    Copy the full SHA
    a894b2d View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2020

  1. Removed error trap as cromwell produced script can produce errors tha… 

    …t are not fatal. Defend against empty files.

Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    e5625d1 View commit details
    Browse the repository at this point in the history

  2. Reduced logging volume 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 22, 2020
    Configuration menu
    Copy the full SHA
    adc525b View commit details
    Browse the repository at this point in the history

Commits on Apr 23, 2020

  1. Merge branch 'develop' into issue/5004

    @markjschreiber
    markjschreiber authored Apr 23, 2020
    Configuration menu
    Copy the full SHA
    6a75311 View commit details
    Browse the repository at this point in the history

  2. Correcting conflict resolution 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 23, 2020
    Configuration menu
    Copy the full SHA
    07fec18 View commit details
    Browse the repository at this point in the history

Commits on Apr 24, 2020

  1. adjustments to script pre-amble 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 24, 2020
    Configuration menu
    Copy the full SHA
    4c90581 View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2020

  1. fixes to file paths required by Centaur tests 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 27, 2020
    Configuration menu
    Copy the full SHA
    941a068 View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2020

  1. corrected delocalization of glob files 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 28, 2020
    Configuration menu
    Copy the full SHA
    577d2dc View commit details
    Browse the repository at this point in the history

  2. check for non-existent input files 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 28, 2020
    Configuration menu
    Copy the full SHA
    ecc24b4 View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2020

  1. correct paths in delocalized tmp files 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 29, 2020
    Configuration menu
    Copy the full SHA
    26add9a View commit details
    Browse the repository at this point in the history

  2. removed un-needed braces and fixed typo in output copy command 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 29, 2020
    Configuration menu
    Copy the full SHA
    1aedd12 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2020

  1. pared back logging to fit in Travis CI limits 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Apr 30, 2020
    Configuration menu
    Copy the full SHA
    1ce2727 View commit details
    Browse the repository at this point in the history

Commits on May 1, 2020

  1. Exclude caching tests. Caching was never correctly implemented in the… 

    … backend and will need refactoring for the new model. This is my next assignment :-)

Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed May 1, 2020
    Configuration menu
    Copy the full SHA
    9935def View commit details
    Browse the repository at this point in the history

Commits on May 8, 2020

  1. updated ARN for new queue 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed May 8, 2020
    Configuration menu
    Copy the full SHA
    d54ae0f View commit details
    Browse the repository at this point in the history

Commits on May 12, 2020

  1. Increased concurrent job limit to 24 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed May 12, 2020
    Configuration menu
    Copy the full SHA
    eef2270 View commit details
    Browse the repository at this point in the history

Commits on May 14, 2020

  1. dialed back logging of s3 downloads 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed May 14, 2020
    Configuration menu
    Copy the full SHA
    994293b View commit details
    Browse the repository at this point in the history

Commits on May 21, 2020

  1. enabled copies of large files using multipart - needed for call cachi… 

    …ng of big files

Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed May 21, 2020
    Configuration menu
    Copy the full SHA
    d48545e View commit details
    Browse the repository at this point in the history

Commits on May 27, 2020

  1. updated queue arn 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed May 27, 2020
    Configuration menu
    Copy the full SHA
    f4622bb View commit details
    Browse the repository at this point in the history

Commits on Jun 2, 2020

  1. Reverted removal of double check 

    Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Jun 2, 2020
    Configuration menu
    Copy the full SHA
    68e3cf3 View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2020

  1. changes from code review

    @markjschreiber
    markjschreiber committed Jun 11, 2020
    Configuration menu
    Copy the full SHA
    e3581d1 View commit details
    Browse the repository at this point in the history

  2. reverted filesystem creation changes

    @markjschreiber
    markjschreiber committed Jun 11, 2020
    Configuration menu
    Copy the full SHA
    d858a9f View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2020

  1. Use terminateJob to abort jobs that are already starting or running. … 

    …Add type declarations to public variables

Signed-off-by: markjschreiber <markjschreiber@gmail.com>
    @markjschreiber
    markjschreiber committed Jun 12, 2020
    Configuration menu
    Copy the full SHA
    bb25190 View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2020

  1. S3FilesystemProvider should return existing filesystem instead of t… 

    …hrowing exception when `newFileSystem` method is called and s3 filesystem already exists [BA-6481]
    Grigoriy Sterin committed Jun 18, 2020
    Configuration menu
    Copy the full SHA
    216a173 View commit details
    Browse the repository at this point in the history