Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WX-906 Sbt Unit Tests as Github Actions #6992

Merged
merged 137 commits into from
Feb 1, 2023
Merged

WX-906 Sbt Unit Tests as Github Actions #6992

merged 137 commits into from
Feb 1, 2023

Conversation

THWiseman
Copy link
Contributor

@THWiseman THWiseman commented Jan 31, 2023 •
edited
Loading

This PR contains the two files necessary for us to run all Cromwell unit tests on Github Actions.

The two YAML files:

.github/workflows/cromwell_unit_tests.yml: A workflow that starts 1 VM, sets up cromwell, and then runs all of cromwell's unit tests.

.github/set_up_cromwell_action/action.yml A reusable/generalized action that executes the steps needed to get Cromwell up-and-running on an ubuntu VM. This is separated from the unit tests so we don't have to duplicate code the next time we want to set up a new Github Action.

@aednichols aednichols self-requested a review January 31, 2023 18:52
aednichols
aednichols approved these changes Jan 31, 2023
View reviewed changes
Copy link
Collaborator

@aednichols aednichols left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is... a lot of test suites to scroll through.

This is clearly forward progress so I approve, but we will probably need to make a second pass later on and somehow consolidate things.

Nice work!

.github/set_up_cromwell_action/action.yml Outdated Show resolved Hide resolved
.github/workflows/cromwell_unit_tests.yml Outdated
jobs:
build-and-test:
strategy:
fail-fast: false #Disabling fail fast means that even if one job in the matrix fails, the others will still try to complete.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

.github/workflows/cromwell_unit_tests.yml
Comment on lines +72 to +74
#This action is using a Github free runner, rather than a Broad self-hosted one.
#This is because the Broad ones don't have sbt installed by default.
runs-on: ubuntu-latest
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's probably fine for these small tests, that are unlikely to ever be the longest-running piece and therefore gating factor to speed.

jgainerdewar
jgainerdewar reviewed Jan 31, 2023
View reviewed changes
Copy link
Collaborator

@jgainerdewar jgainerdewar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Echoing Adam, I think it can be changed in a followon ticket, but I don't think we want to list each project's tests as a top-level action. I think we also don't gain anything in overall test runtime by breaking them up this way, since unit tests are the fast part of the test suite already.

I want to advocate for taking another look at the way it's done in Travis now - breaking the unit tests down in a few big chunks, with one of them being "everything not already handled in another step." That would let us avoid a hard-coded list of projects and also trim down the total number of actions being run. If that's difficult, I think a single action that runs all tests is a more ergonomic "good enough for now" state than one action per sbt project.

.github/workflows/cromwell_unit_tests.yml Outdated Show resolved Hide resolved
.github/workflows/cromwell_unit_tests.yml Outdated Show resolved Hide resolved
jgainerdewar
jgainerdewar approved these changes Jan 31, 2023
View reviewed changes
Copy link
Collaborator

@jgainerdewar jgainerdewar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, go tests go!!!

.github/workflows/cromwell_unit_tests.yml Outdated
with:
cromwell_repo_token: ${{ secrets.BROADBOT_GITHUB_TOKEN }}

#Invoke SBT on a specific subproject, asking it to run all unit tests contained within that subproject.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably want to update this comment 🙂

@THWiseman
Copy link
Contributor Author

The TravisCI test is not passing or finishing, but I am going to bypass that protection and merge anyway. Talked about it with the team: current theory has to do with the fact that Travis is hung up on testing an old branch name (travis_to_workflows_test) that this PR's branch (sbt_unit_tests) was branched from. Alternatively, it could be due to the fact that I am new and might not have some sort of account set up in Travis. Another theory is that Travis can't handle the fact that no cromwell code was changed in this branch.

Anyway....this is only github specific stuff so deemed safe to merge.

@THWiseman THWiseman merged commit a5b0f69 into develop Feb 1, 2023
@THWiseman THWiseman deleted the sbt_unit_tests branch February 1, 2023 21:02
henriqueribeiro added a commit to henriqueribeiro/cromwell that referenced this pull request Mar 20, 2023
@henriqueribeiro @jgainerdewar
@kpierre13 @cjllanwarne @kraefrei @salonishah11 @kshakir @mspector @aednichols @breilly2 @dependabot @JVThomas @tlangs @mcovarr @ekiernan @THWiseman @BMurri
85 release (#28)
8982e07 
* Update cromwell version from 83 to 84

* BW-1255 Implement POST /runs endpoint (broadinstitute#6779)

* Adding route

* Fixing HTTP method error

* All formFields made optional

* A compliling state

* Saving

* Saving

* All three endpoints functioning as expected; updated RESTAPI.md

* Updated response for submission from 200 to 201 to pass tests

* Test submission response

* Moved updated submission response to askSubmit

* test

* updating RESTAPI.md

* saving

* Adding utility file for submitRequest

* cleanup

* Update awssdkv from 2.17.152 to 2.17.194 (broadinstitute#6814)

* BW-1305 Swagger Update (broadinstitute#6818)

* Properly documenting metadataArchiveStatus in WorkflowQueryResult model

* Update docs

* BT-710 Add configs for BlobPathBuilderFactory (broadinstitute#6817)

BT-710 Add configs for BlobPathBuilderFactory

* BW-1305 Make "name" optional in workflow query response (broadinstitute#6821)

* BT-724 Fix BlobPathBuilder failing on retrieving existing filesystem (broadinstitute#6816)

Modify blobPathBuilder to fallback to creating a filesystem if one is not found

* Logging updates: (broadinstitute#6813)

* [BT-698] first pass on BlobTokenGenerator with E2E test (broadinstitute#6824)

* first pass on BlobTokenGenerator with E2E test

* update BlobPathBuilder constructor args in test

* account -> container level client

* [BT-687] specify correct types (broadinstitute#6829)

* specify correct types

* fix test with new type

* remove type declarations in function call

* remove unnecessary sas-token config

* BW-1206 - Combine all Wes Endpoints & add Tests (broadinstitute#6833)

* Add tests, getting frid of WesRunRoutes.scala

* wesWorkflowId fix, ec implicits errors gone

* Refactoring path for GET /runs

* Indentation fix

* Commit to rollback

* Revert "Indentation fix"

This reverts commit 63fc484.

* PR trigger

* Optimize imports

* Missed import

* BW-1354 - Porting CBAS preliminary step (broadinstitute#6837)

* Getting rid of shared utility file; Adding/Updating WES version of submit.

* Edit spec file

* Adding Wes-like error

* BW-1378 Addl CromIAM user enablement checks (broadinstitute#6826)

* Update cromwell version from 84 to 85

* BW-1393 Release doc updates (broadinstitute#6839)

* BT-732 Checksum validation for blobs read by engine (broadinstitute#6838)

* Draft support for optional FileHash

* Draft getMd5 for BlobPath

* Resolve non-parallel IO to fix tests

* Checksum validation for BlobPath

* Nicer error message

* Test for missing Blob hash

* Break attr acquisition into separate method

* Cleanup, comments

* In-progress tests of blob hash command

* Remove test

* Remove unused import

* BT-711 Refresh SAS token for filesystem on expiry (broadinstitute#6831)

* BT-711 Refresh SAS token for filesystem on expiry

* Rough cut of token refresh using exceptions

* Ignore tests, and minor cleanup

* Remove stray line

* Draft of manager class for handling expiring file systems

* Style fixes

* Refactor of blobfilesystemManager and tests covering its functionality

* Refined tests to validate close filesystem as separate unit

* Ignore connected tests

* Clean up of some things

* Refactor BlobFileSystemManager to separate file, and some other cleanup

* Some additional scala-ifying

* Small cleanup

* Correcting imports

* trigger tests

* trigger tests

* Batch 1 of scala steward updates (broadinstitute#6903)

* Batch 1 of scala steward updates

* Rollback snakeYAML

* Attempt 3, with only the passing dependancies

* Revert google API and Big Query udpates

* Winding back other google deps

* rollback remaining google updates

* trigger tests

* trigger tests

* [BW-1398] Migrate PKs to BIGINT (broadinstitute#6907)

* BT-745  Batch 2 of scala steward updates (broadinstitute#6906)

* Update SBT to 2.0.0

* Fix sbt-git import

* Update mouse to 1.0.11

* Update rhino 1.7.14

* SUP-692 Retry with more memory after RC 137 (broadinstitute#6912)

* Reorder execution result checks so 137 can retry with more memory

* Test for memory retry after 137 RC

* Fix test expectations

* Make memory retry checks consistent

* Revert changes to existing test

* Rename retryWithMoreMemory to outOfMemoryDetected

* Scala steward updates batch 3 (broadinstitute#6913)

* Scala steward updates batch 3

* WX-745 Batch 4 scala steward updates (broadinstitute#6916)

* WX-746 Localize all DRS inputs in a single Action (broadinstitute#6914)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-755 Build all images instead of just Cromwell (broadinstitute#6919)

* WX-755 Add `isRelease` option for Docker builds (broadinstitute#6923)

* WX-755 Cromwell/CromIAM automatically board train (broadinstitute#6924)

* WX-755 Fix environment variable syntax (broadinstitute#6926)

* WX-743 Enable TES task creation with BlobPaths (broadinstitute#6921)

* Give blob SAS tokens write permission

* Case class wrapper for subscription id

* Resolve duplicate container name in absolute BlobPath

* Ignored test demonstrating correct absolute path generation

* Update filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobPathBuilderSpec.scala

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* PR feedback

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* [WX-765] Update snakeyaml to 1.33 (broadinstitute#6927)

* update snakeyaml to 1.33

* Don't use deprecated no-arg Constructor constructor

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WM-1414 Refactoring WesRunLog to omit Cromwell's "workflowLog" object (broadinstitute#6925)

* Upgrade Postgres to 42.4.1 (broadinstitute#6932)

* WX-735 Fix incorrect and/or nondeterministic filesystem ordering (broadinstitute#6930)

* WX-772 Update Scala to 2.13.9 (broadinstitute#6928)

* Update Scala to 2.13.9

* Try updating sbt-scoverage

* Does this version exist anywhere we can see?

* This version actually exists

* Update library version to remove conflict

* Codegen version

* Fix fun new 2.13.9 compiler errors

* Resolve warnings

* Newest Scala?

* I guess not

* Does this please Travis?

* force ci

* Back out changes to generated code

Co-authored-by: Adam Nichols <aednichols@gmail.com>

* WX-781 Bump jackson-databind in /CromwellRefdiskManifestCreator (broadinstitute#6935)

Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.13.2.2 to 2.13.4.1.
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)

---
updated-dependencies:
- dependency-name: com.fasterxml.jackson.core:jackson-databind
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* WX-808 Host allowlist for HTTP imports (broadinstitute#6938)

* `hostAllowlist` that allows everything

* Refactor

* Stick allow list in HttpResolver

* Better default config

* Allow list tests

* Make it build

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Update commons text to 1.10.0 (broadinstitute#6937)

* WX-751 Token refresh signal for monitoring (broadinstitute#6939)

* Log messages

* `DEBUG` -> `INFO`

* WX-744 Optionally rewrite blob paths to appear as local paths (broadinstitute#6941)

* Modify blob paths for TES

* Make blob transformation configurable

* Update supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Apply PR feedback in second place

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Update changelog for wdl http allow list (broadinstitute#6944)

* WM-1491 Fixing Cromwell-client (broadinstitute#6943)

* More updated client for use in cbas

* Removing excess code

* Fix client build script (broadinstitute#6945)

* WX-837: Remove CWL references from documentation (broadinstitute#6949)

* wx-837 removed cwl references in markdown doc files

* wx-837 removed cwlParsingOverview.md, updated mkdocs.yml

* wx-837 updated cromwell.yaml, generated new RESTAPI file

* WX-728 Add configurable WSM client to Cromwell (broadinstitute#6948)

* Dependencies

* Compiles but no tests

* Formatting

* Moar exclusions

* Update to latest WSM

* Add additional dependency

* We need some UUID here to make the request

* Formatting

* Clarify what is fake

* Formatting

* Use our own version of Jersey and Jackson stuff

* Port-in Khalid's changes (thank you!)

Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* Test longevity

Don't break the test if someone decides to add a cert to `ws.org`

* Cleanup

* Cleanup

* Cleanup

* Adjust TES config file for CI

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* CROM-6554: Removed PAPIv1 references from doc (broadinstitute#6950)

* crom-6554 removed references to PAPI v1 from doc

* crom-6554 pr feedback, reworded doc to use example conf as a starting point

* WX-833 Real Azure DRS Credentials (broadinstitute#6952)

* Remove B2C reference from name

* Get token for current user rather than getting from KeyVault

* Remove KeyVault config for engine

* Remove KeyVault config for DRSLocalizer

* Remove KeyVault dependency

* Remove KeyVault support from localizer repo template

* Cleaned up and working Azure token acquisition for engine

* Collapse localizer's AccessTokenStrategy into DrsCredentials

* Cleanup

* WX-853 Remove most CWL (broadinstitute#6955)

* WX-696 Enable getting SAS token from WSM (broadinstitute#6954)

* WX-696 Enable getting SAS token from WSM

* Wire container resource id from config

* Move resource-container-id config path

* First pass at config for WSM

* Remove unused singleton config

* Tests for new config

* Fix config parsing

* Modified b2c token to be provided each time

* Remove singletonConfig arg from factory

* Restore types to factory configs

* Clean up comments and empty token default

* Default to config b2c before searching environment

* Fix token default on api client

* Fix test

* Refactor error handling for when there is no token

* Remove token constructor arg for clientProvider

* Move configs to global singleton config

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

* default -> override

* Add override token to test

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Parentheses

* Reduce token timeout

* Move AzureCredentials to separate file

* Make AzureCredentials an object

* WSM token cleanup

* Config refactor (broadinstitute#6960)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Initial blob token documentation

* Refine language in BlobSasTokenGenerator

* Update comment and formatting

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-853 Remove CWL language factory, Centaur runner (broadinstitute#6961)

* WX-842 Add Pact Dependency for Cromwell (broadinstitute#6962)

* WX-842 Add Pact Dependency for Cromwell

* Remove incomplete test spec

* Initial Pact Test

* Fix pact so it compiles

* Add breadcrumb comment and clean up

* ID-125 Add support for drshub, rename all the things (broadinstitute#6959)

* Add support for drshub, rename all the things

* fallback to martha if resolver is not in config

* WX-867 Translate crc32c hashes to b64 for getm (broadinstitute#6970)

* Translate crc32c hashes to b64 for getm

* Update tests

* Remove obsolete b64 handling for md5, centralize hex validation

* Restore old test, fix other test

* WX-843 Workflow failure reason should accurately indicate issues opening blob filesystem (broadinstitute#6965)

* WX-859 Accept workflow execution identity in config (broadinstitute#6967)

* WX-892 Trim down `ValueStore` logging to prevent OOMs (broadinstitute#6981)

* Add Nirvana 3.18.1 reference image test, minor cleanup [VS-705] (broadinstitute#6975)

* WX-863 Turn off Azure NIO logging (broadinstitute#6982)

* Turn off Azure NIO logging

* Poke Travis

* WM-1616: Allow repeating attempts at initialization (take 2) (broadinstitute#6985)

* WX-878 Single shared BlobFileSystemManager (broadinstitute#6986)

* Make BlobFileSystemManager shared across all BlobPathBuilders

* Update TES conf file to reflect new singleton config

* Shell escape reference image  files [VS-796] [WX-910] (broadinstitute#6989)

* WX-769 `disks` compatibility for TES backend (broadinstitute#6991)

* Update FiveMinuteIntro.md (broadinstitute#6994)

* WX-906 Sbt Unit Tests as Github Actions (broadinstitute#6992)

* WX-926 Support falling back to OCI Manifest Format (broadinstitute#7003)

* WX-926 Support falling back to OCI Manifest Forma

* Only mount reference disks if requested [WX-925] (broadinstitute#7001)

* [WM-1646] Add missing fields for `WorkflowDescription` for WomTool /describe endpoint to Swagger (broadinstitute#7004)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status (broadinstitute#6980)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status

* Address feedback

* Address feedback (broadinstitute#6997)

* Address additional feedback (broadinstitute#7000)

* Fix copy/paste error (broadinstitute#7005)

* Address additional feedback

* Fix copy/paste error

* Trigger CI

---------

Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Centaur reference image test should validate symlinks [VS-796] (broadinstitute#6996)

* WX-903 Pre-GHA test suite disablement

* WX-877 Update CHANGELOG for release 85 (broadinstitute#7011)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Katrina P <68349264+kpierre13@users.noreply.github.com>
Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com>
Co-authored-by: Christian Freitas <christian.j.freitas@gmail.com>
Co-authored-by: Saloni Shah <salonipshah11@gmail.com>
Co-authored-by: kshakir <kshakir@broadinstitute.org>
Co-authored-by: mspector <michael.l.spector@gmail.com>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>
Co-authored-by: Brian Reilly <breilly@broadinstitute.org>
Co-authored-by: Adam Nichols <aednichols@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Justin Variath Thomas <JVThomas@users.noreply.github.com>
Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org>
Co-authored-by: Trevyn Langsford <tlangs@broadinstitute.org>
Co-authored-by: Miguel Covarrubias <mcovarr@users.noreply.github.com>
Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com>
Co-authored-by: Tom Wiseman <tom.h.wiseman@gmail.com>
Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>
henriqueribeiro added a commit to henriqueribeiro/cromwell that referenced this pull request May 9, 2023
@henriqueribeiro @jgainerdewar
@kpierre13 @cjllanwarne @kraefrei @salonishah11 @kshakir @mspector @aednichols @breilly2 @dependabot @JVThomas @tlangs @mcovarr @ekiernan @THWiseman @BMurri @geertvandeweyer
Version 85 (#30)
7043c10 
* 85 release (#28)

* Update cromwell version from 83 to 84

* BW-1255 Implement POST /runs endpoint (broadinstitute#6779)

* Adding route

* Fixing HTTP method error

* All formFields made optional

* A compliling state

* Saving

* Saving

* All three endpoints functioning as expected; updated RESTAPI.md

* Updated response for submission from 200 to 201 to pass tests

* Test submission response

* Moved updated submission response to askSubmit

* test

* updating RESTAPI.md

* saving

* Adding utility file for submitRequest

* cleanup

* Update awssdkv from 2.17.152 to 2.17.194 (broadinstitute#6814)

* BW-1305 Swagger Update (broadinstitute#6818)

* Properly documenting metadataArchiveStatus in WorkflowQueryResult model

* Update docs

* BT-710 Add configs for BlobPathBuilderFactory (broadinstitute#6817)

BT-710 Add configs for BlobPathBuilderFactory

* BW-1305 Make "name" optional in workflow query response (broadinstitute#6821)

* BT-724 Fix BlobPathBuilder failing on retrieving existing filesystem (broadinstitute#6816)

Modify blobPathBuilder to fallback to creating a filesystem if one is not found

* Logging updates: (broadinstitute#6813)

* [BT-698] first pass on BlobTokenGenerator with E2E test (broadinstitute#6824)

* first pass on BlobTokenGenerator with E2E test

* update BlobPathBuilder constructor args in test

* account -> container level client

* [BT-687] specify correct types (broadinstitute#6829)

* specify correct types

* fix test with new type

* remove type declarations in function call

* remove unnecessary sas-token config

* BW-1206 - Combine all Wes Endpoints & add Tests (broadinstitute#6833)

* Add tests, getting frid of WesRunRoutes.scala

* wesWorkflowId fix, ec implicits errors gone

* Refactoring path for GET /runs

* Indentation fix

* Commit to rollback

* Revert "Indentation fix"

This reverts commit 63fc484.

* PR trigger

* Optimize imports

* Missed import

* BW-1354 - Porting CBAS preliminary step (broadinstitute#6837)

* Getting rid of shared utility file; Adding/Updating WES version of submit.

* Edit spec file

* Adding Wes-like error

* BW-1378 Addl CromIAM user enablement checks (broadinstitute#6826)

* Update cromwell version from 84 to 85

* BW-1393 Release doc updates (broadinstitute#6839)

* BT-732 Checksum validation for blobs read by engine (broadinstitute#6838)

* Draft support for optional FileHash

* Draft getMd5 for BlobPath

* Resolve non-parallel IO to fix tests

* Checksum validation for BlobPath

* Nicer error message

* Test for missing Blob hash

* Break attr acquisition into separate method

* Cleanup, comments

* In-progress tests of blob hash command

* Remove test

* Remove unused import

* BT-711 Refresh SAS token for filesystem on expiry (broadinstitute#6831)

* BT-711 Refresh SAS token for filesystem on expiry

* Rough cut of token refresh using exceptions

* Ignore tests, and minor cleanup

* Remove stray line

* Draft of manager class for handling expiring file systems

* Style fixes

* Refactor of blobfilesystemManager and tests covering its functionality

* Refined tests to validate close filesystem as separate unit

* Ignore connected tests

* Clean up of some things

* Refactor BlobFileSystemManager to separate file, and some other cleanup

* Some additional scala-ifying

* Small cleanup

* Correcting imports

* trigger tests

* trigger tests

* Batch 1 of scala steward updates (broadinstitute#6903)

* Batch 1 of scala steward updates

* Rollback snakeYAML

* Attempt 3, with only the passing dependancies

* Revert google API and Big Query udpates

* Winding back other google deps

* rollback remaining google updates

* trigger tests

* trigger tests

* [BW-1398] Migrate PKs to BIGINT (broadinstitute#6907)

* BT-745  Batch 2 of scala steward updates (broadinstitute#6906)

* Update SBT to 2.0.0

* Fix sbt-git import

* Update mouse to 1.0.11

* Update rhino 1.7.14

* SUP-692 Retry with more memory after RC 137 (broadinstitute#6912)

* Reorder execution result checks so 137 can retry with more memory

* Test for memory retry after 137 RC

* Fix test expectations

* Make memory retry checks consistent

* Revert changes to existing test

* Rename retryWithMoreMemory to outOfMemoryDetected

* Scala steward updates batch 3 (broadinstitute#6913)

* Scala steward updates batch 3

* WX-745 Batch 4 scala steward updates (broadinstitute#6916)

* WX-746 Localize all DRS inputs in a single Action (broadinstitute#6914)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-755 Build all images instead of just Cromwell (broadinstitute#6919)

* WX-755 Add `isRelease` option for Docker builds (broadinstitute#6923)

* WX-755 Cromwell/CromIAM automatically board train (broadinstitute#6924)

* WX-755 Fix environment variable syntax (broadinstitute#6926)

* WX-743 Enable TES task creation with BlobPaths (broadinstitute#6921)

* Give blob SAS tokens write permission

* Case class wrapper for subscription id

* Resolve duplicate container name in absolute BlobPath

* Ignored test demonstrating correct absolute path generation

* Update filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobPathBuilderSpec.scala

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* PR feedback

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* [WX-765] Update snakeyaml to 1.33 (broadinstitute#6927)

* update snakeyaml to 1.33

* Don't use deprecated no-arg Constructor constructor

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WM-1414 Refactoring WesRunLog to omit Cromwell's "workflowLog" object (broadinstitute#6925)

* Upgrade Postgres to 42.4.1 (broadinstitute#6932)

* WX-735 Fix incorrect and/or nondeterministic filesystem ordering (broadinstitute#6930)

* WX-772 Update Scala to 2.13.9 (broadinstitute#6928)

* Update Scala to 2.13.9

* Try updating sbt-scoverage

* Does this version exist anywhere we can see?

* This version actually exists

* Update library version to remove conflict

* Codegen version

* Fix fun new 2.13.9 compiler errors

* Resolve warnings

* Newest Scala?

* I guess not

* Does this please Travis?

* force ci

* Back out changes to generated code

Co-authored-by: Adam Nichols <aednichols@gmail.com>

* WX-781 Bump jackson-databind in /CromwellRefdiskManifestCreator (broadinstitute#6935)

Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.13.2.2 to 2.13.4.1.
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)

---
updated-dependencies:
- dependency-name: com.fasterxml.jackson.core:jackson-databind
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* WX-808 Host allowlist for HTTP imports (broadinstitute#6938)

* `hostAllowlist` that allows everything

* Refactor

* Stick allow list in HttpResolver

* Better default config

* Allow list tests

* Make it build

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Update commons text to 1.10.0 (broadinstitute#6937)

* WX-751 Token refresh signal for monitoring (broadinstitute#6939)

* Log messages

* `DEBUG` -> `INFO`

* WX-744 Optionally rewrite blob paths to appear as local paths (broadinstitute#6941)

* Modify blob paths for TES

* Make blob transformation configurable

* Update supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Apply PR feedback in second place

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Update changelog for wdl http allow list (broadinstitute#6944)

* WM-1491 Fixing Cromwell-client (broadinstitute#6943)

* More updated client for use in cbas

* Removing excess code

* Fix client build script (broadinstitute#6945)

* WX-837: Remove CWL references from documentation (broadinstitute#6949)

* wx-837 removed cwl references in markdown doc files

* wx-837 removed cwlParsingOverview.md, updated mkdocs.yml

* wx-837 updated cromwell.yaml, generated new RESTAPI file

* WX-728 Add configurable WSM client to Cromwell (broadinstitute#6948)

* Dependencies

* Compiles but no tests

* Formatting

* Moar exclusions

* Update to latest WSM

* Add additional dependency

* We need some UUID here to make the request

* Formatting

* Clarify what is fake

* Formatting

* Use our own version of Jersey and Jackson stuff

* Port-in Khalid's changes (thank you!)

Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* Test longevity

Don't break the test if someone decides to add a cert to `ws.org`

* Cleanup

* Cleanup

* Cleanup

* Adjust TES config file for CI

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* CROM-6554: Removed PAPIv1 references from doc (broadinstitute#6950)

* crom-6554 removed references to PAPI v1 from doc

* crom-6554 pr feedback, reworded doc to use example conf as a starting point

* WX-833 Real Azure DRS Credentials (broadinstitute#6952)

* Remove B2C reference from name

* Get token for current user rather than getting from KeyVault

* Remove KeyVault config for engine

* Remove KeyVault config for DRSLocalizer

* Remove KeyVault dependency

* Remove KeyVault support from localizer repo template

* Cleaned up and working Azure token acquisition for engine

* Collapse localizer's AccessTokenStrategy into DrsCredentials

* Cleanup

* WX-853 Remove most CWL (broadinstitute#6955)

* WX-696 Enable getting SAS token from WSM (broadinstitute#6954)

* WX-696 Enable getting SAS token from WSM

* Wire container resource id from config

* Move resource-container-id config path

* First pass at config for WSM

* Remove unused singleton config

* Tests for new config

* Fix config parsing

* Modified b2c token to be provided each time

* Remove singletonConfig arg from factory

* Restore types to factory configs

* Clean up comments and empty token default

* Default to config b2c before searching environment

* Fix token default on api client

* Fix test

* Refactor error handling for when there is no token

* Remove token constructor arg for clientProvider

* Move configs to global singleton config

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

* default -> override

* Add override token to test

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Parentheses

* Reduce token timeout

* Move AzureCredentials to separate file

* Make AzureCredentials an object

* WSM token cleanup

* Config refactor (broadinstitute#6960)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Initial blob token documentation

* Refine language in BlobSasTokenGenerator

* Update comment and formatting

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-853 Remove CWL language factory, Centaur runner (broadinstitute#6961)

* WX-842 Add Pact Dependency for Cromwell (broadinstitute#6962)

* WX-842 Add Pact Dependency for Cromwell

* Remove incomplete test spec

* Initial Pact Test

* Fix pact so it compiles

* Add breadcrumb comment and clean up

* ID-125 Add support for drshub, rename all the things (broadinstitute#6959)

* Add support for drshub, rename all the things

* fallback to martha if resolver is not in config

* WX-867 Translate crc32c hashes to b64 for getm (broadinstitute#6970)

* Translate crc32c hashes to b64 for getm

* Update tests

* Remove obsolete b64 handling for md5, centralize hex validation

* Restore old test, fix other test

* WX-843 Workflow failure reason should accurately indicate issues opening blob filesystem (broadinstitute#6965)

* WX-859 Accept workflow execution identity in config (broadinstitute#6967)

* WX-892 Trim down `ValueStore` logging to prevent OOMs (broadinstitute#6981)

* Add Nirvana 3.18.1 reference image test, minor cleanup [VS-705] (broadinstitute#6975)

* WX-863 Turn off Azure NIO logging (broadinstitute#6982)

* Turn off Azure NIO logging

* Poke Travis

* WM-1616: Allow repeating attempts at initialization (take 2) (broadinstitute#6985)

* WX-878 Single shared BlobFileSystemManager (broadinstitute#6986)

* Make BlobFileSystemManager shared across all BlobPathBuilders

* Update TES conf file to reflect new singleton config

* Shell escape reference image  files [VS-796] [WX-910] (broadinstitute#6989)

* WX-769 `disks` compatibility for TES backend (broadinstitute#6991)

* Update FiveMinuteIntro.md (broadinstitute#6994)

* WX-906 Sbt Unit Tests as Github Actions (broadinstitute#6992)

* WX-926 Support falling back to OCI Manifest Format (broadinstitute#7003)

* WX-926 Support falling back to OCI Manifest Forma

* Only mount reference disks if requested [WX-925] (broadinstitute#7001)

* [WM-1646] Add missing fields for `WorkflowDescription` for WomTool /describe endpoint to Swagger (broadinstitute#7004)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status (broadinstitute#6980)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status

* Address feedback

* Address feedback (broadinstitute#6997)

* Address additional feedback (broadinstitute#7000)

* Fix copy/paste error (broadinstitute#7005)

* Address additional feedback

* Fix copy/paste error

* Trigger CI

---------

Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Centaur reference image test should validate symlinks [VS-796] (broadinstitute#6996)

* WX-903 Pre-GHA test suite disablement

* WX-877 Update CHANGELOG for release 85 (broadinstitute#7011)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Katrina P <68349264+kpierre13@users.noreply.github.com>
Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com>
Co-authored-by: Christian Freitas <christian.j.freitas@gmail.com>
Co-authored-by: Saloni Shah <salonipshah11@gmail.com>
Co-authored-by: kshakir <kshakir@broadinstitute.org>
Co-authored-by: mspector <michael.l.spector@gmail.com>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>
Co-authored-by: Brian Reilly <breilly@broadinstitute.org>
Co-authored-by: Adam Nichols <aednichols@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Justin Variath Thomas <JVThomas@users.noreply.github.com>
Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org>
Co-authored-by: Trevyn Langsford <tlangs@broadinstitute.org>
Co-authored-by: Miguel Covarrubias <mcovarr@users.noreply.github.com>
Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com>
Co-authored-by: Tom Wiseman <tom.h.wiseman@gmail.com>
Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>

* Develop aws (#29)

* stuck on globbing

* efs works, no callcaching

* update readme

* extended EFS support

* fix for globbing in nested scatters

* updated config for globbing, to prevent issues with empty folders

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Katrina P <68349264+kpierre13@users.noreply.github.com>
Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com>
Co-authored-by: Christian Freitas <christian.j.freitas@gmail.com>
Co-authored-by: Saloni Shah <salonipshah11@gmail.com>
Co-authored-by: kshakir <kshakir@broadinstitute.org>
Co-authored-by: mspector <michael.l.spector@gmail.com>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>
Co-authored-by: Brian Reilly <breilly@broadinstitute.org>
Co-authored-by: Adam Nichols <aednichols@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Justin Variath Thomas <JVThomas@users.noreply.github.com>
Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org>
Co-authored-by: Trevyn Langsford <tlangs@broadinstitute.org>
Co-authored-by: Miguel Covarrubias <mcovarr@users.noreply.github.com>
Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com>
Co-authored-by: Tom Wiseman <tom.h.wiseman@gmail.com>
Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>
Co-authored-by: geertvandeweyer <geertvandeweyer@gmail.com>
henriqueribeiro added a commit to henriqueribeiro/cromwell that referenced this pull request Sep 18, 2023
@henriqueribeiro @jgainerdewar
@kpierre13 @cjllanwarne @kraefrei @salonishah11 @kshakir @mspector @aednichols @breilly2 @dependabot @JVThomas @tlangs @mcovarr @ekiernan @THWiseman @BMurri @geertvandeweyer
Develop AWS (#35)
8395859 
* 85 release (#28)

* Update cromwell version from 83 to 84

* BW-1255 Implement POST /runs endpoint (broadinstitute#6779)

* Adding route

* Fixing HTTP method error

* All formFields made optional

* A compliling state

* Saving

* Saving

* All three endpoints functioning as expected; updated RESTAPI.md

* Updated response for submission from 200 to 201 to pass tests

* Test submission response

* Moved updated submission response to askSubmit

* test

* updating RESTAPI.md

* saving

* Adding utility file for submitRequest

* cleanup

* Update awssdkv from 2.17.152 to 2.17.194 (broadinstitute#6814)

* BW-1305 Swagger Update (broadinstitute#6818)

* Properly documenting metadataArchiveStatus in WorkflowQueryResult model

* Update docs

* BT-710 Add configs for BlobPathBuilderFactory (broadinstitute#6817)

BT-710 Add configs for BlobPathBuilderFactory

* BW-1305 Make "name" optional in workflow query response (broadinstitute#6821)

* BT-724 Fix BlobPathBuilder failing on retrieving existing filesystem (broadinstitute#6816)

Modify blobPathBuilder to fallback to creating a filesystem if one is not found

* Logging updates: (broadinstitute#6813)

* [BT-698] first pass on BlobTokenGenerator with E2E test (broadinstitute#6824)

* first pass on BlobTokenGenerator with E2E test

* update BlobPathBuilder constructor args in test

* account -> container level client

* [BT-687] specify correct types (broadinstitute#6829)

* specify correct types

* fix test with new type

* remove type declarations in function call

* remove unnecessary sas-token config

* BW-1206 - Combine all Wes Endpoints & add Tests (broadinstitute#6833)

* Add tests, getting frid of WesRunRoutes.scala

* wesWorkflowId fix, ec implicits errors gone

* Refactoring path for GET /runs

* Indentation fix

* Commit to rollback

* Revert "Indentation fix"

This reverts commit 63fc484.

* PR trigger

* Optimize imports

* Missed import

* BW-1354 - Porting CBAS preliminary step (broadinstitute#6837)

* Getting rid of shared utility file; Adding/Updating WES version of submit.

* Edit spec file

* Adding Wes-like error

* BW-1378 Addl CromIAM user enablement checks (broadinstitute#6826)

* Update cromwell version from 84 to 85

* BW-1393 Release doc updates (broadinstitute#6839)

* BT-732 Checksum validation for blobs read by engine (broadinstitute#6838)

* Draft support for optional FileHash

* Draft getMd5 for BlobPath

* Resolve non-parallel IO to fix tests

* Checksum validation for BlobPath

* Nicer error message

* Test for missing Blob hash

* Break attr acquisition into separate method

* Cleanup, comments

* In-progress tests of blob hash command

* Remove test

* Remove unused import

* BT-711 Refresh SAS token for filesystem on expiry (broadinstitute#6831)

* BT-711 Refresh SAS token for filesystem on expiry

* Rough cut of token refresh using exceptions

* Ignore tests, and minor cleanup

* Remove stray line

* Draft of manager class for handling expiring file systems

* Style fixes

* Refactor of blobfilesystemManager and tests covering its functionality

* Refined tests to validate close filesystem as separate unit

* Ignore connected tests

* Clean up of some things

* Refactor BlobFileSystemManager to separate file, and some other cleanup

* Some additional scala-ifying

* Small cleanup

* Correcting imports

* trigger tests

* trigger tests

* Batch 1 of scala steward updates (broadinstitute#6903)

* Batch 1 of scala steward updates

* Rollback snakeYAML

* Attempt 3, with only the passing dependancies

* Revert google API and Big Query udpates

* Winding back other google deps

* rollback remaining google updates

* trigger tests

* trigger tests

* [BW-1398] Migrate PKs to BIGINT (broadinstitute#6907)

* BT-745  Batch 2 of scala steward updates (broadinstitute#6906)

* Update SBT to 2.0.0

* Fix sbt-git import

* Update mouse to 1.0.11

* Update rhino 1.7.14

* SUP-692 Retry with more memory after RC 137 (broadinstitute#6912)

* Reorder execution result checks so 137 can retry with more memory

* Test for memory retry after 137 RC

* Fix test expectations

* Make memory retry checks consistent

* Revert changes to existing test

* Rename retryWithMoreMemory to outOfMemoryDetected

* Scala steward updates batch 3 (broadinstitute#6913)

* Scala steward updates batch 3

* WX-745 Batch 4 scala steward updates (broadinstitute#6916)

* WX-746 Localize all DRS inputs in a single Action (broadinstitute#6914)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-755 Build all images instead of just Cromwell (broadinstitute#6919)

* WX-755 Add `isRelease` option for Docker builds (broadinstitute#6923)

* WX-755 Cromwell/CromIAM automatically board train (broadinstitute#6924)

* WX-755 Fix environment variable syntax (broadinstitute#6926)

* WX-743 Enable TES task creation with BlobPaths (broadinstitute#6921)

* Give blob SAS tokens write permission

* Case class wrapper for subscription id

* Resolve duplicate container name in absolute BlobPath

* Ignored test demonstrating correct absolute path generation

* Update filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobPathBuilderSpec.scala

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* PR feedback

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* [WX-765] Update snakeyaml to 1.33 (broadinstitute#6927)

* update snakeyaml to 1.33

* Don't use deprecated no-arg Constructor constructor

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WM-1414 Refactoring WesRunLog to omit Cromwell's "workflowLog" object (broadinstitute#6925)

* Upgrade Postgres to 42.4.1 (broadinstitute#6932)

* WX-735 Fix incorrect and/or nondeterministic filesystem ordering (broadinstitute#6930)

* WX-772 Update Scala to 2.13.9 (broadinstitute#6928)

* Update Scala to 2.13.9

* Try updating sbt-scoverage

* Does this version exist anywhere we can see?

* This version actually exists

* Update library version to remove conflict

* Codegen version

* Fix fun new 2.13.9 compiler errors

* Resolve warnings

* Newest Scala?

* I guess not

* Does this please Travis?

* force ci

* Back out changes to generated code

Co-authored-by: Adam Nichols <aednichols@gmail.com>

* WX-781 Bump jackson-databind in /CromwellRefdiskManifestCreator (broadinstitute#6935)

Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.13.2.2 to 2.13.4.1.
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)

---
updated-dependencies:
- dependency-name: com.fasterxml.jackson.core:jackson-databind
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* WX-808 Host allowlist for HTTP imports (broadinstitute#6938)

* `hostAllowlist` that allows everything

* Refactor

* Stick allow list in HttpResolver

* Better default config

* Allow list tests

* Make it build

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Update commons text to 1.10.0 (broadinstitute#6937)

* WX-751 Token refresh signal for monitoring (broadinstitute#6939)

* Log messages

* `DEBUG` -> `INFO`

* WX-744 Optionally rewrite blob paths to appear as local paths (broadinstitute#6941)

* Modify blob paths for TES

* Make blob transformation configurable

* Update supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Apply PR feedback in second place

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Update changelog for wdl http allow list (broadinstitute#6944)

* WM-1491 Fixing Cromwell-client (broadinstitute#6943)

* More updated client for use in cbas

* Removing excess code

* Fix client build script (broadinstitute#6945)

* WX-837: Remove CWL references from documentation (broadinstitute#6949)

* wx-837 removed cwl references in markdown doc files

* wx-837 removed cwlParsingOverview.md, updated mkdocs.yml

* wx-837 updated cromwell.yaml, generated new RESTAPI file

* WX-728 Add configurable WSM client to Cromwell (broadinstitute#6948)

* Dependencies

* Compiles but no tests

* Formatting

* Moar exclusions

* Update to latest WSM

* Add additional dependency

* We need some UUID here to make the request

* Formatting

* Clarify what is fake

* Formatting

* Use our own version of Jersey and Jackson stuff

* Port-in Khalid's changes (thank you!)

Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* Test longevity

Don't break the test if someone decides to add a cert to `ws.org`

* Cleanup

* Cleanup

* Cleanup

* Adjust TES config file for CI

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* CROM-6554: Removed PAPIv1 references from doc (broadinstitute#6950)

* crom-6554 removed references to PAPI v1 from doc

* crom-6554 pr feedback, reworded doc to use example conf as a starting point

* WX-833 Real Azure DRS Credentials (broadinstitute#6952)

* Remove B2C reference from name

* Get token for current user rather than getting from KeyVault

* Remove KeyVault config for engine

* Remove KeyVault config for DRSLocalizer

* Remove KeyVault dependency

* Remove KeyVault support from localizer repo template

* Cleaned up and working Azure token acquisition for engine

* Collapse localizer's AccessTokenStrategy into DrsCredentials

* Cleanup

* WX-853 Remove most CWL (broadinstitute#6955)

* WX-696 Enable getting SAS token from WSM (broadinstitute#6954)

* WX-696 Enable getting SAS token from WSM

* Wire container resource id from config

* Move resource-container-id config path

* First pass at config for WSM

* Remove unused singleton config

* Tests for new config

* Fix config parsing

* Modified b2c token to be provided each time

* Remove singletonConfig arg from factory

* Restore types to factory configs

* Clean up comments and empty token default

* Default to config b2c before searching environment

* Fix token default on api client

* Fix test

* Refactor error handling for when there is no token

* Remove token constructor arg for clientProvider

* Move configs to global singleton config

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

* default -> override

* Add override token to test

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Parentheses

* Reduce token timeout

* Move AzureCredentials to separate file

* Make AzureCredentials an object

* WSM token cleanup

* Config refactor (broadinstitute#6960)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Initial blob token documentation

* Refine language in BlobSasTokenGenerator

* Update comment and formatting

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-853 Remove CWL language factory, Centaur runner (broadinstitute#6961)

* WX-842 Add Pact Dependency for Cromwell (broadinstitute#6962)

* WX-842 Add Pact Dependency for Cromwell

* Remove incomplete test spec

* Initial Pact Test

* Fix pact so it compiles

* Add breadcrumb comment and clean up

* ID-125 Add support for drshub, rename all the things (broadinstitute#6959)

* Add support for drshub, rename all the things

* fallback to martha if resolver is not in config

* WX-867 Translate crc32c hashes to b64 for getm (broadinstitute#6970)

* Translate crc32c hashes to b64 for getm

* Update tests

* Remove obsolete b64 handling for md5, centralize hex validation

* Restore old test, fix other test

* WX-843 Workflow failure reason should accurately indicate issues opening blob filesystem (broadinstitute#6965)

* WX-859 Accept workflow execution identity in config (broadinstitute#6967)

* WX-892 Trim down `ValueStore` logging to prevent OOMs (broadinstitute#6981)

* Add Nirvana 3.18.1 reference image test, minor cleanup [VS-705] (broadinstitute#6975)

* WX-863 Turn off Azure NIO logging (broadinstitute#6982)

* Turn off Azure NIO logging

* Poke Travis

* WM-1616: Allow repeating attempts at initialization (take 2) (broadinstitute#6985)

* WX-878 Single shared BlobFileSystemManager (broadinstitute#6986)

* Make BlobFileSystemManager shared across all BlobPathBuilders

* Update TES conf file to reflect new singleton config

* Shell escape reference image  files [VS-796] [WX-910] (broadinstitute#6989)

* WX-769 `disks` compatibility for TES backend (broadinstitute#6991)

* Update FiveMinuteIntro.md (broadinstitute#6994)

* WX-906 Sbt Unit Tests as Github Actions (broadinstitute#6992)

* WX-926 Support falling back to OCI Manifest Format (broadinstitute#7003)

* WX-926 Support falling back to OCI Manifest Forma

* Only mount reference disks if requested [WX-925] (broadinstitute#7001)

* [WM-1646] Add missing fields for `WorkflowDescription` for WomTool /describe endpoint to Swagger (broadinstitute#7004)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status (broadinstitute#6980)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status

* Address feedback

* Address feedback (broadinstitute#6997)

* Address additional feedback (broadinstitute#7000)

* Fix copy/paste error (broadinstitute#7005)

* Address additional feedback

* Fix copy/paste error

* Trigger CI

---------

Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Centaur reference image test should validate symlinks [VS-796] (broadinstitute#6996)

* WX-903 Pre-GHA test suite disablement

* WX-877 Update CHANGELOG for release 85 (broadinstitute#7011)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Katrina P <68349264+kpierre13@users.noreply.github.com>
Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com>
Co-authored-by: Christian Freitas <christian.j.freitas@gmail.com>
Co-authored-by: Saloni Shah <salonipshah11@gmail.com>
Co-authored-by: kshakir <kshakir@broadinstitute.org>
Co-authored-by: mspector <michael.l.spector@gmail.com>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>
Co-authored-by: Brian Reilly <breilly@broadinstitute.org>
Co-authored-by: Adam Nichols <aednichols@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Justin Variath Thomas <JVThomas@users.noreply.github.com>
Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org>
Co-authored-by: Trevyn Langsford <tlangs@broadinstitute.org>
Co-authored-by: Miguel Covarrubias <mcovarr@users.noreply.github.com>
Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com>
Co-authored-by: Tom Wiseman <tom.h.wiseman@gmail.com>
Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>

* Develop aws (#29)

* stuck on globbing

* efs works, no callcaching

* update readme

* extended EFS support

* fix for globbing in nested scatters

* updated config for globbing, to prevent issues with empty folders

* efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs (#32)

* Develop aws (#34)

* efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs

* add deployment manual, fix issue with empty disks

* update documentation

* update documentation

* update documentation

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Katrina P <68349264+kpierre13@users.noreply.github.com>
Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com>
Co-authored-by: Christian Freitas <christian.j.freitas@gmail.com>
Co-authored-by: Saloni Shah <salonipshah11@gmail.com>
Co-authored-by: kshakir <kshakir@broadinstitute.org>
Co-authored-by: mspector <michael.l.spector@gmail.com>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>
Co-authored-by: Brian Reilly <breilly@broadinstitute.org>
Co-authored-by: Adam Nichols <aednichols@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Justin Variath Thomas <JVThomas@users.noreply.github.com>
Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org>
Co-authored-by: Trevyn Langsford <tlangs@broadinstitute.org>
Co-authored-by: Miguel Covarrubias <mcovarr@users.noreply.github.com>
Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com>
Co-authored-by: Tom Wiseman <tom.h.wiseman@gmail.com>
Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>
Co-authored-by: geertvandeweyer <geertvandeweyer@gmail.com>
henriqueribeiro added a commit to henriqueribeiro/cromwell that referenced this pull request Jan 8, 2024
@henriqueribeiro @jgainerdewar
@kpierre13 @cjllanwarne @kraefrei @salonishah11 @kshakir @mspector @aednichols @breilly2 @dependabot @JVThomas @tlangs @mcovarr @ekiernan @THWiseman @BMurri @geertvandeweyer @xquek
Develop AWS (#42)
83ca277 
* 85 release (#28)

* Update cromwell version from 83 to 84

* BW-1255 Implement POST /runs endpoint (broadinstitute#6779)

* Adding route

* Fixing HTTP method error

* All formFields made optional

* A compliling state

* Saving

* Saving

* All three endpoints functioning as expected; updated RESTAPI.md

* Updated response for submission from 200 to 201 to pass tests

* Test submission response

* Moved updated submission response to askSubmit

* test

* updating RESTAPI.md

* saving

* Adding utility file for submitRequest

* cleanup

* Update awssdkv from 2.17.152 to 2.17.194 (broadinstitute#6814)

* BW-1305 Swagger Update (broadinstitute#6818)

* Properly documenting metadataArchiveStatus in WorkflowQueryResult model

* Update docs

* BT-710 Add configs for BlobPathBuilderFactory (broadinstitute#6817)

BT-710 Add configs for BlobPathBuilderFactory

* BW-1305 Make "name" optional in workflow query response (broadinstitute#6821)

* BT-724 Fix BlobPathBuilder failing on retrieving existing filesystem (broadinstitute#6816)

Modify blobPathBuilder to fallback to creating a filesystem if one is not found

* Logging updates: (broadinstitute#6813)

* [BT-698] first pass on BlobTokenGenerator with E2E test (broadinstitute#6824)

* first pass on BlobTokenGenerator with E2E test

* update BlobPathBuilder constructor args in test

* account -> container level client

* [BT-687] specify correct types (broadinstitute#6829)

* specify correct types

* fix test with new type

* remove type declarations in function call

* remove unnecessary sas-token config

* BW-1206 - Combine all Wes Endpoints & add Tests (broadinstitute#6833)

* Add tests, getting frid of WesRunRoutes.scala

* wesWorkflowId fix, ec implicits errors gone

* Refactoring path for GET /runs

* Indentation fix

* Commit to rollback

* Revert "Indentation fix"

This reverts commit 63fc484.

* PR trigger

* Optimize imports

* Missed import

* BW-1354 - Porting CBAS preliminary step (broadinstitute#6837)

* Getting rid of shared utility file; Adding/Updating WES version of submit.

* Edit spec file

* Adding Wes-like error

* BW-1378 Addl CromIAM user enablement checks (broadinstitute#6826)

* Update cromwell version from 84 to 85

* BW-1393 Release doc updates (broadinstitute#6839)

* BT-732 Checksum validation for blobs read by engine (broadinstitute#6838)

* Draft support for optional FileHash

* Draft getMd5 for BlobPath

* Resolve non-parallel IO to fix tests

* Checksum validation for BlobPath

* Nicer error message

* Test for missing Blob hash

* Break attr acquisition into separate method

* Cleanup, comments

* In-progress tests of blob hash command

* Remove test

* Remove unused import

* BT-711 Refresh SAS token for filesystem on expiry (broadinstitute#6831)

* BT-711 Refresh SAS token for filesystem on expiry

* Rough cut of token refresh using exceptions

* Ignore tests, and minor cleanup

* Remove stray line

* Draft of manager class for handling expiring file systems

* Style fixes

* Refactor of blobfilesystemManager and tests covering its functionality

* Refined tests to validate close filesystem as separate unit

* Ignore connected tests

* Clean up of some things

* Refactor BlobFileSystemManager to separate file, and some other cleanup

* Some additional scala-ifying

* Small cleanup

* Correcting imports

* trigger tests

* trigger tests

* Batch 1 of scala steward updates (broadinstitute#6903)

* Batch 1 of scala steward updates

* Rollback snakeYAML

* Attempt 3, with only the passing dependancies

* Revert google API and Big Query udpates

* Winding back other google deps

* rollback remaining google updates

* trigger tests

* trigger tests

* [BW-1398] Migrate PKs to BIGINT (broadinstitute#6907)

* BT-745  Batch 2 of scala steward updates (broadinstitute#6906)

* Update SBT to 2.0.0

* Fix sbt-git import

* Update mouse to 1.0.11

* Update rhino 1.7.14

* SUP-692 Retry with more memory after RC 137 (broadinstitute#6912)

* Reorder execution result checks so 137 can retry with more memory

* Test for memory retry after 137 RC

* Fix test expectations

* Make memory retry checks consistent

* Revert changes to existing test

* Rename retryWithMoreMemory to outOfMemoryDetected

* Scala steward updates batch 3 (broadinstitute#6913)

* Scala steward updates batch 3

* WX-745 Batch 4 scala steward updates (broadinstitute#6916)

* WX-746 Localize all DRS inputs in a single Action (broadinstitute#6914)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-755 Build all images instead of just Cromwell (broadinstitute#6919)

* WX-755 Add `isRelease` option for Docker builds (broadinstitute#6923)

* WX-755 Cromwell/CromIAM automatically board train (broadinstitute#6924)

* WX-755 Fix environment variable syntax (broadinstitute#6926)

* WX-743 Enable TES task creation with BlobPaths (broadinstitute#6921)

* Give blob SAS tokens write permission

* Case class wrapper for subscription id

* Resolve duplicate container name in absolute BlobPath

* Ignored test demonstrating correct absolute path generation

* Update filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobPathBuilderSpec.scala

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* PR feedback

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* [WX-765] Update snakeyaml to 1.33 (broadinstitute#6927)

* update snakeyaml to 1.33

* Don't use deprecated no-arg Constructor constructor

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WM-1414 Refactoring WesRunLog to omit Cromwell's "workflowLog" object (broadinstitute#6925)

* Upgrade Postgres to 42.4.1 (broadinstitute#6932)

* WX-735 Fix incorrect and/or nondeterministic filesystem ordering (broadinstitute#6930)

* WX-772 Update Scala to 2.13.9 (broadinstitute#6928)

* Update Scala to 2.13.9

* Try updating sbt-scoverage

* Does this version exist anywhere we can see?

* This version actually exists

* Update library version to remove conflict

* Codegen version

* Fix fun new 2.13.9 compiler errors

* Resolve warnings

* Newest Scala?

* I guess not

* Does this please Travis?

* force ci

* Back out changes to generated code

Co-authored-by: Adam Nichols <aednichols@gmail.com>

* WX-781 Bump jackson-databind in /CromwellRefdiskManifestCreator (broadinstitute#6935)

Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.13.2.2 to 2.13.4.1.
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)

---
updated-dependencies:
- dependency-name: com.fasterxml.jackson.core:jackson-databind
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* WX-808 Host allowlist for HTTP imports (broadinstitute#6938)

* `hostAllowlist` that allows everything

* Refactor

* Stick allow list in HttpResolver

* Better default config

* Allow list tests

* Make it build

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Update commons text to 1.10.0 (broadinstitute#6937)

* WX-751 Token refresh signal for monitoring (broadinstitute#6939)

* Log messages

* `DEBUG` -> `INFO`

* WX-744 Optionally rewrite blob paths to appear as local paths (broadinstitute#6941)

* Modify blob paths for TES

* Make blob transformation configurable

* Update supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Apply PR feedback in second place

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Update changelog for wdl http allow list (broadinstitute#6944)

* WM-1491 Fixing Cromwell-client (broadinstitute#6943)

* More updated client for use in cbas

* Removing excess code

* Fix client build script (broadinstitute#6945)

* WX-837: Remove CWL references from documentation (broadinstitute#6949)

* wx-837 removed cwl references in markdown doc files

* wx-837 removed cwlParsingOverview.md, updated mkdocs.yml

* wx-837 updated cromwell.yaml, generated new RESTAPI file

* WX-728 Add configurable WSM client to Cromwell (broadinstitute#6948)

* Dependencies

* Compiles but no tests

* Formatting

* Moar exclusions

* Update to latest WSM

* Add additional dependency

* We need some UUID here to make the request

* Formatting

* Clarify what is fake

* Formatting

* Use our own version of Jersey and Jackson stuff

* Port-in Khalid's changes (thank you!)

Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* Test longevity

Don't break the test if someone decides to add a cert to `ws.org`

* Cleanup

* Cleanup

* Cleanup

* Adjust TES config file for CI

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* CROM-6554: Removed PAPIv1 references from doc (broadinstitute#6950)

* crom-6554 removed references to PAPI v1 from doc

* crom-6554 pr feedback, reworded doc to use example conf as a starting point

* WX-833 Real Azure DRS Credentials (broadinstitute#6952)

* Remove B2C reference from name

* Get token for current user rather than getting from KeyVault

* Remove KeyVault config for engine

* Remove KeyVault config for DRSLocalizer

* Remove KeyVault dependency

* Remove KeyVault support from localizer repo template

* Cleaned up and working Azure token acquisition for engine

* Collapse localizer's AccessTokenStrategy into DrsCredentials

* Cleanup

* WX-853 Remove most CWL (broadinstitute#6955)

* WX-696 Enable getting SAS token from WSM (broadinstitute#6954)

* WX-696 Enable getting SAS token from WSM

* Wire container resource id from config

* Move resource-container-id config path

* First pass at config for WSM

* Remove unused singleton config

* Tests for new config

* Fix config parsing

* Modified b2c token to be provided each time

* Remove singletonConfig arg from factory

* Restore types to factory configs

* Clean up comments and empty token default

* Default to config b2c before searching environment

* Fix token default on api client

* Fix test

* Refactor error handling for when there is no token

* Remove token constructor arg for clientProvider

* Move configs to global singleton config

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

* default -> override

* Add override token to test

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Parentheses

* Reduce token timeout

* Move AzureCredentials to separate file

* Make AzureCredentials an object

* WSM token cleanup

* Config refactor (broadinstitute#6960)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Initial blob token documentation

* Refine language in BlobSasTokenGenerator

* Update comment and formatting

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-853 Remove CWL language factory, Centaur runner (broadinstitute#6961)

* WX-842 Add Pact Dependency for Cromwell (broadinstitute#6962)

* WX-842 Add Pact Dependency for Cromwell

* Remove incomplete test spec

* Initial Pact Test

* Fix pact so it compiles

* Add breadcrumb comment and clean up

* ID-125 Add support for drshub, rename all the things (broadinstitute#6959)

* Add support for drshub, rename all the things

* fallback to martha if resolver is not in config

* WX-867 Translate crc32c hashes to b64 for getm (broadinstitute#6970)

* Translate crc32c hashes to b64 for getm

* Update tests

* Remove obsolete b64 handling for md5, centralize hex validation

* Restore old test, fix other test

* WX-843 Workflow failure reason should accurately indicate issues opening blob filesystem (broadinstitute#6965)

* WX-859 Accept workflow execution identity in config (broadinstitute#6967)

* WX-892 Trim down `ValueStore` logging to prevent OOMs (broadinstitute#6981)

* Add Nirvana 3.18.1 reference image test, minor cleanup [VS-705] (broadinstitute#6975)

* WX-863 Turn off Azure NIO logging (broadinstitute#6982)

* Turn off Azure NIO logging

* Poke Travis

* WM-1616: Allow repeating attempts at initialization (take 2) (broadinstitute#6985)

* WX-878 Single shared BlobFileSystemManager (broadinstitute#6986)

* Make BlobFileSystemManager shared across all BlobPathBuilders

* Update TES conf file to reflect new singleton config

* Shell escape reference image  files [VS-796] [WX-910] (broadinstitute#6989)

* WX-769 `disks` compatibility for TES backend (broadinstitute#6991)

* Update FiveMinuteIntro.md (broadinstitute#6994)

* WX-906 Sbt Unit Tests as Github Actions (broadinstitute#6992)

* WX-926 Support falling back to OCI Manifest Format (broadinstitute#7003)

* WX-926 Support falling back to OCI Manifest Forma

* Only mount reference disks if requested [WX-925] (broadinstitute#7001)

* [WM-1646] Add missing fields for `WorkflowDescription` for WomTool /describe endpoint to Swagger (broadinstitute#7004)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status (broadinstitute#6980)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status

* Address feedback

* Address feedback (broadinstitute#6997)

* Address additional feedback (broadinstitute#7000)

* Fix copy/paste error (broadinstitute#7005)

* Address additional feedback

* Fix copy/paste error

* Trigger CI

---------

Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Centaur reference image test should validate symlinks [VS-796] (broadinstitute#6996)

* WX-903 Pre-GHA test suite disablement

* WX-877 Update CHANGELOG for release 85 (broadinstitute#7011)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Katrina P <68349264+kpierre13@users.noreply.github.com>
Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com>
Co-authored-by: Christian Freitas <christian.j.freitas@gmail.com>
Co-authored-by: Saloni Shah <salonipshah11@gmail.com>
Co-authored-by: kshakir <kshakir@broadinstitute.org>
Co-authored-by: mspector <michael.l.spector@gmail.com>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>
Co-authored-by: Brian Reilly <breilly@broadinstitute.org>
Co-authored-by: Adam Nichols <aednichols@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Justin Variath Thomas <JVThomas@users.noreply.github.com>
Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org>
Co-authored-by: Trevyn Langsford <tlangs@broadinstitute.org>
Co-authored-by: Miguel Covarrubias <mcovarr@users.noreply.github.com>
Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com>
Co-authored-by: Tom Wiseman <tom.h.wiseman@gmail.com>
Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>

* Develop aws (#29)

* stuck on globbing

* efs works, no callcaching

* update readme

* extended EFS support

* fix for globbing in nested scatters

* updated config for globbing, to prevent issues with empty folders

* efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs (#32)

* Develop aws (#34)

* efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs

* add deployment manual, fix issue with empty disks

* update documentation

* update documentation

* update documentation

* Options to publish status only (#36)

* add options to publish status only

* updated readme.md

---------

Co-authored-by: quekx <quekx@gene.com>

* Fix aws unit tests (#39)

* checkpoint

* fix ecr and batch tests

* fix AwsBatchJobSpec.scala

---------

Co-authored-by: quekx <quekx@gene.com>

* return bucket directly instead of listing and checking it (#38)

Co-authored-by: quekx <quekx@gene.com>

* Add evaluteOnExit for aws batch retry (#40)

Co-authored-by: quekx <quekx@gene.com>

* Improved tagging support (#37)

* efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs

* add deployment manual, fix issue with empty disks

* update documentation

* update documentation

* update documentation

* support for tagging instances and volumes used in jobs, some support for spaces in file paths

* corrected workflow id in tagging

* redirect exit code 137 to retry-with-more-memory routine

---------

Co-authored-by: Henrique Ribeiro <henriqueribeiro@users.noreply.github.com>

* add gpu count (#41)

* add gpu count

* fix typo

---------

Co-authored-by: quekx <quekx@gene.com>
Co-authored-by: Henrique Ribeiro <henriqueribeiro@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Katrina P <68349264+kpierre13@users.noreply.github.com>
Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com>
Co-authored-by: Christian Freitas <christian.j.freitas@gmail.com>
Co-authored-by: Saloni Shah <salonipshah11@gmail.com>
Co-authored-by: kshakir <kshakir@broadinstitute.org>
Co-authored-by: mspector <michael.l.spector@gmail.com>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>
Co-authored-by: Brian Reilly <breilly@broadinstitute.org>
Co-authored-by: Adam Nichols <aednichols@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Justin Variath Thomas <JVThomas@users.noreply.github.com>
Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org>
Co-authored-by: Trevyn Langsford <tlangs@broadinstitute.org>
Co-authored-by: Miguel Covarrubias <mcovarr@users.noreply.github.com>
Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com>
Co-authored-by: Tom Wiseman <tom.h.wiseman@gmail.com>
Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>
Co-authored-by: geertvandeweyer <geertvandeweyer@gmail.com>
Co-authored-by: xquek <46759870+xquek@users.noreply.github.com>
Co-authored-by: quekx <quekx@gene.com>
henriqueribeiro added a commit to henriqueribeiro/cromwell that referenced this pull request Nov 6, 2024
@geertvandeweyer @jgainerdewar
@kraefrei @salonishah11 @breilly2 @aednichols @kpierre13 @dependabot @cjllanwarne @JVThomas @kshakir @tlangs @mcovarr @ekiernan @THWiseman @BMurri @sjfleming @henriqueribeiro @mspector @Ghost-in-a-Jar @ichengchang @dspeck1 @jarroyo28 @jakejennings @dillydally414 @dvoet @xquek @broadbot @beichen1024 @zykonda @rsaperst @jlester-msft @AlexITC @dkj
Develop (#52)
9d22c1a 
* BT-732 Checksum validation for blobs read by engine (broadinstitute#6838)

* Draft support for optional FileHash

* Draft getMd5 for BlobPath

* Resolve non-parallel IO to fix tests

* Checksum validation for BlobPath

* Nicer error message

* Test for missing Blob hash

* Break attr acquisition into separate method

* Cleanup, comments

* In-progress tests of blob hash command

* Remove test

* Remove unused import

* BT-711 Refresh SAS token for filesystem on expiry (broadinstitute#6831)

* BT-711 Refresh SAS token for filesystem on expiry

* Rough cut of token refresh using exceptions

* Ignore tests, and minor cleanup

* Remove stray line

* Draft of manager class for handling expiring file systems

* Style fixes

* Refactor of blobfilesystemManager and tests covering its functionality

* Refined tests to validate close filesystem as separate unit

* Ignore connected tests

* Clean up of some things

* Refactor BlobFileSystemManager to separate file, and some other cleanup

* Some additional scala-ifying

* Small cleanup

* Correcting imports

* trigger tests

* trigger tests

* Batch 1 of scala steward updates (broadinstitute#6903)

* Batch 1 of scala steward updates

* Rollback snakeYAML

* Attempt 3, with only the passing dependancies

* Revert google API and Big Query udpates

* Winding back other google deps

* rollback remaining google updates

* trigger tests

* trigger tests

* [BW-1398] Migrate PKs to BIGINT (broadinstitute#6907)

* BT-745  Batch 2 of scala steward updates (broadinstitute#6906)

* Update SBT to 2.0.0

* Fix sbt-git import

* Update mouse to 1.0.11

* Update rhino 1.7.14

* SUP-692 Retry with more memory after RC 137 (broadinstitute#6912)

* Reorder execution result checks so 137 can retry with more memory

* Test for memory retry after 137 RC

* Fix test expectations

* Make memory retry checks consistent

* Revert changes to existing test

* Rename retryWithMoreMemory to outOfMemoryDetected

* Scala steward updates batch 3 (broadinstitute#6913)

* Scala steward updates batch 3

* WX-745 Batch 4 scala steward updates (broadinstitute#6916)

* WX-746 Localize all DRS inputs in a single Action (broadinstitute#6914)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-755 Build all images instead of just Cromwell (broadinstitute#6919)

* WX-755 Add `isRelease` option for Docker builds (broadinstitute#6923)

* WX-755 Cromwell/CromIAM automatically board train (broadinstitute#6924)

* WX-755 Fix environment variable syntax (broadinstitute#6926)

* WX-743 Enable TES task creation with BlobPaths (broadinstitute#6921)

* Give blob SAS tokens write permission

* Case class wrapper for subscription id

* Resolve duplicate container name in absolute BlobPath

* Ignored test demonstrating correct absolute path generation

* Update filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobPathBuilderSpec.scala

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* PR feedback

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* [WX-765] Update snakeyaml to 1.33 (broadinstitute#6927)

* update snakeyaml to 1.33

* Don't use deprecated no-arg Constructor constructor

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WM-1414 Refactoring WesRunLog to omit Cromwell's "workflowLog" object (broadinstitute#6925)

* Upgrade Postgres to 42.4.1 (broadinstitute#6932)

* WX-735 Fix incorrect and/or nondeterministic filesystem ordering (broadinstitute#6930)

* WX-772 Update Scala to 2.13.9 (broadinstitute#6928)

* Update Scala to 2.13.9

* Try updating sbt-scoverage

* Does this version exist anywhere we can see?

* This version actually exists

* Update library version to remove conflict

* Codegen version

* Fix fun new 2.13.9 compiler errors

* Resolve warnings

* Newest Scala?

* I guess not

* Does this please Travis?

* force ci

* Back out changes to generated code

Co-authored-by: Adam Nichols <aednichols@gmail.com>

* WX-781 Bump jackson-databind in /CromwellRefdiskManifestCreator (broadinstitute#6935)

Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.13.2.2 to 2.13.4.1.
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)

---
updated-dependencies:
- dependency-name: com.fasterxml.jackson.core:jackson-databind
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* WX-808 Host allowlist for HTTP imports (broadinstitute#6938)

* `hostAllowlist` that allows everything

* Refactor

* Stick allow list in HttpResolver

* Better default config

* Allow list tests

* Make it build

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Update commons text to 1.10.0 (broadinstitute#6937)

* WX-751 Token refresh signal for monitoring (broadinstitute#6939)

* Log messages

* `DEBUG` -> `INFO`

* WX-744 Optionally rewrite blob paths to appear as local paths (broadinstitute#6941)

* Modify blob paths for TES

* Make blob transformation configurable

* Update supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Apply PR feedback in second place

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Update changelog for wdl http allow list (broadinstitute#6944)

* WM-1491 Fixing Cromwell-client (broadinstitute#6943)

* More updated client for use in cbas

* Removing excess code

* Fix client build script (broadinstitute#6945)

* WX-837: Remove CWL references from documentation (broadinstitute#6949)

* wx-837 removed cwl references in markdown doc files

* wx-837 removed cwlParsingOverview.md, updated mkdocs.yml

* wx-837 updated cromwell.yaml, generated new RESTAPI file

* WX-728 Add configurable WSM client to Cromwell (broadinstitute#6948)

* Dependencies

* Compiles but no tests

* Formatting

* Moar exclusions

* Update to latest WSM

* Add additional dependency

* We need some UUID here to make the request

* Formatting

* Clarify what is fake

* Formatting

* Use our own version of Jersey and Jackson stuff

* Port-in Khalid's changes (thank you!)

Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* Test longevity

Don't break the test if someone decides to add a cert to `ws.org`

* Cleanup

* Cleanup

* Cleanup

* Adjust TES config file for CI

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* CROM-6554: Removed PAPIv1 references from doc (broadinstitute#6950)

* crom-6554 removed references to PAPI v1 from doc

* crom-6554 pr feedback, reworded doc to use example conf as a starting point

* WX-833 Real Azure DRS Credentials (broadinstitute#6952)

* Remove B2C reference from name

* Get token for current user rather than getting from KeyVault

* Remove KeyVault config for engine

* Remove KeyVault config for DRSLocalizer

* Remove KeyVault dependency

* Remove KeyVault support from localizer repo template

* Cleaned up and working Azure token acquisition for engine

* Collapse localizer's AccessTokenStrategy into DrsCredentials

* Cleanup

* WX-853 Remove most CWL (broadinstitute#6955)

* WX-696 Enable getting SAS token from WSM (broadinstitute#6954)

* WX-696 Enable getting SAS token from WSM

* Wire container resource id from config

* Move resource-container-id config path

* First pass at config for WSM

* Remove unused singleton config

* Tests for new config

* Fix config parsing

* Modified b2c token to be provided each time

* Remove singletonConfig arg from factory

* Restore types to factory configs

* Clean up comments and empty token default

* Default to config b2c before searching environment

* Fix token default on api client

* Fix test

* Refactor error handling for when there is no token

* Remove token constructor arg for clientProvider

* Move configs to global singleton config

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

* default -> override

* Add override token to test

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Parentheses

* Reduce token timeout

* Move AzureCredentials to separate file

* Make AzureCredentials an object

* WSM token cleanup

* Config refactor (broadinstitute#6960)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Initial blob token documentation

* Refine language in BlobSasTokenGenerator

* Update comment and formatting

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-853 Remove CWL language factory, Centaur runner (broadinstitute#6961)

* WX-842 Add Pact Dependency for Cromwell (broadinstitute#6962)

* WX-842 Add Pact Dependency for Cromwell

* Remove incomplete test spec

* Initial Pact Test

* Fix pact so it compiles

* Add breadcrumb comment and clean up

* ID-125 Add support for drshub, rename all the things (broadinstitute#6959)

* Add support for drshub, rename all the things

* fallback to martha if resolver is not in config

* WX-867 Translate crc32c hashes to b64 for getm (broadinstitute#6970)

* Translate crc32c hashes to b64 for getm

* Update tests

* Remove obsolete b64 handling for md5, centralize hex validation

* Restore old test, fix other test

* WX-843 Workflow failure reason should accurately indicate issues opening blob filesystem (broadinstitute#6965)

* WX-859 Accept workflow execution identity in config (broadinstitute#6967)

* WX-892 Trim down `ValueStore` logging to prevent OOMs (broadinstitute#6981)

* Add Nirvana 3.18.1 reference image test, minor cleanup [VS-705] (broadinstitute#6975)

* WX-863 Turn off Azure NIO logging (broadinstitute#6982)

* Turn off Azure NIO logging

* Poke Travis

* WM-1616: Allow repeating attempts at initialization (take 2) (broadinstitute#6985)

* WX-878 Single shared BlobFileSystemManager (broadinstitute#6986)

* Make BlobFileSystemManager shared across all BlobPathBuilders

* Update TES conf file to reflect new singleton config

* Shell escape reference image  files [VS-796] [WX-910] (broadinstitute#6989)

* WX-769 `disks` compatibility for TES backend (broadinstitute#6991)

* Update FiveMinuteIntro.md (broadinstitute#6994)

* WX-906 Sbt Unit Tests as Github Actions (broadinstitute#6992)

* WX-926 Support falling back to OCI Manifest Format (broadinstitute#7003)

* WX-926 Support falling back to OCI Manifest Forma

* Only mount reference disks if requested [WX-925] (broadinstitute#7001)

* [WM-1646] Add missing fields for `WorkflowDescription` for WomTool /describe endpoint to Swagger (broadinstitute#7004)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status (broadinstitute#6980)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status

* Address feedback

* Address feedback (broadinstitute#6997)

* Address additional feedback (broadinstitute#7000)

* Fix copy/paste error (broadinstitute#7005)

* Address additional feedback

* Fix copy/paste error

* Trigger CI

---------

Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Centaur reference image test should validate symlinks [VS-796] (broadinstitute#6996)

* WX-903 Pre-GHA test suite disablement

* WX-877 Update CHANGELOG for release 85 (broadinstitute#7011)

* Update cromwell version from 85 to 86

* WX-905 (broadinstitute#7012)

Co-authored-by: Thomas Wiseman <twiseman@wm275-315.lan>
Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-719 modernize `cromwell-publish` image (broadinstitute#7013)

* WX-930 Add `CODEOWNERS` file (broadinstitute#7015)

* Update FiveMinuteIntro: Java 11 requirement (broadinstitute#6830)

Co-authored-by: Adam Nichols <aednichols@gmail.com>

* WX-952 Fix Cromwell version update in Helm chart (broadinstitute#7014)

* Reference Disk Manifest Builder App Test (broadinstitute#7017)

* [WM-1696] Update Womtool (broadinstitute#7019)

Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org>

* WX-958 write_map() should write its last entry with a newline (broadinstitute#7022)

* Centaur slurm (broadinstitute#7083)

* Remove Deploy Key (broadinstitute#7084)

* WX-950 Set user agent in Java client (broadinstitute#7087)

* WX-950 Upgrade Azure libs to probably fix extraneous log (broadinstitute#7088)

* WX-984 Revert "WX-950 Upgrade Azure libs to probably fix extraneous log" (broadinstitute#7090)

* Add cron run integrations each day of the week (broadinstitute#7089)

* 85 release (#28)

* Update cromwell version from 83 to 84

* BW-1255 Implement POST /runs endpoint (broadinstitute#6779)

* Adding route

* Fixing HTTP method error

* All formFields made optional

* A compliling state

* Saving

* Saving

* All three endpoints functioning as expected; updated RESTAPI.md

* Updated response for submission from 200 to 201 to pass tests

* Test submission response

* Moved updated submission response to askSubmit

* test

* updating RESTAPI.md

* saving

* Adding utility file for submitRequest

* cleanup

* Update awssdkv from 2.17.152 to 2.17.194 (broadinstitute#6814)

* BW-1305 Swagger Update (broadinstitute#6818)

* Properly documenting metadataArchiveStatus in WorkflowQueryResult model

* Update docs

* BT-710 Add configs for BlobPathBuilderFactory (broadinstitute#6817)

BT-710 Add configs for BlobPathBuilderFactory

* BW-1305 Make "name" optional in workflow query response (broadinstitute#6821)

* BT-724 Fix BlobPathBuilder failing on retrieving existing filesystem (broadinstitute#6816)

Modify blobPathBuilder to fallback to creating a filesystem if one is not found

* Logging updates: (broadinstitute#6813)

* [BT-698] first pass on BlobTokenGenerator with E2E test (broadinstitute#6824)

* first pass on BlobTokenGenerator with E2E test

* update BlobPathBuilder constructor args in test

* account -> container level client

* [BT-687] specify correct types (broadinstitute#6829)

* specify correct types

* fix test with new type

* remove type declarations in function call

* remove unnecessary sas-token config

* BW-1206 - Combine all Wes Endpoints & add Tests (broadinstitute#6833)

* Add tests, getting frid of WesRunRoutes.scala

* wesWorkflowId fix, ec implicits errors gone

* Refactoring path for GET /runs

* Indentation fix

* Commit to rollback

* Revert "Indentation fix"

This reverts commit 63fc484.

* PR trigger

* Optimize imports

* Missed import

* BW-1354 - Porting CBAS preliminary step (broadinstitute#6837)

* Getting rid of shared utility file; Adding/Updating WES version of submit.

* Edit spec file

* Adding Wes-like error

* BW-1378 Addl CromIAM user enablement checks (broadinstitute#6826)

* Update cromwell version from 84 to 85

* BW-1393 Release doc updates (broadinstitute#6839)

* BT-732 Checksum validation for blobs read by engine (broadinstitute#6838)

* Draft support for optional FileHash

* Draft getMd5 for BlobPath

* Resolve non-parallel IO to fix tests

* Checksum validation for BlobPath

* Nicer error message

* Test for missing Blob hash

* Break attr acquisition into separate method

* Cleanup, comments

* In-progress tests of blob hash command

* Remove test

* Remove unused import

* BT-711 Refresh SAS token for filesystem on expiry (broadinstitute#6831)

* BT-711 Refresh SAS token for filesystem on expiry

* Rough cut of token refresh using exceptions

* Ignore tests, and minor cleanup

* Remove stray line

* Draft of manager class for handling expiring file systems

* Style fixes

* Refactor of blobfilesystemManager and tests covering its functionality

* Refined tests to validate close filesystem as separate unit

* Ignore connected tests

* Clean up of some things

* Refactor BlobFileSystemManager to separate file, and some other cleanup

* Some additional scala-ifying

* Small cleanup

* Correcting imports

* trigger tests

* trigger tests

* Batch 1 of scala steward updates (broadinstitute#6903)

* Batch 1 of scala steward updates

* Rollback snakeYAML

* Attempt 3, with only the passing dependancies

* Revert google API and Big Query udpates

* Winding back other google deps

* rollback remaining google updates

* trigger tests

* trigger tests

* [BW-1398] Migrate PKs to BIGINT (broadinstitute#6907)

* BT-745  Batch 2 of scala steward updates (broadinstitute#6906)

* Update SBT to 2.0.0

* Fix sbt-git import

* Update mouse to 1.0.11

* Update rhino 1.7.14

* SUP-692 Retry with more memory after RC 137 (broadinstitute#6912)

* Reorder execution result checks so 137 can retry with more memory

* Test for memory retry after 137 RC

* Fix test expectations

* Make memory retry checks consistent

* Revert changes to existing test

* Rename retryWithMoreMemory to outOfMemoryDetected

* Scala steward updates batch 3 (broadinstitute#6913)

* Scala steward updates batch 3

* WX-745 Batch 4 scala steward updates (broadinstitute#6916)

* WX-746 Localize all DRS inputs in a single Action (broadinstitute#6914)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-755 Build all images instead of just Cromwell (broadinstitute#6919)

* WX-755 Add `isRelease` option for Docker builds (broadinstitute#6923)

* WX-755 Cromwell/CromIAM automatically board train (broadinstitute#6924)

* WX-755 Fix environment variable syntax (broadinstitute#6926)

* WX-743 Enable TES task creation with BlobPaths (broadinstitute#6921)

* Give blob SAS tokens write permission

* Case class wrapper for subscription id

* Resolve duplicate container name in absolute BlobPath

* Ignored test demonstrating correct absolute path generation

* Update filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobPathBuilderSpec.scala

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* PR feedback

Co-authored-by: Brian Reilly <breilly@broadinstitute.org>

* [WX-765] Update snakeyaml to 1.33 (broadinstitute#6927)

* update snakeyaml to 1.33

* Don't use deprecated no-arg Constructor constructor

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WM-1414 Refactoring WesRunLog to omit Cromwell's "workflowLog" object (broadinstitute#6925)

* Upgrade Postgres to 42.4.1 (broadinstitute#6932)

* WX-735 Fix incorrect and/or nondeterministic filesystem ordering (broadinstitute#6930)

* WX-772 Update Scala to 2.13.9 (broadinstitute#6928)

* Update Scala to 2.13.9

* Try updating sbt-scoverage

* Does this version exist anywhere we can see?

* This version actually exists

* Update library version to remove conflict

* Codegen version

* Fix fun new 2.13.9 compiler errors

* Resolve warnings

* Newest Scala?

* I guess not

* Does this please Travis?

* force ci

* Back out changes to generated code

Co-authored-by: Adam Nichols <aednichols@gmail.com>

* WX-781 Bump jackson-databind in /CromwellRefdiskManifestCreator (broadinstitute#6935)

Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.13.2.2 to 2.13.4.1.
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)

---
updated-dependencies:
- dependency-name: com.fasterxml.jackson.core:jackson-databind
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* WX-808 Host allowlist for HTTP imports (broadinstitute#6938)

* `hostAllowlist` that allows everything

* Refactor

* Stick allow list in HttpResolver

* Better default config

* Allow list tests

* Make it build

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Update commons text to 1.10.0 (broadinstitute#6937)

* WX-751 Token refresh signal for monitoring (broadinstitute#6939)

* Log messages

* `DEBUG` -> `INFO`

* WX-744 Optionally rewrite blob paths to appear as local paths (broadinstitute#6941)

* Modify blob paths for TES

* Make blob transformation configurable

* Update supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Apply PR feedback in second place

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Update changelog for wdl http allow list (broadinstitute#6944)

* WM-1491 Fixing Cromwell-client (broadinstitute#6943)

* More updated client for use in cbas

* Removing excess code

* Fix client build script (broadinstitute#6945)

* WX-837: Remove CWL references from documentation (broadinstitute#6949)

* wx-837 removed cwl references in markdown doc files

* wx-837 removed cwlParsingOverview.md, updated mkdocs.yml

* wx-837 updated cromwell.yaml, generated new RESTAPI file

* WX-728 Add configurable WSM client to Cromwell (broadinstitute#6948)

* Dependencies

* Compiles but no tests

* Formatting

* Moar exclusions

* Update to latest WSM

* Add additional dependency

* We need some UUID here to make the request

* Formatting

* Clarify what is fake

* Formatting

* Use our own version of Jersey and Jackson stuff

* Port-in Khalid's changes (thank you!)

Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* Test longevity

Don't break the test if someone decides to add a cert to `ws.org`

* Cleanup

* Cleanup

* Cleanup

* Adjust TES config file for CI

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>

* CROM-6554: Removed PAPIv1 references from doc (broadinstitute#6950)

* crom-6554 removed references to PAPI v1 from doc

* crom-6554 pr feedback, reworded doc to use example conf as a starting point

* WX-833 Real Azure DRS Credentials (broadinstitute#6952)

* Remove B2C reference from name

* Get token for current user rather than getting from KeyVault

* Remove KeyVault config for engine

* Remove KeyVault config for DRSLocalizer

* Remove KeyVault dependency

* Remove KeyVault support from localizer repo template

* Cleaned up and working Azure token acquisition for engine

* Collapse localizer's AccessTokenStrategy into DrsCredentials

* Cleanup

* WX-853 Remove most CWL (broadinstitute#6955)

* WX-696 Enable getting SAS token from WSM (broadinstitute#6954)

* WX-696 Enable getting SAS token from WSM

* Wire container resource id from config

* Move resource-container-id config path

* First pass at config for WSM

* Remove unused singleton config

* Tests for new config

* Fix config parsing

* Modified b2c token to be provided each time

* Remove singletonConfig arg from factory

* Restore types to factory configs

* Clean up comments and empty token default

* Default to config b2c before searching environment

* Fix token default on api client

* Fix test

* Refactor error handling for when there is no token

* Remove token constructor arg for clientProvider

* Move configs to global singleton config

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

* default -> override

* Add override token to test

* Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Parentheses

* Reduce token timeout

* Move AzureCredentials to separate file

* Make AzureCredentials an object

* WSM token cleanup

* Config refactor (broadinstitute#6960)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Initial blob token documentation

* Refine language in BlobSasTokenGenerator

* Update comment and formatting

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-853 Remove CWL language factory, Centaur runner (broadinstitute#6961)

* WX-842 Add Pact Dependency for Cromwell (broadinstitute#6962)

* WX-842 Add Pact Dependency for Cromwell

* Remove incomplete test spec

* Initial Pact Test

* Fix pact so it compiles

* Add breadcrumb comment and clean up

* ID-125 Add support for drshub, rename all the things (broadinstitute#6959)

* Add support for drshub, rename all the things

* fallback to martha if resolver is not in config

* WX-867 Translate crc32c hashes to b64 for getm (broadinstitute#6970)

* Translate crc32c hashes to b64 for getm

* Update tests

* Remove obsolete b64 handling for md5, centralize hex validation

* Restore old test, fix other test

* WX-843 Workflow failure reason should accurately indicate issues opening blob filesystem (broadinstitute#6965)

* WX-859 Accept workflow execution identity in config (broadinstitute#6967)

* WX-892 Trim down `ValueStore` logging to prevent OOMs (broadinstitute#6981)

* Add Nirvana 3.18.1 reference image test, minor cleanup [VS-705] (broadinstitute#6975)

* WX-863 Turn off Azure NIO logging (broadinstitute#6982)

* Turn off Azure NIO logging

* Poke Travis

* WM-1616: Allow repeating attempts at initialization (take 2) (broadinstitute#6985)

* WX-878 Single shared BlobFileSystemManager (broadinstitute#6986)

* Make BlobFileSystemManager shared across all BlobPathBuilders

* Update TES conf file to reflect new singleton config

* Shell escape reference image  files [VS-796] [WX-910] (broadinstitute#6989)

* WX-769 `disks` compatibility for TES backend (broadinstitute#6991)

* Update FiveMinuteIntro.md (broadinstitute#6994)

* WX-906 Sbt Unit Tests as Github Actions (broadinstitute#6992)

* WX-926 Support falling back to OCI Manifest Format (broadinstitute#7003)

* WX-926 Support falling back to OCI Manifest Forma

* Only mount reference disks if requested [WX-925] (broadinstitute#7001)

* [WM-1646] Add missing fields for `WorkflowDescription` for WomTool /describe endpoint to Swagger (broadinstitute#7004)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status (broadinstitute#6980)

* WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status

* Address feedback

* Address feedback (broadinstitute#6997)

* Address additional feedback (broadinstitute#7000)

* Fix copy/paste error (broadinstitute#7005)

* Address additional feedback

* Fix copy/paste error

* Trigger CI

---------

Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Centaur reference image test should validate symlinks [VS-796] (broadinstitute#6996)

* WX-903 Pre-GHA test suite disablement

* WX-877 Update CHANGELOG for release 85 (broadinstitute#7011)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Katrina P <68349264+kpierre13@users.noreply.github.com>
Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com>
Co-authored-by: Christian Freitas <christian.j.freitas@gmail.com>
Co-authored-by: Saloni Shah <salonipshah11@gmail.com>
Co-authored-by: kshakir <kshakir@broadinstitute.org>
Co-authored-by: mspector <michael.l.spector@gmail.com>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>
Co-authored-by: Brian Reilly <breilly@broadinstitute.org>
Co-authored-by: Adam Nichols <aednichols@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Justin Variath Thomas <JVThomas@users.noreply.github.com>
Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org>
Co-authored-by: Trevyn Langsford <tlangs@broadinstitute.org>
Co-authored-by: Miguel Covarrubias <mcovarr@users.noreply.github.com>
Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com>
Co-authored-by: Tom Wiseman <tom.h.wiseman@gmail.com>
Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>

* Remove Yaml (broadinstitute#7098)

* Azure blob read/write integration test (broadinstitute#7024)

Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org>

* Goodbye Travis (broadinstitute#7100)

* Single workflow runner (broadinstitute#7106)

* WX-975 Revert special TES path munging (broadinstitute#7103)

* Centaur blob filesystem (broadinstitute#7104)

* WX-1010 Update Github Actions set output syntax (broadinstitute#7111)

* WX-1001 Upgrade JSON Small and Fast Parser (broadinstitute#7116)

* WX-1015 Override glob behavior for TES to use absolute path (broadinstitute#7120)

* WX-990 Make TES request backoff behavior configurable (broadinstitute#7122)

* WX-1103 Revert CI to old TES polling behavior (broadinstitute#7126)

* Develop aws (#29)

* stuck on globbing

* efs works, no callcaching

* update readme

* extended EFS support

* fix for globbing in nested scatters

* updated config for globbing, to prevent issues with empty folders

* WX-1103 Accelerate TES CI (broadinstitute#7130)

* Fix Horicromtal Deadlock Test (broadinstitute#7131)

* WM-1963: Validate PRs begin with Jira tags (broadinstitute#7127)

* WX-1106 Add logging for failed docker manifest pulls (broadinstitute#7135)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* [WX-1108] Disable drs tests (broadinstitute#7145)

* ID-377 Setup pact for consumer contract testing. (broadinstitute#7123)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-1092 Support `size` engine function for public HTTP files (broadinstitute#7128)

* WX-1105 Fix interpretation of full http blob paths (broadinstitute#7138)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* Revert "ID-377 Setup pact for consumer contract testing." (broadinstitute#7146)

* WX-966 Add Cascades, remove Directory from Biscayne (broadinstitute#7105)

* WX-966 Post-merge branch cleanup + test fix (broadinstitute#7149)

* WX-1122 Enable Azure ApplicationInsights (broadinstitute#7143)

* [WX-1108] Add tests back (broadinstitute#7148)

* [WX-1136] Docker Build Test (broadinstitute#7151)

* [WX-1136] Self Hosted (broadinstitute#7152)

* WX-1145 Fix minor regression introduced in WDL 1.1 foundation (broadinstitute#7153)

* ID-377 Setup pact for consumer contract testing. Retry (broadinstitute#7147)

Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org>
Co-authored-by: Ivan <ichang@broadinstitute.org>

* WX-1133 Shorten TES localized blob paths (broadinstitute#7150)

* WX-1154 Fix Slack Messaging  (broadinstitute#7159)

* efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs (#32)

* WX-1160 Fix bad MariaDB version assumptions (broadinstitute#7160)

* WX-1126 Upgrade to modern Python, 3.8 not available in package repo anymore (broadinstitute#7164)

* WX-1122 Use legacy AppInsights to get better control over logging (broadinstitute#7157)

* WX-1137 Replace 4-byte unicode chars in PAPI event descriptions (broadinstitute#7166)

* WX-1101 Update library versions to support batch (broadinstitute#7155)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-1195 Update Azure identity libs (broadinstitute#7173)

* WX-1114 Initial inclusion of Azure NIO 'fork' (broadinstitute#7168)

* WX-1110[risk=low] Added endpoint to fetch failed tasks by root workflow id (broadinstitute#7165)

* WX-1210 Added JIRA ID for Cromwhelm auto commit message (broadinstitute#7180)

* WX-1179 DRS image build updates, remove self-hosted runners (broadinstitute#7179)

* WX-1210-action-fix Use PR title to find JIRA ID for cromwhelm commit message (broadinstitute#7184)

* WX-1110-query-fix Corrected Query to pull in attributes outside of executionStatus and backendStatus (broadinstitute#7185)

* WX-1179 GCP Batch new base (broadinstitute#7177)

Co-authored-by: dspeck <dspeck@burwood.com>
Co-authored-by: jarroyo28 <johnarroyo45@gmail.com>
Co-authored-by: Jacob Jennings <53196131+jakejennings@users.noreply.github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org>
Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com>
Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org>
Co-authored-by: Tristan Garwood <tristan.garwood@gmail.com>

* WM-2100: Update terra-helmfile on publish (broadinstitute#7187)

* WM-2100: Correct commit and PR description of terra-helmfile auto update (broadinstitute#7188)

* WX-1230 Ignore `invalidate_bad_caches_use_good_local.test` (broadinstitute#7197)

* WX-1078 ACR support (broadinstitute#7192)

* WX-1179 Enable GCP Batch Integration Tests (broadinstitute#7199)

Co-authored-by: Adam Nichols <aednichols@gmail.com>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-1179 GCP Batch Docs Update (broadinstitute#7196)

Co-authored-by: Jacob Jennings <53196131+jakejennings@users.noreply.github.com>
Co-authored-by: Adam Nichols <aednichols@gmail.com>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* ID-734 Increase Timeout for DRSHub Communication (broadinstitute#7198)

* [WX-1156] internal_path_prefix for TES 4.4 (broadinstitute#7190)

* [WM-2184] Remove brackets from Jira ID (broadinstitute#7206)

* WX-1153 Azure blob read md5 from metadata for large files (broadinstitute#7204)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-1156 Fix internal_path_prefix (broadinstitute#7208)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-1256 Temporarily turn off engine hashing for blob files (broadinstitute#7209)

Co-authored-by: Adam Nichols <aednichols@gmail.com>

* WX-1173 Reopen filesystem for blob storage outside workspace (broadinstitute#7178)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org>
Co-authored-by: Adam Nichols <aednichols@gmail.com>

* WX-1174 Adjust NIO Copy functionality (broadinstitute#7207)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* [WX-1168] TES Log Paths (broadinstitute#7210)

* Develop aws (#34)

* efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs

* add deployment manual, fix issue with empty disks

* update documentation

* update documentation

* update documentation

* WX-1264 Don't expire an unexpirable filesystem (broadinstitute#7216)

* [WX-495] DRS Parallel Downloads (broadinstitute#7214)

* WX-1225 Print TES error messages to job logger (broadinstitute#7220)

* WX-1217 Workflow completion callback (broadinstitute#7213)

Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com>

* [WM-2199] Changeset for modifying ownership of tables for WORKFLOWS app (broadinstitute#7218)

Co-authored-by: Chris Llanwarne <cjllanwarne@gmail.com>

* [WX-1234] Update CHANGELOG.md (broadinstitute#7227)

Merging past CI since this is doc only.

* WM-2252: Configurable metadata write role (broadinstitute#7225)

Co-authored-by: dvoet <dvoet@users.noreply.github.com>

* Update cromwell version from 86 to 87

* WX-1282 Update failedJobs Query to use `lo_get` instead of INNER JOIN against pg_largeobject (broadinstitute#7228)

* [WX-1234] Update Release Process with Docker Instructions (broadinstitute#7231)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WM-2294: Allow role-setter action to run on change (broadinstitute#7233)

* WM-2296: Callback should supply fully qualified output names (broadinstitute#7234)

* [WX-499] DRS Parallel Downloads Follow-up (broadinstitute#7229)

* WX-1318 gcp batch: Add GPU driver install (broadinstitute#7235)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-1232 Include useful workflow ids in TES tags (broadinstitute#7221)

* Options to publish status only (#36)

* add options to publish status only

* updated readme.md

---------

Co-authored-by: quekx <quekx@gene.com>

* WX-1307 Azure E2E test (broadinstitute#7239)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* Fix aws unit tests (#39)

* checkpoint

* fix ecr and batch tests

* fix AwsBatchJobSpec.scala

---------

Co-authored-by: quekx <quekx@gene.com>

* return bucket directly instead of listing and checking it (#38)

Co-authored-by: quekx <quekx@gene.com>

* WX-1340 GCP Batch: Mount with extra colon issue and multiple zones support (broadinstitute#7240)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-1339 Make `throwExceptionOnExecuteError` false for PAPI aborts (broadinstitute#7245)

* WX-1338 Fix cron invocation of E2E test (broadinstitute#7244)

* WX-1341 Better logging when a runner stops picking up new workflows (broadinstitute#7246)

* Add evaluteOnExit for aws batch retry (#40)

Co-authored-by: quekx <quekx@gene.com>

* [WX-1260] Acquire sas token from task runner (broadinstitute#7241)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* [WX-1183] Ice Lake (broadinstitute#7252)

* WX-1333 Improve logging visibility for load management (broadinstitute#7253)

* [WX-1391] Fix Bash Bug (broadinstitute#7326)

* [WX-1393] Add Content Length to Curl request (broadinstitute#7328)

* [WX-1346] Scalafmt  (broadinstitute#7257)

* WX-1351 Speed up `Centaur Horicromtal PapiV2 Beta` (broadinstitute#7329)

* WX-1351 CI CWL cleanup (broadinstitute#7327)

* WX-1351 Remove slow/misbehaving localization test (broadinstitute#7330)

* WX-1351 Split out restart tests (broadinstitute#7333)

* [WX-1345] Automatic Token Acquisition for TES Config (broadinstitute#7256)

* WX-1387 Remove unused case classes (broadinstitute#7334)

* [WX-1394] Upgrade Logback Core to address CVE-2023-6378 (broadinstitute#7332)

* WX-1351 Another round of CI cleanup (broadinstitute#7336)

* [WX-1395] Scalafmt GitHub Action (broadinstitute#7337)

* WX-1351 SBT compile speedup (broadinstitute#7339)

* WX-1407 Fix contract tests (broadinstitute#7340)

Co-authored-by: Ivan <ichang@broadinstitute.org>

* WX-1409 Java 17 (broadinstitute#7342)

* Revert "WX-1409 Java 17 (broadinstitute#7342)" (broadinstitute#7343)

* WX-1409 Restore Java 17 (broadinstitute#7342) (broadinstitute#7344)

* Revert "WX-1409 Java 17" (broadinstitute#7346)

* [WX-1395] Scala formatter Github Action (broadinstitute#7341)

Co-authored-by: ScalaFmt Fixer <ScalaFmtFixer@fakeDomain.com>
Co-authored-by: Broad Bot <broadbot@broadinstitute.org>

* Improved tagging support (#37)

* efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs

* add deployment manual, fix issue with empty disks

* update documentation

* update documentation

* update documentation

* support for tagging instances and volumes used in jobs, some support for spaces in file paths

* corrected workflow id in tagging

* redirect exit code 137 to retry-with-more-memory routine

---------

Co-authored-by: Henrique Ribeiro <henriqueribeiro@users.noreply.github.com>

* add gpu count (#41)

* add gpu count

* fix typo

---------

Co-authored-by: quekx <quekx@gene.com>
Co-authored-by: Henrique Ribeiro <henriqueribeiro@users.noreply.github.com>

* WX-1385 Reject blob URLs with external SAS tokens as unparsable (broadinstitute#7347)

* WX-1411 Require preinstalled `jq` and `curl` for just-in-time SAS feature (broadinstitute#7350)

* WX-1396 Cromwell client allow additional properties (broadinstitute#7352)

* WM-2428: Include full error context when failing to abort TES jobs (broadinstitute#7354)

* WX-1420 Fix GCP Batch label regex restriction (broadinstitute#7355)

Co-authored-by: Beibei Chen <beibei@formbio.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-1385 Remove SAS tokens from TES input log printouts (broadinstitute#7358)

* [WX-1184] PostgreSQL Docker Image for Local Cromwell (broadinstitute#7172)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-1444 Use MySQL LTS in DBMS tests (broadinstitute#7360)

* WX-1449 Add `latest` Docker tag (broadinstitute#7362)

* WX-767 Upgrade Cloud SDK to 461.0.0 (broadinstitute#7361)

* WX-1443 Adopt `gcloud storage` for localization only (broadinstitute#7359)

* WX-964 suffix() (broadinstitute#7363)

* WX-1417 New database role strategy (broadinstitute#7366)

* [WM-2291] Callback API contract tests between Cromwell and CBAS (broadinstitute#7251)

Co-authored-by: Chris Llanwarne <cjllanwarne@gmail.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* WX-1445 Update docker image regex to handle python:3 correctly (broadinstitute#7367)

* WX-1252 Runtime attributes cleanup – CWL runtime attributes (broadinstitute#7370)

* WX-1485 Upgrade postgres lib (broadinstitute#7371)

* WM-2454: Private GitHub support on describe api (broadinstitute#7365)

* WX-1252 Runtime attributes cleanup – CWL runtime environment (broadinstitute#7369)

* WX-1462 POSIX-flavored sub() (broadinstitute#7374)

* [WX-963] Unzip Engine Function  (broadinstitute#7368)

* WX-1489 Hopefully reduce CI flakiness by modernizing deadlock test (broadinstitute#7376)

* [WX-965] quote() and squote() engine functions. (broadinstitute#7375)

Co-authored-by: Ryan Saperstein <rsaperst@wmd55-b47.hsd1.ma.comcast.net>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* [WX-1317] Remove Akka 'server' header from all HTTP responses (broadinstitute#7379)

* WM-2461] Add support for running private workflows on Azure (broadinstitute#7373)

* WX-1252 Per-backend runtime attributes (broadinstitute#7380)

* ID-347 Remove Martha References (broadinstitute#7384)

* WX-1461 Remove `womtool upgrade` command (broadinstitute#7382)

* WX-1488 Supply cloud platform when making DRS requests (broadinstitute#7381)

* WX-757 Fix workflow stuck in aborting after WDL type error (broadinstitute#7385)

* [WX-1506] Add jobLogger output for subworkflow running (with parent and subworkflow ID) (broadinstitute#7388)

Co-authored-by: jlester-msft <47044104+jlester-msft@users.noreply.github.com>

* WX-757 Fix `stdout`, `stderr` in workflow body causing crashes (broadinstitute#7386)

* WX-1519 Stop leaking DRS Localizer images on every CI run (broadinstitute#7390)

* [WX-1460] WDL 1.1 Struct Literal Parsing (broadinstitute#7391)

Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>

* [WM-2500][WM-2502] Fetch Github token from ECM for importing and running private workflows (broadinstitute#7392)

* WX-1542 Fix workflow cancellation (broadinstitute#7398)

* WX-1550 Don't pointlessly continue running integration tests (broadinstitute#7400)

* WX-1557 Fix default values from Configuring.md and Scaling.md (broadinstitute#7393)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-1557 Fix actor-factory in Batch101.md (broadinstitute#7377)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-1557 Add more tests to the GCP Batch backend (broadinstitute#7394)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* [WX-1531] Struct Literal Type Checking (broadinstitute#7402)

* [WX-1468] Implement `returnCodes` runtime attribute (broadinstitute#7389)

Co-authored-by: Ryan Saperstein <rsaperst@wmd55-b47.hsd1.ma.comcast.net>
Co-authored-by: Ryan Saperstein <rsaperst@wmd55-b47.broadinstitute.org>

* [WM-2555] Cromwell -> ECM contract test (broadinstitute#7405)

* [WX-1568] Bump Akka max-response-reason-length limit (broadinstitute#7406)

* WX-1530 Strip pesky URL bits when creating local paths for HTTP inputs (broadinstitute#7404)

* WX-1538 Add more tests to the GCP Batch backend (broadinstitute#7410)

* [WX-1605] Fix Codecov Report Uploads (broadinstitute#7418)

* [WX-1410] Sanitize 4 byte UTF-8 characters before inserting into METADATA_ENTRY (broadinstitute#7414)

* WX-1566 Special Docker build for debugging (broadinstitute#7417)

Run `sbt -Dproject.isDebug=true server/docker`

* [WX-1448] Add verbose logging and timeout for getm (broadinstitute#7416)

Co-authored-by: Adam Nichols <anichols@broadinstitute.org>

* WX-1566 Fix Morgan's call cache file hash CPU thrash Cromwell crash (broadinstitute#7419)

* WX-1566 Fix RTD so this ticket's docs get published (broadinstitute#7420)

* fix womFileType coercion (#47)

Co-authored-by: quekx <quekx@gene.com>

* PR #44: Multi-aserisk globbing by @kevinRenaersBio.  Reapplied selection of changes on aws_develop branch

* PR #45 reapplied to aws_develop: support for sharedMemory. skipped EFS fixes for now while waiting for feedback

* upstream PR broadinstitute#7488 : fix compilation error due to swagger2markup

* Fix issue with caching on EFS by replacing touch with existOrThrow routine

* make sure that md5 sibling file of EFS data is newer than main file to be valid

* revise sharedmemory attribute to take reguluar GB/MB specification

* updated AWS README file

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Janet Gainer-Dewar <jdewar@broadinstitute.org>
Co-authored-by: Christian Freitas <christian.j.freitas@gmail.com>
Co-authored-by: Saloni Shah <salonipshah11@gmail.com>
Co-authored-by: Brian Reilly <breilly@broadinstitute.org>
Co-authored-by: Adam Nichols <anichols@broadinstitute.org>
Co-authored-by: Katrina P <68349264+kpierre13@users.noreply.github.com>
Co-authored-by: Adam Nichols <aednichols@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Llanwarne <cjllanwarne@users.noreply.github.com>
Co-authored-by: Justin Variath Thomas <JVThomas@users.noreply.github.com>
Co-authored-by: Khalid Shakir <kshakir@broadinstitute.org>
Co-authored-by: Christian Freitas <cfreitas@broadinstitute.org>
Co-authored-by: Trevyn Langsford <tlangs@broadinstitute.org>
Co-authored-by: Miguel Covarrubias <mcovarr@users.noreply.github.com>
Co-authored-by: ekiernan <55763654+ekiernan@users.noreply.github.com>
Co-authored-by: Tom Wiseman <tom.h.wiseman@gmail.com>
Co-authored-by: Blair Murri <BMurri@users.noreply.github.com>
Co-authored-by: Thomas Wiseman <twiseman@wm275-315.lan>
Co-authored-by: Tom Wiseman <twiseman@broadinstitute.org>
Co-authored-by: Stephen Fleming <sjfleming@users.noreply.github.com>
Co-authored-by: Henrique Ribeiro <henriqueribeiro@users.noreply.github.com>
Co-authored-by: mspector <michael.l.spector@gmail.com>
Co-authored-by: Tristan Garwood <tristan.garwood@gmail.com>
Co-authored-by: Ivan <ichang@broadinstitute.org>
Co-authored-by: dspeck1 <dspeck@burwood.com>
Co-authored-by: jarroyo28 <johnarroyo45@gmail.com>
Co-authored-by: Jacob Jennings <53196131+jakejennings@users.noreply.github.com>
Co-authored-by: Dillon Scott <67511512+dillydally414@users.noreply.github.com>
Co-authored-by: Chris Llanwarne <cjllanwarne@gmail.com>
Co-authored-by: dvoet <dvoet@users.noreply.github.com>
Co-authored-by: xquek <46759870+xquek@users.noreply.github.com>
Co-authored-by: quekx <quekx@gene.com>
Co-authored-by: ScalaFmt Fixer <ScalaFmtFixer@fakeDomain.com>
Co-authored-by: Broad Bot <broadbot@broadinstitute.org>
Co-authored-by: Beibei Chen <beibei@formbio.com>
Co-authored-by: Maria Yazykova <95451512+zykonda@users.noreply.github.com>
Co-authored-by: rsaperst <156102201+rsaperst@users.noreply.github.com>
Co-authored-by: Ryan Saperstein <rsaperst@wmd55-b47.hsd1.ma.comcast.net>
Co-authored-by: jlester-msft <47044104+jlester-msft@users.noreply.github.com>
Co-authored-by: Alexis Hernandez <github@alexitc.com>
Co-authored-by: dkj <david.jackson@sanger.ac.uk>
Co-authored-by: Ryan Saperstein <rsaperst@wmd55-b47.broadinstitute.org>
Co-authored-by: Alexis Hernandez <alexis22229@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jgainerdewar jgainerdewar jgainerdewar approved these changes

@aednichols aednichols aednichols approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@THWiseman @aednichols @jgainerdewar