dxCompiler 2.10.7

• dxda update to v0.5.10 which adds a retry for file download. This fixes the "context canceled" error which was
  thrown due to the dxda behavior.
• Added support for new region in Azure London, specifically for OFH projects

Dependency updates

wdlTools 0.17.15

• Minor fixes and refactoring

dxCompiler 2.10.6

• Support for Directory outputs in the WDL development version. Note that the dxCompiler support for the development version of WDL is under development and not yet production-ready.
• Minor fixes of type coercion. It includes a fix to the "Value V_String(1.0 GiB) could not be coerced to one of Vector(T_Int)" error when setting Runtime.memory as a string in a development version task, for example:

task {
  ...
  runtime {
    memory: "1 GiB"
  }
}

Dependency updates

wdlTools 0.17.14

• Support for Directory outputs in the WDL development version
• Minor fixes of type coercion

dxCompiler 2.10.5

• Fix for callable recompilation if the Struct composition was changed

Dependency updates

wdlTools 0.17.13

• Fix for V_Directory in Struct to Directory input

dxCompiler 2.10.4

• CWL: fixed target tool id when overriding docker requirement. File ID of an image provided in DockerRequirement.dockerLoad
  in a CWL workflow is now correctly mapped and detected upon override.
• WDL: manifest mode correctly handles non-fully qualified file IDs (updated dxScala: api)

Dependency updates

dxApi 0.13.3

• Handling of non-fully qualified file IDs for bulk search/describe. Now for the files provided without the project ID,
  the describe response will be returned only for current workspace/project. If a file was cloned to other projects, they
  will be ignored. Non-fully qualified file IDs are not allowed when searching files in other projects.
• Regression tests for API calls to platform

dxCommon 0.11.4

• Fix for handling 503 error in dx CLI when API requests are throttled

dxFileAccessProtocols 0.5.5

• Dependency update

wdlTools 0.17.12

• Optional types are preserved when using outputs from the standard library functions without explicit declaration.

dxCompiler 2.10.3

• WDL: (WDL >= 1.1) Fix for nested workflows when compiled in the unlocked mode: optional inputs with None as default are coerced correctly.
• CWL: Fix for merging optional source inputs. If an input is a collection (e.g. an array), where some items are the type of "null", it is correctly merged when MultipleInputFeatureRequirement is specified.
• CWL: Fix for making target step argument for cwltool.
• CWL: Added support for "metadata" field for input files in CWL under development (pending inclusion in CWL standard).
• CWL: fix casting cwl types from scatter to scatter.
• CWL: Fix for parameter mapping for output stage of the workflow.

dxCompiler 2.10.2

• WDL: Fix to native app(let)s instance override with system requirements (cpu/memory/disks) in case of the direct calls and executions within fragments. Only -instanceTypeSelection static (default) is supported. If compiled with dynamic - default instances will be used.
• CWL: Partial implementation of task/fragment applets and workflows reuse. Same as in WDL, the DocContents attribute, which is the part of source code that defines each executable, is now used for checksum calculation and comparison. However, since the CWL source code is packed as a nested JSON file (instead of standalone blocks in WDL), fragment applets and workflows will also include the code of all underlying processes in their DocContents, and they will not be reused if the wrapped applets/subworkflows have changed.
• CWL: Fix evaluating scatter inputs during workflow execution. It would no longer raise an error if optional inputs of the underlying task are not declared at the scatter level.
• CWL & WDL: Keep the Docker container after a task finishes running, so users can debug Docker related job failures.
• CWL: Fix to the pattern handling from secondaryFiles at workflow level during compilation

Dependency updates

wdlTools 0.17.11

• Keep the Docker container after a task finishes running, so users can debug Docker related job failures.
• Runtime object is aware if it was created with default system requirements

dxCommon 0.11.3

• Fix JsUtils.makeDeterministic to handle JsArrays sorting

cwlScala 0.8.3

• Updated cwljava with fixes to secondaryFiles with pattern field when parsing workflow-level parameters and the helper function utils.Uris.shortname to generate enum symbols without namespaces
• Updated cwljava to 1.0

dxCompiler 2.10.1

• WDL: Fragments and blocks reuse applets as well, i.e. they are not rebuilt if the code corresponding to them hasn't been updated in the WDL source file. Previously only tasks were reused. Breaking: can break the logic of any App reuse for CWL (even tasks maybe are not reused), because ApplicationCompiler and WorkflowCompiler now look at the DocContents for checksum, and SourceCode attribute is now ignored.
• CWL: NetworkAccess, WorkReuse and ToolTimeLimit hints are now supported
• WDL: Update custom reorg applet (used for custom handling of your workflow outputs) example in documentation.
• WDL: Fix to the order of precedence for the different job reuse settings. The ignoreReuse setting specified in the extras.json file should override the dx_ignore_reuse setting specified in the runtime section of the WDL file.
• Updates to the documentation, esp. sections about delay workspace destruction, DNAnexus-specific runtime settings, and outputing DNAnexus files.
• WDL: Fix to configuring network access to tasks (applets). Now you can disable network access to your tasks with the runSpec.access setting in the extras.json file:

{
  "defaultTaskDxAttributes" : {
    "runSpec": {
        "access" : {
          "network": []
        }
      }
  }
}

However, if the task needs network access since it uses a Docker image from an external registry (e.g. DockerHub), the setting will be overwritten and full network access will be given to the task. You can prevent this by storing the Docker image as a file in your DNAnexus project.

Note that the setting above is under defaultTaskDxAttributes, which means it will be the default setting for all tasks; follow these instructions to configure a specific task.

Dependency updates

cwlScala 0.8.1

• Fixes CWL default requirement classnames NetworkAccess, WorkReuse and ToolTimeLimit so the corresponding hints can be recognized by dxCompiler (instead of being defined as GenericHints which are not interpreted during compilation).

wdlTools 0.17.9

• TAT.Workflow has a source attribute in analogy to TAT.Document to be used for checksum calculation for App/Job reuse
• Fixes evaluation of structs as input parameters of workflow/scatter: hash inputs are coerced from object to struct, and their optional elements are assigned to Null if not specified in job inputs.

dxCompiler 2.10.0

• CWL: All tasks and workflows conformance tests now pass
• CWL: Allow a valid CWL identifier that starts with a number or other characters disallowed by the DNAnexus platform
  • Valid CWL IDs starting with disallowed characters will be padded with an extra "___" when generating the full encoded name, which will still be a valid CWL ID (as an URI) when decoded
• CWL: Fix to an error with execution of nested workflow step due to incorrect step identifiers. Previously it prevented running of workflows with the --single-process flag.
• WDL: Minor fixes with directory path handling when passing -separateOutputs flag
  • When a workflow was compiled with the -separateOutputs flag, it would result in an error at runtime saying "InvalidInput: Folder path must start with /."
• WDL: Fix an issue with optional identifiers not found when they were a part of an unused expression
• Fix an issue where a dbcluster object stored in a project in which a workflow was run was causing a runtime error, if the workflow was compiled with the -reorg flag.

Dependency updates

dxApi 0.13.2

• Added database and dbcluster to the list of DNAnexus data objects that are recognized and can be described

dxFileAccessProtocols 0.5.4

• Fix to the issue where -separateOutputs option was causing a dx API runtime error due to a missing leading / in folder path

wdlTools 0.17.8

• Input variables are now evaluated and added to the dependency graph, even when they are optional and not used. They need to be a part of the graph in the case when they are assigned to other (unused) variables for a proper expression evaluation.

dxCompiler 2.9.1

• Exception messages of subprocesses are propagated to the exception message of the main process.
• Fixes a bug where inputs to a WDL block were being ignored if they were Optional
• CWL: run input default is now used when not declared in step
• Minor updates to Developing docs

Dependency updates

dxCommon 0.11.2

• Minor changes to JSON formatting

dxCompiler 2.9.0

2.9.0 2022-02-08

• Command file is now echoed to stderr rather than stdout
• Fixes a bug where directory support was broken for WDL; directories in WDL are represented as strings
• Switches from using cwltool to cwlpack
• Fix an issue with default values in CWL
• Adds support for publishing global workflows from dxCompiler-generated WDL workflows; see documentation
• Fix an error message detecting unsupported CWL version
• Update sbt to 1.6.1

Dependency updates

cwlScala 0.8.0

• Breaking
  • Parser API has changed substantially, with parameters added, removed, and rearranged
  • Parser.parse method is now private - use parseString or parseFile instead
  • Removes all options to modify IDs during parsing
• Handles workflows packed by cwlpack
• Adds Process.simpleName method to return simplified process name from ID automatically generated by cwlpack --add-ids
• Adds Identifiable.copySimplifyIds method to deep-copy objects with simplified IDs
• Adds CwlEnum.symbolNames function for getting enum symbols without any namespace prefixes
• Adds coerce option to Evaluator.evaluate to actually perform type coercion, rather than just checking that the result is coercible to the specified type
• Trims StringValue when coercing to primitive types
• Fixes Evaluator.finalizeInputValue for compound and optional types
• Evaluator.finalizeInputValue loads file contents from remote file source if file does not exist locally
• CwlType.coerceTo now returns both the coerced-to type and value
• Added CwlType.CwlGenericRecord, which is coercible to either CwlInputRecord or CwlOutputRecord
• Fixes evaluation of values with multiple possible types