Using task-level output_data results in: UnboundLocalError: local variable \'include\' referenced before assignment

[ghost]

Problem Description

I am trying to egress stdout.txt and stderr.txt files for my tasks to azure blob storage by using task-level output_data feature.

However, during jobs add command I get the following error:
UnboundLocalError: local variable \'include\' referenced before assignment

More details below.

Batch Shipyard Version

3.8.1

Steps to Reproduce

Run jobs add command with jobs.yaml below.

Expected Results

Tasks are run, output files are stored on the azure blob account.

Actual Results

Error message below.

Redacted Configuration

job_specifications:
- id: myjobid
  tasks:
  - id: mytaskid
    docker_image: mydockerimage
    command: mycommand
    output_data:
      azure_storage:
      - storage_account_settings: my_storage_account
        remote_path: myfolder/myjobid/mytaskid
        local_path: $AZ_BATCH_TASK_DIR
        include:
        - stdout.txt
        - stderr.txt

Additional Logs

Error message and stack trace:

[20304] Failed to execute script shipyard
Traceback (most recent call last):
  File "shipyard.py", line 3135, in <module>
  File "site-packages\\click\\core.py", line 764, in __call__
  File "site-packages\\click\\core.py", line 717, in main
  File "site-packages\\click\\core.py", line 1137, in invoke
  File "site-packages\\click\\core.py", line 1137, in invoke
  File "site-packages\\click\\core.py", line 956, in invoke
  File "site-packages\\click\\core.py", line 555, in invoke
  File "site-packages\\click\\decorators.py", line 64, in new_func
  File "site-packages\\click\\core.py", line 555, in invoke
  File "shipyard.py", line 1970, in jobs_add
  File "convoy\\fleet.py", line 3974, in action_jobs_add
  File "convoy\\batch.py", line 5580, in add_jobs
  File "convoy\\batch.py", line 4736, in _construct_task
  File "convoy\\data.py", line 424, in process_output_data
  File "convoy\\data.py", line 356, in _process_storage_output_data
UnboundLocalError: local variable \'include\' referenced before assignment

[ghost]

Thanks for the fix, the error is gone now. Just a quick follow-up question:

I have noticed that stdout.txt and stderr.txt are copied directly to base container myfolder, instead of myfolder/myjobid/mytaskid. I am copying to Azure Blob Storage. Is this expected? Underlying Azure Batch OutputFile* API seems to support copying to a sub-folder of a container.

[alfpark]

This is yet another bug - apologies for the churn. I'll try to have this fixed and you'll be able to use develop to workaround until the next hotfix.

[ghost]

Great, no worries - it is not urgent, thanks!

[alfpark]

This has been fixed in develop. Short-term workarounds until the next hotfix release are:

1. Pull develop if you've used the git clone install method
2. Use develop-cli Docker image (once the DevOps build completes)
3. Use a non-native pool