Task directory_out

[DavidGOrtega]

• closes task destroy, need of being able to specify output folder #307
• closes task recovering two tasks specifying a different folder ends in error #300
• closes task destroy multiple tasks may end up in transfer errors that makes impossible the task destruction #306

Created the nested structure using a TypeSet because TypeMap can not set predefined keys so we were not able in that case to setup the ForceNew strategy.

Example

resource "iterative_task" "tpi-examples-basic2" {
    provider = iterative2

    name      = "tpi-examples-basic2"
    cloud     = "az"
    region    = "us-east"
    machine   = "Standard_D2S_v3"
    #spot      = 0

    workdir {
        input  = "."
        output = "./task2"
    }
    
    script = <<-END
    #!/bin/bash
    echo 'hello tpi' > data.txt
    printenv >> data.txt
    END
}

[DavidGOrtega]

@iterative/cml what do you think about: directory and directory_out ?

1

In my opinion I prefer to have them this way opposed to a nested structure as

directory : {
in:
out
}

[DavidGOrtega]

2

and probably prefer dir and dir_out

[0x2b3bfa0]

Voting in favor of nested blocks:

resource "iterative_task" "example" {
  ...
  storage {
    input_directory = "."
    output_directory = "output"
    ...
  }
}

[0x2b3bfa0]

No strong preference either. 🤷🏼‍♂️

[DavidGOrtega]

Voting in favor of nested blocks:

Just downvote 1

[dacbd]

I think I would be in favor of a storage block, It would be a convenient place to put a potential addition of a pre-made / specified object-storage path

[DavidGOrtega]

I would not say storage is the best word @iterative/cml

[DavidGOrtega]

👆 updated PR desc

Im doing the integration tests in another PR.
I have successfully tested this with:

1 Nothing set

resource "iterative_task" "tpi-examples-basic2" {
    provider = iterative2

    name      = "tpi-examples-basic2"
    cloud     = "az"
    region    = "us-east"
    machine   = "Standard_D2S_v3"
    #spot      = 0

    script = <<-END
    #!/bin/bash
    echo 'hello tpi' > data.txt
    printenv >> data.txt
    END
}

2 directory set then change directory_out just refreshes

resource "iterative_task" "tpi-examples-basic2" {
    provider = iterative2

    name      = "tpi-examples-basic2"
    cloud     = "az"
    region    = "us-east"
    machine   = "Standard_D2S_v3"
    #spot      = 0
    storage {
        directory = "."
    }

    script = <<-END
    #!/bin/bash
    echo 'hello tpi' > data.txt
    printenv >> data.txt
    END
}

and then updating the storage

resource "iterative_task" "tpi-examples-basic2" {
    provider = iterative2

    name      = "tpi-examples-basic2"
    cloud     = "az"
    region    = "us-east"
    machine   = "Standard_D2S_v3"
    #spot      = 0

    storage {
        directory = "."
        directory_out = "./task2"
    }
    
    script = <<-END
    #!/bin/bash
    echo 'hello tpi' > data.txt
    printenv >> data.txt
    END
}

3 directory set then change directory destroys and create

resource "iterative_task" "tpi-examples-basic2" {
    provider = iterative2

    name      = "tpi-examples-basic2"
    cloud     = "az"
    region    = "us-east"
    machine   = "Standard_D2S_v3"

    script = <<-END
    #!/bin/bash
    echo 'hello tpi' > data.txt
    printenv >> data.txt
    END
}

and then updating the storage

resource "iterative_task" "tpi-examples-basic2" {
    provider = iterative2

    name      = "tpi-examples-basic2"
    cloud     = "az"
    region    = "us-east"
    machine   = "Standard_D2S_v3"

    storage {
        directory = "."
        directory_out = "./task2"
    }
    
    script = <<-END
    #!/bin/bash
    echo 'hello tpi' > data.txt
    printenv >> data.txt
    END
}

[0x2b3bfa0]

Doesn't contemplate droste effect prevention nor input exclusion as per #307 (comment), but... rhinoceroses definitely can be farm animals. 😄

[0x2b3bfa0]

Nitpick: any preference for directory and directory_out instead of input_directory and output_directory or input and output for the sake of brevity? In theory, APIs are forever.

[DavidGOrtega]

Nitpick: any preference for directory and directory_out instead of input_directory and output_directory or input and output for the sake of brevity? In theory, APIs are forever.

directory is better than input_directory because is the default thing, probably users won't need the output unless they want to run many times... indeed to me sounds better workdir. I do not like input nor output because they are meaningless inside the storage structure... if we were using directory instead of storage I would agree

[DavidGOrtega]

@0x2b3bfa0 please review 🙏

[DavidGOrtega]

@0x2b3bfa0 any chance?

[casperdcl]

Having tested the previous version (just directory) and finding it very painful and counter-intuitive mostly due to the name, I'd strongly suggest workdir:

workdir {
  in(put): ...
  out(put): ...
}

Because the remote instance really uses these as (current) working directories.

[DavidGOrtega]

@iterative/cml @casperdcl I feel the same about workdir
Changing it

[DavidGOrtega]

@iterative/cml i'm changing just only the interface for the internal test. I we like it II will adapt then the code internally.
I will create a technical_debt issue

[DavidGOrtega]

@iterative/cml please review and accept if ok 🙏

[DavidGOrtega]

@casperdcl any luck here? 🙏

[casperdcl]

Just pushed a couple of documentation updates, please review before merge 🙏

[DavidGOrtega]

Looks great @casperdcl

[casperdcl]

Awesome. Do we have a follow-up issue for some of the outstanding items like #340 (comment)?

[0x2b3bfa0]

We don't, and we probably should. Or, at least, move it to Notion.

[DavidGOrtega]

All solved! We are not going to solve 1,2,3 as ideally we thought because adds complexity and the solution is not right anyway.

Please read the comment why 1,2,3 are discarded in favour of just error if the output folder is not empty