tool: inconsistent result when looping the creation of multiple files & folders using multiple values

[dsaintilma-flinks]

Hello,

Cue version:

$ cue version
cue version v0.4.2 linux/amd64

Issue:
I'm trying to create multiple files & folders using a for loop but the they're getting created only after multiple retries and it's never the same amount of retries. Not sure what's happening. Is it perhaps a case of race condition?

Here's the code that was executed:

$ cat genfile_tool.cue 
package kube

import (
  // "tool/exec"
	"tool/file"
	"encoding/yaml"
)

command: genfiles: {
  for key, val in objects {
    task: "mkdir-\(key)": file.MkdirAll & {
      path: "manifests/\(cluster_metadata.project)/\(cluster_metadata.name)/\(val.metadata.namespace)"
    }
    task: "write-\(key)": file.Create & { 
      filename: "manifests/\(cluster_metadata.project)/\(cluster_metadata.name)/\(val.metadata.namespace)/\(val.metadata.name).yaml"
      contents: yaml.Marshal(val)
    }
  }
}

Ran from the cli

$ cue genfiles ./configs/dev/app01
task failed: open manifests/dev/app01/argocd/monitoring-app01.yaml: no such file or directory
code 1

$ cue genfiles ./configs/dev/app01
task failed: open manifests/dev/app01/argocd/monitoring-app01.yaml: no such file or directory
code 1

$ cue genfiles ./configs/dev/app01
task failed: open manifests/dev/app01/argocd/monitoring-app01.yaml: no such file or directory
code 1

$ cue genfiles ./configs/dev/app01
task failed: open manifests/dev/app01/argocd/monitoring-app01.yaml: no such file or directory
code 1

$ cue genfiles ./configs/dev/app01
task failed: open manifests/dev/app01/argocd/monitoring-app01.yaml: no such file or directory
code 1

$ cue genfiles ./configs/dev/app01 <=== OK

Thanks in advance for the help.

[verdverm]

I suspect your code is trying to make files before directories. Try making an explicit dependency by referencing a fake field in mkdir in write, like dep: task["mkdir-\(key)"].dummy inside of the write task, just before filename

(edit): I ended up writing out a number of gotchyas and patterns around cue/flow | the cue cmd

https://cuetorials.com/patterns/scripts-and-tasks/

[dsaintilma-flinks]

Yep, it works. Thanks!

[dsaintilma-flinks]

Example of the addition of the dependency:

command: genfiles: {
  for key, val in objects {
    task: "mkdir-\(key)": file.MkdirAll & {
      path: "manifests/\(cluster_metadata.project)/\(cluster_metadata.name)/\(val.metadata.namespace)"
    }
    task: "write-\(key)": file.Create & { 
      filename: "manifests/\(cluster_metadata.project)/\(cluster_metadata.name)/\(val.metadata.namespace)/\(val.metadata.name).yaml"
      contents: yaml.Marshal(val)
      dep: task["mkdir-\(key)"].dummy
    }
  }
}

[eonpatapon]

You can reference the task itself, this should work:

task: "write-\(key)": file.Create & {
  after: task["mkdir-\(key)"]
  filename: ...
}

Probably a nicer way to do this is to use the MkDirAll path in file.Create, so that you have a reference and the order is correct but it also factorize the code:

command: genfiles: {
  for key, val in objects {
    task: "mkdir-\(key)": file.MkdirAll & {
      path: "manifests/\(cluster_metadata.project)/\(cluster_metadata.name)/\(val.metadata.namespace)"
    }
    task: "write-\(key)": file.Create & { 
      filename: "\(task["mkdir-\(key)"].path)/\(val.metadata.name).yaml"
      contents: yaml.Marshal(val)
    }
  }
}

[verdverm]

I've been building custom tasks on top of cue/flow and this does not work in the general case.

• with only referencing the task, it may not run if it was defined outside of the scope of the root value used to construct the DAG
• referencing values which are concrete prior to running a task does not enforce the ordering, i.e. it is already concrete so downstream tasks are free to run.

Generally, I would reference a value produced by a task, but in tasks which do not produce a value, a field which will always be incomplete is sufficient to produce the desired dependency, hence a "dummy" field in this case.

Here's an example for cue cmd: https://github.com/hofstadter-io/cuetils/tree/main/flow/tests/repro/cuecmd

Note for using reusable tasks, local or imported, you have to make sure all tasks referenced are within the root value passed to the flow tool. In the case of cue cmd, that root value will be the command: <cmd-name>: .... ( I may be slightly off here, in that if they are in the same package, it might work because of the config to flow that cue cmd uses. Going to do a test now, I will update this once known)

So you may find this pattern necessary: https://github.com/hofstadter-io/cuetils/blob/main/flow/tests/repro/cuecmd/do_tool.cue#L14