[ideas] Support targets writing directly to the store

[multimeric]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

There are a number of cases where it makes sense to work with files on disk in Targets. For example, you might want to run some command line tool that uses file paths as input and output. Targets somewhat supports this via the format = "file", but this doesn't feel particularly idiomatic because:

• The results of the pipeline now live in two or more separate places: the targets _store and a user defined path where we put our files
• The user must think about the absolute location of their files, where normally in a targets pipeline this is abstracted away. Hiding this is a nice feature of frameworks such as Nextflow.

To resolve these problems, I think it would be great if there was an officially sanctioned way to generate a target by writing directly to the store. For example, let's say that tar_output_path() returned the path to the current target, and also marked the result of the current target as irrelevant (the return value from system2 here doesn't need to be saved):

tar_target(
    name = alignment,
    command = {
        system2("bwa", some_fastq_path, stdout=tar_output_path())
    }
)

This example is similar to the following, where we explicitly capture the stdout as a string and let targets save it to the store.
However, letting our subprocess write directly to the filesystem will be vastly more performant and memory efficient because everything can be streamed:

tar_target(
    name = alignment,
    command = {
        system2("bwa", some_fastq_path, stdout=TRUE)
    }
)

I'm aware of the existence of tar_path_target, but it's not clear to me what will happen if I write to this file inside the current target. I suspect the return value of the target will be saved to this file afterwards, overwriting my output. Therefore, I think we still need a new function for this purpose.

[brownag]

If I understand correctly, I think you may want a custom target storage format with tar_format().

A custom format would allow you to deal with objects in the the target store via custom read/write methods. The target command itself would be the input path or object, without a system call/subprocess, the write function would involve system call that writes the output to target path, the read function could use a custom method to read from the target store, return the path to the file that was written in the write step, or similar

[multimeric]

I suppose it could be implemented like that, but it doesn't feel natural. It would mean the actual target's "logic", which in this case is a subprocess call, needs to be moved into the format's write function rather than the command where you might expect it. Also a format is supposed to be re-usable while the system call will vary from target to target, so I'm not sure this makes sense.

[multimeric]

The other thing is that I want to be able to write this file target from R in some cases. Otherwise every IO function in R needs to be wrapped in its own format. Let's say I want to write a CSV in one target but a TSV in another, but they need to be treated as files by targets, I don't think I should need to touch tar_format which is quite a low level function.

[joelnitta]

I think this is a great idea. I have personally felt this pain quite keenly when constructing pipelines that call external programs that make use of file input/output (as is quite common in bioinformatics pipelines). If we could abstract away file paths, that would be awesome.

I think a concrete example may help illustrate: https://github.com/joelnitta/targets_vcf_example/blob/main/_targets.R

As you can see, I depend heavily on tar_file() but have to keep track of all the file paths myself.

[multimeric]

Perfect, it helps to have a fleshed out example of what I mean. I can see that you're writing data to store subdirectories, I guess to keep everything together (my first point).

[joelnitta]

Yep, I put them all in ./_targets/user/. Which was really an attempt to abstract them away, in a sense. All of my custom functions take file paths as input and return file paths as output.

[wlandau]

I don't talk about this much, but storage = "none" leaves the responsibility of saving the target completely to the user. Likewise with retrieval = "none" for loading the target. It might help you here, but there are subtle gotchas with cloud storage and dynamic branching. Example _targets.R:

library(targets)
tar_option_set(storage = "none", retrieval = "none")
fs::dir_create("_targets/objects")
list(
  tar_target(
    name = x,
    command = saveRDS(2, tar_path_target())
  ),
  tar_target(
    name = y,
    command = saveRDS(readRDS(tar_path_target("x")) + 1, tar_path_target())
  )
)

[multimeric]

Wonderful, thank you! I'd be happy to document this pattern with the use cases described in this thread. Is there part of the book that you think it would suit?