Banning tar_read() tar_meta() etc during pipeline execution breaks safe multi-plan workflows [general]

[MilesMcBain]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Hi Will,

It looks like recent work you undertook to speed up metadata writes (#1055) has had the side effect of breaking our approach to shared-targets among multi-plans.

Pretty much all of our projects use multiple plans with _targets.yaml. This is to do with the fact that this aligns with distinct project phases, billing milestones etc. It has worked nicely.

We built ourselves some targets factories that smooth over the sharing of data between plans. tar_external()reads and stores the data from the provided store into the current store, and tar_external_pointer() stores a lazy object that points to a target in another store (with a hash that can be used to determine validity). The former is for small stuff you're fine to copy, the latter is for larger stuff.

Implementation details are probably not relevant (but see here if you are curious), suffice to say these targets factories make use of tar_meta() and tar_read() to read from the store of another targets plan - i.e. one that is definitely not the same store as the currently running plan.

So with the latest version we now see:

tar_external():  Last error: target refresh_date attempted to run targets::tar_read() during a pipeline, which is unsupported except when format %in% c("file", "file_fast") and repository == "local". This is because functions like tar_read() attempt to access or modify the local data store, which may not exist or be properly synced in certain situations. Also, please be aware that some functions like tar_make() and tar_destroy() should never run inside a target. Please find a different workaround for your use case.

I feel like what we are doing is safe. Could the assertions used to block the tar_read and tar_meta et.al. calls be modified to check the current store is different to the supplied store argument? It seems like they are in a position to have that information.

[wlandau]

Yeah, I think it would be feasible to check the store and use it to make a decision.

[wlandau]

Should be fixed in c9127ba. If you set the store argument of tar_read() or tar_meta() to a data store that the current pipeline is not using, then development targets should not throw an error.

[MilesMcBain]

Thanks Will, that gets us out of a jam 👍 😅