Turn pushing off for intermediate results

[brnd42]

My original request:
Is there a way to automatically never push certain intermediate results? For example if I had a script extract_features.py which outputs the features as hdf5 and than use those hdf5 files for train.py, I would like to never push those hdf5 files, but keep them in the local cache, since we don't want to backup files that can be reproduced exactly. But I also don't want to reproduce these hdf5 files every time.
I could also just gc the remote from time to time, but I don't really want to delete anything there :/

ruslan wrote on Discord:
Hi @brnd42 ! Unfortunately there is no special feature for that right now, and as I understand you still want to cache them locally, so --outs-no-cache doesn't work for you :slight_frown: It would be a very good feature though! To implement it, we would need to introduce something like push: false to dvc.yaml for specific outputs and then just take it into considersation when collecting cache in get_used_cache. It is pretty straightforward and simple. Could you create a feature request on our github, please?

[karajan1001]

Share a same flag with #4581?

[karajan1001]

Two kinds of jobs had DVC done to the data, one is backup and sharing (for security), another is local caching (for speed). We currently mixed them up?

[charlesbaynham]

#4581 introduces all the machinery required: the only thing missing is a flag for the user interface. That PR has already grown quite a lot from the initial scope however: maybe the best way is to wait until it's merged then add a flag in a subsequent, small PR.

[brnd42]

@charlesbaynham Any updates on this issue? As the ticket is still open, I assume the changes have not been released yet?

[charlesbaynham]

The PR is ready, but I think @efiop wanted to do some refactoring of some of the code which it alters, since some of it was getting quite complex as more options were added. @efiop, any updates?

[efiop]

@charlesbaynham @brnd42 Sorry for the delay, guys. Likely the import + dvc.yaml move is not going to happen before 2.0, so we'll likely get back to @charlesbaynham PR to merge it in the current form. But --outs-no-cache might actually work for @brnd42 for now, as it still saves the results to the run-cache. @brnd42 Could you give it a try?

[brnd42]

According to the documentation here: https://dvc.org/doc/command-reference/run
--outs-no-cache would lead to regenerating the data every time, which would be suboptimal, since the data generation/feature extraction usually takes several hours or days in our setup. But there is also the option of --outs-persist-no-cache <path> is probably suitable in our case, since we don't need to track the intermediate data, but still want to generate it in a dvc stage.
@charlesbaynham @efiop Sorry for the late response, do you agree, that --outs-persist-no-cache <path> would work here or do you see any edge cases in its functionality, that might be obstacles for us?

[brnd42]

For anyone reading this --outs-persist-no-cache works for my use case

[pmrowla]

This can now be accomplished by setting push: false for outputs in dvc.yaml and in .dvc files. There is currently no CLI flag for dvc run or dvc stage add to set this field, it must be manually added to the dvc.yaml or .dvc file.