add: --to-remote needed? OR --external needed?

[jorgeorpinel]

Follow up to #5198 (comment), #5301, and iterative/dvc.org#2172 (comment):

Question

add --to-remote is a bit strange because normally add doesn't move target data, rather tracks it in-place (analog to git add). But --to-remote implies that external data will be moved into the workspace at some point, which we skip for now but "pre-push" (transfer) it to remote storage (for later pull/fetch).

As of now add --to-remote has a similar result to get-url + add + push + remove, gc. So OK, maybe it's nice to have a shortcut to all that, but we already have import-url (--to-remote) to achieve the same.

The only difference vs. importing is that the data source is not recorded as a dependency in the .dvc file. So you can't update it or unfreeze+repro it. However I don't see any use cases where you would want to prevent the .dvc from having this dep, as you can simply never update or unfreeze it.

TLDR: I think import-url --to-remote is enough and what we should recommend for these situations. And add --to-remote breaks the Git analogy. Cc @dberenbaum

Improvement

• But if we keep it, an improvement would be to NOT require the --external flag with it (cc @isidentical). This saves the user from typing a flag that is always needed, but also make sense since the data is not actually being treated as external in the sense that it won't be tracked/controlled in it's original location (requiring external cache, etc.).

---

• Finish or close Example for dvc add --to-remote dvc.org#2172 when this is decided.

[jorgeorpinel]

This probably applies to add --to-cache add --out (to-cache) too BTW. Not sure (we haven't documented that one yet).

[shcheklein]

My 2cs:

• add is more natural to find (I need to add data to DVC but it's too large to do it with a regular dvc add on this machine - etc - I'll be searching for something like "How do add large data to DVC" or something.
• import-url has a different semantics (.dvc file is different). So, I am not sure why would keep only one, even though they are similar.
• agreed on removing --external constraint, --to-remote should imply it. Since it's probably almost always the case that we need both.

[jorgeorpinel]

add is more natural to find (I need to add data... searching for something like "How do add large data to DVC

I get the semantics part but that can probably addressed with good docs. Or maybe import could have a better name or become a flag in add (just a thought).

I still find add --to-remote unintuitive, esp. for people used to git add which never moves/copies files. Maybe if we had another way to use add that moves external data into the workspace like add --import, then this could be expressed as add --import --skip-local/--push (again rhetorical, not an actual suggestion).

Also, when you add /external/path it gives you a hint that you need --external. It could hint instead/additionally that you probably want import-url --to-remote.

.dvc file is different

When users ask, which one are we going to recommend in most cases? IMO import-url has added benefits (frozen external dependency) that even if you don't need now, can hardly get in the way, and may come handy later.

agreed on removing --external constraint, --to-remote should imply it

Cc @isidentical that's the only action point so far 😬

[dberenbaum]

I haven't been tracking this issue closely, so I lack some context on both the command and related commands and their intended uses, which maybe is helpful so I can provide more of an outsider perspective?

Between get, add, and import, plus get-url and import-url, it's pretty overwhelming and confusing. If we also have --external, --to-remote, and --to-cache flags for several (but not all) of these commands, as well as run --external, it makes it hard for any user to understand, so my first instinct is to minimize and consolidate the number of different commands and flags as much as we can.

Is add --external needed at all if import-url --to-cache/--to-remote exists? I see some notes above on the differences between them, so maybe that will become more obvious when I spend some time with the commands.

[dberenbaum]

When users ask, which one are we going to recommend in most cases? IMO import-url has added benefits (frozen external dependency) that even if you don't need now, can hardly get in the way, and may come handy later.

Agreed. Playing around with it a bit, I'm struggling to see how to meaningfully differentiate between:

• dvc import-url --to-cache and dvc add --external (would --to-cache replace --external or is there some difference between them?)
• dvc import-url --to-remote and dvc add [--external] --to-remote.

[isidentical]

Cc @isidentical that's the only action point so far grimacing

I am still new to the concept of --external, though I don't see why it would be needed in case of to-cache or to-remote;

(.venv) (Python 3.10.0a4+) [ 11:16ÖS ]  [ isidentical@desktop:/tmp/test_dvc(master✗) ]
 $ dvc add /home/isidentical/t.py -o t.py
100% Add|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|1/1 [00:00,  1.95file/s]
                                                                                                                                                                                                                                                   
To track the changes with git, run:

        git add t.py.dvc
(.venv) (Python 3.10.0a4+) [ 11:16ÖS ]  [ isidentical@desktop:/tmp/test_dvc(master✗) ]
 $ cat t.py.dvc
outs:
- md5: 239e65e4b720cc7a7f06cf129d060d53
  nfiles: 1
  path: t.py

(.venv) (Python 3.10.0a4+) [ 11:19ÖS ]  [ isidentical@desktop:/tmp/test_dvc(master✗) ]
 $ dvc add /home/isidentical/notes --to-remote -o notes
100% Add|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|1/1 [00:00,  4.47file/s]
                                                                                                                                                                                                                                                   
To track the changes with git, run:

        git add notes.dvc
(.venv) (Python 3.10.0a4+) [ 11:19ÖS ]  [ isidentical@desktop:/tmp/test_dvc(master✗) ]
 $ dvc pull notes
A       notes                                                                                                                                                                                                                                      
1 file added and 1 file fetched                                                                                                                                                                                                                    

In case of you don't supply -o notes with --to-remote (you can't do this with to-cache, since having -o is obligatory, though you can try to set the output destination outside of repo, which you need --external only then) it will try to add /home/isidentical/notes under the control of dvc, which I thought would be the reason why added this commands in the first place (unless I am missing something, which seems probable so examples are much appreciated for me to understand this situation better).

[dberenbaum]

Examples definitely help, @isidentical!

$ dvc add /home/isidentical/notes --to-remote -o notes

Wouldn't the following be pretty similar (except for the difference in the .dvc file noted above)?

$ dvc import-url --to-remote /Users/dave/notes notes
$ cat notes.dvc
frozen: true
deps:
- path: /Users/dave/notes
outs:
- md5: d41d8cd98f00b204e9800998ecf8427e
  nfiles: 1
  path: notes

On the other hand, I guess there's no import-url equivalent for:

$ dvc add --external --to-remote /Users/dave/notes -o /Users/dave/notes_new_location
$ cat notes_new_location.dvc
outs:
- md5: d41d8cd98f00b204e9800998ecf8427e
  nfiles: 1
  path: /Users/dave/notes_new_location

[jorgeorpinel]

minimize and consolidate

Agree @dberenbaum , that would be ideal. Too many ways to deal with external data IMO

Is add --external needed at all if import-url --to-cache/--to-remote exists?
still new to the concept of --external, though I don't see why it would be needed

Yeah they're different. Story: add always accepted external paths (which end up as external outputs in the .dvc file) but we started realizing over time we shouldn't encourage that use, so we started requiring the explicit --external flag as kind of a barrier (cc @isidentical). Now we also have a scary warning in https://dvc.org/doc/user-guide/managing-external-data.

add --to-remote and -o results in something completely new for add which is to move an external target to the workspace (which happens after doing that + pull).

set the output destination outside of repo, which you need --external

This is also new to me: that you can output an external file/dir to some other external location. I wonder whether there's a use case for that or if it's just a side effects of providing --out.

[shcheklein]

I see that there is a lot of confusion (as usual with any data located externally) because for some reason --external got involved. My take on this - we should forget about --external completely here. They are completely different features. For some (unfortunate) reason (to my mind) we forgot to disable --external for --to-remote. But it should be disabled to keep all things related to the "external outputs and dependencies" feature separate from the regular dvc add --to-remote.

--to-remote is in fact a utility for the regular dvc add, regular dvc import-url, etc. It has nothing to my mind has to do with "external outputs and dependencies".

I get the semantics part but that can probably addressed with good docs.
IMO import-url has added benefits (frozen external dependency) that even if you don't need now, can hardly get in the way, and may come handy later.

@jorgeorpinel could you clarify this? I'm not sure I understand, to be honest, why do we compare import and add? They are two different operations to my mind. --to-remote makes sense for both of them (or neither of them if we can find an alternative).

I still find add --to-remote unintuitive

It might be not ideal, but it was the best we were able to come with. At least w/o introducing an additional top level command. We decided against it (top level command) mostly for a reason that this is expected to be an advanced utility to help people add large data files in specific cases, not an alternative to a general workflow, not a global scenario.

[jorgeorpinel]

nothing to my mind has to do with "external outputs and dependencies".

True. I'm referring to the more general concept of handling "external data" (found outside the project). But yes, other than not requiring --external with --to-cache/remote, we can forget about --external in this discussion.

IMO import-url has added benefits

could you clarify this?

Sure: my argument was that when people have a scenario for a --to-cache/remote operation, I believe I'll hardly ever have an incentive to recommend add over import-url. I expect the latter to be superior in the vast majority of cases.

I could be wrong of course, this is hypothetical. And even if I'm right the worst case scenario is that no one ends up using add --to-... which isn't really a big deal.

why do we compare import and add? --to-remote makes sense for both of them

Originally --to-remote was only going to be introduced to import-url IIRC. Then for some reason we decided to also put it in add (in the same core PR) which may be why we're still talking about them in tandem.