user-guide: clarify differences and use cases for external storage mechanisms?

[jorgeorpinel]

I've noticed that in general the different mechanisms DVC has to handle external storage are confusing. This includes DVC remotes, external dependencies/outputs/cache, and dvc import, dvc import-url commands, or even more general concepts like "remote location" and "external data source".

We have independent docs explaining each one but no single document (or consistent references throughout) so users can clearly identify which mechanism does what or solves which problem – In fact I wonder whether the same external data problem could be solved with one or another mechanism (which could add to the confusion).

Well, we do have this page https://dvc.org/doc/user-guide/managing-external-data (and issue #143) that try to combine external outputs and external cache into a single doc but I think the effort should probably encompass all of these mechanisms.

I'm not sure exactly how to address this so just opening this possible issue and cc @iterative/engineering for a discussion. Thanks

[jorgeorpinel]

UPDATE: Also related #103, maybe #143, #497, and #563 (specially given #563 (comment) and #563 (comment)).

[ghost]

@jorgeorpinel , I don't have any solution to this but I agree that having a page explaining how to deal with external storage would be helpful.

A good next step would be to have several questions that we want this page to answer.
Maybe searching on Discord for keywords like external storage and remote or something that will help to figure out the FAQs from the community.

What I've identified (from skimming Discord) is that there are a lot of questions related to remote configuration and usage. Currently, there's no single page to refer to for tips and tricks when setting up a remote. Users need to go through the remote sub commands and click on the expandable boxes to know about it. There was an issue already (currently closed) but I think it might be related #499

Another frequent question is about NFS / CIFS we have no entry about this in our docs (as you already noted in your comment)

[jorgeorpinel]

Closing in favor of epic #520

[casperdcl]

Shouldn't we keep this open? This (#566) is explicitly a checklist item in #520. We shouldn't close epic sub-tasks afaik.

[jorgeorpinel]

I'm still not sure exactly how to address this so it probably doesn't make sense to reopen it unless there's an actionable plan.

[casperdcl]

I'd say remove the checklist item in #520 then?

[jorgeorpinel]

We could remove the link (although the context may be useful) but the idea there is pretty generic "Clarify differences and use cases for external storage mechanisms" not exactly = this issue necessarily.