Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

concepts: create DVC Remote page #2174

Closed
iesahin opened this issue Feb 10, 2021 · 3 comments
Closed

concepts: create DVC Remote page #2174

iesahin opened this issue Feb 10, 2021 · 3 comments
Labels
A: docs Area: user documentation (gatsby-theme-iterative) status: research Writing concrete steps for the issue status: stale You've been groomed! type: enhancement Something is not clear, small updates, improvement suggestions

Comments

@iesahin
Copy link
Contributor

iesahin commented Feb 10, 2021

that describes its differences from external dependencies?

In External Dependencies there is a line

Note that remote storage is a different
feature.

but also in remote command ref

What is a "local remote" ?

While the term may seem contradictory, it doesn't have to be. The "local" part
refers to the type of location where the storage is: another directory in the
same file system. "Remote" is how we call storage for DVC projects.
It's essentially a local backup for data tracked by DVC.

AFAICT the difference between remotes and dependencies is this: Remotes are actually containers that can store external dependencies and external outputs. A file can be a dependency to a stage or an output from a stage, can may be stored in a remote or in .dvc/cache. Remote is a location designation, it means not stored in .dvc/cache. Dependency/output is a role designation, a file can be in dependency role for a stage, in output role in another stage, both if a stage reads and writes to the same file or neither if the file is not involved in any pipeline.

Is there any error in my understanding? @jorgeorpinel @shcheklein

@shcheklein shcheklein added question A: docs Area: user documentation (gatsby-theme-iterative) type: enhancement Something is not clear, small updates, improvement suggestions labels Feb 10, 2021
@jorgeorpinel
Copy link
Contributor

jorgeorpinel commented Feb 11, 2021

@iesahin you got it mostly right. It's a confusion with terminology: The terms "remote" and "external" can have general meanings that are pretty similar. See #608 and #1127

A DVC Remote or what we most often call remote storage in DVC is a storage unit you configure for the project where (parts of) the cache can be pushed/pulled to/from.

But then on top of that we support external data in different ways (mainly as dependencies or outputs) and to indicate its location you can use the some of same URLs that remote storage configuration supports. There's even a special format like remote://myremote/path to locate external files/dirs.

See also #520

both if a stage reads and writes to the same file

p.s. on this that's more of a Q about deps and out. I don't think you can define the same file as dep and out of a single state. There's other constraints like no overlapping outputs among stages.

@jorgeorpinel
Copy link
Contributor

jorgeorpinel commented Feb 11, 2021

That said I think we definitely want a concept page (and probably some guides) that extracts all the info about remotes. But not sure we need to focus on a separate page to disambiguate external deps. Let's just keep that in mind as we address #520?

@jorgeorpinel jorgeorpinel changed the title basic concepts: Add a remote document that describes its differences from external dependencies concepts: create DVC Remote page Feb 11, 2021
@iesahin iesahin added status: research Writing concrete steps for the issue and removed question labels Oct 14, 2021
@jorgeorpinel jorgeorpinel added the status: stale You've been groomed! label Jun 29, 2022
@jorgeorpinel
Copy link
Contributor

We will address this as part of #550

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A: docs Area: user documentation (gatsby-theme-iterative) status: research Writing concrete steps for the issue status: stale You've been groomed! type: enhancement Something is not clear, small updates, improvement suggestions
Projects
None yet
Development

No branches or pull requests

3 participants