Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

api: fallback to external_repo if not a DVC repository #3222

Merged
merged 2 commits into from
Jan 23, 2020

Conversation

skshetry
Copy link
Member

@skshetry skshetry commented Jan 23, 2020

Fixes #3221. Now, it will just fallback to the external_repo if Repo() could not be instantiated.
Another idea would have been to check for the existence of .dvc directory, but, I went with this, as it feels correct and robust.

  • ❗ Have you followed the guidelines in the Contributing to DVC list?

  • 📖 Check this box if this PR does not require documentation updates, or if it does and you have created a separate PR in dvc.org with such updates (or at least opened an issue about it in that repo). Please link below to your PR (or issue) in the dvc.org repo.

  • ❌ Have you checked DeepSource, CodeClimate, and other sanity checks below? We consider their findings recommendatory and don't expect everything to be addressed. Please review them carefully and fix those that actually improve code or fix bugs.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

@skshetry skshetry requested review from Suor, efiop, a user and pared January 23, 2020 05:38
@skshetry skshetry self-assigned this Jan 23, 2020
@@ -128,10 +128,10 @@ def pull_to(self, path, to_info):
raise PathMissingError(path, self.url)

@contextmanager
def open_by_relpath(self, path, mode="r", encoding=None):
def open_by_relpath(self, path, mode="r", encoding=None, **kwargs):
Copy link
Member Author

@skshetry skshetry Jan 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

open_by_relpath on the Repo also takes more arguments such as remote, etc which does not make sense here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it works it should stay as is. Let's not add ignored arguments, this will lead to silent errors.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does not as it also gets unexpected arguments from Repo.open_by_relpath:

def open_by_relpath(self, path, remote=None, mode="r", encoding=None):

Copy link
Member Author

@skshetry skshetry Jan 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could though add remote, or not even send one, but, I prefer this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand. This line is never called from Repo.open_by_relpath().

Copy link
Member Author

@skshetry skshetry Jan 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's called by dvc.api.open() here with following args:

dvc/dvc/api.py

Lines 70 to 80 in 4903996

def open(path, repo=None, rev=None, remote=None, mode="r", encoding=None):
"""Context manager to open a file artifact as a file object."""
args = (path,)
kwargs = {
"repo": repo,
"remote": remote,
"rev": rev,
"mode": mode,
"encoding": encoding,
}
return _OpenContextManager(_open, args, kwargs)

And, successively, _open calls this line:

dvc/dvc/api.py

Lines 94 to 100 in 4903996

def _open(path, repo=None, rev=None, remote=None, mode="r", encoding=None):
with _make_repo(repo, rev=rev) as _repo:
with _repo.open_by_relpath(
path, remote=remote, mode=mode, encoding=encoding
) as fd:
yield fd

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, then we should add only remote param not **kwargs and assert remote is None. This way things won't be silently ignored.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Suor assert is not needed, as the point of that method is to provide the same interface for git and dvc repos, so it should be silently ignored there. Or am I missing something?

try:
abs_path = os.path.join(self.root_dir, path)
with open(abs_path, mode, encoding) as fd:
with open(abs_path, mode, encoding=encoding) as fd:
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Third argument is buffering.

@skshetry skshetry force-pushed the fallback-external-repo branch from e825f64 to fcd0bc8 Compare January 23, 2020 05:44
@skshetry skshetry changed the title api: fallback to external_repo on error api: fallback to external_repo if not a DVC repository Jan 23, 2020
@efiop
Copy link
Contributor

efiop commented Jan 23, 2020

@skshetry Please check the tests, they've failed.

dvc/api.py Outdated Show resolved Hide resolved
@codecov
Copy link

codecov bot commented Jan 23, 2020

Codecov Report

Merging #3222 into master will decrease coverage by 0.33%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3222      +/-   ##
==========================================
- Coverage   91.67%   91.33%   -0.34%     
==========================================
  Files         139      139              
  Lines        8681     8681              
==========================================
- Hits         7958     7929      -29     
- Misses        723      752      +29
Impacted Files Coverage Δ
dvc/scm/git/__init__.py 88.39% <ø> (ø) ⬆️
dvc/system.py 70.17% <0%> (-17.55%) ⬇️
dvc/analytics.py 93.15% <0%> (-2.74%) ⬇️
dvc/daemon.py 54.23% <0%> (-1.7%) ⬇️
dvc/external_repo.py 88.61% <0%> (-1.63%) ⬇️
dvc/utils/fs.py 96.15% <0%> (-0.77%) ⬇️
dvc/utils/__init__.py 92.59% <0%> (-0.62%) ⬇️
dvc/stage.py 96.27% <0%> (-0.38%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update da6399d...2e0b143. Read the comment docs.

@skshetry skshetry force-pushed the fallback-external-repo branch from 908f5c6 to 566aa57 Compare January 23, 2020 08:56
dvc/api.py Show resolved Hide resolved
dvc/api.py Outdated Show resolved Hide resolved
@@ -128,10 +128,10 @@ def pull_to(self, path, to_info):
raise PathMissingError(path, self.url)

@contextmanager
def open_by_relpath(self, path, mode="r", encoding=None):
def open_by_relpath(self, path, mode="r", encoding=None, **kwargs):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it works it should stay as is. Let's not add ignored arguments, this will lead to silent errors.

@skshetry skshetry force-pushed the fallback-external-repo branch from 566aa57 to 003b5c3 Compare January 23, 2020 08:59
@skshetry skshetry force-pushed the fallback-external-repo branch from cdd9d0d to 2e0b143 Compare January 23, 2020 09:43
@efiop efiop merged commit 1a7eb8f into iterative:master Jan 23, 2020
@skshetry skshetry deleted the fallback-external-repo branch January 23, 2020 11:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

api: does not work with git only local repository
4 participants