Pull dataset from studio if not available locally #901
Conversation
If the following case are met, this will pull dataset from Studio. - User should be logged in to Studio. - The dataset or version doesn't exist in local - User has not pass studio=False to from_dataset. In such case, this will pull the dataset from studio before continuing further. The test is added to check for such behavior. Closes #874
Deploying datachain-documentation with
|
| Latest commit: |
a479a1b
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://0302fb58.datachain-documentation.pages.dev |
| Branch Preview URL: | https://amrit-from-dataset.datachain-documentation.pages.dev |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #901 +/- ##
==========================================
- Coverage 87.69% 87.65% -0.05%
==========================================
Files 130 130
Lines 11665 11690 +25
Branches 1586 1590 +4
==========================================
+ Hits 10230 10247 +17
- Misses 1038 1043 +5
- Partials 397 400 +3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
| @@ -1112,6 +1135,21 @@ def __iter__(self): | |||
| def __or__(self, other): | |||
| return self.union(other) | |||
|
|
|||
| def pull_dataset(self, name: str, version: Optional[int] = None) -> "DatasetRecord": | |||
| print("Dataset not found in local catalog, trying to get from studio") | |||
There was a problem hiding this comment.
let's use logger here in debug mode? @skshetry what is your take?
There was a problem hiding this comment.
I tried using logger with info here at beginning. But print seemed consistent with other similar messages. And also, we definitely want user to know we are trying to get it from Studio so that they can expect the delay in execution of the code.
If the following case are met, this will pull dataset from Studio.
In such case, this will pull the dataset from studio before continuing
further.
The test is added to check for such behavior.
Closes #874