Fix for instantiation of pulled dataset

[ilongin]

Adding a fix for instantiated datasets pulled from Studio.
In addition, it adds small improvement for avoiding going to Studio if dataset with specified version is already present locally

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.87%. Comparing base (b0e3a32) to head (24f28a0).
Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #573      +/-   ##
==========================================
+ Coverage   87.83%   87.87%   +0.04%     
==========================================
  Files         100      100              
  Lines        9993    10007      +14     
  Branches     1356     1357       +1     
==========================================
+ Hits         8777     8794      +17     
+ Misses        873      871       -2     
+ Partials      343      342       -1     

Flag	Coverage Δ
datachain	87.81% <100.00%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[shcheklein]

Thanks @ilongin ! Can we add a test for this?

[shcheklein]

Btw, do we have any docs? what is the command that was broken?

[mattseddon]

[Q] Is there some validation that happens to confirm that dataset versions are "the same" between local and remote?

[mattseddon]

E.g. my_dataset@v1 on remote is full of PDFs and my_dataset@v1 locally is images.

[ilongin]

Good question. No, there is no such thing ATM. Currently it's up to user to make sure he doesn't "shadow" remote dataset. In order to make this kind of validation we need to have equal function between datasets (I will create an issue for this) and my first feeling is that it's not that trivial to implement it (maybe I'm wrong, idk)

[ilongin]

Actually, it seems like a simple uuid should be enough for this, which is trivial

[cloudflare-workers-and-pages]

Deploying datachain-documentation with    Cloudflare Pages

Latest commit:	24f28a0
Status:	✅  Deploy successful!
Preview URL:	https://64b5c9ed.datachain-documentation.pages.dev
Branch Preview URL:	https://ilongin-566-fix-dataset-pull.datachain-documentation.pages.dev

View logs

[ilongin]

Thanks @ilongin ! Can we add a test for this?

Added a test

[ilongin]

Btw, do we have any docs? what is the command that was broken?

Command that was broken was datachain pull ds://some-dataset-from-studio . By default this command pulls dataset to local DB but it also starts instantiation of it and that instantiation was broken. Instantiation can be avoided by adding --no-cp flag.
We can maybe think of removing instantiation as default option and add it as optional (to have --cp flat instead of --no-cp).

So currently if user wants to pull dataset and instantiate it locally, it can be done in two ways:

datachain pull ds://my-bucket -o output_dir  # does pulling and instantiating in one command

datachain pull ds://my-bucket --no-cp  # does only pulling
datachain cp ds://my-bucket output_dir  # instantiating local dataset

[shcheklein]

We can maybe think of removing instantiation as default option and add it as optional (to have --cp flat instead of --no-cp).

sounds reasonable to me. Or do a separate command to instantiate it?