-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gdrive remote strange behaviour #3230
Comments
Hi @RomanVeretenov ! We are still actively working on gdrive, and have recently released some important changes. Please try upgrading to 0.82.2 and check if the issue persists. 🙂 |
@RomanVeretenov could you try to specify |
Seems that it's a regular personal drive (but with unlimited storage). At least I login to regular google account via standart google login page. If you know how I could check it more detailed, please let me know. |
So I have updated dvc to 0.82.3 and added no_traverse true Now I'm getting following error
But if I follow the link, I see that quotas aren't exceeded |
@RomanVeretenov could you please try the latest version (remove @RomanVeretenov not sure what's up with rate limits. Will try to reproduce. 👀 Btw, I would also recommend to remove the files on the google drive remote you don't need anymore (from the time you was pushing them one by one). Just create a clean one and push again, or use |
I will try ti reproduce it on latest DVC version with new gdrive folder and without |
So it still does not work
|
@RomanVeretenov could you please check what version of PyDrive2 is installed? ( |
I'm still not sure what is happening wit the exceeded limits. btw ... still looking into it and trying to reproduce |
|
After updating PyDrive2 to 1.4.4,
and the cache value differ always. I have limited the --jobs to 4, but it does not help. I'm working on a remote Ubuntu pc via ssh, the pc is located somewhere in USA, but I newer faced any problems with network connection on this pc. Usual gdrive upload/download speed it there 10 MBytes/sec. |
@RomanVeretenov thanks, I think I know what the problem is with the "Network is unreachable" (I hit it myself already, it leaks connections) and I know already how fix this, we will prepare and release and new version this weekend. Thank you for the feedback and your patience. |
@RomanVeretenov I think, @efiop has released a new version of the DVC (and the PyDrive) - it should be more stable. Would you mind to give it a try again, please? There will be the next iteration released soon, with some major performance improvements and stability when we deal with a lot of files, |
I have updated dvc and pydrive
But it still doesn't work as expected I do What I get is a huge list of missing caches like
Also I can't find the 'bbbc75ce995855d3e965f958fbd584bb' in my google drive. |
@RomanVeretenov what does Also I can't find the 'bbbc75ce995855d3e965f958fbd584bb' in my google drive. It should be something like |
@RomanVeretenov also, we've just released a new version with some major changes to the GDrive, could you please try to check it out as well. |
bc75ce995855d3e965f958fbd584bb persist in gdrive |
@RomanVeretenov and at the same moment if you run |
Yes. I think there was a mess with local remote and gremote. I will try once more with clean gdrive storage paying attention to not ot forget to use -r option |
Ok, closing this one. We've fixed a bunch of issues here and it should be way more stable. The next one is to try to reproduce and fix #3098 Then optimizations to make it work way faster with large amounts of files - V3 API, etc. @RomanVeretenov please keep us updates about your experience with it. |
DVC version - 0.80.0, Installed via pip
Ubuntu 18.04.2 LTS
To continue situation described in this issue:
I've created 2 remotes in .dvc/config
Syncing with local remote works fine.
Syncing with gdrive remote behaves in some strange way:
Situation 1:
I do
dvc push -r gremote
, wait until all data is uploaded (it really appears in gdrive folder) and after it dogit clean -dxf
and thendvc pull -r gremote
(also I can clone the git repo to another place, behaviour will be the same)Expected: dvc progress bar appears and pull begins.
Got:
WARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
name: plates/***/706.zip, md5: ebd148595ce875c5754e95604d05d550
and so on and here follows list of all files that must be pulled
ERROR: failed to pull data from the cloud - Checkout failed for following targets:
plates/***/706.zip
and so on
Did you forget to fetch?
dvc fetch
says the sameWARNING: Some of the cache files do not exist neither locally nor on remote. Missing cache files:
As the result, no files are pulled
Situation 2:
I do
dvc push -r gremote
and then repeat it.Expected:
everything is up to date
messageGot: push runs again and takes same time as on first run.
The text was updated successfully, but these errors were encountered: