-
Notifications
You must be signed in to change notification settings - Fork 86
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
download while on AWS creates one file #562
Comments
@itcarroll, I suspect this is not specific to cryointhecloud, nor AWS in general. You likely need to first create the target directory (I don't know if it will be created for you, which may be the problem). Delete the erroneous file, create the If that still doesn't work, then in addition, specify Please report back your findings. Whatever your findings, this appears to be behavior that we must correct (and likely clearly document as well). |
Oh, I should have mentioned that the function works as documented, even creating the directory if it doesn't exist, without direct access. |
Oh, very interesting. Would you mind trying my suggestions above, just to see if/how things are affected in that environment, just so we have more information about how things work/don't work? |
👀 Good catch!!! |
Creating the folder first causes expected behavior (i.e. multiple files in the folder). Using "test/" as the argument is equivalent to "test". It's possible, based on fsspec's
although that's a bit flimsy looking. |
Cool. Would you mind trying one more test? Delete the files in the |
If a "test" folder exists, download works with "test" and "test/". If there is nothing called "test", one file called "test" is created for either argument. Conversion to |
After a bit of digging into fsspec, I see the problem, which boils down to this: either the local (destination) directory must exist (and be a directory) OR the specified path must include a trailing slash. Otherwise, the destination is interpreted as a file rather than a directory, thus resulting in the reported behavior in this issue. |
@itcarroll, I don't have a CryoCloud account (I just submitted a request), but I did attempt to reproduce the error in the MAAP environment. For reference, I noticed this in your reported output:
Here's the output that I got:
That resulted in the expected However, I noticed that my output was different than yours, and it turns out that when I ran the code, earthaccess did not correctly determine that I was "in region," thus did not use the S3 download URLs, and instead used the HTTPS URLs. Therefore, I removed the earthaccess.__store__.in_region = True Then I ran the download again, and this time I reproduced the behavior you reported:
This resulted in a single file named The difference is that when earthaccess thinks it's not running "in region," it takes a different path in the code, and calls the |
That all checks out with my understanding, and I like the explicit fix. Thanks! I don't know what the MAAP is, but perhaps you are also hitting #444. |
Sorry, FYI, here's MAAP: https://maap-project.org/ I suspect I'm indeed hitting #444. |
The
earthaccess.download
function appears to be (over)writing all data received to a single file, rather than to separate files in the provided directory, when working on AWS. This example uses earthaccess 0.9.0 on cryointhecloud.Instead of a "test" folder, I get one "test" file.
Expected behavior would be either to
raise "STAHP! Wut R u DoINg?"
or just download the files.The text was updated successfully, but these errors were encountered: