Upload or put object in S3 failing silently #1067
Comments
|
Could you share a santized version of your debug logs? That should have more information about what's going on. You can enable debug logs by adding |
jamesls
added
the
response-requested
|
OK, here are the debug logs. 2017-04-24 09:27:29,491 botocore.credentials [DEBUG] Looking for credentials via: env Mon, 24 Apr 2017 16:27:29 GMT 2017-04-24 09:27:32,687 botocore.hooks [DEBUG] Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x7fc9ee9b8050> |
|
I'm experiencing the same behavior. Please let me know if you would like my debug logs as well. |
|
Any update on this? |
|
I have the same issue with files that were meant to be uploaded by a background process not being available. Is there any way of knowing if an upload has been successful, without making another S3 request? |
|
Same problem here - is there any solution to that ? |
|
So just for your information (and perhaps this will help someone figure out what's up,) using the Service Resource instead of the client works fine. The upload into S3 "folders" works perfectly. Here's some example code: So at least there is a workaround for the problem with the client. |
|
@michellemurrain will this raise an error if upload fails ? |
|
It raises an error if, for instance, the bucket doesn't exist, or there is something wrong with the key or the file, but to my knowledge that wasn't a problem with the client API either. It hasn't failed to do an upload in normal circumstances using "folders" (that is, with a valid bucket, key and file), so I can't say whether it will raise an error if it does fail (I'll definitely update this ticket if I run into that.) |
|
The client will raise an error in those scenarios as well. The problem that we have had is different. I haven't actually seen it while happening ( always in background processes), but the upload process is completed without error, and the file is not there. At this stage I can only make assumption that this may be related on connection issues, but it feels unlikely as the code is running on EC2, same region as the S3.. In any case needs more investigation on our end as well. |
|
The only situation in which I've ever had that behavior (upload fails, no error) was using the client to store files in S3 in a folder structure. Storing files in S3 using the client w/o a folder, or using the service resource hasn't led to any issues in file uploads. We're also using an EC2 to run this code. |
|
hm, we are storing the files inside folders as well, using client('s3'), will try to get time and switch to resource and then see if the issue persists. |
|
In my case, it seems as if uploads were not failing. Rather, there was a gap between the time when the client reported success and the files actually appeared in S3. Not sure if this is a network issue or perhaps something related to background processes not firing immediately as @latusaki referred to. |
|
I’m having a very similar silent failing upload issue, was it any help using resources @latusaki? |
|
We added logic to trigger re-fetch and upload on the endpoint that downloads S3 files instead of the upload. Upload is done using |
|
So it was not using |
|
I do think of using resource instead of client as a work-around, rather than a fix. I don't know what's going on at the code level to make using client a problem. I'm hoping to actually dig into this sometime in the next couple of weeks. |
|
In my case locally there were no issues with files up to 100MB, which is failing in our staging environment. The difference is our s3 uploads is triggered via django views, which are also wrapped by gunicorn workers and they also seem to have a default 30 second timeout window. Which could be our root cause, not the S3 or boto library itself. |
|
Having a leading slash in your path to the object will create a folder with a blank name in your bucket root, which holds the uploads. path = "/uploads/"
s3.Object(my_bucket, path + filename).put(Body=data)will create / /uploads in your bucket root (note the blank folder) and put the object in there rather than in /uploads. Since boto was also spamming the bucket root with odd files, I didn't notice the empty folder until much later and lost a lot of time. Could be your problem, too? |
|
@maxpearl #1067 (comment) saved my day. But I still think put_object not working properly must be considered as a known bug. Interesting and weird thing is that this issue is intermittent with s3_put_object |
|
+1 I am also getting timeouts for calls to s3 using boto3. I tried all the permutations mentioned above with no luck. i.e., uploading as resource, object etc. Even |
|
my issue was caused by lambda disable internet access when explicit vpc setting. It has nothing to do with s3 or boto3. Sorry for the confusion! https://medium.com/@philippholly/aws-lambda-enable-outgoing-internet-access-within-vpc-8dd250e11e12 |
|
I'm experiencing the same problem when both using client and resource objects. Have you experienced a similar issue on your workaround as well @maxpearl ? |
|
Hi @cagriar - the issue only arose using the client, and since we switched to using the resource, we haven't run into this again at all. |
|
If method PUT, HTTP response status code 400: Could this be raised as |
|
In my case it will fail silently, won't create the object, no exception, unless the object's key has a dot in it, for example However, changing the key to |
|
Had the same issue and using resource solve the issue. |
swetashre
added
s3
and removed
response-requested
|
@maxpearl- Following up on this issue. According to this code snippet key_name = folder + '/'
s3_connect = boto3.client('s3', s3_bucket_region,)
# upload File to S3
for filename in os.listdir(folder):
s3_name = key_name + filename
print folder, filename, key_name, s3_name
upload = s3_connect.upload_file(
s3_name, s3_bucket, key_name,
)Here So if you replace And this code snippet import boto3
testfile = 'test.csv'
bucket_name = 'bucket-name-here'
folder_name = 'folder-name'
key = folder_name + '/' + testfile
s3 = boto3.resource('s3')
bucket = s3.Bucket(bucket_name)
bucket.upload_file(testfile, key)is working because here you have mentioned Please let me know if you have any questions. |
swetashre
added
the
closing-soon
|
You are correct, the code way above (a few years old now) is incorrect. I did so much testing of the issue - I have no idea which iteration of testing that particular code is from. Theoretically, though, it shouldn't have failed silent - it should have just created the folder (which it did not do in my testing). I did testing just now, and I have been unable at this time, with either python 2 or 3, to re-create the issue I had before - either with correct or incorrect code - a key name with a folder without a file will correctly create the folder, and the client creates a file just fine. I will close this now. |
no-response
bot
removed
the
closing-soon
|
In my case, using boto3.client, I have observed that if any intermediate folder is missing then client upload fails silently to output the file. For example I am uploading files like |
|
Having an issue with this now. it seems I had this before and referenced this issue. At the time, changing from using client to resource resolved the issue for me. But now I'm experiencing the same thing with the bucket Resource. I'm trying to upload a single file with no directory in the key name. just |
|
For anyone that comes across this thread, I was having the same issues. I thought that upload_file was failing silently, but the reason I could not see the key after uploading it was because I was using list_objects to list all my keys. list_objects has a limit on how many keys it will return, so I fixed this by using paginator and was able to see my keys after. |
|
Why am I back here. Seeing the same issue now with s3client.upload_file() , and this isn't new code. Changed from using Also tried to set the region via |
|
same issue here! I also tried: does anybody have an update here? |
|
Facing this issue in prod! Setting custom checks to raise exceptions. Can we please reopen this issue?! |
|
any solution for this, i am having the same issue |
|
@Rahulreddy1020 could you create a new issue for further investigation? Please share a code snippet for reproducing the issue and debug logs (with any sensitive info redacted) which you can get by adding |
|
Make sure your key's folder name doesn't start with a slash, it should be a relative path. That was my mistake. |
Removing the leading slash in the key also solved my issue (previously |
and others
I've been trying to upload files from a local folder into folders on S3 using Boto3, and it's failing kinda silently, with no indication of why the upload isn't happening.
Printing upload just says "None", with no other information. No upload happens. I've also tried using put_object:
and I get an HTTPS response code of 200 - but no files upload.
First, I'd love to solve this problem, but second, it seems this isn't the right behavior - if an upload doesn't happen, there should be a bit more information about why (although I imagine this might be a limitation of the API?)
The text was updated successfully, but these errors were encountered: