[Tracking] Getting TemporaryRedirect when using aiobotocore but not when using botocore

[chuck1]

I have a recently created a new bucket and am trying to read and write to it. If I understand correctly, when a bucket is created, you can get TemporaryRedirect errors for a while. But I have found that I only get that error when using aiobotocore, not when using botocore. Below is my actual test program. The first half, the botocore part, runs with no problems and I get good responses. But the put_object call in the aiobotocore part raises An error occurred (TemporaryRedirect) when calling the PutObject operation: Please re-send this request to the specified temporary endpoint. Continue to use the original request endpoint for future requests. I am using aiobotocore version 0.4.0 and python 3.5.

I thought that the two should produce the same result. What could be going on here?

import asyncio
import os
import sys

import botocore.session

conf = # object with my access and secret keys and other info

session = botocore.session.get_session()

client = session.create_client(
        's3', 
        region_name=conf.REGION,
        aws_secret_access_key=conf.SECRET_ACCESS_KEY,
        aws_access_key_id=conf.ACCESS_KEY_ID)

resp = client.put_object(
        Bucket=conf.BUCKET,
        Key='test',
        Body=b'hello')

print(resp)
print()

resp = client.get_object(
        Bucket=conf.BUCKET,
        Key='test')

print(resp)
print()


import aiobotocore

loop = asyncio.get_event_loop()

session = aiobotocore.session.get_session()

async def go():

    async with session.create_client(
            's3', 
            region_name=conf.REGION,
            aws_secret_access_key=conf.SECRET_ACCESS_KEY,
            aws_access_key_id=conf.ACCESS_KEY_ID) as client:

        resp = await client.put_object(
                Bucket=conf.BUCKET,
                Key='test',
                Body=b'hello')

        print(resp)
        print()
        
        resp = await client.get_object(
                Bucket=conf.BUCKET,
                Key='test')

        print(resp)
        print()

loop.run_until_complete(go())

[thehesiod]

interesting, looking into this

[thehesiod]

btw for tests like these I suggest using AWS_PROFILE: http://boto3.readthedocs.io/en/latest/guide/configuration.html#shared-credentials-file

so I just repro'd with this more complete test:

import asyncio
import botocore.session
import aiobotocore

TEST_BUCKET = 'thehesiod-dummy2'
TEST_KEY = 'blah'
CREATE_REGION = 'eu-west-1'
QUERY_REGION = 'us-west-2'


create_client = botocore.session.get_session().create_client('s3', region_name=CREATE_REGION)


def boto_test():
    print('boto_test')
    session = botocore.session.get_session()

    client = session.create_client('s3', region_name=QUERY_REGION)

    resp = client.put_object(
            Bucket=TEST_BUCKET,
            Key=TEST_KEY,
            Body=b'hello')

    print(resp)
    print()

    resp2 = client.get_object(
            Bucket=TEST_BUCKET,
            Key=TEST_KEY)

    print(resp2)
    print()
    return resp, resp2


def aioboto_test():
    print('aioboto_test')
    loop = asyncio.get_event_loop()
    session = aiobotocore.session.get_session()

    async def go():
        async with session.create_client('s3', region_name=QUERY_REGION) as client:
            resp = await client.put_object(
                    Bucket=TEST_BUCKET,
                    Key=TEST_KEY,
                    Body=b'hello')

            print(resp)
            print()

            resp2 = await client.get_object(
                    Bucket=TEST_BUCKET,
                    Key=TEST_KEY)

            print(resp2)
            print()

        return resp, resp2

    loop.run_until_complete(go())

create_client.create_bucket(Bucket=TEST_BUCKET,
                          CreateBucketConfiguration={'LocationConstraint': CREATE_REGION})

try:
    boto_test()
    aioboto_test()
finally:
    create_client.delete_object(Bucket=TEST_BUCKET, Key=TEST_KEY)
    create_client.delete_bucket(Bucket=TEST_BUCKET)

digging into it

[thehesiod]

I have a feeling this was introduced with e44d9ba I'm still trying to figure out why it was done

[thehesiod]

hah, reverting that fixes this testcase but breaks the other one, I'll need to investigate the old testcase now

[thehesiod]

ahh, IIRC sometimes AWS puts the redirect location in the body and not the header (where it belongs), which means that if we tell aiohttp to follow redirects it will fix one case, if we turn it off botocore will handle the other case...argh, this will be interesting to fix.

[thehesiod]

fascinating, when run with botocore the "Location" header is set on the redirect response, so we need to figure out how to get it to be populated with aiohttp

[thehesiod]

aha! So apparently requests does not treat the response as a redirect if the "location" is not in the headers. https://github.com/boto/botocore/blob/develop/botocore/vendored/requests/sessions.py#L99 ends up calling https://github.com/boto/botocore/blob/develop/botocore/vendored/requests/models.py#L643 which means it won't fail if the "location" tag is missing, and instead will bump it up to botocore and botocore then issues a new request. I'll probably add an issue to aiohttp to be able to support this scenario.

[thehesiod]

ok so I created a work-around until we get an official fix from aiohttp. I'm going to do some more testing and if @jettify approves will roll this into a new release.

[jettify]

Good catch! Please do release :)

[thehesiod]

thanks @jettify and thanks @chuck1 for bringing this to our attention! I'll try to get a release out today with the fix

[jettify]

Thanks @thehesiod ! Hope one day you will tell me how you manage to find/fix bugs so quickly :)

[thehesiod]

my secret is pycharm :) Then just run both cases in parallel.

[thehesiod]

oh, and be high on caffeine, lol

[thehesiod]

@chuck1 ok aiobotocore 0.4.1 released with fix

[thehesiod]

btw we're going to leave this issue open to remind us to remove the hack when aiohttp releases a fix

[thehesiod]

already reverted