moto fails to mock newer boto resulting in AWS being accessed even if boto3.setup_default_session used

[pskowronek]

This is a consequence of my comment to #2076. Here is the minimal sample code to demonstrate that moto may not mocking everything that is necessary to stop boto from reaching AWS.

s3_test.py:

from moto import mock_s3
import boto3
boto3.setup_default_session() # doesn't work

import unittest

@mock_s3
class SimpleTest(unittest.TestCase):

    def setUp(self):
        boto3.setup_default_session()   # doesn't work either
        self.s3 = boto3.resource('s3', region_name='us-east-1')
        self.s3_bucket = self.s3.create_bucket(Bucket='some-bucket')

    def test_sth(self):
        self.assertTrue(True, "should arrive here")

requirements.txt:

boto3 == 1.9.115
botocore == 1.12.115
moto == 1.3.7

To run the test pls execute: python3 -m unittest discover -s . -p "*_test.py".

The result as I see is:
botocore.errorfactory.BucketAlreadyExists: An error occurred (BucketAlreadyExists) when calling the CreateBucket operation: The requested bucket name is not available. The bucket namespace is shared by all users of the system. Please select a different name and try again.

But if you don't have AWS auth in the environment the error will be: botocore.exceptions.NoCredentialsError: Unable to locate credentials

The code above worked perfectly when using moto 1.3.7, boto3 1.7.84 and botocore 1.10.84 (the same was stated in this issue: #1815), for anything newer of boto causes moto to fail to mock properly. I've just spotted #1793 - very similar code sample and exactly the same problems.

[kgutwin]

After reproducing your issue in an isolated Docker container (docker run -ti --rm -v $PWD:/app python:3 bash), I believe the problem you're seeing was reported in #1924 and fixed in PR #1952. That PR has been merged to master, but there are some lingering concerns with its approach (#2058).

Could you try reinstalling moto from git?

pip install --upgrade git+https://github.com/spulec/moto.git

[pskowronek]

@kgutwin Yes, the master fixes the problem - the test above passes successfully.

Can someone shed some light on ETA of the next moto release? Maybe it's time for a beta release at least?

[pskowronek]

@kgutwin - that's strange... on another machine (also OSX) version of moto just right from git does not fix the problem. I assume that some other dependencies may interfere?

On dockerized version of this test it works, but when I modify the code to use SQS (self.request = self.sqs.create_queue(QueueName='queue')) I get:

botocore.errorfactory.QueueNameExists: An error occurred (QueueAlreadyExists) when calling the CreateQueue operation: The specified queue already exists.

So, even moto master has still problems with accessing AWS - this time for SQS and probably some other services, too.

[adamdavis40208]

Moto on pypi hasn't been updated in quite awhile: https://pypi.org/project/moto/#history

:(

edit: Looks like that was fixed. Thanks!

[chris-erickson]

Is Moto dead? The last few months have really been rough, and mocking AWS stuff is becoming miserable. Credit where credit is due of course (AWS changing everything) - what do we do? Needing to use the deprecated decorators to prevent network access is salt in the wound.

[adamdavis40208]

localstack has also been rough lately. both projects need some fresh blood, sweat and tears.

[aizelauna]

Same issue with moto 1.3.8 and with these boto versions:

boto 2.49.0 Amazon Web Services Library
boto3 1.9.169 The AWS SDK for Python
botocore 1.12.169 Low-level, data-driven core of boto 3.

[krtkvrm]

@aizelauna I solved it using

botocore==1.12.85
git+https://github.com/spulec/moto.git@df493ea18de22b4533073eee3e296686019911c2

Installing via commit till moto>1.3.8

[aizelauna]

thanks a lot for the information!

[aizelauna]

Unfortunately, moto declares that it is compatible with botocore 1.12.86 and above which prevents a tool like poetry to resolve dependencies... And no override option is available to fix this case: python-poetry/poetry#697

[spulec]

A new release has been cut. Feel free to open an issue if there are still problems.

[yitzikc]

A new release has been cut. Feel free to open an issue if there are still problems.

I can still reproduce the issue with S3 and SQS tests on
boto3 = "==1.9.189" botocore = "==1.12.189" moto = "==1.3.13"
Same with moto = "==1.3.10"
Could you please give an example of acombination of versions that's expected to work?

[artis3n]

I am also still seeing this issue on

python_version = "3.7"
moto==1.3.13
boto3==1.9.181
botocore==1.12.181

[spulec]

Is it the same code as above that is failing or something different?

The above code passes for me on the latest commit in moto master branch and boto3==1.9.183 and botocore==1.12.1911

[artis3n]

I see the error botocore.exceptions.NoCredentialsError: Unable to locate credentials unless I pass in fake creds as env vars.

[spulec]

Are you using the decorators or server-mode? Can you add the code you are using?

[artis3n]

I'm using server-mode. I have a discrete process that runs outside a test suite and needs AWS services pre-seeded with data for the mock test. It's code from work so I can get you a snippet on Monday. I'm mocking S3.

[spulec]

Got it. With server-mode, you will need to configure the client to point at your server. Near the end of this, we show how to do it either through the config or through passing an endpoint_url to the client. Let me know if you are doing one of those.

[artis3n]

Yup, using endpoint_url. Some code:

pipenv run moto_server s3 -p ${mocks3_port} &
MOTOSERVER_PID=$!
# Moto is sort of broken right now and does not stop Boto3 from failing if AWS env creds do not exist, even if
# moto doesn't actually use them.
AWS_ACCESS_KEY_ID='1111' AWS_SECRET_ACCESS_KEY='2222' pipenv run python ${PWD}/library/test/seed_s3.py <flags, redacted> -e "${mocks3_endpoint}"

The python code:

s3_client = boto3.client(service_name='s3', endpoint_url=s3_url)
s3_client.create_bucket(Bucket=s3_bucket)
[seed_values(path, path_metadata, s3_bucket, s3_namespace, s3_client) for path, path_metadata in to_examine]

This works fine with Moto with the manual inclusion of fake AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, but without manually specifying I'll get an error.

botocore.exceptions.NoCredentialsError: Unable to locate credentials

My understanding is that moto is supposed to inject fake creds for boto3 to be satisfied.

You know what, this seems slightly different than the error from the issue, although both get the credentials error. Let me know if you'd like me to file a separate issue.

[spulec]

Got it.

So, this is actually expected behavior. In the standalone server mode, moto has no influence over the client. Think of the server as a fake AWS web server. You are responsible for configuring the client to point at that server and any other requirements.

In non-server mode, we have the ability to change the client accordingly because the decorator gives us that access. In server mode though, we have no ability to modify the client since the moto code isn't being run.

[artis3n]

Ahh, good to know! There may be an opportunity to make that clearer in the docs.

[spulec]

Yes, we could definitely be clearer in the docs. I'll take a look.

Thanks!