TST/CI: Use moto to mock S3 calls #17325
Description
https://github.com/spulec/moto
I think most of our tests hitting s3 can be mocked now that we're using s3fs and boto3. We don't have any code doing network requests, auth, etc. As an example:
import pytest
import pandas as pd
from moto import mock_s3
import boto3
f = open("pandas/tests/io/data/tips.csv", 'rb')
@mock_s3
def test_foo():
conn = boto3.resource("s3", region_name="us-east-1")
conn.create_bucket(Bucket="pandas-test")
conn.Bucket("pandas-test").put_object(Key="not-a-real-key.csv", Body=f)
result = pd.read_csv("s3://pandas-test/not-a-real-key.csv")
assert isinstance(result, pd.DataFrame)
with pytest.raises(Exception):
result = pd.read_csv("s3://pandas-test/for-real-not-a-real-key.csv")
It'd probably be good to have a few integration tests that actually hit S3, but I think most can be replaced with mocked calls.