BUG: s3 reads from public buckets not working #34626

ayushdg · 2020-06-07T07:22:00Z

I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
(optional) I have confirmed this bug exists on the master branch of pandas.

Code Sample

# Your code here
import pandas as pd
df = pd.read_csv("s3://nyc-tlc/trip data/yellow_tripdata_2019-01.csv")

Error stack trace

Traceback (most recent call last):
  File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/pandas/io/s3.py", line 33, in get_file_and_filesystem
    file = fs.open(_strip_schema(filepath_or_buffer), mode)
  File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/fsspec/spec.py", line 775, in open
    **kwargs
  File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/s3fs/core.py", line 378, in _open
    autocommit=autocommit, requester_pays=requester_pays)
  File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/s3fs/core.py", line 1097, in __init__
    cache_type=cache_type)
  File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/fsspec/spec.py", line 1065, in __init__
    self.details = fs.info(path)
  File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/s3fs/core.py", line 530, in info
    Key=key, **version_id_kw(version_id), **self.req_kw)
  File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/s3fs/core.py", line 200, in _call_s3
    return method(**additional_kwargs)
  File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/client.py", line 622, in _make_api_call
    operation_model, request_dict, request_context)
  File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/client.py", line 641, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/signers.py", line 160, in sign
    auth.add_auth(request)
  File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials
During handling of the above exception, another exception occurred:
Traceback (most recent call last):

File "", line 1, in 

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/pandas/io/parsers.py", line 676, in parser_f

return _read(filepath_or_buffer, kwds)

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/pandas/io/parsers.py", line 431, in _read

filepath_or_buffer, encoding, compression

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/pandas/io/common.py", line 212, in get_filepath_or_buffer

filepath_or_buffer, encoding=encoding, compression=compression, mode=mode

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/pandas/io/s3.py", line 52, in get_filepath_or_buffer

file, _fs = get_file_and_filesystem(filepath_or_buffer, mode=mode)

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/pandas/io/s3.py", line 42, in get_file_and_filesystem

file = fs.open(_strip_schema(filepath_or_buffer), mode)

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/fsspec/spec.py", line 775, in open

**kwargs

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/s3fs/core.py", line 378, in _open

autocommit=autocommit, requester_pays=requester_pays)

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/s3fs/core.py", line 1097, in init

cache_type=cache_type)

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/fsspec/spec.py", line 1065, in init

self.details = fs.info(path)

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/s3fs/core.py", line 530, in info

Key=key, **version_id_kw(version_id), **self.req_kw)

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/s3fs/core.py", line 200, in _call_s3

return method(**additional_kwargs)

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/client.py", line 316, in _api_call

return self._make_api_call(operation_name, kwargs)

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/client.py", line 622, in _make_api_call

operation_model, request_dict, request_context)

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/client.py", line 641, in _make_request

return self._endpoint.make_request(operation_model, request_dict)

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request

return self._send_request(request_dict, operation_model)

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/endpoint.py", line 132, in _send_request

request = self.create_request(request_dict, operation_model)

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/endpoint.py", line 116, in create_request

operation_name=operation_model.name)

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit

return self._emitter.emit(aliased_event_name, **kwargs)

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit

return self._emit(event_name, kwargs)

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit

response = handler(**kwargs)

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/signers.py", line 90, in handler

return self.sign(operation_name, request)

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/signers.py", line 160, in sign

auth.add_auth(request)

File "/home/conda/envs/pandas-test/lib/python3.7/site-packages/botocore/auth.py", line 357, in add_auth

raise NoCredentialsError

Problem description

Reading directly from s3 public buckets (without manually configuring the anon parameter via s3fs) is broken with pandas 1.0.4 (worked with 1.0.3).

Looks like reading from public buckets requires anon=True while creating the filesystem. This 22cf0f5 seems to have introduced the issue, where anon=False is passed when the noCredentialsError is encountered.

Output of `pd.show_versions()`

INSTALLED VERSIONS

commit : None
python : 3.7.7.final.0
python-bits : 64
OS : Linux
OS-release : 4.15.0-55-generic
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.0.4
numpy : 1.18.1
pytz : 2020.1
dateutil : 2.8.1
pip : 20.0.2
setuptools : 47.1.1.post20200604
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 0.15.1
pytables : None
pytest : None
pyxlsb : None
s3fs : 0.4.2
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None

The text was updated successfully, but these errors were encountered:

jorisvandenbossche · 2020-06-07T13:22:41Z

@ayushdg thanks for the report!

cc @simonjayhawkins @alimcmaster1 for 1.0.5, it might be safer to revert #33632, and then target the fixes (like #34500) to master

alimcmaster1 · 2020-06-07T14:15:06Z

Agree @jorisvandenbossche - do you want me to open a PR to revert #33632 on 1.0.x branch? Apologies for this change it didn’t go as planned. I’ll check why our test cases didn’t catch the above!

jorisvandenbossche · 2020-06-07T14:18:46Z

do you want me to open a PR to revert #33632 on 1.0.x branch?

Yes, that sounds good

Apologies for this change it didn’t go as planned.

No, no, nobody of us had foreseen the breakages ;)

alimcmaster1 · 2020-06-07T22:01:03Z

Can't seem to reproduce this using moto... Potentially related: https://github.com/dask/s3fs/blob/master/s3fs/tests/test_s3fs.py#L1089

(I can repo locally using the s3 URL above - if I remove AWS Creds from my environment)

alimcmaster1 · 2020-06-09T23:05:45Z

The fix for this to target 1.1 is to set ‘anon=True’ in S3FileSystem https://github.com/pandas-dev/pandas/pull/33632/files#diff-a37b395bed03f0404dec864a4529c97dR41

I’ll wait as we are moving to fsspec which gets rid of this logic #34266 - but we should definitely trying using moto to test this.

TomAugspurger · 2020-06-12T16:18:41Z

Can anyone summarize the status here?

1.0.3: worked
1.0.4: broken
master: broken?
master+#34266: broken?

Do we have a plan in place to restore this? IIUC the old way was to

try with the default (which I think looks up keys based on env vars)
If we get an error, retry with anon=True

jorisvandenbossche · 2020-06-12T17:03:24Z

Yep, it broke in 1.0.4, and will be fixed in 1.0.5 by reverting the patch that broke it.
That means that master is still broken, and thus we first need to write a test for it, and check whether #34266 actually fixes it already, or otherwise still fix it differently.

The old way was indeed to try with anon=True if it first failed. I suppose we can "simply" restore that logic? (in case it's not automatically fixed with fsspec)

TomAugspurger · 2020-06-12T17:35:41Z

Thanks

in case it's not automatically fixed with fsspec

It's not. So we'll need to do that explicitly. Long-term we might want to get away from this logic by asking users to do read_csv(..., storage_options={"requester_pays": False}). But for 1.1 we'll want to restore the old implicit retry behavior if possible.

Closes pandas-dev#34626 This works in 1.0.4 I think, so no whatsnew.

jorisvandenbossche · 2020-07-08T12:05:57Z

Long-term we might want to get away from this logic

On the other hand, it seems nice that reading from a public bucket just works out of the box without needing the pass any option?

ayushdg added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Jun 7, 2020

jorisvandenbossche added IO Parquet parquet, feather Regression Functionality that used to work in a prior pandas version and removed Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Jun 7, 2020

jorisvandenbossche added this to the 1.0.5 milestone Jun 7, 2020

alimcmaster1 self-assigned this Jun 7, 2020

alimcmaster1 mentioned this issue Jun 7, 2020

Revert backport of #33632: Parquet & s3 I/O changes #34632

Merged

dchigarev mentioned this issue Jun 9, 2020

Updated required python version to 3.6.1+ modin-project/modin#1551

Merged

5 tasks

jorisvandenbossche mentioned this issue Jun 10, 2020

RLS: 1.0.5 #34684

Closed

simonjayhawkins mentioned this issue Jun 15, 2020

CI testing for s3 reads from public buckets on 1.0.x - DO NOT MERGE #34793

Closed

jorisvandenbossche modified the milestones: 1.0.5, 1.1 Jun 15, 2020

jorisvandenbossche added the Blocker Blocking issue or pull request for an upcoming release label Jun 15, 2020

TomAugspurger mentioned this issue Jun 15, 2020

BUG: read_csv stopped working with s3 file system #34519

Closed

TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Jun 18, 2020

REGR: Fixed reading from public S3 buckets with credentials

9d38c57

Closes pandas-dev#34626 This works in 1.0.4 I think, so no whatsnew.

TomAugspurger mentioned this issue Jun 18, 2020

[WIP]REGR: Fixed reading from public S3 buckets with credentials #34866

Closed

alimcmaster1 mentioned this issue Jun 19, 2020

BUG/TST: Read from Public s3 Bucket Without Creds #34877

Merged

1 task

TomAugspurger closed this as completed in #34877 Jul 15, 2020

prutskov mentioned this issue Oct 29, 2021

FIX-#2301: Add support of storage_options in read_* modin-project/modin#3552

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: s3 reads from public buckets not working #34626

BUG: s3 reads from public buckets not working #34626

ayushdg commented Jun 7, 2020

INSTALLED VERSIONS

jorisvandenbossche commented Jun 7, 2020

alimcmaster1 commented Jun 7, 2020

jorisvandenbossche commented Jun 7, 2020

alimcmaster1 commented Jun 7, 2020 •

edited

Loading

alimcmaster1 commented Jun 9, 2020

TomAugspurger commented Jun 12, 2020

jorisvandenbossche commented Jun 12, 2020 •

edited

Loading

TomAugspurger commented Jun 12, 2020

jorisvandenbossche commented Jul 8, 2020

BUG: s3 reads from public buckets not working #34626

BUG: s3 reads from public buckets not working #34626

Comments

ayushdg commented Jun 7, 2020

Code Sample

Problem description

Output of pd.show_versions()

INSTALLED VERSIONS

jorisvandenbossche commented Jun 7, 2020

alimcmaster1 commented Jun 7, 2020

jorisvandenbossche commented Jun 7, 2020

alimcmaster1 commented Jun 7, 2020 • edited Loading

alimcmaster1 commented Jun 9, 2020

TomAugspurger commented Jun 12, 2020

jorisvandenbossche commented Jun 12, 2020 • edited Loading

TomAugspurger commented Jun 12, 2020

jorisvandenbossche commented Jul 8, 2020

Output of `pd.show_versions()`

alimcmaster1 commented Jun 7, 2020 •

edited

Loading

jorisvandenbossche commented Jun 12, 2020 •

edited

Loading