Skip to content

Commit

Permalink
Clarify S3FileSystem Credentials chain for EC2
Browse files Browse the repository at this point in the history
The C implementation of `DefaultAWSCredentialsProviderChain` not only reads the environment variable when trying to resolve AWS credentials, but also looks at profile config and the EC2 Instance Metadata Service.

I want to document this behavior in the documentation.
The `s3fs` documentation mentions the specific case for EC2
  • Loading branch information
kevinjqliu authored and bluefishdata committed May 10, 2023
1 parent 5de5692 commit a2748ff
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 5 deletions.
5 changes: 3 additions & 2 deletions docs/source/python/filesystems.rst
Original file line number Diff line number Diff line change
Expand Up @@ -153,8 +153,9 @@ PyArrow implements natively a S3 filesystem for S3 compatible storage.
The :class:`S3FileSystem` constructor has several options to configure the S3
connection (e.g. credentials, the region, an endpoint override, etc). In
addition, the constructor will also inspect configured S3 credentials as
supported by AWS (for example the ``AWS_ACCESS_KEY_ID`` and
``AWS_SECRET_ACCESS_KEY`` environment variables).
supported by AWS (such as the ``AWS_ACCESS_KEY_ID`` and
``AWS_SECRET_ACCESS_KEY`` environment variables, AWS configuration files,
and EC2 Instance Metadata Service for EC2 nodes).


Example how you can read contents from a S3 bucket::
Expand Down
12 changes: 9 additions & 3 deletions python/pyarrow/_s3fs.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -140,14 +140,20 @@ cdef class S3FileSystem(FileSystem):
"""
S3-backed FileSystem implementation
If neither access_key nor secret_key are provided, and role_arn is also not
provided, then attempts to initialize from AWS environment variables,
otherwise both access_key and secret_key must be provided.
AWS access_key and secret_key can be provided explicitly.
If role_arn is provided instead of access_key and secret_key, temporary
credentials will be fetched by issuing a request to STS to assume the
specified role.
If neither access_key nor secret_key are provided, and role_arn is also not
provided, then attempts establish the credentials automatically.
S3FileSystem will try the following methods, in order:
* `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_SESSION_TOKEN` environment variables
* configuration files such as `~/.aws/credentials` and `~/.aws/config`
* for nodes on Amazon EC2, the EC2 Instance Metadata Service
Note: S3 buckets are special and the operations available on them may be
limited or more expensive than desired.
Expand Down

0 comments on commit a2748ff

Please sign in to comment.