GH-35409: [Python][Docs] Clarify S3FileSystem Credentials chain for EC2

[kevinjqliu]

Rationale for this change

When resolving AWS credentials on EC2 hosts, the underlying AWS SDK also looks at the EC2 Instance Metadata Service.

I want to document this behavior for pyarrow. The s3fs documentation mention this specific case for EC2.

What changes are included in this PR?

Documentation for the behavior described above.

Technical Details

S3FileSystem uses the CS3Options.Defaults() option when no credentials are passed into the constructor. It utilizes the Aws::Auth::DefaultAWSCredentialsProviderChain

The C++ implementation of DefaultAWSCredentialsProviderChain not only reads the environment variable when trying to resolve AWS credentials, but also looks at profile config and the EC2 Instance Metadata Service.

Are these changes tested?

No, just documentation changes

Are there any user-facing changes?

Yes, changing public documentation

• Closes: [Docs] Document S3FileSystem behavior in EC2  #35409

Render Changes

Render the changes locally via Building the doc:
docs/source/python/filesystems.rst:

python/pyarrow/_s3fs.pyx:

[github-actions]

• Closes: [Docs] Document S3FileSystem behavior in EC2  #35409

[github-actions]

⚠️ GitHub issue #35409 has been automatically assigned in GitHub to PR creator.

[jorisvandenbossche]

@kevinjqliu thanks for the contribution!

We have a section about S3FileSystem in the user guide as well: https://arrow.apache.org/docs/dev/python/filesystems.html#s3. Could you have a look to see if that might require an equivalent update?

[kevinjqliu]

@jorisvandenbossche thanks for the review. I've updated the formatting issues and made changes to the S3 section of filesystems.html docs as well.

PTAL

[kevinjqliu]

@jorisvandenbossche fixed a formatting issue. there was something wrong with CI too, hence the force pushes.

[pitrou]

@jorisvandenbossche Would you like to take another look?

[kevinjqliu]

hey, @jorisvandenbossche wanted to ping on this again. thanks!

[github-actions]

⚠️ GitHub issue #35409 has been automatically assigned in GitHub to PR creator.

[kou]

@github-actions crossbow submit docs-preview

[github-actions]

Unable to match any tasks for `docs-preview`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/5709116257

[kou]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: a2748ff

Submitted crossbow builds: ursacomputing/crossbow @ actions-4f25ed291d

Task	Status
preview-docs

[kevinjqliu]

thanks, @kou I've applied your suggestions. Do the commits need to be squashed?

[kou]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: 4cdd0d2

Submitted crossbow builds: ursacomputing/crossbow @ actions-889dfc34d1

Task	Status
preview-docs

[kevinjqliu]

@kou built the changes locally, I've attached a screenshot above

[kou]

Thanks but we have 2 changed pages but the screenshot shows only 1 page.

[kevinjqliu]

The visible change is from docs/source/python/filesystems.rst.
python/pyarrow/_s3fs.pyx is a code comment change.

[kou]

We use it for API reference: https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html#pyarrow.fs.S3FileSystem

[kevinjqliu]

We use it for API reference: https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html#pyarrow.fs.S3FileSystem

I see. I'm following Building a single directory for dev purposes without all the pre-requisites to render the changes for docs/source/python.

The changes in python/pyarrow/_s3fs.pyx does not show up. How can I render that part?

[kou]

We need pyarrow to build documents in python/pyarrow/_s3fs.pyx. So we can't use the approach.

We need to use the full (at least including pyarrow) build approach.
https://arrow.apache.org/docs/developers/documentation.html#building-with-docker is easier than https://arrow.apache.org/docs/developers/documentation.html#building .

[kou]

Could you rebase on main to use #36943 ?

[kevinjqliu]

@kou I've rebased the branch and was able to build it locally using docker. Updated the description above to render changes to both files.

Thanks for guiding me through the process

[kou]

@github-actions crossbow submit preview-docs

[github-actions]

Revision: f4b66d7

Submitted crossbow builds: ursacomputing/crossbow @ actions-84183cdf1d

Task	Status
preview-docs

[kou]

http://crossbow.voltrondata.com/pr_docs/35312/

• http://crossbow.voltrondata.com/pr_docs/35312/python/filesystems.html#s3
• http://crossbow.voltrondata.com/pr_docs/35312/python/generated/pyarrow.fs.S3FileSystem.html

[kou]

+1

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 334b46d.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.