object_store: different behavior from aws cli for default profile

[Renkai]

Describe the bug
Object store doesn't use the default profile in ~/.aws/config

To Reproduce

• compile with flag aws_profile
• Have [default] section in ~/.aws/config
• No AWS_PROFILE value
• object store ignored the content in [default]

Expected behavior
object store uses the [default] section for s3 even if there is no AWS_PROFILE in the environment variables

Additional context
Maybe related code position:
https://github.com/apache/arrow-rs/blob/master/object_store/src/aws/mod.rs#L1040

[tustvold]

What happens if you set AWS_PROFILE=default, is this an acceptable workaround?

I'm struggling to devise a way to achieve this without potentially breaking downstreams in unpredictable ways, we only want to use the aws-config credentials (used to provide AWS_PROFILE) if explicitly opted into by the user. As feature flags should be strictly additive, they are insufficient to gate this functionality.

[Renkai]

It's acceptable workaround, but I also hope there is a warn message when I expect the default profile without the environment variable set but object_store decide not.

[tustvold]

FYI #4163 might provide a mechanism for users to opt-in to AWS CLI compatibility, by using the CLI to generate the credentials

[tustvold]

label_issue.py automatically added labels {'object-store'} from #4238