Plugin cannot connect, dispite AWS CLI connecting without issue

[PhilipBehrenberg]

As stated in the title, the S3 plugin is not authorized to connect to S3 to do any action, init or add repo. On the other hand, the AWS CLI on the same account/machine is able to cp the index.yaml or list/download files in the same bucket.

The commands are being run on an instance within the same AWS account, connecting to a private S3 bucket owned by the account logged into the CLI. Using an "identical" (as far as I can tell, and I've looked very closely) setup on a separate AWS account, everything works exactly as expected. The working account was using version 0.13, so I even tried downgrading that version to no avail. Both the s3 bucket and the instance are in the same region.

The ~/.aws/credentials and ~/.aws/config files both exist and are populated with default region, key, and secret. I even tried manually populating the ENV vars AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION, again to no avail.

AWS CLI, working as expected

$ aws s3 cp index.yaml s3://project-charts/
upload: ./index.yaml to s3://project-charts/index.yaml

$ aws s3 ls s3://project-charts
2022-09-20 18:54:53         27 index.yaml

Helm S3 Plugin, failing

$ helm s3 init s3://project-charts
Error: upload index to s3: upload index to S3 bucket: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
        status code: 403, request id: C3ESKMTBKA2Z0QW2, host id: Q88z9IFQnHAcF9RwP+hHUyHbmA/k7TccSWH36q08V96pv3ebuk374rMF+SezisHEr/HqnK+qVrc=
Error: plugin "s3" exited with error

helm repo add project s3://project-charts
Error: fetch from s3 url=s3://project-charts/index.yaml: fetch object from s3: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
        status code: 403, request id: AR03X9M5CPT7B2PD, host id: ukx8Rc+p/SlvZB4Be2WHofk5T/gtDp0ZW038OposBjF7IS82T7R7MBMjF4unAcu+R4VCikKFP/A=
Error: looks like "s3://project-charts" is not a valid chart repository or cannot be reached: plugin "bin/helm-s3 download" exited with error

Software/OS versions:

$ cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

$ aws --version
aws-cli/2.7.33 Python/3.9.11 Linux/4.14.287-215.504.amzn2.x86_64 exe/x86_64.amzn.2 prompt/off

$ helm version
version.BuildInfo{Version:"v3.9.4", GitCommit:"dbc6d8e20fe1d58d50e6ed30f09a04a77e4c68db", GitTreeState:"clean", GoVersion:"go1.17.13"}

$ helm plugin list | grep s3                                                                              
s3      0.14.0  Provides AWS S3 protocol support for charts and repos. https://github.com/hypnoglow/helm-s3

[mechaHarry]

Also facing the same issue after our plugin was bumped from 0.10.0 to anything 0.11.0 and onwards. Same steps, same errors/results.

[stek29]

this might be caused by introduction of DynamicBucketRegion in #146, which was released in 0.11.0

DynamicBucketRegion is always called, it's always called with hard-coded credentials and hard-coded endpoint

it also breaks the plugin in environments with restricted access to internet, and leaks internal bucket names to amazon AWS in non-restricted environments, since the dynamic bucket region "guessing" is always performed on amazon s3, no matter what is set in .aws files or environment variables.