Add proposal for configuring endpoints per service

[kdaily]

This pull request contains a proposal for improvements to configure endpoint URLs for AWS services independently.

For a Markdown rendered version of this proposal, view it here.

⚠️ Important note for community members

This pull request is a mechanism to engage our community to collect feedback for the design of this specific cross-SDK feature. There is no commitment to using this mechanism for future cross-SDK features.

The proposal reflects the initial direction we are considering for this cross-SDK feature. There may be conversations that occur outside of this pull request that change the design direction, and we will update the pull request to reflect those changes to allow for follow up feedback.

🗒️ Providing feedback

After reading the proposal and comments, you can give feedback in the following ways:

1. If you generally approve direction, add 👍 reaction to top-level comment.
2. If you prefer one of the alternatives listed, comment on the line where an alternative is proposed that you prefer. If there is already a comment for the alternative, add a 👍 to the original comment.
3. If there are any other suggestions/alternatives that you'd prefer, add a comment describing the suggestion and alternative. 👍 the comment if you agree with it.
4. Comment with any additional questions.

Thank you!

[Type1J]

This is a highly desired feature. Can someone at AWS please assign reviewers?

[kdaily]

Hi @Type1J,

Thanks for your comment. We're planning to keep this proposal open for public comments, questions, and feedback for a total of three weeks. After this, we will review it internally and publish any updates. The tracking issue (#229) will be open until we've completed implementation.

If you have any feedback of your own, we would be happy to consider it. Similarly, if you have any questions, we'll answer them here as well. See this comment (#230 (comment)) for more information on how to provide feedback. Thanks!

[srgg]

It seems that the more universal way is to combine some alternatives with the proposal.

Configuration via environment variables

Set the global endpoint URL for all services and, nevertheless, have the ability to override particular one with service specific endpoint url variable:

export AWS_ENDPOINT_URL=http://localhost:5678
export AWS_ELASTIC_BEANSTALK_ENDPOINT_URL=https://localhost:5567

Shared configuration file

[profile some-profile]
endpoint_url = http://localhost:5678
beanstalk = 
  endpoint_url = https://localhost:4567

[kjenney]

I'm really interested in this feature as I'm in the process of moving to FIPS endpoints for a number of services. Storing and using configuration in a repeatable way would make this work significantly easier.

[sanudatta11]

Really Interested in this feature

[jamesls]

Hi, I was recently pointed to this proposal, and I think this would be a really useful feature to have. That being said, I have some feedback on the proposed design.

Config file

I would strongly discourage creating per-service keys in the config file. This creates an unbounded set of configuration keys that is continuously growing that we don't control. This not only limits the set of potential config keys we can use in the future, it also has the potential to conflict with existing keys any time a new service is released. Granted, most of the existing configuration keys seem innocuous, I'd rather not have that as a possibility to begin with. For example, if a new region service launches (seems unlikely, but we did launch a config service so it wouldn't surprise me at all), we've potentially broken everyone if we have validation that we'd expected a dictionary type (looking for an endpoint_url key) but instead we found the string us-west-2.

As an alternative, consider having an endpoint_urls key with services as the subkeys:

[profile testing]
output = json
endpoint_urls = 
    s3 = https://localhost:4567/
    dynamodb = https://localhost:5678

Environment variables

While I understand the motivation for wanting parity between env vars and the config file, I'd suggest starting with a more modest approach of a single AWS_ENDPOINT_URL that is the exact same as the --endpoint-url arg. For one, we don't even have that yet, and two I'm not sure the env var workflow of multiple services with custom endpoint urls has the same ergonomics as the config file.

For example, a common use case I've seen (and also encountered myself) is that when you're developing/interacting with a service, it's tedious to have to specify --endpoint-url https://.../ everytime you run a CLI command. It's easier to just set the env var once and then run various CLI commands. In the case where you actually have multiple services you're interacting with, it tends to be grouped by service anyways, such that there's not really a big difference between:

AWS_IAM_ENDPOINT_URL=https://localhost:54321/
AWS_LAMBDA_ENDPOINT_URL=https://localhost:12345/
aws iam foo
aws iam bar
aws iam more-iam-stuff
aws lambda create-function
aws lambda invoke ...
aws lambda more-lambda-stuff

vs.

AWS_ENDPOINT_URL=https://localhost:54321/
aws iam foo
aws iam bar
aws iam more-iam-stuff
AWS_ENDPOINT_URL=https://localhost:12345/
aws lambda create-function
aws lambda invoke ...
aws lambda more-lambda-stuff

Also consider the common scenario of having a dev vs. prod profile as cited in the Motivation section above. You'd essentially have:

# Dev mode
export AWS_<SERVICE1>_ENDPOINT_URL=https://...../
export AWS_<SERVICE2>_ENDPOINT_URL=https://..../
... potentially dozens more ...

Then when you want to switch to prod mode, you'd have to unset all the env vars you set previously in dev mode, or potentially have to re-export all the env vars to the prod endpoints (if prod also has custom endpoints). If you miss even a single env var, you're potentially talking to a mix of dev and prod endpoints. This would be bad.

Compare this to the config file approach where you'd just have a dev profile and a prod profile, and switching between them requires flipping a single env var. Impossible to accidentally mix dev and prod endpoints.

We can always add service-specific env vars later if we find in practice the config file approach is not sufficient.

[jamesls]

Does this "subkey" syntax have precedent in AWS config files?

Yeah, I added it back in 2014 so it's been there a while (originally used for s3 config). FWIW the original proposal still suggests using the nested syntax, just keyed off of services:

[profile s3-minio]
s3 = 
  endpoint_url = https://play.min.io:9000

[omercnet]

Is there an update here? Any plans to implement this?

[pkoch]

@kdaily What are the next steps here? Is there maybe a deadline to get feedback?

[kdaily]

Hi @pkoch,

This proposal is being discussed and worked on internally. I'll be sure to share an update when we've finalized more on the implementation!

Feel free to leave any feedback that you have. Thanks!

[mihir-karbelkar-wiz]

Any workarounds for this till it gets deployed?

[nlongton]

In addition to specifying the URL for each endpoint we need to disable SSL verification as the certificate from the service does not have a name that matches the URL.

[fcossio]

I am also interested in using this feature :) Is there a way I can contribute?

[ljmc-github]

EDIT (2022-12-22)

While this proposal is being evaluated, I have implemented the proposal in a workaround package: boto-endpoint-url-shim.

Base usage for service-specific environment variable and service subsection in shared configuration file:

import boto_endpoint_url_shim
import boto3

boto_endpoint_url_shim.proposed_endpoint_url_resolution()

s3 = boto3.resource("s3")  # uses the custom endpoint

This is my first public python package and it is in early stages, but it implements all the proposed endpoint resolution methods. If you have any feedback, feel free to drop me a message or open an issue on gitlab.

[aude]

Here is a simple version that does the same:

import os
import boto3

s3 = boto3.resource(
    "s3",
    endpoint_url=os.getenv("AWS_S3_ENDPOINT_URL") or os.getenv("AWS_ENDPOINT_URL"),
)
dynamodb = boto3.resource(
    "dynamodb",
    endpoint_url=os.getenv("AWS_DYNAMODB_ENDPOINT_URL") or os.getenv("AWS_ENDPOINT_URL"),
)

@ljmc-github, it's impressive that you managed to build the shim 🙂 It's helpful to have a workaround in the meantime while this proposal is being evaluated. I wanted to avoid an extra package if possible, from both a maintainability and security perspective.

[joarobles]

Hi @kdaily ! Any updates on this?

[kdaily]

Hi! Work on this proposal is still ongoing. I just pushed up the latest revision. Highlights include:

1. Details were updated on the shared configuration file specification. The configuration of service-specific endpoint URLs will be made in a services section in the shared config file instead of directly in the profile section. In this way a services section can be shared between profiles. See the "Examples" section for details.
2. The feature will have a separate configuration option to completely opt out and return to the existing behavior. Opting out would ignore any configured endpoint URLs provided from environment variables or the shared configuration file.

Feedback is appreciated!

[CodePint]

Hi! Work on this proposal is still ongoing. I just pushed up the latest revision. Highlights include:

1. Details were updated on the shared configuration file specification. The configuration of service-specific endpoint URLs will be made in a services section in the shared config file instead of directly in the profile section. In this way a services section can be shared between profiles. See the "Examples" section for details.
2. The feature will have a separate configuration option to completely opt out and return to the existing behavior. Opting out would ignore any configured endpoint URLs provided from environment variables or the shared configuration file.

Feedback is appreciated!

Let me just start by saying:

I appreciate all the work that has been done by the maintainers here and recognize the amount of thought that has been put into the proposal, as well as the complexity of the wider issue...however we have to admit that the fact this issue has taken 8 years to reach this point indicates something has gone wrong within the SDLC.

As someone who has interacted with this issue many times within different companies, languages and frameworks over the past few years...may I propose that we try to be more 'agile' with our approach to this and other future features requests which arise.

Whilst many people want and require per service endpoint configuration (myself included), the vast majority would be served by just providing parity with the aws endpoint_url parameter.

Why not release an agile minimal solution which addresses this basic problem even if it has to be very much ad-hoc for each languages sdk. Even officially endorsing and documenting one or more of the user suggested workarounds would be great.

The implementation could be separate from the rest of the cli/sdk config by having a language or environment specific env vars, i.e: AWS_JAVA_ENDPOINT_URL , AWS_PYTHON_ENDPOINT_URL,
AWS_LOCAL_ENDPOINT_URL

While not an ideal or long-term it would reduce the scope of any changes and provide isolation from potential bugs.

Would you please be able to provide a timeline for how long it would take to implement a simple imperfect agile solution like those proposed by myself and other users vs a comprehensive long-term 'waterfall' solution proposed here

Regardless of the outcome of this issue, I really hope this is something that is taken onboard going forward.

[pkulev]

Great work, thanks! Is it really should take more than a year to add configurable endpoints though?

[waldman]

So... is this happening?

[cartermckinnon]

Seems this train has already left the station: https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html

Can this be merged or closed?

[tim-finnigan]

I think this can be closed now. The corresponding tracking issue (#229) has already been closed. As was mentioned in that issue:

I'm excited to announce that we have published an AWS Developer Blog post about this feature. Let us know in the feedback links on the post what you think!

https://aws.amazon.com/blogs/developer/new-improved-flexibility-when-configuring-endpoint-urls-with-the-aws-sdks-and-tools/