Add environment variable to override endpoint_url (#2099) #2746
Conversation
|
Thanks to raise this PR. I am waiting for this featuer as well (#1375 #2099) But I have a question. In latest localstack, it has an improved feature to combine all endpoint url with one port (it used to be different ports for different services). So with your change, it supposes the port is always same for all services. But will this be always the case? If in the future, someone need support with different ports, looks this PR hardcode without different port, and it would be hard to update to support different ports for different services. |
|
I agree @ozbillwang that your suggestion would be an improvement. |
|
I am also interested in this PR - it would be very useful for localstack. |
|
We really need this feature, in conjunction with the AWS_CONFIG_FILE and AWS_SHARED_CREDENTIALS_FILE env vars it would allow us to change the cloud backend outside of the code. but this pr has almost 3 months and no reviewer... is there a problem including it ? |
|
Should I ask someone to review it? Not sure what the process is to get a PR into core |
|
It looks like you did everything right according to their contributions
instructions:
https://github.com/boto/boto3/blob/develop/CONTRIBUTING.rst
…On Fri, 16 Apr 2021, 19:20 Rachel Willmer, ***@***.***> wrote:
Should I ask someone to review it? Not sure what the process is to get a
PR into core
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#2746 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAJGLDUMBAA5V3S6EJQ5BVDTJB5VRANCNFSM4W2YEVCA>
.
|
|
If anyone else needs a workaround for localstack while waiting for the PR, I have one using a reverse proxy that intercepts the calls made to the default AWS endpoints and redirects them to localstack on 4566. This means no code changes are required in the lambdas. https://github.com/luminescent/localstack-reverse-proxy |
boto3/session.py
Outdated
| try: | ||
| endpoint_url = os.environ["AWS_ENDPOINT_URL"] | ||
| except KeyError: | ||
| pass |
There was a problem hiding this comment.
This could do a .get() which returns None if the env var is not set.
| try: | |
| endpoint_url = os.environ["AWS_ENDPOINT_URL"] | |
| except KeyError: | |
| pass | |
| endpoint_url = os.environ.get("AWS_ENDPOINT_URL") |
There was a problem hiding this comment.
Are we able to make this change and get the PR merged?
|
+1 this would be incredibly use for me |
|
Any chance of getting this merged? |
|
+1 this would be incredibly use for me |
|
This is a very nice, minimally invasive solution to a very common problem. Please do consider it. |
|
@rwillmer are you able to make the changes requested in the code review? If you do not have time or I don't hear back, I am happy to branch off and make the requested changes. |
|
For anyone looking for an interim solution to use with PynamoDB import os
from botocore.session import Session
from pynamodb.models import Model
class BaseModel(Model):
class Meta:
if os.environ['APP_ENV'] in ['local', 'test']:
host = os.environ['AWS_ENDPOINT_URL']
region = os.environ['AWS_DEFAULT_REGION']
else:
region = Session().get_config_variable('region') |
|
watchig. |
|
@AidanDelaney @CodePint I've made the suggested change |
|
The last two people to merge pull requests were @nateprewitt and @joguSD. Maybe they can offer some guidance on this? |
|
Bumping the issue. It would be greatly appreciated to have this merged. |
|
I'm the original author of the PR and AFAIK I've made the suggested change. |
|
@nateprewitt Have you seen this? |
|
@kdaily Have you seen this? |
|
+1 for this! Really needed! Today I ran into same issue when I needed to invoke SAM lambda and SAM API simultaneously. As all lambdas are using boto3, it will be beneficial instead of changing the code. |
|
I'm the OP. I'm seeing 2 messages:
I don't have access to request any more reviews - so could someone, who does, ask a maintainer to approve this? assuming that the 2 existing reviewers aren't maintainers I don't have write access to this repo, so cannot merge this PR. Could someone, who does, merge this? In the meantime, I'll try to keep my fork up-to-date with the current boto3/develop branch, in the hope that this would be useful to folk. |
|
Just to clarify, this doesn't cover cases where we use different local components, right? If we need to set, for example, an endpoint for a local DynamoDB and another for minio at the same time, we would still need to implement a customized call to |
| client._endpoint.host == ENDPOINT_URL, | ||
| "AWS_ENDPOINT_URL env var not used when set", | ||
| ) | ||
| del os.environ["AWS_ENDPOINT_URL"] |
There was a problem hiding this comment.
If AWS_ENDPOINT_URL is actually set on the environment before this test, del os.environ["AWS_ENDPOINT_URL"] could mess up with it for other tests.
Maybe a setUp method like this would be better, to restore the os.environ to its original state after each test run?
def restore_dict(self, dictionary: dict, original_data: dict):
dictionary.clear()
dictionary.update(original_data)
def setUp(self):
original_env = os.environ.copy()
self.addCleanup(lambda: self.restore_dict(os.environ, original_env))
…ery useful for localstack while waiting for upstream PR from boto3: boto/boto3#2746
) * Add ability to set endpoint_url from AWS_ENDPOINT_URL env variable. Very useful for localstack while waiting for upstream PR from boto3: boto/boto3#2746 * add test * rm print * lint
) (#195) * Add ability to set endpoint_url from AWS_ENDPOINT_URL env variable. Very useful for localstack while waiting for upstream PR from boto3: boto/boto3#2746 * add test * rm print * lint Co-authored-by: Kabir Khan <kabirkhan1137@gmail.com>
|
I am literally confused - so simple and so wanted fix was not yet merged. Do not want to press on maintainers, but mates, if you have no time to merge, maybe give some others active contributors write access? Or just enable something like auto-merge (anyone can merge PR as soon as approved and CI checks passed) |
|
And...i'm running into this again with localstack and S3 buckets. |
|
After looking at the code more, there is a strong argument that this change should be integrated with botocore config for the sake of consistency with other session variables. boto3 Would it be worth trying to ping the botocore maintainers and working at this from both sides? botocore references: With this being said, the PR here is very lightweight and would be of great use as is. |
|
Implementation of this PR using a boto3 session subclass, will leave this here in case it's useful to anyone else: import os
from boto3.session import Session
class BotoSession(Session):
def client(self, *args, **kwargs):
if kwargs.get('endpoint_url') is None:
kwargs['endpoint_url'] = os.environ.get("AWS_ENDPOINT_URL")
return super().client(*args, **kwargs)
# s3_client = BotoSession().client('s3') |
|
I'm the OP. I offered this 2-line change 15 months ago. Zero interest from the maintainers in taking a PR, which included tests. I've given up expecting that this change will be added to boto/boto3. |
|
Fwiw, The more effective way to lobby for this change is via your account
managers and reference this issue. Likely the change needs to be
coordinated across other language sdks for consistent usage.
…On Tue, Apr 19, 2022 at 6:08 PM Rachel Willmer ***@***.***> wrote:
I'm the OP. I offered this 2-line change 15 months ago.
Zero interest from the maintainers in taking a PR, which included tests.
I've given up expecting that this change will be added to boto/boto3.
—
Reply to this email directly, view it on GitHub
<#2746 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAFJEU7BLRAFWSHJ4DJBVDVF4VF3ANCNFSM4W2YEVCA>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
What account manager is that? I'm a freelancer, offering a wee bit of code that I thought would be useful, with tests. It's turned out to be impossible to get this tiny PR merged. |
|
Is it possible that this PR is not merged just because it allows developers to use other S3-like storage? Maintainers, please tell me that I am wrong. |
I think this was already addressed in a different issue (can't seem to find it currently). Basically there are use cases within the AWS official ecosystem where an |
This is a good idea, I will see If I can raise this through our AWS support channels. |
Then this needs to be communicated by the maintainers, so we can stop wasting time here and focus on the other sdks |
|
Thank you for posting your feedback here, and our apologies that we’ve been thinking this over for a long time without much forward motion. There are similar requests to implement this feature in a few of the AWS SDKs and the AWS CLI, so in order to coordinate those teams - and hopefully make the discussions a little easier to follow - we’ve created a new issue in aws/aws-sdk here: aws/aws-sdk#229 |
|
Hi all, We recently added a pull request (aws/aws-sdk#230) that contains a proposal based on community comments and suggestions and our own discussions. This document proposes to extend the options for configuring the endpoint to allow users to provide an endpoint URL independently for each AWS service via an environment variable or a profile subsection in the shared configuration file. You can read the proposal here. For more information on how to give feedback, please see this comment on the aws/aws-sdk repository: Thanks! |
|
I'm happy to announce that the ability to configure the endpoint URL via the shared configuration file and environment variables is now available in the AWS Python SDK! You can now specify the endpoint to use for all service requests through the shared configuration file and environment variables, as well as specify the endpoint URL for individual AWS services. To start using this feature, install To read more about this feature, see the documentation page "Service-specific Endpoints" in the AWS SDKs and Tools Reference Guide: https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html Look forward to a blog post demonstrating the use of this feature on the AWS Developer Tools Blog! |
1st time contributing to this project, let me know if I need to change anything.