Provide an Option to Use Path-Style-Access with S3 Repo #41966

original-brownbear · 2019-05-08T19:36:16Z

As discussed, added the option to use path style access back again and
deprecated it.
Defaulted to falling back to the SDK defaults
Added warning to docs + breaking changes entry
Closes Path-style S3 access is deprecated #41816

* As discussed, added the option to use path style access back again and deprecated it. * Defaulted to `false` * Added warning to docs * Closes elastic#41816

elasticmachine · 2019-05-08T19:36:19Z

Pinging @elastic/es-distributed

original-brownbear · 2019-05-09T12:48:56Z

plugins/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3ClientSettings.java

@@ -95,6 +95,10 @@
    static final Setting.AffixSetting<Boolean> USE_THROTTLE_RETRIES_SETTING = Setting.affixKeySetting(PREFIX, "use_throttle_retries",
        key -> Setting.boolSetting(key, ClientConfiguration.DEFAULT_THROTTLE_RETRIES, Property.NodeScope));

+    /** Whether the s3 client should use path style access. */
+    static final Setting.AffixSetting<Boolean> USE_PATH_STYLE_ACCESS = Setting.affixKeySetting(PREFIX, "path_style_access",
+        key -> Setting.boolSetting(key, false, Property.NodeScope, Property.Deprecated));


Now that AWS changed course on this one and has announced continued support for this access pattern for existing/older buckets (https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story), maybe there's no need to change the default here before 8.0?

ywelsch · 2019-05-21T18:55:54Z

plugins/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3ClientSettings.java

@@ -127,9 +131,13 @@
    /** Whether the s3 client should use an exponential backoff retry policy. */
    final boolean throttleRetries;

+    /** Whether the s3 client should use path style access. */
+    final boolean pathStyleAccess;


The AWS SDK uses this as a ternary setting (true, false, null), with null being the default, which enables the path-style setting based on the configured endpoint. I think we should do treat this setting exactly the same way, allowing path-style setting both being explicitly enabled and disabled.

Sounds good, so we break behavior after all by changing the default to null? See my comment above, this may not be necessary (at least in 7.x)?

It's unclear what the behaviour of setPathStyleAccessEnabled(Boolean.FALSE) is from the SDK docs. I think we're doing the right thing here, either enabling it or leaving it unset.

…ccess

original-brownbear · 2019-06-03T14:18:41Z

@ywelsch Adjusted this to a ternary setting now and default to the SDK behavior as discussed. Take a look when you have a chance :)

ywelsch · 2019-06-04T08:46:40Z

docs/plugins/repository-s3.asciidoc

+`path_style_access`::
+
+   Whether to use the path style access pattern. If `true`, the path style access pattern will be used if false, DNS style access will
+   be used. Defaults to letting the AWS S3 SDK decide the right access pattern dynamically.


AWS Java SDK

s/right access pattern/path style access/

Let's add a link to the relevant SDK docs

ywelsch · 2019-06-04T08:47:23Z

docs/plugins/repository-s3.asciidoc

@@ -145,6 +145,18 @@ settings belong in the `elasticsearch.yml` file.
    Whether retries should be throttled (i.e. should back off). Must be `true`
    or `false`. Defaults to `true`.

+`path_style_access`::
+
+   Whether to use the path style access pattern. If `true`, the path style access pattern will be used if false, DNS style access will


this sentence reads a bit funny with the if false not being delimited

ywelsch · 2019-06-04T08:51:51Z

docs/plugins/repository-s3.asciidoc

+   AWS S3 will stop supporting the path style access pattern for new buckets from
+   https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story[September 30th, 2020].
+   Any setups using the path style access pattern should be switched to the domain style access pattern as soon as possible to ensure
+   continued compatibility with the S3 repository plugin.


Technically it depends on how long AWS is going to support path-style-access on their Java SDK. I couldn't find any information on that. I'm not sure whether we should word this here as strongly, but instead point out that we rely on the SDK and that if that configuration option is going away, we will not be able to support this anymore.

Isn't toning it down a little inconsistent with us now making a breaking change in 7.x?
My thinking was that we're doing this breaking change because we fear that future versions of the SDK won't support path style access. If we don't think this is an imminent enough issue to strongly push users towards adjusting their environment asap then why make a breaking change and not just wait for 8.0?

Isn't toning it down a little inconsistent with us now making a breaking change in 7.x?

It's just going back to the pre-7.0 behavior, which was the more sensible default, especially given the current strategy of Amazon not to support path-style access in the future anymore, and there being no way to explicitly force-disable it in the SDK (i.e. it would still use path-style access with IP-based endpoint). This here makes it sound like we would break Minio support with path-style access at any moment in time, which sounds perhaps a bit overdramatic.

ywelsch · 2019-06-04T08:52:55Z

docs/reference/migration/migrate_8_0/snapshots.asciidoc

+==== The S3 repository plugin uses the DNS style access pattern by default
+
+Starting in version 7.3 using the path style access pattern with the S3 repository is deprecated.
+Previously the S3 repository plugin was exclusively using the path style access pattern. This is a breaking


Perhaps qualify this by "In 7.0, 7.1 and 7.2"

ywelsch · 2019-06-04T08:58:21Z

plugins/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3ClientSettings.java

+    /** Whether the s3 client should use path style access. */
+    static final Setting.AffixSetting<S3ClientSettings.PathStyleAccess> USE_PATH_STYLE_ACCESS = Setting.affixKeySetting(
+        PREFIX, "path_style_access",
+        key -> new Setting<>(key, s -> "default", s -> Boolean.parseBoolean(s) ? PathStyleAccess.ENABLED : PathStyleAccess.DISABLED,


this PR is missing tests, and I doubt that the parsing here is correct given that it never yields PathStyleAccess.DEFAULT

My idea was to simply only allow parsing true and false here and let the DEFAULT represent the setting not being set (I didn't see much value in allowing to explicitly set "default" or "null"). Not a good idea? :)

We can make this a boolean setting (see my comment above) and use S3ClientOptions.DEFAULT_PATH_STYLE_ACCESS, which is false, as the default value. We should also add a test to make sure this default never changes.

ywelsch · 2019-06-04T09:03:43Z

docs/reference/migration/migrate_8_0/snapshots.asciidoc

+[float]
+==== The S3 repository plugin uses the DNS style access pattern by default
+
+Starting in version 7.3 using the path style access pattern with the S3 repository is deprecated.


I'm not sure we should deprecate this at this time. AWS's SDK is going to support path-style access for a very long time to come (even after Sept. 2020, given that the SDK will still be used to connect to older repos).

Same point as in the above, why make a breaking change then (that we know for a fact will break some environments)?

The focus of this section should be on the fact that the behavior changes between 7.2- and 7.3+, going back to the 6.x behavior, and that the 7.0-7.2 behavior can be restored by setting this new setting. The deprecation of that new setting is secondary. As said, given that we don't have a time frame for removing this setting, I would prefer to defer the deprecation to a later time.

ywelsch · 2019-06-04T09:04:05Z

plugins/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3ClientSettings.java

+    static final Setting.AffixSetting<S3ClientSettings.PathStyleAccess> USE_PATH_STYLE_ACCESS = Setting.affixKeySetting(
+        PREFIX, "path_style_access",
+        key -> new Setting<>(key, s -> "default", s -> Boolean.parseBoolean(s) ? PathStyleAccess.ENABLED : PathStyleAccess.DISABLED,
+            Property.NodeScope, Property.Deprecated));


As said above, I don't think we should log deprecation warnings.

…ccess

original-brownbear · 2019-06-04T11:38:05Z

@ywelsch all points addressed :), left some questions though.

original-brownbear · 2019-06-05T10:41:20Z

Jenkins run elasticsearch-ci/1

ywelsch

I have added a few more replies. Also, still missing tests.

ywelsch · 2019-06-11T10:19:25Z

docs/plugins/repository-s3.asciidoc

+`path_style_access`::
+
+   Whether to use the path style access pattern. If `true`, the path style access pattern will be used. If set to`false`, DNS style access will
+   be used. Defaults to letting the AWS Java SDK decide whether to use the path style access pattern dynamically


AFAICS, the SDK is implementing something different. If set to false, it is equivalent to the SDK's default of determining the path style access pattern depending on the configured endpoint (an IP will result in path-style access). The SDK only allows forcing this to true, which means that all requests are forced to use path-style access. Implementation and docs should reflect this.

RIght ... I should've looked on step deeper S3ClientOptions uses boolean already ... will simplify this PR then :)

ywelsch · 2019-06-11T10:34:11Z

plugins/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3ClientSettings.java

+    /** Whether the s3 client should use path style access. */
+    static final Setting.AffixSetting<S3ClientSettings.PathStyleAccess> USE_PATH_STYLE_ACCESS = Setting.affixKeySetting(
+        PREFIX, "path_style_access",
+        key -> new Setting<>(key, s -> "default", s -> Boolean.parseBoolean(s) ? PathStyleAccess.ENABLED : PathStyleAccess.DISABLED,


We can make this a boolean setting (see my comment above) and use S3ClientOptions.DEFAULT_PATH_STYLE_ACCESS, which is false, as the default value. We should also add a test to make sure this default never changes.

ywelsch · 2019-06-11T11:09:47Z

docs/plugins/repository-s3.asciidoc

+   AWS S3 will stop supporting the path style access pattern for new buckets from
+   https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story[September 30th, 2020].
+   Any setups using the path style access pattern should be switched to the domain style access pattern as soon as possible to ensure
+   continued compatibility with the S3 repository plugin.


Isn't toning it down a little inconsistent with us now making a breaking change in 7.x?

It's just going back to the pre-7.0 behavior, which was the more sensible default, especially given the current strategy of Amazon not to support path-style access in the future anymore, and there being no way to explicitly force-disable it in the SDK (i.e. it would still use path-style access with IP-based endpoint). This here makes it sound like we would break Minio support with path-style access at any moment in time, which sounds perhaps a bit overdramatic.

ywelsch · 2019-06-11T11:11:38Z

docs/reference/migration/migrate_8_0/snapshots.asciidoc

+
+Starting in version 7.3 using the path style access pattern with the S3 repository is deprecated.
+In versions 7.0, 7.1, and 7.2 the S3 repository plugin was exclusively using the path style access pattern. This is a breaking
+change for deployments that do not also allow for the DNS style access pattern. For short-term compatibility with these deployments users


only those that are not on AWS and that do not use IP-address based endpoint.

ywelsch · 2019-06-11T11:15:23Z

docs/reference/migration/migrate_8_0/snapshots.asciidoc

+[float]
+==== The S3 repository plugin uses the DNS style access pattern by default
+
+Starting in version 7.3 using the path style access pattern with the S3 repository is deprecated.


The focus of this section should be on the fact that the behavior changes between 7.2- and 7.3+, going back to the 6.x behavior, and that the 7.0-7.2 behavior can be restored by setting this new setting. The deprecation of that new setting is secondary. As said, given that we don't have a time frame for removing this setting, I would prefer to defer the deprecation to a later time.

…ccess

ywelsch

I've left a few smaller doc changes, but looks good o.w.
@DaveCTurner can you have a look as well?

ywelsch · 2019-06-17T08:14:58Z

docs/plugins/repository-s3.asciidoc

+   (See https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/AmazonS3Builder.html#setPathStyleAccessEnabled-java.lang.Boolean-[AWS documentation] for details).
+   Defaults to `false`.
+
+NOTE: In versions `7.0` and `7.1`, all bucket operations were using the path style access pattern in every situation without an option to


7.0, 7.1 and 7.2

docs/plugins/repository-s3.asciidoc

…ccess

original-brownbear · 2019-06-24T10:13:33Z

@DaveCTurner ping :)

DaveCTurner

Sorry, somehow I missed this past the 7.3 feature freeze. In repentance, I've updated the docs to refer back to 7.0-7.3 and cleaned up the wording and formatting a bit too, see f76a14e. This LGTM now.

DaveCTurner · 2019-07-04T12:17:40Z

plugins/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3ClientSettings.java

@@ -127,9 +131,13 @@
    /** Whether the s3 client should use an exponential backoff retry policy. */
    final boolean throttleRetries;

+    /** Whether the s3 client should use path style access. */
+    final boolean pathStyleAccess;


It's unclear what the behaviour of setPathStyleAccessEnabled(Boolean.FALSE) is from the SDK docs. I think we're doing the right thing here, either enabling it or leaving it unset.

original-brownbear · 2019-07-04T15:37:22Z

thanks @DaveCTurner !

* Provide an Option to Use Path-Style-Access with S3 Repo * As discussed, added the option to use path style access back again and deprecated it. * Defaulted to `false` * Added warning to docs * Closes elastic#41816

) * Provide an Option to Use Path-Style-Access with S3 Repo * As discussed, added the option to use path style access back again and deprecated it. * Defaulted to `false` * Added warning to docs * Closes #41816

Add support for elastic/elasticsearch#41966

Add support for S3 repo settings: path style access: elastic/elasticsearch#41966

Add support for S3 repo settings: path style access: elastic/elasticsearch#41966 (cherry picked from commit 6b0afaa)

original-brownbear added 2 commits May 8, 2019 21:23

Provide an Option to Use Path-Style-Access with S3 Repo

713698a

* As discussed, added the option to use path style access back again and deprecated it. * Defaulted to `false` * Added warning to docs * Closes elastic#41816

add breaking changes entry

13e3311

original-brownbear added >breaking :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v8.0.0 v7.2.0 labels May 8, 2019

original-brownbear commented May 9, 2019

View reviewed changes

original-brownbear requested review from ywelsch and DaveCTurner May 9, 2019 12:49

ywelsch reviewed May 21, 2019

View reviewed changes

gwbrown mentioned this pull request May 22, 2019

Update Deprecation Info API checks for 8.0 #42404

Closed

80 tasks

original-brownbear added 3 commits May 31, 2019 05:54

Merge remote-tracking branch 'elastic/master' into fix-s3-path-styl-a…

4530b79

…ccess

Merge remote-tracking branch 'elastic/master' into fix-s3-path-styl-a…

897516a

…ccess

default to SDK defaults

5c85170

original-brownbear requested a review from ywelsch June 3, 2019 14:18

ywelsch added v7.3.0 and removed v7.2.0 labels Jun 4, 2019

ywelsch suggested changes Jun 4, 2019

View reviewed changes

original-brownbear added 2 commits June 4, 2019 13:09

Merge remote-tracking branch 'elastic/master' into fix-s3-path-styl-a…

ad1ead9

…ccess

CR: Adjust docs

22ecda1

original-brownbear requested a review from ywelsch June 4, 2019 11:38

ywelsch suggested changes Jun 11, 2019

View reviewed changes

original-brownbear added 4 commits June 11, 2019 13:22

Merge remote-tracking branch 'elastic/master' into fix-s3-path-styl-a…

0979112

…ccess

boolean setting

3110d3c

Merge remote-tracking branch 'elastic/master' into fix-s3-path-styl-a…

9582749

…ccess

CR: tone down language :)

42fa7e4

ywelsch approved these changes Jun 17, 2019

View reviewed changes

original-brownbear added 2 commits June 17, 2019 11:15

Merge remote-tracking branch 'elastic/master' into fix-s3-path-styl-a…

c865f8d

…ccess

CR: fix docs

c13bf0c

wujinhu mentioned this pull request Jun 28, 2019

[Distributed/Snapshot/Restore] Add configurations: use_path_style && use_chunked_encoding #43738

Closed

jpountz added v7.4.0 and removed v7.3.0 labels Jul 3, 2019

DaveCTurner added 3 commits July 4, 2019 13:13

Docs fixes post 7.3 FF

f76a14e

Merge branch 'master' into fix-s3-path-styl-access

a2b8d8e

Wording tweak

5ba5ac4

DaveCTurner approved these changes Jul 4, 2019

View reviewed changes

original-brownbear merged commit d6e23e9 into elastic:master Jul 4, 2019

original-brownbear deleted the fix-s3-path-styl-access branch July 4, 2019 15:37

original-brownbear added backport pending and removed backport pending labels Jul 4, 2019

original-brownbear mentioned this pull request Jul 7, 2019

Provide an Option to Use Path-Style-Access with S3 Repo (#41966) #44046

Merged

original-brownbear mentioned this pull request Jul 20, 2019

Docs: Path-style access is currently not working #34439

Closed

codebrain mentioned this pull request Oct 14, 2019

7.4 meta ticket elastic/elasticsearch-net#4133

Closed

56 tasks

Mpdreamz added a commit to elastic/elasticsearch-net that referenced this pull request Oct 14, 2019

S3 repo settings: path style access

c3ce205

Add support for elastic/elasticsearch#41966

Mpdreamz mentioned this pull request Oct 14, 2019

S3 repo settings: path style access elastic/elasticsearch-net#4136

Merged

russcam pushed a commit to elastic/elasticsearch-net that referenced this pull request Oct 15, 2019

S3 repo settings: path style access (#4136)

6b0afaa

Add support for S3 repo settings: path style access: elastic/elasticsearch#41966

russcam pushed a commit to elastic/elasticsearch-net that referenced this pull request Oct 15, 2019

S3 repo settings: path style access (#4136)

96563a6

Add support for S3 repo settings: path style access: elastic/elasticsearch#41966 (cherry picked from commit 6b0afaa)

jakelandis mentioned this pull request Feb 22, 2021

DRAFT [META] REST Compatible API V7 completeness #68905

Closed

This comment has been minimized.

Sign in to view

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

Provide an Option to Use Path-Style-Access with S3 Repo #41966

Provide an Option to Use Path-Style-Access with S3 Repo #41966

Conversation

original-brownbear commented May 8, 2019 • edited Loading

elasticmachine commented May 8, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

original-brownbear commented Jun 3, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

original-brownbear commented Jun 4, 2019

original-brownbear commented Jun 5, 2019

ywelsch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ywelsch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

original-brownbear commented Jun 24, 2019

DaveCTurner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

original-brownbear commented Jul 4, 2019

This comment has been minimized.

This comment has been minimized.

original-brownbear commented May 8, 2019 •

edited

Loading