docs: clarify the use of "/" as a prefix_rewrite #3160

dio · 2018-04-22T00:31:51Z

docs: clarify the use of "/" as a prefix_rewrite

This patch clarifies the use of "/" as a prefix_rewrite in route and redirect prefix rewriting. And also a note on the use of trailing slashes as match value.

Risk Level: Low

Testing: add more input samples to RedirectPrefixRewrite test.

Docs Changes:

Update route.proto doc regarding path_rewrite both for redirect and route.

Release Notes:
N/A

Fixes #2956

Signed-off-by: Dhi Aurrahman <dio@rockybars.com>

mattklein123 · 2018-04-23T16:04:42Z

@zuercher can you review this one?

zuercher · 2018-04-23T19:47:44Z

It's unfortunate that we can't distinguish between prefix_rewrite = "" and unset because the real problem is that you can't rewrite a prefix to the empty string.

zuercher · 2018-04-23T18:40:40Z

source/common/config/utility.cc

+std::string Utility::normalizePath(absl::string_view path) {
+  std::vector<absl::string_view> segments = StringUtil::splitToken(path, "/", false);
+
+  // Removes dot and double-dot segments https://tools.ietf.org/html/rfc3986#section-5.2.4.


The algorithm described at that URL does more than simply delete ".." segments.

I think if you're going to do more than handle the case where the user wants to remove a prefix from the path (which just amounts to dealing with "//" at the start of the rewritten path, then the normalization should handle ".." correctly.

In either case, the docs for the prefix_rewrite field should be updated to describe what happens.

@zuercher I think I will remove that partial support for now.

A side question/clarification,

It's unfortunate that we can't distinguish between prefix_rewrite = "" and unset because the real problem is that you can't rewrite a prefix to the empty string.

Do you think it is better to allow users to set prefix_rewrite as "" rather than "/"?

Signed-off-by: Dhi Aurrahman <dio@rockybars.com>

zuercher

Thanks. Can you also update api/envoy/api/v2/route/route.proto with this new behavior?

zuercher · 2018-04-23T22:54:59Z

source/common/config/utility.h

@@ -263,6 +263,13 @@ class Utility {
  factoryForGrpcApiConfigSource(Grpc::AsyncClientManager& async_client_manager,
                                const envoy::api::v2::core::ApiConfigSource& api_config_source,
                                Stats::Scope& scope);
+
+  /**
+   * Normalize URI path. Currently it only handles to remove repeated slashes.


Nit: "Currently it only handles removing repeated slashes."

Signed-off-by: Dhi Aurrahman <dio@rockybars.com>

dio · 2018-04-23T23:26:49Z

@zuercher I added an explanation on this as an example. If you think we need more please let me know. Thanks!

mattklein123

LGTM from a functionality perspective, wondering if we can do better with perf?

mattklein123 · 2018-04-24T22:07:11Z

source/common/config/utility.cc

@@ -228,5 +228,11 @@ Grpc::AsyncClientFactoryPtr Utility::factoryForGrpcApiConfigSource(
  return async_client_manager.factoryForGrpcService(grpc_service, scope, false);
 }

+std::string Utility::normalizePath(absl::string_view path) {
+  std::vector<absl::string_view> segments = StringUtil::splitToken(path, "/", false);


nit: const here and next line

mattklein123 · 2018-04-24T22:08:52Z

source/common/config/utility.cc

+std::string Utility::normalizePath(absl::string_view path) {
+  std::vector<absl::string_view> segments = StringUtil::splitToken(path, "/", false);
+  bool has_trailing_slash = path.back() == '/';
+  return absl::StrCat("/", absl::StrJoin(segments, "/"), has_trailing_slash ? "/" : "");


Q: I feel like this is a pretty heavy hammer from a perf perspective for solving a specific issue (duplicate '/' after rewrite). Could we detect the duplicate '/' and copy strings in a way in which this would perform better? cc @jmarantz who likes these types of things. :)

Interesting. Probably most of the time the use-case is to have duplicates '/' at the beginning of the path.

actually I have functional concerns about this PR first.

jmarantz · 2018-04-25T01:35:26Z

source/common/config/utility.cc

+std::string Utility::normalizePath(absl::string_view path) {
+  std::vector<absl::string_view> segments = StringUtil::splitToken(path, "/", false);
+  bool has_trailing_slash = path.back() == '/';
+  return absl::StrCat("/", absl::StrJoin(segments, "/"), has_trailing_slash ? "/" : "");


actually I have functional concerns about this PR first.

jmarantz · 2018-04-25T01:38:05Z

test/common/config/utility_test.cc

+  // Maintain the trailing slash.
+  EXPECT_EQ("/ok/a/b/c/", Utility::normalizePath("/ok/a/b/c/"));
+  EXPECT_EQ("/ok/a/b/c/", Utility::normalizePath("///ok/a/b/c///"));
+  EXPECT_EQ("/ok/a/b/c/", Utility::normalizePath("///ok////a////b////c///"));


I don't think normalization should be this aggressive, messing with the slashes in the middle of the path.

I think the beginning of the path is a special case, and "scheme://origin/" should be treated the same as "scheme://origin". But I don't think that URL normalizers usually mess with anything after the "/".

The motivation force in the bug the fixes was really the behavior of replacing prefix "/foo" with "/". WDYT of handling this at the parsing of that command, where a trailing slash is appended to src or dest if they don't have them? Then the swap can be done fairly cleanly.

+1, that's basically my thinking as well.

Sorry for overthinking it. What do you think of handling this “/“ specifically: treating that as replacing with empty string? https://github.com/dio/envoy/blob/fix-2956/source/common/router/config_impl.cc#L375

Signed-off-by: Dhi Aurrahman <dio@rockybars.com>

jmarantz · 2018-04-25T12:15:30Z

In the description you describe a scenario:

- match: { prefix: "/ignore-this"}
  redirect: { prefix_rewrite: "/" }

The reason that produces unexpected results is that the match string doesn't end with a slash and the redirect does. You could solve this problem by either adding a slash to the match-string or removing the slash from the redirect string. Which you choose depends on what you want the result of a URL "/ignore-this-maybe/foo" to be. Do you want that to map to "/-maybe/foo"? If the answer is you'd prefer that not trigger the matching logic at all, then you want to make the match more specific by appending a slash.

jmarantz · 2018-04-25T12:26:24Z

Another alternative is to just change doc (and not code) to suggest that if you want to match a whole segment, then include the trailing slash in the match string.

dio · 2018-04-25T19:15:35Z

@jmarantz thanks for clearing that up.

Sorry for not catching that earlier! Will keep the code and convert this PR to just contain doc and additional test input samples for that RedirectPrefixRewrite case. Thanks!

Signed-off-by: Dhi Aurrahman <dio@rockybars.com>

dio · 2018-04-26T01:40:59Z

@jmarantz updated. If you have time please help to take a look. Thanks!

Side note: for now to accomplish that common use-case (when you want to rewrite e.g. /prefix, /prefix/ok, /prefix/ok/1/2 to /, /ok and /ok/1/2 respectively) we need to define two entries in the config.

jmarantz · 2018-04-25T21:04:11Z

api/envoy/api/v2/route/route.proto

-  //   it rewrites :code:`/prefix` to :code:`/` and :code:`/prefix/foo` to :code:`/foo`.
+  //   it rewrites :code:`/prefix` to :code:`/`, but :code:`/prefix/ok` to :code:`//ok`, which
+  //   most of the time is not the intended result. However, it can be useful for the case of
+  //   replacing substring of a path segment. For example to rewrite :code:`/prefixok` to :code:`/ok`.


s/replacing substring/replacing a substring/

jmarantz · 2018-04-26T02:04:45Z

api/envoy/api/v2/route/route.proto

+  //         prefix: "/prefix"
+  //       route:
+  //         prefix_rewrite: "/"
+  //


I think the examples below will be clearer if the two prefix-substitutions above use different match strings.

Could you clarify?

The point of this is to tell that to achieve such behavior (rewriting by replacing a segment of a URL, in this case: "prefix" and attach the rest to the root path) is by declaring two match rules (with and without trailing "/", in this case "/prefix" and "/prefix/"). The below example tells that if we remove the rule with "match" value with trailing slash, it gives a different result (rewrites "/prefix/ok" to "//ok").

jmarantz · 2018-04-26T02:05:43Z

api/envoy/api/v2/route/route.proto

+    //     - match:
+    //         prefix: "/prefix"
+    //       redirect:
+    //         prefix_rewrite: "/"


I actually don't have the whole context for this file, but I wonder if you could point to the first instance of this example rather than repeating it.

jmarantz · 2018-04-26T02:14:06Z

test/common/router/config_impl_test.cc

+    redirect->rewritePathHeader(headers);
+    EXPECT_EQ("http://redirect.lyft.com/", redirect->newPath(headers));
+  }
+  {


It's worth some comments here to describe what how these two cases are different.

OK, will add some comments here.

zuercher · 2018-04-26T20:32:01Z

api/envoy/api/v2/route/route.proto

+  //
+  // .. note::
+  //
+  //   Take a careful attention to the use of trailing slash in match prefix value. A common


I think if you write something like the following, you can delete the examples.

"Pay careful attention to the use of trailing slashes in the RouteMatch's prefix value. Stripping a prefix from a path requires multiple Routes to handle all cases. For example, rewriting /prefix to / and /prefix/etc to /etc cannot be done in a single Route."

Thanks @zuercher, I have updated this PR as suggested.

Signed-off-by: Dhi Aurrahman <dio@rockybars.com>

mattklein123 · 2018-04-29T06:58:54Z

api/envoy/api/v2/route/route.proto

+  //   :ref:`route's match <envoy_api_field_route.Route.match>` prefix value.
+  //   Stripping a prefix from a path requires multiple Routes to handle all cases. For example,
+  //   rewriting */prefix* to */* and */prefix/etc* to */etc*
+  //   cannot be done in a single :ref:`Route <envoy_api_msg_route.Route>`.


Can we give an explicit example of what the user would configure? I think that would be helpful.

Also, Q: I lost track of the conversation but isn't there a targeted fix possible here where we just adjust/normalize the slash where the prefix ends and the rest of the route begins?

Can we give an explicit example of what the user would configure? I think that would be helpful.

I added an example in my previous commit, https://github.com/dio/envoy/blob/d46c26e4ede389971fbfeca7bd2d2e48716ea2d4/api/envoy/api/v2/route/route.proto#L369-L393 if you think this is good enough I can add it back.

Also, Q: I lost track of the conversation but isn't there a targeted fix possible here where we just adjust/normalize the slash where the prefix ends and the rest of the route begins?

We have this case like the following:

{ Http::TestHeaderMapImpl headers = genRedirectHeaders("redirect.lyft.com", "/ignore-this/use/", false, false); const DirectResponseEntry* redirect = config.route(headers, 0)->directResponseEntry(); redirect->rewritePathHeader(headers); EXPECT_EQ("http://redirect.lyft.com/use/", redirect->newPath(headers)); }

Where the config is:

- match: { prefix: "/ignore-this"} redirect: { prefix_rewrite: "/" }

We decided to accommodate this behavior. Hence no code change for this.

Yeah my thinking here is that slashes have no special role in the URL after the first 3. E.g.
scheme://origin/foo/bar
and
scheme://origin/foo//bar
are both different valid URLs and origin servers can interpret them differently. Moreover, proxies may see arbitrary slashes in URL parameters. If they originate from a browser they'd be escaped, but not if they originate from some other client.

You could normalize the match strings, but you only lose generality doing so, and it's conceivable someone may want a URL rewriter that strips off a prefix that does not necessarily end with a slash.

Authoritative spec is https://www.w3.org/Addressing/URL/url-spec.txt which has this (IMO) somewhat ambiguous statement:

The path is interpreted in a manner dependent on the scheme being
used. Generally, the reserved slash "/" character (ASCII 2F hex)
denotes a level in a hierarchical structure, the higher level part
to the left of the slash.

So in typical use (e.g. mapping to a file-system) might correspond to a file-system where normalization makes sense, that's not required by the spec and in practice I think it's better not to mess with URLs too much.

I think what I'm suggesting is to add an explicit example above in the docs where you talk about how it cannot be done in a single route rule. Can you list out the two rules that cover the example case so the user better understands?

@mattklein123 OK. I have it here: a4c60d5.

Thank you @dio! Very clear.

Signed-off-by: Dhi Aurrahman <dio@rockybars.com>

docs: clarify the use of "/" as a prefix_rewrite This patch clarifies the use of "/" as a prefix_rewrite in route and redirect prefix rewriting. And also a note on the use of trailing slashes as match value. Risk Level: Low Testing: add more input samples to RedirectPrefixRewrite test. Docs Changes: Update route.proto doc regarding path_rewrite both for redirect and route. Release Notes: N/A Fixes envoyproxy#2956 Signed-off-by: Dhi Aurrahman <dio@rockybars.com> Signed-off-by: Rama <rama.rao@salesforce.com>

dio added 2 commits April 22, 2018 07:17

utility: add normalize URL's path helper

65d01cc

Signed-off-by: Dhi Aurrahman <dio@rockybars.com>

config: normalize URL path after prefix rewrite

ae407fd

Signed-off-by: Dhi Aurrahman <dio@rockybars.com>

mattklein123 assigned zuercher Apr 22, 2018

zuercher requested changes Apr 23, 2018

View reviewed changes

Remove partial support for url normalization

64bf6ad

Signed-off-by: Dhi Aurrahman <dio@rockybars.com>

zuercher requested changes Apr 23, 2018

View reviewed changes

dio added 2 commits April 24, 2018 06:19

Add examples as notes

a88cebb

Signed-off-by: Dhi Aurrahman <dio@rockybars.com>

Fix comment

e5bf03b

Signed-off-by: Dhi Aurrahman <dio@rockybars.com>

zuercher previously approved these changes Apr 24, 2018

View reviewed changes

mattklein123 reviewed Apr 24, 2018

View reviewed changes

jmarantz reviewed Apr 25, 2018

View reviewed changes

Remove normalize path util

9541fb6

Signed-off-by: Dhi Aurrahman <dio@rockybars.com>

dio dismissed zuercher’s stale review via 9541fb6 April 25, 2018 04:39

dio changed the title ~~config: normalize URL path after prefix rewriting~~ router config: handle / in prefix rewrite as replace with empty string Apr 25, 2018

dio changed the title ~~router config: handle / in prefix rewrite as replace with empty string~~ router config: handle / in prefix rewriting as replacing with empty string Apr 25, 2018

dio changed the title ~~router config: handle / in prefix rewriting as replacing with empty string~~ router config: handle “/“ in prefix rewriting as replacing with empty string Apr 25, 2018

Add more samples for RedirectPrefixRewrite

dc34878

Signed-off-by: Dhi Aurrahman <dio@rockybars.com>

dio changed the title ~~router config: handle “/“ in prefix rewriting as replacing with empty string~~ docs: clarify the use of "/" as a prefix_rewrite Apr 25, 2018

dio added 2 commits April 26, 2018 03:21

Add more input samples

ddf2a2e

Signed-off-by: Dhi Aurrahman <dio@rockybars.com>

Update docs

d46c26e

Signed-off-by: Dhi Aurrahman <dio@rockybars.com>

jmarantz approved these changes Apr 26, 2018

View reviewed changes

zuercher reviewed Apr 26, 2018

View reviewed changes

Update doc and comments

b47c282

Signed-off-by: Dhi Aurrahman <dio@rockybars.com>

zuercher previously approved these changes Apr 28, 2018

View reviewed changes

mattklein123 reviewed Apr 29, 2018

View reviewed changes

Add example of config entries

a4c60d5

Signed-off-by: Dhi Aurrahman <dio@rockybars.com>

dio dismissed zuercher’s stale review via a4c60d5 April 30, 2018 10:20

mattklein123 approved these changes Apr 30, 2018

View reviewed changes

zuercher approved these changes Apr 30, 2018

View reviewed changes

zuercher merged commit d20c12d into envoyproxy:master Apr 30, 2018

docs: clarify the use of "/" as a prefix_rewrite #3160

docs: clarify the use of "/" as a prefix_rewrite #3160

Conversation

dio commented Apr 22, 2018 • edited Loading

mattklein123 commented Apr 23, 2018

zuercher commented Apr 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zuercher left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dio commented Apr 23, 2018

mattklein123 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dio Apr 25, 2018 • edited Loading

Choose a reason for hiding this comment

jmarantz commented Apr 25, 2018

jmarantz commented Apr 25, 2018

dio commented Apr 25, 2018 • edited Loading

dio commented Apr 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dio Apr 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dio Apr 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dio Apr 29, 2018 • edited Loading

Choose a reason for hiding this comment

jmarantz Apr 29, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dio commented Apr 22, 2018 •

edited

Loading

dio Apr 25, 2018 •

edited

Loading

dio commented Apr 25, 2018 •

edited

Loading

dio commented Apr 26, 2018 •

edited

Loading

dio Apr 26, 2018 •

edited

Loading

dio Apr 26, 2018 •

edited

Loading

dio Apr 29, 2018 •

edited

Loading

jmarantz Apr 29, 2018 •

edited

Loading