feature regression: no longer possible to rewrite authority

[howardjohn]

Title: no longer possible to rewrite authority

Description:
#14747 changed the behavior of header modifications that are allowed. Previously, we relied on this to rewrite the authority on a per-destination basis.

After the change, this is no longer possible. While we can do a rewrite at the route level, we cannot do it on a per-weighted-cluster basis, unlike all other headers.

For us this represents a breaking change, so we will likely set the runtime flag for now. But we want to understand a long term plan.

cc @alyssawilk

[optional Relevant Links:]
Istio specific issue: istio/istio#33226

[alyssawilk]

Yeah, this was a bit of a bug, because it allowed rewrites specific for HTTP/1 and not HTTP/2 so would cause inconsistent behavior.

Would any of
"host_rewrite_literal": "...",
"auto_host_rewrite": "{...}",
"host_rewrite_header": "...",
"host_rewrite_path_regex": "{...}",

work for you?
https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route_components.proto#envoy-v3-api-field-config-route-v3-routeaction-regex-rewrite

If not happy to discuss the use case further.

[howardjohn]

We do use host_rewrite_literal, but the issue is it cannot be set inside a WeightedCluster_ClusterWeight. So we can set it at the RouteAction level but if we have multiple destinations we can only use one rewrite between all of them

[alyssawilk]

ah, gotcha. So if host_rewrite_specifier were added to ClusterWeight that'd work for you?
cc @envoyproxy/api-shepherds too since I'd like their input as well.

Also you did catch this validation is runtime guarded, yes? Ideally you should just be able to flip the runtime guard back while we figure out a long term solution.

[howardjohn]

Yep, I was planning to set the runtime flag in the meantime. Thanks for adding that

ah, gotcha. So if host_rewrite_specifier were added to ClusterWeight that'd work for you?

Yep

[alyssawilk]

@adisuissa @lizan thoughts?

[alyssawilk]

@htuch can you comment on this?
We allow header tweaks in many places - if we're game for adding host rewrite to more places I can go do that.

[htuch]

I guess this is reasonable to ask for behavior wise. What I'm missing a bit is what are the set of hierarchically overridable things we want to make available more generally. We already have request/response headers to add, now we add host rewrite, what does the full picture here look like?

We don't need to boil the ocean, but at the same time, building an understandable/maintainable/consistent API should have some notion of what makes sense to have as hierarchically overridable. I have this feeling that if we had rewritten the API from scratch with these requirements, it would have a dedicated message that could be composed across levels.

[alyssawilk]

@howardjohn should be fixed - lmk if this doesn't work for all your use cases and I'll keep iterating :-)