Servers with a gRPC proxyProtocol and HTTPRoutes cannot be upgraded past edge-24.7.1

[andrew-gropyus]

What is the issue?

We are running edge 24.5.3 and upgrading to 24.11.8.

We use a default deny policy.

We currently run a number of grpc servers (Servers that have proxyProtocol: "gRPC") combined with http routes and authorization policies for fine grained authorization.

In 24.5.3 the behavior of linkerd when building the server model internally was to collect the http routes for grpc server.

In #12785 (released in edge-24.7.1) this behavior was changed to use grpc routes for a grpc server:

-            ProxyProtocol::Grpc => Some(proto::proxy_protocol::Kind::Http2(
-                proto::proxy_protocol::Http2 {
-                    routes: http::to_route_list(&srv.http_routes, cluster_networks),
+            ProxyProtocol::Grpc => Some(proto::proxy_protocol::Kind::Grpc(
+                proto::proxy_protocol::Grpc {
+                    routes: grpc::to_route_list(&srv.grpc_routes, cluster_networks),
+                },
+            )),

When a grpc server does not have any grpc routes, it defaults to a default grpc route.

The same PR also added support for adding grpc routes as target refs for authorization policies. Before this PR, an authorization policy that had a target ref pointing to a grpc route would fail in the policy admission controller.

As I understand it, this means that the following combination of components can not be upgraded with out downtime:

• default deny policy
• servers with proxy protocol grpc
• http routes
• authorization policies

How can it be reproduced?

Before upgrading to 24.7.1 have a server, http route, and authorization policy that look like the following:

apiVersion: policy.linkerd.io/v1beta2
kind: Server
metadata:
  name: api-grpc
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/instance: api
  port: grpc
  proxyProtocol: gRPC
---
apiVersion: policy.linkerd.io/v1beta3
kind: HTTPRoute
metadata:
  name: api-grpc-health
spec:
  parentRefs:
    - group: policy.linkerd.io
      kind: Server
      name: api-grpc
  rules:
    - matches:
        - path:
            value: "/grpc.health.v1.Health/Check"
---
apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata:
  name: api-grpc-health-allow-anonymous
spec:
  targetRef:
    group: policy.linkerd.io
    kind: HTTPRoute
    name: api-grpc-health
  requiredAuthenticationRefs: []

Then upgrade to or past 24.7.1.

Note that the server has not detected the http route, and is instead relying on a default route, with a default policy to authorize requests.

Logs, error output, etc

No logs

output of linkerd check -o short

Only up to date warnings

Environment

• Kubernetes Version: 1.30.5
• Kubernetes Environment: AKS
• Host OS: Ubuntu 22.04
• Linkerd version: edge-24.5.3

Possible solution

• When building the internal server model for grpc servers, allow falling back to http routes if grpc routes are not present.
• Switch from proxy protocol gRPC to HTTP/2. This would work with the http routes but would likely impact the metrics that are being collected. I have not looked deeply into this.

Additional context

We are looking for an online migration path, we would rather not bring services down while we switched them from http routes to grpc routes.

Would you like to work on fixing this bug?

None

[andrew-gropyus]

We are also experimenting with changing the proxy protocol for grpc to http/2. Although we are not fully sure of the implications.

From what we understand, load balancing http/2 should be the same as load balancing grpc. There are some differences in metrics. In particular the grpc_status_code label would not be published. The classification label would work the same as far as we understand, since grpc error status codes map to http 4xx or 5xx status codes.

[adleong]

Thanks for reporting this @andrew-gropyus. We're looking into this so that we can advise you on the best way to do this upgrade without downtime.

[adleong]

Hey @andrew-gropyus thanks for your patience on this one.

I would recommend switching the proxyPrototol to HTTP/2 as you identified. This will allow you to continue to use the HTTPRoute resources that you have configured and it won't change the way that failure classification or load balancing works. Once the proxyProtocol is set to HTTP/2, you should be free to upgrade Linkerd versions without issue.

Once upgraded, you can either keep the HTTPRotues and HTTP/2 proxy protocol or switch over to gRPC. If you do switch over, make sure to change the proxyProtocol to gRPC and to create GRPCRoute resources to replace your current HTTPRoute resources all in one operation. If done at the same time, you won't incur any downtime. Note that if you have both GRPCRoute and HTTPRoute resources targeting the same Server, the GRPCRoute resources will take precedence over the HTTPRoutes but if the proxyProtocol is still set to HTTP/2, this means that all routes will be ignored. This is why it is important to change the proxyProtocol to gRPC at the same time as creating the GRPCRoutes.

Alternatively, there is no major downside to continuing to use the HTTPRoutes and HTTP/2 proxyProtocol after upgrading.

[andrew-gropyus]

Thanks for the investigation @adleong. We're going to take the proxyProtocol: HTTP2 path.

It's also good to know that there is a path to switch back over to grpc without downtime if needed.