HTTP3 and QUIC support

[howardjohn]

QUIC recently went to "version 1" as https://datatracker.ietf.org/doc/html/rfc9000.

This issue tracks support for HTTP 3 and QUIC in the API.

These are similar to HTTP and TCP in terms of routing functionality, but they always have TLS and are on UDP instead of TCP.
Envoy, Google cloud, and Nginx all support HTTP3 today.

Typically, sites will want to expose HTTP3 and HTTP1/2 on the same domain/port. Unlike HTTP2 which can be negotiated using ALPN, HTTP3 is on a completely different protocol (QUIC), so this is not possible. Instead, clients connect using HTTP1/2 initially then upgrade to HTTP3 after receiving the alt-svc header from the server.

Kubernetes makes this a bit more difficult, as mixing protocols is not allowed in LoadBalancer Services. This is supported by the 1.20 alpha feature MixedProtocolLBService: Support mixed protocols in service.type=loadbalancer. However, it has minimal adoption so far, with only MetalLB support as far as I know. As a result, users will most likely not be able to actually achieve HTTP2+3 on the same domain/port for a while. However, the API can still support this, enabling users to move to the same port/service once their clusters support it. We may additionally choose to automate the alt-svc headers, or leave it up to the users to explicitly define it in the API

[jpeach]

Could implementations transparently enable HTTP3 on a gateway listener it the protocol in "HTTPS"? Leaving aside whether the underlying cluster networking supports that, I think that would be OK from an API perspective?

[howardjohn]

Without thinking too much about the implications, I am thinking yes? But it may be helpful to have the user specify intent here, as adding HTTP3 on the listener may not be free?

A similar concept applies for HTTP/2 support really.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[hbagdi]

/lifecycle frozen

[shaneutt]

Despite this issue being quite old, we the maintainers are still pretty convinced that we want to have this functionality in a future release. We are marking this help wanted as we're looking for contributors with strong use cases to help champion and drive this forward.

[alibo]

Mixed Protocol

the current status of the mixed protocol feature in k8s:

1. MixedProtocolLBService graduated to beta in v1.24: Feature flag change when moving from alpha to beta kubernetes/kubernetes#109213
2. Apparently, Azure also supports it, but the adoption is slow for other cloud providers: Delete deprecated mixed procotol annotation kubernetes/kubernetes#97096

---

Status of HTTP3 in current ingress/gateway-api controllers

Emissary-ingress/Ambassador and Traefik are the early adopters of http3 (downstream-mode) among ingress controllers:

• [v3.0] add downstream support for HTTP/3 emissary-ingress/emissary#4246
• Add HTTP3 support (experimental) traefik/traefik#7724

they also have workarounds for the major cloud providers regarding the mixed-protocol issue:

• HTTP/3 with AKS
• HTTP/3 with EKS
• HTTP/3 with GKE
• How to use Traefik Proxy and HTTP/3 on AWS EKS

Related open issues in other ingress/gateway-api controllers for supporting http3/quic:

• Kong: HTTP3/QUIC Support Kong/insomnia#3691
• Contour: Support HTTP/3 clients projectcontour/contour#2582
• Nginx ingress: [FEATURE REQUEST] HTTP/3 support kubernetes/ingress-nginx#4760
• Apisix: feat: apisix support QUIC+HTTP3 apache/apisix#6403

(I couldn't find anything related to http3/quic for the rest of controllers listed here: https://gateway-api.sigs.k8s.io/implementations/)

---

Usage

Traefik

1. You need to enable it using cli flags or config file in traefik pods, first:

• CLI:

--experimental.http3=true 
--entrypoints.name.http3.advertisedport=443

• Config file:

experimental:
  http3: true

entryPoints:
  name:
    http3:
      advertisedPort: 443

complete Deployment file: https://github.com/traefik-tech-blog/traefik-http3/blob/master/traefik/2-deployment.yaml#L27-L30

2. It's completely hidden from user side and handled implicitly by traefik itself:

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: dashboard
  namespace: default
spec:
  entryPoints:
    - websecure
  routes:
    - kind: Rule
      match: PathPrefix(`/dashboard`) || PathPrefix(`/api`)
      services:
        - name: api@internal
          kind: TraefikService
  tls: {}

https://github.com/traefik-tech-blog/traefik-http3/blob/master/traefik/dashboard/4-ingressroute.yaml

Emissary-Ingress/Ambassador

1. You need to create a dedicated Listener for UDP/HTTP3 traffic:

# This is a standard Listener that leverages TCP to serve HTTP/2 and HTTP/1.1 traffic.
# It is bound to the same address and port (0.0.0.0:8443) as the UDP listener.
apiVersion: getambassador.io/v3alpha1
kind: Listener
metadata:
  name: emissary-ingress-https-listener
  namespace: emissary
spec:
  port: 8443
  protocol: HTTPS
  securityModel: XFP
  hostBinding:
    namespace:
      from: ALL
---
# This is a Listener that leverages UDP and HTTP to serve HTTP/3 traffic.
# NOTE: Raw UDP traffic is not supported. UDP and HTTP must be used together.
apiVersion: getambassador.io/v3alpha1
kind: Listener
metadata:
  name: emissary-ingress-https-listener-udp
  namespace: emissary
spec:
  port: 8443
  # Order is important here. HTTP is required.
  protocolStack:
    - TLS
    - HTTP
    - UDP
  securityModel: XFP
  hostBinding:
    namespace:
      from: ALL

https://www.getambassador.io/docs/emissary/3.1/topics/running/http3/

2. The rest is the same as http2/http1.1 Host object:

apiVersion: getambassador.io/v3alpha1
kind: Host
metadata:
  name: my-domain-host
spec:
  hostname: your-hostname
  # acme isn't required but just shown as an example of how to manage a valid TLS cert
  acmeProvider:
    email: your-email@example.com
    authority: https://acme-v02.api.letsencrypt.org/directory
  tls:
    # QUIC requires TLS v1.3 version. Verify your client supports it.
    min_tls_version: v1.3
    # Either protocol can be upgraded, but http/2 is recommended.
    alpn_protocols: h2,http/1.1

https://www.getambassador.io/docs/emissary/3.1/topics/running/http3/#configuring-the-host-resource

---

Proposed API Design for Gateway API

IMHO, it should be enabled and configured in Gateway object and be transparent to users.

1. Mixed with HTTPS protocol, but using a new field to configure:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: mixed-http3-and-https
spec:
  gatewayClassName: http3-enabled-gateway
  listeners:
  - name: foo-https
    protocol: HTTPS
    port: 443
    hostname: foo.example.com
    http3:
      enabled: true
      advertisedPort: 443
    tls:
      certificateRefs:
      - kind: Secret
        group: ""
        name: foo-example-com-cert
      options:
        minTlsVersion: 1.3

2. A new listener with a dedicated protocol:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: separated-listener-for-http3
spec:
  gatewayClassName: http3-enabled-gateway
  listeners:
  - name: http3
    protocol: HTTP3
    port: 443 # can be used for alt-svc as well
    hostname: foo.example.com
    tls:
      certificateRefs:
      - kind: Secret
        group: ""
        name: foo-example-com-cert
      options:
        minTlsVersion: 1.3
  - name: https
    protocol: HTTPS
    port: 443
    hostname: foo.example.com
    tls:
      certificateRefs:
      - kind: Secret
        group: ""
        name: foo-example-com-cert

---

Challenages/Questions

1. TLSRoute / TLS Passthrough: TLS passthrough and HostSNI rules for HTTP/3 UDP traefik/traefik#9050
2. Upstream HTTP3: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/http/http3#http3-upstream - too experimental! - it sounds like the previous one, but in this mode, termination happens in the gateway-api itself.
3. Should user be able to set other parameters such as persist for the alt-svc header? https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Alt-Svc#syntax

---

Usecases

Despite this issue being quite old, we the maintainers are still pretty convinced that we want to have this functionality in a future release. We are marking this help wanted as we're looking for contributors with strong use cases to help champion and drive this forward.

@shaneutt You may find good use-cases for supporting http3 in the comprehensive and well-written articles by Robin Marx:

• https://www.smashingmagazine.com/2021/08/http3-core-concepts-part1/
• https://www.smashingmagazine.com/2021/08/http3-performance-improvements-part2/
• https://www.smashingmagazine.com/2021/09/http3-practical-deployment-options-part3/

but to give you a practical and real-world example, our company is a ride-hailing company with a large number of moving drivers, we're observing many TCP reconnections as they move constantly and connect to different cell towers every so often, so the connection is dropped and clients need to reconnect to the server. As a result, we're observing a noticeable delay in receiving data on both sides + TLS handshake has so much overhead even with TLS 1.3 when the number of requests is high. Connection ID / Connection migration is a great feature of http3/quic we can utilize to mitigate this issue.

[shaneutt]

Thanks for the detailed write-up @alibo. Given the depth of your comment I would encourage you to take this issue if that's something that interests you. Otherwise for anyone else who's interested in taking on this issue it would appear @alibo's write-up above ☝️ provides a lot of good context and thoughts to get started, definitely read it over.

[alibo]

@shaneutt I'm glad it was helpful, my colleague ( @m-yosefpor ) and I are interested in working on this feature, but any contributions are also welcomed :)

I guess in the first step, we need to prepare a GEP? https://gateway-api.sigs.k8s.io/contributing/enhancement-requests/

[alibo]

Also, I've investigated some popular load balancers/reverse proxies to see how they've supported http3/quic:

Haproxy (2.6+)

frontend mysite
 bind :80
 bind :443  ssl crt /etc/haproxy/certs/foo.com/cert.pem alpn h2

 # enables HTTP/3 over QUIC
 bind quic4@:443 ssl crt /etc/haproxy/certs/foo.com/cert.pem alpn h3

 # Redirects to HTTPS
 http-request redirect scheme https unless { ssl_fc }

 # 'Alt-Svc' header invites client to switch to the QUIC protocol
 # Max age (ma) is set to 15 minutes (900 seconds), but
 # can be increased once verified working as expected
 http-response set-header alt-svc "h3=\":443\";ma=900;"

 default_backend webservers

it can be enabled in the same frontend HTTP and HTTPS configured by adding a new bind directive.

More info: https://www.haproxy.com/blog/announcing-haproxy-2-6/#http-3-over-quic

Nginx (cloudflare/quiche)

http {
    server {
        # Enable QUIC and HTTP/3.
        listen 443 quic reuseport;

        # Enable HTTP/2 (optional).
        listen 443 ssl http2;

        ssl_certificate      cert.crt;
        ssl_certificate_key  cert.key;

        # Enable all TLS versions (TLSv1.3 is required for QUIC).
        ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;

        # Add Alt-Svc header to negotiate HTTP/3.
        add_header alt-svc 'h3=":443"; ma=86400';
    }
}

It can be enabled by adding a new listen directive with quic option. More info:

• https://github.com/cloudflare/quiche/tree/master/nginx
• https://blog.cloudflare.com/experiment-with-http-3-using-nginx-and-quiche/

Nginx (experimental, but official nginx-quic branch)

    http {
        log_format quic '$remote_addr - $remote_user [$time_local] '
                        '"$request" $status $body_bytes_sent '
                        '"$http_referer" "$http_user_agent" "$http3"';

        access_log logs/access.log quic;

        server {
            # for better compatibility it's recommended
            # to use the same port for quic and https
            listen 8443 http3 reuseport;
            listen 8443 ssl;

            ssl_certificate     certs/example.com.crt;
            ssl_certificate_key certs/example.com.key;+
−
            ssl_protocols       TLSv1.3;

            location / {
                # required for browsers to direct them into quic port
                add_header Alt-Svc 'h3=":8443"; ma=86400';
            }
        }
    }

It can be enabled by adding a new listen directive with http3 option. More info:

• https://hg.nginx.org/nginx-quic/file/quic/README
• https://www.nginx.com/blog/our-roadmap-quic-http-3-support-nginx/

It also provides many HTTP and SERVER directives to configure quic and http3 protocols:

• ssl_early_data on; - To enable 0-RTT
• quic_retry on; - To enable address validation
• quic_gso on; - To enable GSO (Generic Segmentation Offloading)
• quic_mtu <size>; - To limit maximum UDP payload size on receive path
• quic_host_key <filename>; - To set host key for various token
• http3_stream_buffer_size
• http3_max_concurrent_pushes
• http3_max_concurrent_streams
• http3_push
• http3_push_preload
• http3_hq (requires NGX_HTTP_V3_HQ macro)

I'm not sure it's a good idea to expose these parameters to Gateway API objects. I think they can be easily configured in GatewayClass using parametersRef field if needed or they can be configured in a new options field (similar to tls)

Caddy Server

{
    auto_https off
    servers {
        protocol {
            experimental_http3
        }
    }
}

yoursite.com {
    tls /caddy.crt /caddy.key
    reverse_proxy * https://host-gateway {
        transport http {
            tls_insecure_skip_verify
        }
    }
}

More info: https://dev.to/syncsynchalt/safe-http3-experimentation-with-caddy-447f

Traefik

Already discussed in #687 (comment)

It can be enabled using cli flags or config file:

CLI:

--experimental.http3=true 
--entrypoints.name.http3.advertisedport=443

Config file:

experimental:
  http3: true

entryPoints:
  name:
    http3:
      advertisedPort: 443

more info:

• https://github.com/traefik-tech-blog/traefik-http3
• https://doc.traefik.io/traefik/routing/entrypoints/#http3

Envoyproxy (downstream mode)

 static_resources:
  listeners:
  - name: listener_tcp
    address:
      socket_address:
        protocol: TCP
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - transport_socket:
        name: envoy.transport_sockets.tls
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
          common_tls_context:
            tls_certificates:
            - certificate_chain:
                filename: certs/servercert.pem
              private_key:
                filename: certs/serverkey.pem
      filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: HTTP2
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              response_headers_to_add:
              - header:
                  key: alt-svc
                  value: h3=":10000"; ma=86400, h3-29=":10000"; ma=86400
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  host_rewrite_literal: www.envoyproxy.io
                  cluster: service_envoyproxy_io
          http3_protocol_options:
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

  - name: listener_udp
    address:
      socket_address:
        protocol: UDP
        address: 0.0.0.0
        port_value: 10000
    udp_listener_config:
      quic_options: {}
      downstream_socket_config:
        prefer_gro: true
    filter_chains:
    - transport_socket:
        name: envoy.transport_sockets.quic
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.transport_sockets.quic.v3.QuicDownstreamTransport
          downstream_tls_context:
            common_tls_context:
              tls_certificates:
              - certificate_chain:
                  filename: certs/servercert.pem
                private_key:
                  filename: certs/serverkey.pem
      filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          codec_type: HTTP3
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  host_rewrite_literal: www.envoyproxy.io
                  cluster: service_envoyproxy_io
          http3_protocol_options:
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

http3 options:

{
  "quic_protocol_options": {...},
  "override_stream_error_on_invalid_http_message": {...},
  "allow_extended_connect": ...
}

quic options:

{
  "max_concurrent_streams": {...},
  "initial_stream_window_size": {...},
  "initial_connection_window_size": {...},
  "num_timeouts_to_trigger_port_migration": {...},
  "connection_keepalive": {...}
}

It requires a dedicated listener with a specific codec and transport socket. You can also see the emissary's implementation in the following link to understand how it configures http3 in envoy: emissary-ingress/emissary#4246

More info:

• https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/http/http3
• https://github.com/envoyproxy/envoy/blob/main/configs/envoyproxy_io_proxy_http3_downstream.yaml
• https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#config-core-v3-http3protocoloptions
• https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/core/v3/protocol.proto#envoy-v3-api-msg-config-core-v3-quicprotocoloptions

[shaneutt]

I guess in the first step, we need to prepare a GEP? https://gateway-api.sigs.k8s.io/contributing/enhancement-requests/

Yep sounds good, and looks like you can add much of the context already posted here in this issue. Let us know if there's any help you need putting together a GEP. Thank you!

/assign @alibo

Also @m-yosefpor if I get a confirmation from you that you'd like to be assigned to this as well, I'm happy to add you on the issue just let me know.

[m-yosefpor]

@shaneutt Sure. I would be glad to work on this issue.

[shaneutt]

Thank you!

/assign @m-yosefpor

[howardjohn]

FYI https://github.com/istio/istio/wiki/Experimental-QUIC-and-HTTP-3-support-in-Istio-gateways also supports it. I think you could get it working with gateway-api today as well, but its not currently documented (more of a coincidence that it would work really, since HTTP3 isn't in the spec yet)

…

On Tue, Aug 23, 2022 at 7:14 AM Shane Utt ***@***.***> wrote: Thank you! /assign @m-yosefpor <https://github.com/m-yosefpor> — Reply to this email directly, view it on GitHub <#687 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXPAMFGGLZNSCUOREWLV2TMEHANCNFSM455CRDQA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[alibo]

@howardjohn cool, thank you for the heads up. It's also one of my questions, if http3 becomes the mainstream technology like http2 is right now, then I'm not seeing any reason to have a specific field/type for just http3 in a more general API like Gateway API. What if somewhere in the future http4 is released, should we add another type/field for it?!

[howardjohn]

The tricky thing with HTTP3 IMO is that its on a completely different protocol (UDP). This implies opening 2 ports (not supported in many cases), alt-svc for negotiation, etc. Its possible to just do that automagically, but I could see very reasonable reasons for wanting to opt-in/opt-out of that.

…

On Tue, Aug 23, 2022 at 9:24 AM Ali Borhani ***@***.***> wrote: @howardjohn <https://github.com/howardjohn> cool, thank you for the heads up. It's also one of my questions, if http3 becomes the mainstream technology like http2 is right now, then I'm not seeing any reason to have a specific field/type for just http3 in a more general API like Gateway API. What if somewhere in the future http4 is released, should we add another type/field for it?! — Reply to this email directly, view it on GitHub <#687 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXJXO3IKWGJSCL7J4KLV2T3NXANCNFSM455CRDQA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[youngnick]

Those are some seriously great write-ups @alibo. Nice work.

In your GEP, I'd recommend picking whichever one of the two options you suggest here you think is better, and document the other one in "Alternaatives Considered". (On an initial read, I think I'm slightly preferring adding a new Protocol and having a separate listener, but it seems like defining how the alt-svc thing would work across listeners might be tricky and would definitely need to be included in the GEP).

For this one as well, I'd put a little background in the Introduction (or add an appendix) that explains how the alt-svc works and why it's important.

[alibo]

Sorry for my delayed response. I had a chance to look at this again and I realized a few blockers/assumptions are changed and they might affect the future of this feature.

1. MixedProtocolLBService became GA in K8s 1.26: KEP-1435 Mixed Protocol values in LoadBalancer Service GA kubernetes/kubernetes#112895
2. Support for HTTPS DNS RR in clients (especially browsers)

Supporting HTTPS DNS RR is already added to Chrome and Firefox for different use cases - including http3 protocol upgrade (chrome: chromium/chromium@d811496 | firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1721132). It's not enabled by default in Chrome, but you can enable it in chrome://flags/#use-dns-https-svcb-alpn. (it's already enabled in Firefox, check network.dns.upgrade_with_https_rr and network.dns.use_https_rr_as_altsvc variables in about:config)

I think it will be the default option to upgrade to http3 pretty soon as it's faster and it's going to be used for other important use cases as well:

• ECH (encrypted-client-hello):
• HTTPS upgrade (+ HSTS)
• Protocol upgrade (alt-svc: h2, h3, ipv6, ...) and configuration

$ q -t HTTPS cloudflare.com

cloudflare.com. 5m0s HTTPS 1 . alpn="h3,h3-29,h2" ipv4hint="104.16.132.229,104.16.133.229" ipv6hint="2606:4700::6810:84e5,2606:4700::6810:85e5"

$ q -t HTTPS cloudflare-quic.com

cloudflare-quic.com. 5m0s HTTPS 1 . alpn="h3,h3-29,h2" ipv4hint="104.22.8.38,104.22.9.38,172.67.9.235" ech="AEX+..." ipv6hint="2606:4700:10::6816:826,2606:4700:10::6816:926,2606:4700:10::ac43:9eb"

---

Considering the new changes, I guess, this feature can be delegated to the controller itself.

[shaneutt]

@alibo and @m-yosefpor I notice you are assigned, but it seems unclear given the time that's elapsed whether you're both working on this still. Are you still working on this, or should we unassign you? Let us know how we can help! 🖖

[m-yosefpor]

@shaneutt As explained in #687 (comment) , we think there are no changes required in APIs to support HTTP3. So we can delegate this to controller implementations. Do you still think we need to somehow express http3 explicitly in API gateway spec?

[shaneutt]

I understood the suggestion, but it was not entirely clear to me if we were all on the same page. I think for the time being, given that it seems both of you assigned seem to agree there's no action to take right now, we can close this one and keep it for posterity.

As time moves on if contributors reach this issue and do want some further and specific action taken regarding HTTP3/QUIC let's create some new and distinct issues (OR re-open this if that's preferred).