Override default Envoy Proxy Bootstrap Configuration

[arkodg]

#24 highlights the need for users to override xDS resources configured by
Envoy Gateway. Raising this issue to track how do you express this intent for the Envoy Proxy Bootstrap Config using the Gateway API

here's how popular OSS projects do it

• Istio - this & this
• Contour allows the user to attach a new bootstrap file
• Emissary allows the user to override a subset of the bootstrap resources

Updated

• Design doc xds Bootstrap Design and API #1070
• API Design xds Bootstrap Design and API #1070
• EnvoyProxy Bootstrap validation validation for bootstrap within EnvoyProxy res #1109
• Implement API Implement Bootstrap API #1094
• Update GatewayClass Status based on EnvoyProxy validation Update GatewayClass status based on EnvoyProxy validation #1107
• Implement egctl subcommand to generate default bootstrap Add EnvoyProxy resource validation to egctl translate #1281
• Docs docs: for xds bootstrap #1333

[arkodg]

Some options, we could provide some or all of these knobs

• Support custom Envoy Proxy Bootstrap patches/configs using a field within the Envoy Gateway Bootstrap Config
• Support Support custom Envoy Proxy Bootstrap patches/configs using a field within the Gateway Class Parameter Reference Object

[danehans]

From #24 (comment), I like all EG or all BYO bootstrap.

[danehans]

Raising this issue to track how do you express this intent for the Envoy Proxy Bootstrap Config using the Gateway API

Currently, the bootstrap config is managed as a go template rendered by the Infra Manager and referenced as --config-yaml by the Envoy pod. As the Envoy Gateway Configuration API Design states:

... the managed Envoy proxies are configured dynamically through Kubernetes resources, primarily Gateway API objects. Optionally, the Envoy proxies can be configured by referencing the EnvoyProxy custom resource (CR) through spec.parametersRef of the managed GatewayClass.

I suggest extending the EnvoyProxy API type to support this use case. For example:

// EnvoyProxySpec defines the desired state of EnvoyProxy.
type EnvoyProxySpec struct {
	// Bootstrap defines the desired state of the Envoy bootstrap config file.
        // If unspecified, default bootstrap config parameters are used.
	//
	// +optional
	Bootstrap *Bootstrap `json:"bootstrap,omitempty"`
}

// Bootstrap defines the desired state of the Envoy bootstrap configuration file.
type Bootstrap struct {
	// Xds defines xDS configuration parameters...
        // If unspecified, default bootstrap config parameters are used.
	//
	// +optional
	Xds *Xds `json:"xds,omitempty"`
}

// Xds defines desired xDS configuration parameters.
type Xds struct {
	// Address defines the address of the xDS server...
        // If unspecified, "envoy-gateway" is used.
	//
	// +optional
	Address *XdsAddress `json:"address,omitempty"`

	// Port defines the port of the xDS server...
        // If unspecified, 18000 is used.
	//
	// +optional
	Port *XdsPortNumber `json:"port,omitempty"`
}

type XdsAddress string
type XdsPortNumber int32

A user would then create the EnvoyProxy CRD and a GatewayClass that references it. For example:

apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: example
  namespace: envoy-gateway-system
spec:
  bootstrap:
    xds:
      address: host.docker.internal # used to run EG locally
      port: 1234
  ...
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: config.gateway.envoyproxy.io
    kind: EnvoyProxy
    name: example
    namespace: envoy-gateway-system

Envoy Gateway will need to add support for gatewayclass.spec.parametersRef, get the referrenced EnvoyProxy, and validate its fields. For example, the EnvoyProxy should be rejected and surface appropriate status conditions if bootstrap.xds.address and/or bootstrap.xds.port do not match Envoy Gateway's config.

A drawback of this approach is that EG manages only one GatewayClass and all Gateways will use the same bootstrap config.

[arkodg]

@danehans I agree with where the Bootstrap content must be specified - within EnvoyProxySpec
But I do think we should start by allowing the user to completely specify a new bootstrap spec, rather than just edit the xds server info
To make this easier, envoy-gateway default xds-bootstrap (something similar) could help the user understand what is used by default

[danehans]

@arkodg thanks for providing feedback.

The following are a few concerns that come to mind for "... allowing the user to completely specify a new bootstrap spec":

1. If we will eventually support managing bootstrap config through EnvoyProxySpec, EG will have two ways to accomplish the same goal.
2. Since EG CLI flags should also be supported by the EG config file, we would pollute the EnvoyGateway API with Envoy Proxy-specific config. The EnvoyGateway API should be for configuring Envoy Gateway and not the managed Envoy proxies.
3. Supporting a full-blown replacement of the bootstrap config seems error-prone and a support burden on maintainers.

[arkodg]

3. Supporting a full-blown replacement of the bootstrap config seems error-prone and a support burden on maintainers.

I think we should better understand what the (advanced) users would like and what the project can provide.
#31 (comment) suggests we were planning on starting with all BYO bootstrap

[danehans]

I think we should better understand what the (advanced) users would like and what the project can provide.
#31 (comment) suggests we were planning on starting with all BYO bootstrap

I consider ^ a discussion more than a plan. v0.3.0 planning will take place post-KubeCon NA 2022. Let's discuss this issue during planning to try and achieve a concensensouse on the path forward.

[arkodg]

sg @danehans
also linking the extensibility goals section https://github.com/envoyproxy/gateway/blob/main/GOALS.md#extensibility here which should help drive the discussion, and would need to be revised if its no longer valid

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.

[arkodg]

keeping this open until we wrap up docs