Support NginxProxy at the Gateway level

[kate-osborn]

Proposed changes

Problem: When the control plane and data planes are split, the user will need the ability to specify data plane settings on a per-Gateway basis. To allow this, we need to support NginxProxy at the Gateway level in addition the the GatewayClass level. In practice, this means a user can reference an NginxProxy resource via the spec.infrastructure.parametersRef field on the Gateway resource. We still want to support referencing an NginxProxy at the GatewayClass level. If a Gateway and its GatewayClass reference distinct NginxProxy resources, the settings must be merged. Settings specified on a Gateway NginxProxy must override those set on the GatewayClass NginxProxy.

Solution:

NginxProxy API Changes

The NginxProxy API is now at version v1alpha2.

Breaking Changes:

• Change the scope of the CRD to Namespaced.
  • This change makes the parametersRef.namespace field on the GatewayClass required. If not specified, the NginxProxy resource will not be configured and an error message is left on the GatewayClass status.
• Make DisableHTTP2 and Telemetry.Exporter.Endpoint optional.
  • This allows Gateway users to inherit the values of these fields from the GatewayClass NginxProxy resource so they do not need to specify them again in the Gateway NginxProxy.

New fields:

• Telemetry.DisabledFeatures: allows users to explicitly disable telemetry features. It is a list with one supported entry: DisableTracing. More features may be added in future releases.

Other changes:

• Remove the listType=Map kubebuilder annotation from the RewriteClientIP.TrustedAddresses field. This listType is incorrect since TrustedAddresses can have duplicate keys.

Code changes

The graph now stores NginxProxies that are referenced by the winning GatewayClass and Gateway. This will need to be updated once we support multiple Gateways. The graph is also responsible for merging the NginxProxies when necessary. The result of this is stored on the graph's Gateway object in the field EffectiveNginxProxy. The EffectiveNginxProxy on the Gateway is used to build the NGINX configuration.

Testing:

Test description	Behavior
Create NP with no reference	NGF ignores
Update GC to reference NP	NGF processes and applies to all Gateways. GatewayClass has ResolvedRefs:True status
Update NP	NGF updates config accordingly
Create new NP with conflicting settings	NGF ignores because it is not referenced
Update GW to reference new NP	NGF calculates and applies effective NP config, favoring GW values. Gateway status contains ResolvedRefs:True
Update GW NP and add a net-new value	NGF updates config accordingly
Update GW NP to disable telemetry using new DisabledFeatures field	Telemetry is disabled
Update GW NP and remove DisabledFeatures field	Telemetry is re-enabled
Delete GC NP	NGF updates config, removing any settings inherited by the GC NP. GC status has ResolvedRefs: false and a warning for Accepted:True
Delete GW NP	NGF removes all NP related config. GW status has ResolvedRefs:false and a warning for Accepted:True
Helm install with config specified	Helm creates NP in release namespace and references it in the GC parametersRef field

Please focus on (optional): If you any specific areas where you would like reviewers to focus their attention or provide
specific feedback, add them here.

Closes #2990

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes

If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.

Add support for Gateway Infrastructure ParametersRef. Configure data plane settings for individual Gateways by referencing an NginxProxy resource from the `spec.infrastructure.parametersRef` field on the Gateway resource.

[kate-osborn]

This doc: https://github.com/nginx/nginx-gateway-fabric/blob/main/site/content/how-to/data-plane-configuration.md needs to be updated to reflect the new API, but I decided to add this in a separate PR since this PR is already pretty large.

[codecov]

Codecov Report

Attention: Patch coverage is 95.52573% with 20 lines in your changes missing coverage. Please review.

Project coverage is 88.77%. Comparing base (6337c97) to head (b0d18b5).
Report is 9 commits behind head on change/control-data-plane-split.

Files with missing lines	Patch %	Lines
internal/mode/static/state/graph/nginxproxy.go	88.80%	12 Missing and 2 partials ⚠️
internal/mode/static/manager.go	53.84%	6 Missing ⚠️

Additional details and impacted files

@@                         Coverage Diff                         @@
##           change/control-data-plane-split    #3058      +/-   ##
===================================================================
- Coverage                            89.74%   88.77%   -0.98%     
===================================================================
  Files                                  109      111       +2     
  Lines                                11150    11495     +345     
  Branches                                50       50              
===================================================================
+ Hits                                 10007    10205     +198     
- Misses                                1083     1235     +152     
+ Partials                                60       55       -5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sjberman]

Looks like codecov is reporting a few misses in configuration.go that can probably be addressed.

[salonichf5]

lgtm except for compatibility doc mentioning the version change.

I know you are handling that in another PR?

[kate-osborn]

lgtm except for compatibility doc mentioning the version change.

I know you are handling that in another PR?

Good callout! I totally forgot about the compat doc. I'll update that as part of the docs PR

[bjee19]

🚀