Build and use custom nginx container

[sjberman]

Is your enhancement request related to a problem? Please describe.
Right now our deployment simply uses the publicly available nginx image. While convenient, there are some downsides of this.

• Users have to create a ConfigMap containing our custom http matching module, which gets mounted as a volume into the nginx container. This pattern would be repeated for any new modules that we would add. If the ConfigMap is forgotten, the Pod fails to start.
• We have to run an init container to set the proper permissions on the /etc/nginx directory so that it can be written to.

What would you like to be added:
The remedy for this is to package and distribute our own data plane container image. We would use the public nginx image as the base, and build on top of that.

Why this is needed:
This solves the issues mentioned under the problem above.

Additional context

• Custom modules are packaged directly into the image, removing the need for ConfigMaps and volume mounts
• Directory permissions are defined in the image itself, removing the need for an init container

See discussion #754 for more context.

A/C:

• build custom data plane image, using public nginx image as the base (can we use alpine nginx to reduce size?)
• image should package modules and set directory permissions
• image should be built and distributed in the pipeline the same way that the control plane image is handled
• remove init container and unnecessary volume mounts
• The custom nginx container does NOT run as root
• Ensure unix capabilities are the minimum required
• update documentation
  • remove references to modules ConfigMap
  • update any docs that reference the data plane image if necessary
  • update the architecture doc