feat: remove pagespeed module from our nginx container (#1150)

* refactor: use standard nginx container
* introduce shellspec testing
* add shellspec tests job

BREAKING CHANGES: The [pagespeed module](https://www.modpagespeed.com) of NGINX has been removed without replacement.

.github/workflows/integration.yml

@@ -161,3 +161,14 @@ jobs:
         with:
           name: videos
           path: e2e/cypress/videos
+
+  Shellspec:
+    needs: [CancelPrevious]
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+      - name: Execute shellspec tests
+        run: |
+          docker run --rm -v "${{ github.workspace }}/nginx/docker-entrypoint.d:/src" shellspec/shellspec

.prettierignore

@@ -44,6 +44,7 @@ node_modules
 nginx.conf
 /nginx/**/*.conf
 /nginx/**/*.tmpl
+/nginx/docker-entrypoint.d/.shellspec
 /.idea/
 /charts/**/templates/*
 *~

docker-compose.yml

@@ -48,8 +48,8 @@ services:
     environment:
       UPSTREAM_PWA: 'http://pwa:4200'
       # DEBUG: 1
+      NGINX_ENTRYPOINT_QUIET_LOGS: ANYVALUE
       CACHE: 0
-      PAGESPEED: 0
       SSR: 1
       # PROMETHEUS: 1
       # COMPRESSION: 0

docs/concepts/pwa-building-blocks.md

@@ -34,7 +34,6 @@ Each of these is separately configurable (see [Building and Running nginx Docker
 Nginx enables the following features to be used in an Intershop PWA deployment:
 
 - Uncomplicated caching of PWA server-side rendering responses provided by the upstream Angular Universal server.
-- Integration of the [PageSpeed Module](https://www.modpagespeed.com/) for access to different browser optimizations.
 - Handling of multiple channels via URL parameters in conjunction with SSR (see [Multi-Site Handling](multi-site-handling.md)).
 - Customizable compression for downstream services
 - Device type detection to ensure a correct pre-render, adapted to the incoming user agent.

docs/guides/migrations.md

@@ -15,6 +15,8 @@ All uses of the `(deferLoad)` directive in custom code need to be replaced.
 We removed the unmaintained `angular2-uuid` library in favor of the standard `uuid` library that is already included as an Angular dependency.
 In order to match our changes, replace all occurrences of `angular2-uuid` in your custom code (see #1203).
 
+The [pagespeed module](https://www.modpagespeed.com) of NGINX has been removed without replacement.
+
 ## 2.3 to 2.4
 
 The PWA 2.4 contains an Angular update to version 13.3.10 and many other dependencies updates.

docs/guides/nginx-startup.md

@@ -7,7 +7,7 @@ kb_sync_latest_only
 
 # Building and Running NGINX Docker Image
 
-We provide a docker image based on [nginx](https://www.nginx.com/) for the [PWA deployment](../concepts/pwa-building-blocks.md#pwa---nginx).
+We provide a docker image based on [nginx](https://nginx.org/) for the [PWA deployment](../concepts/pwa-building-blocks.md#pwa---nginx).
 
 ## Building
 
@@ -23,6 +23,10 @@ For HTTP, the server will run on default port 80.
 If HTTPS is chosen as an upstream, it will run on default port 443.
 In the latter case the files `server.key` and `server.crt` have to be supplied in the container folder `/etx/nginx` (either by volume mapping with `docker run` or in the image itself by `docker build`).
 
+We're using the standard NGinx Docker image.
+Therefore we inherit all their configuration capabilities.
+For further information please refer to [the official NGinx Docker image page](https://hub.docker.com/_/nginx?tab=description)
+
 ### Basic Auth
 
 For deploying to test environments that should not be indexed by search bots or should not be accessible by the public, the nginx container can be set up with basic authentication.
@@ -117,15 +121,10 @@ If no environment variable is set, this feature is disabled.
 
 ### Other
 
-The page speed configuration can also be overridden:
-
-- Set the environment variable `NPSC_ENABLE_FILTERS` to a comma-separated list of active [Page Speed Filters](https://www.modpagespeed.com/examples/) to override our carefully chosen defaults. Do this at your own risk.
-
 Built-in features can be enabled and disabled:
 
 - `SSR=off` effectively disables SSR rendering for browsers (default `on`)
 - `CACHE=off` disables caching (default `on`)
-- `PAGESPEED=off` disables pagespeed optimizations (default `on`)
 - `COMPRESSION=off` disables compression (default `on`)
 - `DEVICE_DETECTION=off` disables user-agent detection (default `on`)
 - `PROMETHEUS=on` enables [Prometheus](https://prometheus.io) metrics exports on port `9113` (default `off`)
@@ -140,7 +139,6 @@ The feature name must only contain word characters (letters, numbers and undersc
 ### Cache
 
 If the cache feature is switched off, all caching for pre-rendered pages is disabled.
-If the cache should also be disabled for static resources, the page speed feature has to be switched off as well as it caches optimized images individually.
 
 The cache duration for pre-rendered pages can be customized using `CACHE_DURATION_NGINX_OK` (for successful responses) and `CACHE_DURATION_NGINX_NF` (for 404 responses).
 The value supplied must be in the `time` format that is supported by [nginx proxy_cache_valid](http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_valid)
@@ -152,3 +150,4 @@ The value supplied must be in the `time` format that is supported by [nginx prox
 - [Concept - Logging](../concepts/logging.md)
 - [Concept - Single Sign-On (SSO) for PWA](../concepts/sso.md)
 - [Guide - Monitoring with Prometheus](./prometheus-monitoring.md)
+- [README of official NGinx Docker image](https://hub.docker.com/_/nginx?tab=description)

nginx/Dockerfile

@@ -1,41 +1,21 @@
-FROM ubuntu:focal as buildstep
-RUN apt-get update
-RUN apt-get install -y curl build-essential zlib1g-dev libpcre3-dev unzip wget uuid-dev sudo openssl libssl-dev
-RUN curl -kfL -sS https://ngxpagespeed.com/install > install.sh
-RUN bash install.sh --ngx-pagespeed-version v1.13.35.2-stable --nginx-version 1.21.6 --additional-nginx-configure-arguments '--with-http_ssl_module --with-http_stub_status_module'
-
-FROM scratch as configstep
-COPY --from=nginx:mainline /etc/nginx /etc/nginx
-COPY nginx.conf /etc/nginx/
+FROM nginx:1.20
+RUN apt-get update \
+  && apt-get install --no-install-recommends --no-install-suggests -y apache2-utils
+COPY nginx.conf /etc/nginx/nginx.conf
 COPY features /etc/nginx/features/
 COPY templates /etc/nginx/templates/
-COPY entrypoint.sh *.yaml /
+COPY docker-entrypoint.d/*.sh /docker-entrypoint.d/
+COPY *.yaml /
 COPY 50x.html /usr/share/nginx/html/
 ADD https://github.com/hairyhenderson/gomplate/releases/download/v3.8.0/gomplate_linux-amd64-slim /gomplate
-
-FROM ubuntu:focal
-RUN apt-get update && \
-  apt-get install -y gettext-base libssl1.1 apache2-utils && \
-  apt-get -y autoremove && \
-  apt-get clean && \
-  rm -r /var/cache/apt /var/lib/apt/lists
-COPY --from=buildstep /usr/local/nginx /usr/local/nginx
-COPY --from=nginx/nginx-prometheus-exporter:0.9.0 /usr/bin/nginx-prometheus-exporter /nginx-prometheus-exporter
-COPY --from=configstep / /
-RUN chmod 777 /gomplate
-ENV NPSC_ENABLE_FILTERS=in_place_optimize_for_browser,prioritize_critical_css,inline_preview_images,lazyload_images,rewrite_images,rewrite_css,remove_comments,move_css_to_head,move_css_above_scripts,collapse_whitespace,combine_javascript,extend_cache
-ENV NPSC_JsPreserveURLs=off
-ENV NPSC_ImagePreserveURLs=on
-ENV NPSC_ForceCaching=off
+RUN chmod 700 /gomplate
+COPY --from=nginx/nginx-prometheus-exporter:0.10.0 /usr/bin/nginx-prometheus-exporter /nginx-prometheus-exporter
 ENV CACHE=on
 ENV COMPRESSION=on
-ENV PAGESPEED=on
 ENV DEVICE_DETECTION=on
 ENV SSR=on
 ENV CACHE_DURATION_NGINX_OK=10m
 ENV CACHE_DURATION_NGINX_NF=60m
 ENV LOGFORMAT=main
 
 EXPOSE 80 443 9113
-
-ENTRYPOINT [ "sh", "entrypoint.sh" ]

nginx/docker-entrypoint.d/.shellspec

@@ -0,0 +1,12 @@
+--require spec_helper
+
+## Default kcov (coverage) options
+# --kcov-options "--include-path=. --path-strip-level=1"
+# --kcov-options "--include-pattern=.sh"
+# --kcov-options "--exclude-pattern=/.shellspec,/spec/,/coverage/,/report/"
+
+## Example: Include script "myprog" with no extension
+# --kcov-options "--include-pattern=.sh,myprog"
+
+## Example: Only specified files/directories
+# --kcov-options "--include-pattern=myprog,/lib/"

nginx/docker-entrypoint.d/00-envcheck.sh

@@ -0,0 +1,24 @@
+#!/bin/sh
+
+set -e
+
+if [ -f "/etc/nginx/conf.d/default.conf" ]
+then
+  rm /etc/nginx/conf.d/default.conf
+fi
+
+if [ -z "$UPSTREAM_PWA" ]
+then
+  echo "UPSTREAM_PWA is not set"
+  exit 1
+fi
+
+if [ -z "$ICM_BASE_URL" ]
+then
+  echo "ICM_BASE_URL is not set. Cannot use sitemap proxy feature."
+fi
+
+if [ -z "$OVERRIDE_IDENTITY_PROVIDERS" ]
+then
+  echo "OVERRIDE_IDENTITY_PROVIDERS is not set. Cannot use override identity provider feature."
+fi

nginx/docker-entrypoint.d/40-features.sh

@@ -0,0 +1,5 @@
+#!/bin/sh
+
+set -e
+
+find /etc/nginx/features/*.conf -print0 | xargs -0 -I{} echo {} | sed -e "s%.*\/\(\w*\).conf%\1%" | grep -E '^\w+$' | while read feature; do echo "# $feature" ; env | grep -iqE "^$feature=(on|1|true|yes)$" && echo "include /etc/nginx/features/${feature}.conf;" || echo "include /etc/nginx/features/${feature}-off[.]conf;" ; done >/etc/nginx/conf.d/features.conf

nginx/docker-entrypoint.d/40-gomplate.sh

@@ -0,0 +1,32 @@
+#!/bin/sh
+
+set -e
+if [ -z "$MULTI_CHANNEL_SOURCE" ]
+then
+  if [ -z "$MULTI_CHANNEL" ]
+  then
+    MULTI_CHANNEL_SOURCE="./multi-channel.yaml"
+  else
+    MULTI_CHANNEL_SOURCE="env:///MULTI_CHANNEL?type=application/yaml"
+  fi
+fi
+
+if [ -z "$OVERRIDE_IDENTITY_PROVIDERS_SOURCE" ]
+then
+  if [ -n "$OVERRIDE_IDENTITY_PROVIDERS" ]
+  then
+    OVERRIDE_IDENTITY_PROVIDERS_SOURCE="env:///OVERRIDE_IDENTITY_PROVIDERS?type=application/yaml"
+  fi
+fi
+
+if [ -z "$CACHING_IGNORE_PARAMS_SOURCE" ]
+then
+  if [ -z "$CACHING_IGNORE_PARAMS" ]
+  then
+    CACHING_IGNORE_PARAMS_SOURCE="./caching-ignore-params.yaml"
+  else
+    CACHING_IGNORE_PARAMS_SOURCE="env:///CACHING_IGNORE_PARAMS?type=application/yaml"
+  fi
+fi
+
+/gomplate -d "domains=$MULTI_CHANNEL_SOURCE" -d "overrideIdentityProviders=$OVERRIDE_IDENTITY_PROVIDERS_SOURCE" -d "cachingIgnoreParams=$CACHING_IGNORE_PARAMS_SOURCE" -d 'ipwhitelist=env:///BASIC_AUTH_IP_WHITELIST?type=application/yaml' --input-dir="/etc/nginx/templates" --output-map='/etc/nginx/conf.d/{{ .in | strings.ReplaceAll ".conf.tmpl" ".conf" }}'

nginx/docker-entrypoint.d/entrypoint.sh

@@ -0,0 +1,20 @@
+#!/bin/sh
+
+set -e
+# set -x
+
+if [ -n "$BASIC_AUTH" ]
+then
+  htpasswd -bc /etc/nginx/.htpasswd $(echo "$BASIC_AUTH" | sed 's/:/ /')
+fi
+
+if env | grep -iqE "^DEBUG=(on|1|true|yes)$"
+then
+  find /etc/nginx -name '*.conf' -print -exec cat '{}' \;
+  nginx -V
+fi
+
+if env | grep -iqE "^PROMETHEUS=(on|1|true|yes)$"
+then
+  (sleep 5 && /nginx-prometheus-exporter)&
+fi

nginx/docker-entrypoint.d/spec/00-envcheck_spec.sh

@@ -0,0 +1,58 @@
+Describe "envcheck behavior"
+  Describe 'Mandatory parameters'
+    It 'Should exit with error if value is missing'
+      When run script 00-envcheck.sh
+      The status should be failure
+      The output should eq 'UPSTREAM_PWA is not set'
+
+    End
+
+    It 'Should not exit with error if value is provided'
+      BeforeRun "export UPSTREAM_PWA=sdf"
+      When run script 00-envcheck.sh
+      The status should be success
+      The output should not eq 'UPSTREAM_PWA is not set'
+    End
+  End
+
+  Describe "OVERRIDE_IDENTITY_PROVIDERS"
+    It 'Should issue a warning message if value is missing'
+      BeforeRun "export UPSTREAM_PWA=a ICM_BASE_URL=b"
+      When run script 00-envcheck.sh
+      The status should be success
+      The output should eq 'OVERRIDE_IDENTITY_PROVIDERS is not set. Cannot use override identity provider feature.'
+    End
+
+    It 'Should continue with success without messages if value is set'
+      BeforeRun "export UPSTREAM_PWA=a ICM_BASE_URL=b OVERRIDE_IDENTITY_PROVIDERS=c"
+      When run script 00-envcheck.sh
+      The status should be success
+      The output should not eq 'OVERRIDE_IDENTITY_PROVIDERS is not set. Cannot use override identity provider feature.'
+    End
+  End
+
+  Describe "ICM_BASE_URL"
+    It 'Should issue a warning message if value is missing'
+      BeforeRun "export UPSTREAM_PWA=a OVERRIDE_IDENTITY_PROVIDERS=b"
+      When run script 00-envcheck.sh
+      The status should be success
+      The output should eq 'ICM_BASE_URL is not set. Cannot use sitemap proxy feature.'
+    End
+
+    It 'Should continue with success without messages if value is set'
+      BeforeRun "export UPSTREAM_PWA=a ICM_BASE_URL=b OVERRIDE_IDENTITY_PROVIDERS=c"
+      When run script 00-envcheck.sh
+      The status should be success
+      The output should not eq 'OVERRIDE_IDENTITY_PROVIDERS is not set. Cannot use override identity provider feature.'
+    End
+  End
+
+  Describe "Default NGinx Config File"
+    It 'Should be removed if existing to prevent defaults being applied'
+      BeforeRun "mkdir -p /etc/nginx/conf.d && touch /etc/nginx/conf.d/default.conf && export UPSTREAM_PWA=a ICM_BASE_URL=b OVERRIDE_IDENTITY_PROVIDERS=c"
+      When run script 00-envcheck.sh
+      The status should be success
+      The path '/etc/nginx/conf.d/default.conf' should not be exist
+    End
+  End
+End

nginx/docker-entrypoint.d/spec/spec_helper.sh

@@ -0,0 +1,24 @@
+# shellcheck shell=sh
+
+# Defining variables and functions here will affect all specfiles.
+# Change shell options inside a function may cause different behavior,
+# so it is better to set them here.
+# set -eu
+
+# This callback function will be invoked only once before loading specfiles.
+spec_helper_precheck() {
+  # Available functions: info, warn, error, abort, setenv, unsetenv
+  # Available variables: VERSION, SHELL_TYPE, SHELL_VERSION
+  : minimum_version "0.28.1"
+}
+
+# This callback function will be invoked after a specfile has been loaded.
+spec_helper_loaded() {
+  :
+}
+
+# This callback function will be invoked after core modules has been loaded.
+spec_helper_configure() {
+  # Available functions: import, before_each, after_each, before_all, after_all
+  : import 'support/custom_matcher'
+}