NGINX Plus UBI Builds show `[error] 37#37: *9 open() "/etc/nginx/html/api"` in logs

[shaun-nx]

Overview

When accessing the NGINX Plus API endpoint, UBI base images for NGF will show this error log

[error] 37#37: *9 open() "/etc/nginx/html/api" failed (2: No such file or directory), client: unix:, server: , request: "GET /api HTTP/1.1", host: "nginx-plus-api"• [76.726 seconds]

As of NGF v2.1.2, this does not appear to be affecting any existing functionality.
All existing functional and conformance tests pass even with this error.

This issue is here to ensure the error is documented in the event we see a genuine issue related to this error in the future.

[shaun-nx]

Myself and @CVanF5 did some digging into this one on Friday.

Since any call to the NGINX Plus API is sent to Agent from NGF, we enabled Agent Debug logs to see if there was any kind of error or discrepancy between the behavior in UBI and Alpine.

Below are details on how this error was tested

UBI Images

For UBI based images, the select log lines below show the section in which the error is shown, as well as related Agent debug logs (note: the timestamp and level are removed from the start of the agent logs for ease of reading)

msg="Received management plane action request" server_type=command
msg="Resource plugin received api action request message" correlation_id=eebed631-ad6b-451f-8436-a74b59b7f0dc server_type=command
msg="Updating http upstream servers" request="http_upstream_name:\"default_tea_80\"  servers:{fields:{key:\"server\"  value:{string_value:\"10.244.0.8:8080\"}}}" correlation_id=eebed631-ad6b-451f-8436-a74b59b7f0dc server_type=command
2025/11/03 08:54:25 [error] 56#56: *264 open() "/etc/nginx/html/api" failed (2: No such file or directory), client: unix:, server: , request: "GET /api HTTP/1.1", host: "nginx-plus-api"
2025/11/03 08:54:25 [notice] 12#12: using the "epoll" event method
unix: - - [03/Nov/2025:08:54:25 +0000] "GET /api/9/http/upstreams/default_tea_80/servers HTTP/1.1" 200 136 "-" "Go-http-client/1.1" "-"
msg="Successfully updated http upstream servers" http_upstream_name=default_tea_80 add=0 update=0 delete=0 correlation_id=eebed631-ad6b-451f-8436-a74b59b7f0dc server_type=command
msg="Processing command" correlation_id=eebed631-ad6b-451f-8436-a74b59b7f0dc server_type=command
msg="Command plugin received data plane response message" correlation_id=eebed631-ad6b-451f-8436-a74b59b7f0dc server_type=command
msg="Sending data plane response message" message="Successfully updated HTTP Upstreams" status=COMMAND_STATUS_OK correlation_id=eebed631-ad6b-451f-8436-a74b59b7f0dc server_type=command
msg="Sending data plane response" response="message_meta:{message_id:\"b2c62597-b892-11f0-bbfc-e2a99340c7c0\"  correlation_id:\"eebed631-ad6b-451f-8436-a74b59b7f0dc\"  timestamp:{seconds:1762160065  nanos:255157924}}  command_response:{status:COMMAND_STATUS_OK  message:\"Successfully updated HTTP Upstreams\"}  instance_id:\"e8d1bda6-397e-3b98-a179-e500ff99fbc7\"" correlation_id=eebed631-ad6b-451f-8436-a74b59b7f0dc server_type=command

Lets break these down

From these logs lines, we can see that Agent first receives a request from NGF specifically to update an upstream in NGINX. It is here where we see. the error as NGINX attempts to access a file called api in /etc/nginx/html

msg="Received management plane action request" server_type=command
msg="Resource plugin received api action request message" correlation_id=eebed631-ad6b-451f-8436-a74b59b7f0dc server_type=command
msg="Updating http upstream servers" request="http_upstream_name:\"default_tea_80\"  servers:{fields:{key:\"server\"  value:{string_value:\"10.244.0.8:8080\"}}}" correlation_id=eebed631-ad6b-451f-8436-a74b59b7f0dc server_type=command
2025/11/03 08:54:25 [error] 56#56: *264 open() "/etc/nginx/html/api" failed (2: No such file or directory), client: unix:, server: , request: "GET /api HTTP/1.1", host: "nginx-plus-api"

However, we can see right after, that the upstreams were updated successfully

msg="Successfully updated http upstream servers" http_upstream_name=default_tea_80 add=0 update=0 delete=0 correlation_id=eebed631-ad6b-451f-8436-a74b59b7f0dc server_type=command

Lets bring back in the timestamps now to see the delay between when the request was sent, when the error occurred, and when the up streams were updated:

time=2025-11-03T08:54:25.254Z level=INFO source=command/command_plugin.go:306 msg="Received management plane action request" server_type=command
time=2025-11-03T08:54:25.254Z level=DEBUG source=resource/resource_plugin.go:141 msg="Resource plugin received api action request message" correlation_id=eebed631-ad6b-451f-8436-a74b59b7f0dc server_type=command
time=2025-11-03T08:54:25.254Z level=DEBUG source=resource/resource_plugin.go:196 msg="Updating http upstream servers" request="http_upstream_name:\"default_tea_80\"  servers:{fields:{key:\"server\"  value:{string_value:\"10.244.0.8:8080\"}}}" correlation_id=eebed631-ad6b-451f-8436-a74b59b7f0dc server_type=command
2025/11/03 08:54:25 [error] 56#56: *264 open() "/etc/nginx/html/api" failed (2: No such file or directory), client: unix:, server: , request: "GET /api HTTP/1.1", host: "nginx-plus-api"
2025/11/03 08:54:25 [notice] 12#12: using the "epoll" event method
unix: - - [03/Nov/2025:08:54:25 +0000] "GET /api/9/http/upstreams/default_tea_80/servers HTTP/1.1" 200 136 "-" "Go-http-client/1.1" "-"
time=2025-11-03T08:54:25.255Z level=DEBUG source=resource/nginx_plus_actions.go:116 msg="Successfully updated http upstream servers" http_upstream_name=default_tea_80 add=0 update=0 delete=0 correlation_id=eebed631-ad6b-451f-8436-a74b59b7f0dc server_type=command

We see the initial action request sent from the management plane was sent at 2025-11-03T08:54:25.254Z.
We then see the upstreams were successfully updated at 2025-11-03T08:54:25.255Z.

This is a time differences of 0.005ms

Alpine images

Here are the same set of log lines from an Alpine based image
Since we know what log lines we are looking at now, we'll keep the timestamp and log level here

time=2025-10-29T12:08:29.114Z level=DEBUG source=resource/resource_plugin.go:141 msg="Resource plugin received api action request message" correlation_id=89ba6e2f-b982-4320-9c1f-07a68ae3b654 server_type=command
time=2025-10-29T12:08:29.114Z level=DEBUG source=resource/resource_plugin.go:196 msg="Updating http upstream servers" request="http_upstream_name:\"default_tea_80\"  servers:{fields:{key:\"server\"  value:{string_value:\"10.244.0.8:8080\"}}}" correlation_id=89ba6e2f-b982-4320-9c1f-07a68ae3b654 server_type=command
time=2025-10-29T12:08:29.121Z level=DEBUG source=resource/nginx_plus_actions.go:116 msg="Successfully updated http upstream servers" http_upstream_name=default_tea_80 add=1 update=0 delete=0 correlation_id=89ba6e2f-b982-4320-9c1f-07a68ae3b654 server_type=command
2025/10/29 12:08:29 [info] 53#53: *5 client unix: closed keepalive connection
time=2025-10-29T12:08:29.121Z level=DEBUG source=command/command_plugin.go:108 msg="Processing command" correlation_id=89ba6e2f-b982-4320-9c1f-07a68ae3b654 server_type=command
time=2025-10-29T12:08:29.121Z level=DEBUG source=command/command_plugin.go:219 msg="Command plugin received data plane response message" correlation_id=89ba6e2f-b982-4320-9c1f-07a68ae3b654 server_type=command

Here we see the initial action request sent from the management plane was sent at 2025-10-29T12:08:29.114Z.
We then see the upstreams were successfully updated at 2025-10-29T12:08:29.121Z.

In this case, even with no error recorded, there was a time difference of 0.008ms
While marginal compared to our UBI times, we can be confident that we don't see any meaningful delays between when the API action request it send, and the update is made.

What this means

While there is still uncertainty as to why NGINX is attempting to make a call to /etc/nginx/html/api specifically in UBI based images, we can feel confident that it is not causing a disruption to performance and traffic.

To be sure, there will be a follow up post here with tracing data to see if we observe any meaningful traffic disruption.

[shaun-nx]

Traffic Disruption Test

To ensure this error does not impact user traffic, a test was ran that would send x amount of curl requests to an endpoint.
The script used to run this test would scale the coffee deployment up in increments of 5 after every 5 requests. This was done to trigger repeated updates to the upstream using the NGINX Plus API.

The deployment in tests was configured to with Tracing using our existing docs to configure tracing: https://docs.nginx.com/nginx-gateway-fabric/monitoring/tracing/

Below is the script used to query the /coffee endpoint x number of times.
In this test, we set 100 queries:

#!/usr/bin/env bash
set -euo pipefail

if [[ $# -lt 1 ]]; then
  echo "Usage: $0 <count>"
  exit 1
fi

COUNT="$1"
if ! [[ "$COUNT" =~ ^[0-9]+$ ]] || [[ "$COUNT" -le 0 ]]; then
  echo "Error: count must be a positive integer"
  exit 1
fi

HOST="${HOST:-cafe.example.com}"
PORT="${PORT:-8080}"
IP="${IP:-127.0.0.1}"
URI="${URI:-/coffee}"
DELAY="${DELAY:-0}"

SCALE_INCREMENT=5
REPLICAS=0

for ((i=1; i<=COUNT; i++)); do
  echo "[$i/$COUNT] curl --resolve ${HOST}:${PORT}:${IP} http://${HOST}:${PORT}${URI}"
  curl --fail --silent --show-error --resolve "${HOST}:${PORT}:${IP}" "http://${HOST}:${PORT}${URI}"

  if [[ "$DELAY" -gt 0 ]] && [[ "$i" -lt "$COUNT" ]]; then
    sleep "$DELAY"
  fi

  if (( i % SCALE_INCREMENT == 0 )); then
    REPLICAS=$((REPLICAS + SCALE_INCREMENT))
    echo "Scaling coffee deployment to ${REPLICAS} replicas"
    kubectl scale deployment coffee --replicas="${REPLICAS}"
  fi
done

echo "Resetting coffee deployment to 1 replica"
kubectl scale deployment coffee --replicas=1

UBI Results

Attached here are the JSON tracing results when querying the /coffee deployment against a deployment of NGF based off of UBI
ubi-traces.json

The jq query command below can be used to inspect the file for any non-200 http status codes.
This will output No non-200 http.status_code entries if this is true.

jq -r '([.data[].spans[].tags[] | select(.key=="http.status_code" and .value!=200)] | length) as $n | if $n == 0 then "No non-200 http.status_code entries" else "Found ($n) non-200 entries" end' ubi-traces.json
No non-200 http.status_code entries

What this means

While it would still be helpful to know why exactly NGINX is making a call to /etc/nginx/html/api specifically in UBI based imaged, we can feel confident that this error will not cause a disruption in traffic for our users based on these results