NGINX Controller integration from container logs

[motizuki]

Hey team,

I've been trying to set up the 'Nginx Ingress Controller Logs' Fleet integration, but I've found a few issues that I'd like some advice on.

We are using the nginx-ingress helm chart to run our controllers, this chart is configured to send the access logs to stdout and error logs to stderr, which will be logged in the container logs.

The container log entries have the following format:

{"log":"94.139.46.156 - - [06/Oct/2022:03:36:42 +0000] \"GET /files/legaldeposit/blablabla.pdf HTTP/1.1\" 200 475146 \"-\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.74 Safari/537.36 Edg/79.0.309.43\" 785 0.096 [-] [] 3.5.166.123:443 475146 0.076 200 da1250a3e808288fa9e6bda34df7fba5\n","stream":"stdout","time":"2022-10-06T03:36:42.501610815Z"}

I've tried to set up the 'Nginx Ingress Controller Logs' integration to the container log file /var/log/containers/ingress-nginx-alb*.log however, the file is being ignored because it's a symlink, and I couldn't find a way to enable symlink via the integration configuration.

I've managed to work around the symlink issue, but then, the ingest pipeline doesn't work quite well with the log entries from the container logs, as it expect the log file to be formatted as the access.log

Does anyone have any advice on this?

Thank you!

[vnirocha]

Hey @motizuki,

Let me please add one more question to yours. Dear elastic team, what is the purpose of having two nginx integrations (nginx & nginx ingress controller) if both of them tails access.log and error.log?

In my scenario here, my team deployed a rke2 cluster having nginx 1.2.1 deployed as the default ingress controller. I'm having the same issue as @motizuki... access.log and error.log are being directed to stdout of container and its not possible to read its logs.

Appreciate your attention.

Thanks

[vnirocha]

@motizuki

Worked well for me using "access log" path as /var/log/pods/kube-system_rke2-ingress-nginx-controller*/rke2-ingress-nginx-controller/*.log. Access logs and dashboards seems all ok.

I'm using kibana 8.4.3 with ingress controller integration version v1.2.0.

Unfortunately I coundnt make error logs work, as error and access are being redirected to the same output (stdout of container)

Hope it helps

[motizuki]

This has been fix with this #4855
Thanks @gsantoro!