Dump logs of API and Pipeline services

[JenySadadia]

On the staging instance, we are running Dozzle container for publishing API and Pipeline service logs.
i.e. https://staging.kernelci.org:9088/

One problem with it is whenever services restart, we lose all the previous logs and hence also lose a trail of failures as well.
It's not possible to catch all failures without log dump. Hence, enable a mechanism to dump logs to log files to debug the services at a later time.

[JenySadadia]

Some comments based on an initial investigation:

• We can tweak Logger configurations to dump logs to file instead of stdout here https://github.com/kernelci/kernelci-pipeline/blob/main/src/logger.py#L26
• The issue is Dozzle doesn't work well with log files and is compatible with stream outputs instead. So, we won't be able to see any logs on Dozzle if we redirect them to log file. We would still be able to see docker logs there. See https://dozzle.dev/guide/log-files-on-disk#following-log-files-on-disk
• However, there is a workaround to get logs from log file on Dozzle. See https://dozzle.dev/guide/log-files-on-disk#mounting-local-log-files-with-alpine. But I was thinking for creating a new log file for each container on every re-start and I am not sure that would work well with the log file mounting shown in the link I mentioned before.
• Another solution is to drop Dozzle and look for another alternative that works well with log files.

[nuclearcat]

Finally i found approach that helped me in past. In general terms it is described here: https://docs.python.org/3/howto/logging-cookbook.html

You can add multiple handlers with different log levels to Logging, so this way you can save in file verbose submission info, and leave for stdout (dozzle/kubernetes) generic info, simultaneously.