Enable log file support

[JenySadadia]

Closes kernelci/kernelci-project#334

Enable multiple log streams for Logger.

Create a file handler to dump logs to files. Use stream and file handlers for all the pipeline services to redirect logs to both the streams including stdout and log file.
Update the logger config file.

[JenySadadia]

Staging seems working and creating log files for pipeline services:

kernelci@kernelci-staging:~/kernelci-deploy/checkout/kernelci-pipeline$ ls -lrt logs
total 68
-rw-r--r-- 1 kernelci kernelci  159 Jul  3 13:04 monitor_2024_07_03-13:03:55.log
-rw-r--r-- 1 kernelci kernelci  104 Jul  3 13:08 result_summary_2024_07_03-13:08:05.log
-rw-r--r-- 1 kernelci kernelci  164 Jul  3 13:10 test_report_2024_07_03-13:08:10.log
-rw-r--r-- 1 kernelci kernelci  159 Jul  3 13:10 monitor_2024_07_03-13:08:08.log
-rw-r--r-- 1 kernelci kernelci  642 Jul  3 13:10 timeout-closing_2024_07_03-13:08:10.log
-rw-r--r-- 1 kernelci kernelci  159 Jul  3 13:10 regression_tracker_2024_07_03-13:08:09.log
-rw-r--r-- 1 kernelci kernelci  175 Jul  3 13:10 timeout-holdoff_2024_07_03-13:08:10.log
-rw-r--r-- 1 kernelci kernelci  174 Jul  3 13:10 scheduler_2024_07_03-13:08:05.log
-rw-r--r-- 1 root     root      318 Jul  3 13:10 scheduler_2024_07_03-13:08:08.log
-rw-r--r-- 1 kernelci kernelci 8926 Jul  3 13:10 trigger_2024_07_03-13:08:10.log
-rw-r--r-- 1 kernelci kernelci 9832 Jul  3 13:10 timeout_2024_07_03-13:08:06.log
-rw-r--r-- 1 kernelci kernelci 1962 Jul  3 13:10 scheduler_2024_07_03-13:08:10.log
-rw-r--r-- 1 kernelci kernelci 1410 Jul  3 13:10 tarball_2024_07_03-13:08:10.log

@nuclearcat Let me know if the file naming convention needs to be tweaked.

[JenySadadia]

@nuclearcat We also need to add some kind of purge service that will discard the log files older than some decided number of days. Otherwise, It will create storage issues on VM.

[pawiecz]

I must be missing something obvious - please help me understand:

Currently config/logger.conf handles 11 services even though there are 16 overall [1]. Are the potentially missing ones (lava-callback, patchset, scheduler-{docker,k8s,lava}) already covered implicitly somehow?

I also don't get the convention for passing service name as an argument - it's used like that for schedulers, other services have it set by their constructor parameter or even hardcoded directly in the constructor (e.g. timeout-s: 1, 2, 3).

Any additional pointers will be much appreciated.

[1] Compose files in use for staging deployment converted to a single configuration file for inspection:

$ docker-compose -f docker-compose.yaml -f docker-compose-kcidb.yaml -f docker-compose-lava.yaml -f docker-compose-production.yaml config > staging.yaml

[JenySadadia]

Hey,

Currently config/logger.conf handles 11 services even though there are 16 overall [1]. Are the potentially missing ones (lava-callback, patchset, scheduler-{docker,k8s,lava}) already covered implicitly somehow?

Thanks for pointing this out. Whenever a logger section is not found with the particular name, it simply uses root configs.
Bdw I have added missing logger sections to the config file. For lava-callback, we need to integrate Logger module first.

I also don't get the convention for passing service name as an argument - it's used like that for schedulers, other services have it set by their constructor parameter or even hardcoded directly in the constructor (e.g. timeout-s: 1, 2, 3).

For timeout services, we have different class for all 3 of them. And that's why we can use static service name for them.
In the case of scheduler, it just has a single class implementation. So it will not be possible to identify which service (docker, lava, or k8s) invoked the class. That's why I introduced --name argument in docker-compose to use it to pass the service name.

[pawiecz]

LGTM

[nuclearcat]

I was going to ask, is it going to work in kubernetes? as likely logs directory wont be created yet (it needs some changes in pipeline yaml specifications), as i planned production update today

[JenySadadia]

I was going to ask, is it going to work in kubernetes? as likely logs directory wont be created yet (it needs some changes in pipeline yaml specifications), as i planned production update today

Oh, sorry. I already merged it.
I will check k8s config files and get back to you.

[nuclearcat]

Just make sure fallback scenario - if directory doesn't exist - it should not crash service, and in such case it just wont log to file, only to console, for example, so i can plan log volume properly during this week.