Duplicated agent logs and not cleaning up correctly

[mudler]

Kairos version:
3.3.1, but also olders

CPU architecture, OS, and Version:

Describe the bug
Seems we are logging both to /var/log/kairos and /run/kairos. The problem is that while /var/log/kairos logs are rotated, /run isn't rotated and supposedly populated only with sentinel or env config file. Additionally, our current policy seems to be very relaxed and keeps too many files around. The default logrotate policy has rotate till 3 instead of delete using maxage. This is an host almost 2mo old:

To Reproduce
Boot a kairos system

Expected behavior
To have logs in /run/kairos only if /var/log/kairos is not available or immutable

Logs

root@edge-d01b3842ae8bd3e4138e6cc9273671c2:/var/log# cat /run/kairos/agent-20250217103734.1535.log 
2025-02-17T10:37:34Z INF Kairos Agent version=v2.13.9
2025-02-17T10:37:34Z INF Kairos System version=v3.1.3
2025-02-17T10:37:34Z INF creating a runtime
2025-02-17T10:37:34Z INF detecting boot state
2025-02-17T10:37:34Z INF Boot Mode boot_mode=active_boot
2025-02-17T10:37:34Z INF Boot in uki mode result=false
root@edge-d01b3842ae8bd3e4138e6cc9273671c2:/var/log# cat /run/kairos/agent-20250217124237.3715.log
2025-02-17T12:42:37Z INF Kairos Agent version=v2.13.9
2025-02-17T12:42:37Z INF Kairos System version=v3.1.3
2025-02-17T12:42:37Z INF creating a runtime
2025-02-17T12:42:37Z INF detecting boot state
2025-02-17T12:42:37Z INF Boot Mode boot_mode=active_boot
2025-02-17T12:42:37Z INF Boot in uki mode result=false
2025-02-17T12:42:37Z INF Running stage: reconcile.before

2025-02-17T12:42:37Z INF Done executing stage 'reconcile.before'

2025-02-17T12:42:37Z INF Running stage: reconcile

2025-02-17T12:42:37Z INF Done executing stage 'reconcile'

2025-02-17T12:42:37Z INF Running stage: reconcile.after

2025-02-17T12:42:37Z INF Done executing stage 'reconcile.after'

2025-02-17T12:42:37Z INF Running stage: reconcile.before

2025-02-17T12:42:37Z INF Done executing stage 'reconcile.before'

2025-02-17T12:42:37Z INF Running stage: reconcile

2025-02-17T12:42:37Z INF Done executing stage 'reconcile'

2025-02-17T12:42:37Z INF Running stage: reconcile.after

2025-02-17T12:42:37Z INF Done executing stage 'reconcile.after'

Additional context

[mudler]

See also related issue: #3207

[jimmykarily]

Fixed here: kairos-io/kairos-sdk#566

[Itxaka]

will now log to /var/log/kairos to a single file, appending, thread safe.

If journald is available it will do to jorunal instead.

If logging to file, it will prepend the PID to the log line so its easier to track