Change logrotate compress to delaycompress to prevent fluentd log tailing from getting stuck

[r4j4h]

Fluent log tailing fails sporadically, seemingly due to logrotate setings.

Source of the setting: fluent/fluentd#780 (comment)

If this the wrong way and there are other ideas on the cause of this issue they are more than welcome! We can change this PR accordingly.

---

This change is 

[k8s-ci-robot]

Hi @r4j4h. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[chrislovecnm]

@k8s-bot ok to test

[justinsb]

/lgtm

Thanks! This makes intuitive sense to me.

Was there an error message @r4j4h or did it just stop silently?

[r4j4h]

No error messages at all, in fact I was only made aware by a team's integration test that verified a given transaction id appears in all app's logs. After a weekend or so only one pod would strangely go silent. I tracked it down by echo'ing directly into the log and knew there was an issue when sending invalid lines didn't even cause a formatting error in the log.

Initially we found rebooting fluentd did not help but restarting the pod in question did, but a patch by okkez seemed to fix that for a while. Then it happened again, but this time with fluentd restarts being enough.

I am happy to report that we haven't seen the issue since I applied the change to each node. I will report back here if it happens again.

Thanks for the quick turn around!! :)

[r4j4h]

To follow up, this has not completely solved the problem for us. It largely has, as it does not seem as frequent, but we occasionally do still find some tail processes getting stuck. :(

[nsidhaye]

If you are using copytruncate then it should not hamper in compress. Am I right?

Ref: https://manpages.debian.org/jessie/logrotate/logrotate.8.en.html