Support log rotation

[mintiayields]

Support traditional Linux log rotation. To make this work we need to be able to configure neard to save logs to a file and then have a mechanism to tell it to close the log file descriptor and reopen it.

The way this traditionally works on Linux servers is:

1. Daemon saves logs to ‘log-file’.
2. logrotate renames ‘log-file’ to ‘log-file.1’.
3. logrotate sends SIGHUP to the daemon.
4. Daemon closes its log file descriptor and opens ‘log-file’ again.
5. logrotate can do whatever it needs to with ‘log-file.1’.

PR #6666 added handler for SIGHUP which reconfigures log levels and we might be able to leverage that work. What remains is:

• Add option to configure log file (e.g. --log-output=<file> flag though now that we have log_config.json and SIGHUP handler which reads it adding output file there might be best option).
• Add code to existing SIGHUP handler which does the rotation.

I (@mina86) think the hardest part of this is figuring out tracing crate. We need to tell it to save to a log file and then also we need to be able to tell it to reopen the file. But this opinion comes from ignorance of tracing crate so perhaps it’s actually trivial to do.

— >8 — original report by @mintiayields below

When logs are redirected in the service file like, for instance: StandardOutput=append:/home//.near/neard.log StandardError=inherit

at this point nearcore keeps the file open for writing all the time and will not allow any log rotation. It will not even allow to truncate it to a certain size. I played around with bash scripts in an attempt to truncate the contents or rotate the log. The results are as follows:

1. Erasing the log file completely will/may cause a software error to nearcore and result in a break in validation; restart needed.
2. Stopping the daemon, rotating/truncating log, starting it back up will cause a lapse in validation. If you are lucky you don't have to catch up much and do not miss blocks, but most of the time you will. I got proposals for the case:
3. Shall the developers make a method in neard binary itself to rotate logs? I see that nearup somehow rotates its logs, can't figure out how.
4. Will it be possible to release the log file for native linux logrotate daemon to do its job?
5. Will the developers offer a way to rotate logs without any code mods needed?

[bowenwang1996]

I don't think log rotation will be implemented in nearcore itself. It should be done on the devops side. @mhalambek can provide more details.

[mintiayields]

Should I address @mhalambek directly or wait for instructions here? Here would be better as this is a publicly available thread.

[pmnoxx]

@mintiayields @bowenwang1996 I think we should address the performance of the logger issue first. Current logger is too slow by a factor of 30, and that's affecting our service. See #6072

My plan is to change, the way we use logging, and then adding features like log rotation would be easy to do.

[stale]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months.
It will be closed in 7 days if no further activity occurs.
Thank you for your contributions.

[mina86]

I am actually surprised that append:... doesn’t work. With that it should be possible to copy and truncate the file (i.e. logrotate’s copytruncate behaviour). Another solution is to leave out StandardOutput and StandardError and let the logs end up in systemd journal which can be rotated regardless of nerad.

Ideally neard would handle SIGHUP and reopen the log file but that is not currently supported.

Erasing the log file completely will/may cause a software error to nearcore and result in a break in validation; restart needed.

This will only remove the dentry. The fill will remain on the filesystem but it will be inaccessible. So as neard runs the log will accumulate in size.

I see that nearup somehow rotates its logs, can't figure out how.

That’s not quite true. nearup rotates logs but only on startup. It does not support rotating while the daemon is running.

[nagisa]

tracing_appender has a writer that supports rotating on a fixed schedule with RollingFileAppender. This is not quite what is described here, but if we insist on SIGHUP approach, we'd probably just copy over the implementation and make the modifications we need (possibly publishing our own crate and/or submitting our suggested changes upstream)

[mina86]

Handling SIGHUP would have an advantage that one could configure logrotate to rotate the logs based on size rather than schedule. Also, if we do rotation inside neard than we need to expose the schedule option to the user and if we just handle SIGHUP than configuring the schedule goes into logrotate configuration and we don’t need to care about it.