Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logfiles are always generated with a '-{timestamp}.log' suffix? #11061

Closed
henrikjohansen opened this issue Aug 18, 2021 · 5 comments · Fixed by #11070
Closed

Logfiles are always generated with a '-{timestamp}.log' suffix? #11061

henrikjohansen opened this issue Aug 18, 2021 · 5 comments · Fixed by #11070

Comments

@henrikjohansen
Copy link

According to the docs (here and here) it should be possible to configure Nomad to output agent logs and audit logs to a specific file. This seems to be slightly broken as I only ever see logs being generated with a -{timestamp}.log suffix ?

It does respect the file name (see the examples below) ... but it always adds the -{timestamp}.log suffix even though this should, according to the docs, only happen upon file rotation.

Nomad version is v1.1.2+ent.

Configuration of the ordinary agent logs in nomad.hcl 👇

# log settings
...
log_file  = "/var/log/nomad.log"
...

Configuration of the Enterprise audit logs in nomad.hcl 👇

sink "audit" {
    ...
    path   = "/var/log/nomad_audit.log"
    ...
}

The result of this config ...

root@xxx:/var/log# ls -alh /var/log/nomad*
-rw-r----- 1 root root 8.3K Aug 18 11:22 /var/log/nomad-1629283904922740456.log
-rw-r----- 1 root root 7.7K Aug 18 11:22 /var/log/nomad-1629285757963036138.log
-rw------- 1 root root 2.1M Aug 18 11:22 /var/log/nomad_audit-1629283907665719584.log
-rw------- 1 root root 1.1M Aug 18 11:42 /var/log/nomad_audit-1629285759194447584.log

Likewise, trying to change the file permissions for the audit log file (per these instructions) does not seem to work as the files are always created with 0600 - I am not sure if the 2 issues are related but I thought I might as well mention it here.

@lgfa29
Copy link
Contributor

lgfa29 commented Aug 19, 2021

Thanks @henrikjohansen. The timestamped files are part of the log rotation process, but if I understand correctly, what you would expect to see is that log_file would set the name of the current active log?

So in your example, you would want to see:

root@xxx:/var/log# ls -alh /var/log/nomad*
-rw-r----- 1 root root 8.3K Aug 18 11:22 /var/log/nomad.log
-rw-r----- 1 root root 7.7K Aug 18 11:22 /var/log/nomad-1629285757963036138.log
-rw------- 1 root root 2.1M Aug 18 11:22 /var/log/nomad_audit.log
-rw------- 1 root root 1.1M Aug 18 11:42 /var/log/nomad_audit-1629285759194447584.log

@henrikjohansen
Copy link
Author

Hi @lgfa29.

Correct - the audit docs at least give the impression that this should be the case:

By default Nomad will use its configured data_dir for a combined path of /data_dir/audit/audit.log. If rotate_bytes or rotate_duration are set file rotation will occur. In this case the filename will be post-fixed with a timestamp "filename-{timestamp}.log"

@shantanugadgil
Copy link
Contributor

possibly related to #9582 ?

@lgfa29
Copy link
Contributor

lgfa29 commented Aug 20, 2021

Thanks @shantanugadgil! That does seem like the same issue with regards to log_level.

@henrikjohansen for audit logs we use go-eventlogger, which is also used by other HashiCorp tools, so a change there could impact them as well. Would you mind opening an issue for the file name format there?

Also, for the file mode, this feature was recently added in 1.1.3+ent. Would you mind checking using that version instead?

Thanks!

schmichael added a commit to hashicorp/go-eventlogger that referenced this issue Aug 31, 2021
This patch updates the file_sink to *not* append the timestamp on the
current log file. The timestamp is only appended when the file is
rotated.

This matches the behavior:

- Implemented in Nomad v1.1.4 with hashicorp/nomad#11070
- Requested in hashicorp/nomad#11061
- Referenced in Consul's documentation:
  https://www.consul.io/docs/enterprise/audit-logging

> The following example configures a destination called "My Sink" which
> stores audit events at the file `/tmp/audit.json`.

Nomad's documentation is a little vague but implies the prior behavior
of always appending a timestamp.

While this is a *backward incompatible change* I think it's worth it for
usability and matching Consul's documentation. Nomad's documentation can
be fixed and a notice placed in the Upgrade Guide.
schmichael added a commit to hashicorp/go-eventlogger that referenced this issue Sep 16, 2021
Backward compatible version of #62 -- This version does *not* change the
default behavior. Users must opt in to new behavior.

This patch updates the file sink to allow *not* appending the timestamp
on the current log file. In this mode the timestamp is only appended
when the file is rotated.

This matches the behavior:

- Implemented in Nomad v1.1.4 with hashicorp/nomad#11070
- Requested in hashicorp/nomad#11061
- Referenced in Consul's documentation: https://www.consul.io/docs/enterprise/audit-logging

> The following example configures a destination called "My Sink" which
> stores audit events at the file `/tmp/audit.json`.

Nomad's documentation is a little vague but implies the prior behavior
of always appending a timestamp.

While this is a *backward incompatible change* I think it's worth it for
usability and matching Consul's documentation. Nomad's documentation can
be fixed and a notice placed in the Upgrade Guide.

In a bid to try to win your support I labeled this PR a bug despite the
tests and comments being very clear the old behavior (always append a
timestamp) *was* intentional.
schmichael added a commit to hashicorp/go-eventlogger that referenced this issue Sep 16, 2021
Backward compatible version of #62 -- This version does *not* change the
default behavior. Users must opt in to new behavior.

This patch updates the file sink to allow *not* appending the timestamp
on the current log file. In this mode the timestamp is only appended
when the file is rotated.

This matches the behavior:

- Implemented in Nomad v1.1.4 with hashicorp/nomad#11070
- Requested in hashicorp/nomad#11061

> The following example configures a destination called "My Sink" which
> stores audit events at the file `/tmp/audit.json`.

Nomad's documentation is a little vague but implies the prior behavior
of always appending a timestamp.
schmichael added a commit to hashicorp/go-eventlogger that referenced this issue Sep 17, 2021
* allow not appending timestamp to current log file

Backward compatible version of #62 -- This version does *not* change the
default behavior. Users must opt in to new behavior.

This patch updates the file sink to allow *not* appending the timestamp
on the current log file. In this mode the timestamp is only appended
when the file is rotated.

This matches the behavior:

- Implemented in Nomad v1.1.4 with hashicorp/nomad#11070
- Requested in hashicorp/nomad#11061

> The following example configures a destination called "My Sink" which
> stores audit events at the file `/tmp/audit.json`.

Nomad's documentation is a little vague but implies the prior behavior
of always appending a timestamp.

Also fixed one flaky test:

* time may be either the same of after

While Go compares time using a monotonic clock that only prevents time from
moving *backward,* it does *not* guarantee time will move forward
between subsequent calls to `time.Now()`. The After assertion failed in
a test run on Github because it is missing the `Equal` comparison on the
preceeding assertion.
@github-actions
Copy link

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants