Logs to stdout or stderr stops after a few minutes #8955
Comments
|
That is interesting. Thank you for the detailed description. We (Yahoo) have properties that run with stderr logging and we haven't had anyone report seeing the logging stall. We should be able to get this to work, therefore. Since this happens so quickly, I suggest enabling the Note: Use |
|
Thanks for the prompt reply. First, I set debug.enabled to 1 in part because this is a dev server so the perf impact is irrelevant to me, but mostly because with 3 I wasn't seeing debug logs being printed at all 😅 However after a few tests I was surprisingly not always able to reproduce the issue. Looking at the git history (as I tried quite a few things yesterday), this is the piece of records.config that causes the issue: When both lines are removed, logs work fine (at least for a while longer; it's not impossible the issue just takes longer to show in such case; currently started without both and waiting to find that out). Commenting out Attached is either way the full diags.log when both are present (and I set diags to log to a file for the occasion, ofc) The last successful stderr output line was at 17:15:42, and the relevant (I suspect) bit is unfortunately not very informative: is about the first error information in there however I don't see anything else that'd be particularly obvious |
|
Nevermind the remark about the two lines removal fixing it; it does happen without them, just after approximately 7 minutes instead of 3. |
|
Tried a few more things to run the container as "standardly" as possible. Notably, starting traffic_manager (pid 1) as root, to no avail. However something interesting seems to be that the traffic server process seems to get restarted around when it happens: Here's the "extended" version of it, from the last Details |
|
If someone else encounters this, for now I'm just resorting to the old-and-dirty tricks of disabling log rotation and symlinking log files to /dev/stdout or /dev/stderr... ie |
|
@Tristan971 Is it possible for you to make a Docker container, so we can reproduce the issue? We haven't been able to reproduce the issue ourselves. |
|
This issue has been automatically marked as stale because it has not had recent activity. Marking it stale to flag it for further consideration by the community. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
While trying to run ATS in a Docker container, I thought I'd make use of #7937 as a nice way to handle logs.
So I built branch 9.2.x, specifically 15bea4d and set the following:
In records.config:
In logging.yaml:
Then I use
traffic_managerwithout arguments as (non-root) PID 1 in the container.At first things look alright, but after approximately 3 minutes, the logs just stop being output. Moving back error and diags to a file I can see:
Looking into the container, I also notice the following:
As one would expect, the first two commands cause "test" to be printed on the container logs, and the third does nothing.
It seems like stderr gets disconnected entirely at some point? A
traffic_ctl server restartdoes bring logs back for another 3 minutes, but then the issue happens again. (note that the other ones work just fine)At first I thought it would be some issue with ATS trying to "roll" the pipe or something akin to that, but adding
CONFIG proxy.config.log.rolling_enabled INT 0to records.config doesn't make a difference, so I guess it's not that.If it is relevant, this was compiled and ran on a Debian Bullseye image, and the image in question is
registry.gitlab.com/mangadex-pub/trafficserver:9.2.x-bullseye-dbcecd20.Also, ATS otherwise keeps running just fine while that happens.
I doubt this will be particularly relevant, but here's the logs until they stop (when all set to stderr):
Logs
Hopefully this helps track down the issue
The text was updated successfully, but these errors were encountered: