podman logs crash #5131
Comments
openshift-ci-robot
added
the
kind/bug
|
Eek. We may be reading the entire file into memory... |
|
@mheon we did that intentionally ... waiting for and hoping this would never be the case. ill take this one |
|
@mheon lets discuss this on monday ... it's a reasonably deep hole we will need to go in to fix this. I am not discounting the bug, more on how we want to fix it. |
|
some thoughts before i loose them ... the failure here is fortunately not with the tail pkg we use. It is specifically with the function to print out out the previous entries of the log. When getting the logs of a container, we print N number of last log entries. The default is to print them all but N can also be provided (like show me the last 10 lines). To do this, we were reading the whole file and then working backwards to find N number of lines. A line is something that ends in \n. A log line has must both end in \n AND fit a certain format or it is NOT considered a log (line). Reading of the whole file is what tripped when the file size was larger than the available memory. To fix this,we will need to generally open the file, seek the end. Then read backwards finding the beginning and ending of a line (between \n's). That line needs to then be qualified as log line (has the right format). then, increment the number of found lines if N is set. the additional quirk here is that we currently return this information to the caller method as a []logline. Given that we are here to fix low memory constraints, it is entirely possible that the []logline could exceed memory capacity (in the case of a large log) so we would only be moving the memory problem from the read to the return. there is a log channel where we normally send stuff. the caller of the method in question, usually iterates the returned []logline and sends it to output. i think to do this correctly, the log channel will need to be provided deeper into the methods to immediately output and keep memory usage ideal. |
|
One thing we might want to add is a way to truncate the logs, rather then allowing them to grow forever. |
in cases where the log file exceeds the available memory of a system, we had a bug that triggered an oom because the entire logfile was being read when the tail parameter was given. this reads in chunks and is more or less memory safe. fixes: containers#5131 Signed-off-by: Brent Baude <bbaude@redhat.com>
in cases where the log file exceeds the available memory of a system, we had a bug that triggered an oom because the entire logfile was being read when the tail parameter was given. this reads in chunks and is more or less memory safe. fixes: containers#5131 Signed-off-by: Brent Baude <bbaude@redhat.com>
github-actions
bot
added
the
locked - please file new issue/PR
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
Sometimes,
podman logscrashes saying it has run out of memory. Looks like it's trying to load the whole file in RAM, and crashes when it doesn't fit.Steps to reproduce the issue:
# podman logs -f --tail=10 some-containerDescribe the results you received:
Describe the results you expected:
Additional information you deem important (e.g. issue happens only occasionally):
Output of
podman version:Output of
podman info --debug:Package info (e.g. output of
rpm -q podmanorapt list podman):Additional environment details (AWS, VirtualBox, physical, etc.):
OrangePi Zero
Memory at the time of the occurrence:
Size of the
ctr.logat that same time:The text was updated successfully, but these errors were encountered: