Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log files with same inode number in different block Device caused missing log file #4326

Open
mioyui88 opened this issue Oct 17, 2023 · 3 comments
Labels
bug Something isn't working enhancement Feature request or improve operations

Comments

@mioyui88
Copy link

Describe the bug

I have two log files in different block device. Two log files have same file name and same inode number.

The first file with inode 17 in device fh72h/64882d 
stat /var/lib/kubelet/pods/pod-uid-1/volumes/kubernetes.io~csi/local-pv-id-1/mount/xx/xx/file.log
       Device: fh72h/64882d                 Inode:17 

The second file with inode 17 in device fd53h/64851d 
stat /var/lib/kubelet/pods/pod-uid-2/volumes/kubernetes.io~csi/local-pv-id-2/mount/xx/xx/file.log
       Device: fd53h/64851d                 Inode:17 

When I want collect the two file.log, I find fluentd only collect one file, the other one doesn't show the fluentd tail target and existing list.

tailing paths: target =  /var/lib/kubelet/pods/pod-uid-1/volumes/kubernetes.io~csi/local-pv-id-1/mount/xx/xx/file.log |
  existing = /var/lib/kubelet/pods/pod-uid-1/volumes/kubernetes.io~csi/local-pv-id-1/mount/xx/xx/file.log

To Reproduce

  1. mount two block device to different mountpoint
  2. cd mountpoint, make new file, I think default inode number is same
  3. start fluentd , set config follow_inodes true

Expected behavior

Fluentd can identify inode number in different block device, and collect log.

Your Environment

- Fluentd version:1.16.2
- TD Agent version:
- Operating system: Centos7
- Kernel version:3.10.0-1160.92.1.el7.x86_64

Your Configuration

<source>
  @type tail
  read_from_head true
  skip_refresh_on_startup true
  path /var/lib/kubelet/pods/*/volumes/kubernetes.io~csi/*/mount/namespace/*/*/*.log
  pos_file /var/log/csi_namespace.log.pos
  refresh_interval 5s
  tag "csi_namespace"
  limit_recently_modified 7d
  format /?<message>.*/
</source>

Your Error Log

There is no error log, just can't get the log. And Fluentd print existing just include one log file.

Additional context

No response

@mioyui88
Copy link
Author

mioyui88 commented Oct 18, 2023

In addition, I use config
follow_inodes false, enable_stat_watcher true, enable_watch_timer true, open_on_every_update true , still have some problem.
When I mount new volume to /var/lib/kubelet/pods/pod-new/volumes/kubernetes.io~csi/local-pv-new/mount ,
and config tail path is /var/lib/kubelet/pods/*/volumes/kubernetes.io~csi/*/mount/namespace/*/*/*.log
Fluentd can't collect new log in /var/lib/kubelet/pods/pod-new/volumes/kubernetes.io~csi/local-pv-new/mount/namespace/*/*/file.log.
After I restart Fluentd agent, then the log can be collected to Fluentd.

So how can I collect the log file that stored in new mounted path without restarting Fluentd agent.

**Update**
There is no problem with config follow_inodes false, enable_stat_watcher true, enable_watch_timer true, open_on_every_update true.
The reason caused log file missing is because K8S pod get the node mountpoint need config
mountPropagation: HostToContainer

@mioyui88
Copy link
Author

@daipom @ashie @repeatedly

@ashie ashie added feature request *Deprecated Label* Use enhancement label in general bug Something isn't working and removed waiting-for-triage labels Dec 8, 2023
@ashie
Copy link
Member

ashie commented Dec 8, 2023

Thanks for your report!
Hmm, it seems a specification bug.
The ideal solution is adding device ID to pos file, but doing it is not so easy with keeping backward compatibility.

Possible work around is separating <source> section per devices, although it might be hard to use in this case since mount point is determined dynamically.

@kenhys kenhys added enhancement Feature request or improve operations and removed feature request *Deprecated Label* Use enhancement label in general labels Jul 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement Feature request or improve operations
Projects
None yet
Development

No branches or pull requests

3 participants