Count each unique mountpoint once in fsstasts

[zyrill]

• Version: # metricbeat version 5.0.0-alpha4 (amd64), libbeat 5.0.0-alpha4
• Operating System: CentOS Linux release 7.2.1511 (Core)
• Steps to Reproduce: activate fsstat metricset, switch logging to debug, compare with df output

Following the full info.
metricbeat Config:

###################### Metricbeat Configuration Example #######################
#==========================  Modules configuration ============================
metricbeat.modules:
- module: system
  metricsets:
    - cpu
    #- core
    - diskio
    #- filesystem
    - fsstat
    - memory
    - network
    #- process
  enabled: true
  period: 10s
  processes: ['.*']
  cpu_ticks: false

- module: nginx
  metricsets: ['stubstatus']
  enabled: true
  period: 10s
  hosts: ['http://127.0.0.1']
  server_status_path: '/nginx_status'

#================================ General =====================================
#name:
#tags: ["service-X", "web-tier"]
#env: staging

#================================ Outputs =====================================
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  #hosts: ["localhost:9200"]
  #template.name: "metricbeat"
  #template.path: "metricbeat.template.json"
  #template.overwrite: false

#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["10.255.210.5:5044"]
  tls.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
  #tls.certificate: "/etc/pki/client/cert.pem"
  #tls.certificate_key: "/etc/pki/client/cert.key"

#================================ Logging =====================================
#logging.level: debug
logging.level: error
logging.to_files: true
logging.to_syslog: false
logging.files:
  path: /var/log/metricbeat/
  name: metricbeat.log
  keepfiles: 7

A relevant entry of the metricbeat log:

2016-07-21T14:33:34+02:00 DBG  Publish: {
  "@timestamp": "2016-07-21T12:33:34.783Z",
  "beat": {
    "hostname": "CCT-WEB008-C7V",
    "name": "CCT-WEB008-C7V"
  },
  "metricset": {
    "module": "system",
    "name": "fsstat",
    "rtt": 379
  },
  "system": {
    "fsstat": {
      "count": 32,
      "total_files": 1410425,
      "total_size": {
        "free": 23922294784,
        "total": 35769786368,
        "used": 11847491584
      }
    }
  },
  "type": "metricsets"
}

The corresponding output from the same machine of df:

# df -h
Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
/dev/sda3        16G    5,5G  8,4G   40% /
devtmpfs        483M       0  483M    0% /dev
tmpfs           492M       0  492M    0% /dev/shm
tmpfs           492M     19M  473M    4% /run
tmpfs           492M       0  492M    0% /sys/fs/cgroup
/dev/sda2       494M    210M  285M   43% /boot
/dev/sda1       200M    9,5M  191M    5% /boot/efi
tmpfs            99M       0   99M    0% /run/user/0

As you can see, the metricset reports 32 fs instead of 8 (or, if we're picky, the correct 3). It also reports free space of 23922294784 instead of the correct ~8.4GB.

[ruflin]

Could you run metricbeat with the filesystem module enabled. This should show you what the 32 file systems are that were counted.

[zyrill]

I did, info following (I converted the JSON into a slightly better readable table):

avail, devicename, files, free, free_files, mountpoint, total, used, bytes, pct
8915763200, rootfs, 0, 10636902400, 0, "/", 16443768832, 5806866432, 0.3531
0, "sysfs", 0, 0, 0, "/sys", 0, 0, 0
0, "proc", 0, 0, 0, "/proc", 0, 0, 0
505581568, "devtmpfs", 123433, 505581568, 123119, "/dev", 505581568, 0, 0
0, "securityfs", 0, 0, 0, "/sys/kernel/security", 0, 0, 0
515457024, "tmpfs", 125844, 515457024, 125843, "/dev/shm", 515457024, 0, 0
0, "devpts", 0, 0, 0, "/dev/pts", 0, 0, 0
489467904, "tmpfs", 125844, 489467904, 125513, "/run", 515457024, 25989120, 0.0504
515457024, "tmpfs", 125844, 515457024, 125831, "/sys/fs/cgroup", 515457024, 0, 0
0, "cgroup", 0, 0, 0, "/sys/fs/cgroup/systemd", 0, 0, 0
0, "pstore", 0, 0, 0, "/sys/fs/pstore", 0, 0, 0
0, "efivarfs", 0, 0, 0, "/sys/firmware/efi/efivars", 0, 0, 0
0, "cgroup", 0, 0, 0, "/sys/fs/cgroup/net_cls", 0, 0, 0
0, "cgroup", 0, 0, 0, "/sys/fs/cgroup/blkio", 0, 0, 0
0, "cgroup", 0, 0, 0, "/sys/fs/cgroup/hugetlb", 0, 0, 0
0, "cgroup", 0, 0, 0, "/sys/fs/cgroup/freezer", 0, 0, 0
0, "cgroup", 0, 0, 0, "/sys/fs/cgroup/devices", 0, 0, 0
0, "cgroup", 0, 0, 0, "/sys/fs/cgroup/cpu,cpuacct", 0, 0, 0
0, "cgroup", 0, 0, 0, "/sys/fs/cgroup/memory", 0, 0, 0
0, "cgroup", 0, 0, 0, "/sys/fs/cgroup/perf_event", 0, 0, 0
0, "cgroup", 0, 0, 0, "/sys/fs/cgroup/cpuset", 0, 0, 0
0, "configfs", 0, 0, 0, "/sys/kernel/config", 0, 0, 0
8915763200, "/dev/sda3", 0, 10636902400, 0, "/", 16443768832, 5806866432, 0.3531
0, "selinuxfs", 0, 0, 0, "/sys/fs/selinux", 0, 0, 0
0, "systemd-1", 0, 0, 0, "/proc/sys/fs/binfmt_misc", 0, 0, 0
0, "hugetlbfs", 0, 0, 0, "/dev/hugepages", 0, 0, 0
0, "debugfs", 0, 0, 0, "/sys/kernel/debug", 0, 0, 0
0, "mqueue", 0, 0, 0, "/dev/mqueue", 0, 0, 0
298393600, "/dev/sda2", 512000, 298393600, 511965, "/boot", 517713920, 219320320, 0.4236
199618560, "/dev/sda1", 0, 199618560, 0, "/boot/efi", 209489920, 9871360, 0.0471
0, "binfmt_misc", 0, 0, 0, "/proc/sys/fs/binfmt_misc", 0, 0, 0
103092224, "tmpfs", 416404, 103092224, 416403, "/run/user/0", 103092224, 0, 0

[andrewkroh]

Link to Discuss topic: https://discuss.elastic.co/t/all-numbers-reported-by-metricbeat-too-high-bug/56094

[monicasarbu]

Closing the issue, as I think all the questions were answered under discuss.

[andrewkroh]

It seems like a bug to me because Metricbeat's fsstats is reporting the wrong value. In the data posted by @zyrill there are two devices (rootfs and /dev/sda3) that are sharing the / mountpoint. They are the same device.

df seems to know enough to ignore one of the devices so probably Metricbeat should too. I'm just not sure what logic df uses for this.

[monicasarbu]

Re-open the issue in order to add the logic in fsstats to count each unique mountpoint once.

[andrewkroh]

Closing. This should now be fixed in master (you can test the fix by running the snapshot build).