[0.10.2] netstat input. Only reports 0 established connections #656

jonaz · 2016-02-06T10:15:12Z

netstat -anp | grep EST | wc -l shows 43 right now.

But the netstat input only report 0.

OS:

Distributor ID: Ubuntu
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Codename: trusty

The text was updated successfully, but these errors were encountered:

sparrc · 2016-02-06T16:34:11Z

please provide OS information

jonaz · 2016-02-06T18:48:12Z

Provided

sparrc · 2016-02-06T18:59:25Z

Can you also provide:

full output of netstat -anp | grep EST
full output of sudo -u telegraf netstat -anp | grep EST (if you're running as a service)

jonaz · 2016-02-06T19:32:52Z

1 and 2 are both the same:


sudo -u telegraf netstat -anp | grep EST
(No info could be read for "-p": geteuid()=997 but you should be root.)
tcp        0      0 127.0.0.1:53243         127.0.0.1:8282          ESTABLISHED -               
tcp        0      0 192.168.1.1:3483       192.168.1.50:17975     ESTABLISHED -               
tcp        0      0 192.168.1.1:55361      192.168.1.1:8086       ESTABLISHED -               
tcp        0      0 127.0.0.1:53244         127.0.0.1:8282          ESTABLISHED -               
tcp        0      0 127.0.0.1:47374         127.0.0.1:8091          ESTABLISHED -               
tcp        0      0 127.0.0.1:9090          127.0.0.1:48238         ESTABLISHED -               
tcp        0      0 127.0.0.1:53245         127.0.0.1:8282          ESTABLISHED -               
tcp        0      0 192.168.1.1:36807      192.168.1.142:8009     ESTABLISHED -               
tcp        0      0 127.0.0.1:8091          127.0.0.1:47374         ESTABLISHED -               
tcp        0      0 5.1.243.162:43340      195.140.202.142:6667    ESTABLISHED -               
tcp        0      0 5.1.243.162:49907      94.125.182.252:6667     ESTABLISHED -               
tcp        0      0 5.1.243.162:37505      80.65.1.26:6667        ESTABLISHED -               
tcp        0      0 127.0.0.1:53242         127.0.0.1:8282          ESTABLISHED -               
tcp        0      0 192.168.1.1:55360      192.168.1.1:8086       ESTABLISHED -               
tcp    83373      0 5.1.243.162:33441      46.21.106.168:8000      ESTABLISHED -               
tcp        0      0 5.1.243.162:57746      173.194.65.125:443      ESTABLISHED -               
tcp        0      0 127.0.0.1:48920         127.0.0.1:58846         ESTABLISHED -               
tcp        0      0 127.0.0.1:47372         127.0.0.1:8091          ESTABLISHED -               
tcp        0      0 192.168.1.1:445        192.168.1.96:42166     ESTABLISHED -               
tcp        0      0 5.1.243.162:49555      90.230.29.9:58351       ESTABLISHED -               
tcp        0      0 127.0.0.1:8091          127.0.0.1:47372         ESTABLISHED -               
tcp        0      0 5.1.243.162:22         217.10.117.170:39830    ESTABLISHED -               
tcp        0      0 5.1.243.162:55583      82.94.168.13:443        ESTABLISHED -               
tcp        0      0 192.168.1.1:22         192.168.1.96:46894     ESTABLISHED -               
tcp        0      0 192.168.1.1:445        192.168.1.96:42168     ESTABLISHED -               
tcp        0      0 127.0.0.1:48238         127.0.0.1:9090          ESTABLISHED -               
tcp        0      0 127.0.0.1:58846         127.0.0.1:48920         ESTABLISHED -               
tcp6       0      0 ::1:50264               ::1:8086                ESTABLISHED -               
tcp6       0      0 127.0.0.1:8282          127.0.0.1:53244         ESTABLISHED -               
tcp6       0      0 ::1:50266               ::1:8086                ESTABLISHED -               
tcp6       0      0 192.168.1.1:8282       192.168.1.56:43675     ESTABLISHED -               
tcp6       0      0 127.0.0.1:8282          127.0.0.1:53242         ESTABLISHED -               
tcp6       0      0 192.168.1.1:8086       192.168.1.1:55361      ESTABLISHED -               
tcp6       0      0 127.0.0.1:8282          127.0.0.1:53243         ESTABLISHED -               
tcp6       0      0 ::1:8086                ::1:50266               ESTABLISHED -               
tcp6       0      0 ::1:8086                ::1:50264               ESTABLISHED -               
tcp6       0      0 127.0.0.1:8282          127.0.0.1:53245         ESTABLISHED -               
tcp6       0      0 192.168.1.1:8086       192.168.1.1:55360      ESTABLISHED -               
tcp6       0      0 192.168.1.1:8080       192.168.1.136:39084    ESTABLISHED -               
udp6       0      0 ::1:53788               ::1:39626               ESTABLISHED -               
udp6       0      0 ::1:39626               ::1:53788               ESTABLISHED -

sparrc · 2016-02-07T06:03:53Z

gopsutil gathers this info by running lsof, can you paste the output from running: lsof -i -a -n -P?

https://github.com/shirou/gopsutil/blob/master/net/net_unix.go#L17

jonaz · 2016-02-07T15:33:50Z

That explains it. I have to run it as root to get any data. When i run it with sudo -u telegraf it shows nothing.

This is the same on all machines i tried with. A few ubuntu and one arch linux and a few cent os 7.

sparrc · 2016-02-07T15:59:11Z

I see, gopsutil using lsof has been a recurring problem. Hopefully it will get changed soon: shirou/gopsutil#142

workaround might be to use setcap to allow lsof to run non-root, setcap net_cap_admin=+ep /bin/lsof should do it.

bwolf · 2016-02-12T14:09:57Z

I also run into this issue on debian machines (8.x on amd64 and armhf). lsof gives no output without root privileges.

cdgraff · 2016-02-20T15:02:00Z

after research and compare PSOF with NETSTAT, i found that psof only show connection status for connections that has a process attached, but TIME_WAIT and other are not related to a process

Example:


tcp        0      0 149.255.37.10:80        213.189.169.76:52780    ESTABLISHED 15201/flussonic
tcp        0      0 149.255.37.10:80        73.247.70.26:39532      ESTABLISHED 15201/flussonic
tcp        0      0 149.255.37.10:80        78.240.140.50:63539     TIME_WAIT   -
tcp        0      0 149.255.37.10:80        88.74.110.0:54418       TIME_WAIT   -
tcp        0      0 149.255.37.10:80        175.140.250.35:54992    TIME_WAIT   -
tcp        0      0 149.255.37.10:80        91.75.74.154:65238      TIME_WAIT   -
tcp        0      0 149.255.37.10:80        88.105.69.212:29333     FIN_WAIT2   -
tcp        0      0 149.255.37.10:80        217.225.68.181:55689    TIME_WAIT   -
tcp        0  40095 149.255.37.10:80        87.184.142.129:63302    ESTABLISHED 15201/flussonic

root@ubuntu# netstat -nap | awk '{print $6}' | sort -n | uniq -c

    116 CLOSE_WAIT
     55 CLOSING
     19 CONNECTED
   4726 ESTABLISHED
    110 FIN_WAIT1
    257 FIN_WAIT2
     61 LAST_ACK
      8 LISTEN
      1 SEQPACKET
      3 STREAM
     57 SYN_RECV
   7744 TIME_WAIT

root@ubuntu# lsof -i -a -n -P | awk '{print $10}' | sort -n | uniq -c

    115 (CLOSE_WAIT)
   4728 (ESTABLISHED)
      8 (LISTEN)

shirou · 2016-02-20T15:25:37Z

I tried to read connection infomation from /proc/*/fd and /proc/*/net but I can not get net information from other users process due to a permission denied. I check pssutil (http://pythonhosted.org/psutil/) and it also get only the user's connection. but netstat can get all information (suid is set?).

I think in order to get all users network information, gopsutil should use netstat instead of lsof. I will try.

shirou · 2016-02-20T15:42:19Z

I got it. netstat read /proc/net/* which contains network connection information of all users. But it contains only inode, so other user can not get which process is use that connection. I try to use /proc/net but it seems very complicated cause there are UNIX/TCP/TCP6. So please wait.

cdgraff · 2016-02-20T16:20:45Z

btw, look like output inside telegraf plugin be wrong too, don't report more than
tcp_established=9386i (always less than 9900)

root@ubuntu# lsof -i -a -n -P | awk '{print $10}' | sort -n | uniq -c

    2
    625 (CLOSE_WAIT)
  13229 (ESTABLISHED)
      8 (LISTEN)

telegraf -test -input-filter netstat -config /etc/telegraf/telegraf.conf

* Plugin: netstat, Collection 1
> netstat tcp_close=0i,tcp_close_wait=459i,tcp_closing=0i,tcp_established=9386i,tcp_fin_wait1=0i,tcp_fin_wait2=0i,tcp_last_ack=0i,tcp_listen=8i,tcp_none=0i,tcp_syn_recv=0i,tcp_syn_sent=0i,tcp_time_wait=0i,udp_socket=1i 1455984992992170502

sparrc · 2016-03-07T12:23:34Z

This will be fixed when gopsutil no longer uses lsof, see shirou/gopsutil#164,

will close this case when Telegraf has updated that dependency

closes #656

lswith · 2016-03-23T00:49:21Z

This issue is still occurring with nightly build: 0.12.0~n1458634430. I believe that this nightly build has the new gopsutil dependency.

sparrc · 2016-03-23T04:45:46Z

@lswith what is your OS?

sparrc · 2016-03-23T14:08:59Z

Actually this won't be available until a new nightly gets made tomorrow morning at ~8am GMT. The nightly build process was broken for a moment because we now require Go 1.6 with the gdm dependencies.

abarban · 2016-03-30T16:00:32Z

downloaded the nightly build rpm!
It is no longer compatible with RedHat EL5?

sparrc · 2016-03-30T16:05:16Z

@abarban yes centos 5 does not support 4096 bit keys, see #734

jonaz changed the title ~~[0.10.2] netstat input. Only reports 0 connections~~ [0.10.2] netstat input. Only reports 0 established connections Feb 6, 2016

sparrc mentioned this issue Feb 18, 2016

Netstat report incorrect values #719

Closed

sparrc added the bug unexpected problem or unintended behavior label Mar 7, 2016

sparrc added a commit that referenced this issue Mar 21, 2016

Update gopsutil dependency

7ae7ca5

closes #656

sparrc mentioned this issue Mar 21, 2016

Update gopsutil dependency #902

Merged

sparrc added a commit that referenced this issue Mar 21, 2016

Update gopsutil dependency

830b76d

closes #656

sparrc added a commit that referenced this issue Mar 21, 2016

Update gopsutil dependency

f697406

closes #656

sparrc added a commit that referenced this issue Mar 21, 2016

Update gopsutil dependency

2a3cb27

closes #656

sparrc added a commit that referenced this issue Mar 21, 2016

Update gopsutil dependency

2e11e36

closes #656

sparrc added a commit that referenced this issue Mar 21, 2016

Update gopsutil dependency

2c88d9e

closes #656

sparrc added a commit that referenced this issue Mar 22, 2016

Update gopsutil dependency

e3035f4

closes #656

sparrc added a commit that referenced this issue Mar 22, 2016

Update gopsutil dependency

4f5f676

closes #656

sparrc closed this as completed in #902 Mar 22, 2016

lswith mentioned this issue Mar 31, 2016

add http_response plugin #943

Closed

danielnelson mentioned this issue May 4, 2017

procstat permission denied #517

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[0.10.2] netstat input. Only reports 0 established connections #656

[0.10.2] netstat input. Only reports 0 established connections #656

jonaz commented Feb 6, 2016

sparrc commented Feb 6, 2016

jonaz commented Feb 6, 2016

sparrc commented Feb 6, 2016

jonaz commented Feb 6, 2016

sparrc commented Feb 7, 2016

jonaz commented Feb 7, 2016

sparrc commented Feb 7, 2016

bwolf commented Feb 12, 2016

cdgraff commented Feb 20, 2016

shirou commented Feb 20, 2016

shirou commented Feb 20, 2016

cdgraff commented Feb 20, 2016

sparrc commented Mar 7, 2016

lswith commented Mar 23, 2016

sparrc commented Mar 23, 2016

sparrc commented Mar 23, 2016

abarban commented Mar 30, 2016

sparrc commented Mar 30, 2016

[0.10.2] netstat input. Only reports 0 established connections #656

[0.10.2] netstat input. Only reports 0 established connections #656

Comments

jonaz commented Feb 6, 2016

sparrc commented Feb 6, 2016

jonaz commented Feb 6, 2016

sparrc commented Feb 6, 2016

jonaz commented Feb 6, 2016

sparrc commented Feb 7, 2016

jonaz commented Feb 7, 2016

sparrc commented Feb 7, 2016

bwolf commented Feb 12, 2016

cdgraff commented Feb 20, 2016

shirou commented Feb 20, 2016

shirou commented Feb 20, 2016

cdgraff commented Feb 20, 2016

sparrc commented Mar 7, 2016

lswith commented Mar 23, 2016

sparrc commented Mar 23, 2016

sparrc commented Mar 23, 2016

abarban commented Mar 30, 2016

sparrc commented Mar 30, 2016