haproxy.py feature request

[neerajtiwar]

I am using tcollector in my infrastructure to push the metrics to tsdb, i am using haproxy.py onmy some of the haproxy machines, but haproxy.py is unable to push the metrics to tsdb.
I tried to find out and found that my haproxy servers are running multiple threads instead of running a single thread (which is default) so thats why haproxy.py is unable to detect the correct pid and thus is unable to push.

I recommend adding new feature to be added in haproxy.py to capture any number of threads if running, and can send the metrics accordingly.

haproxy is running like the below in my machines:-
haproxy 37114 1 0 Aug26 ? 00:11:28 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
haproxy 37115 1 0 Aug26 ? 00:11:21 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
haproxy 37116 1 0 Aug26 ? 00:11:07 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
haproxy 37117 1 0 Aug26 ? 00:11:11 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
haproxy 37118 1 0 Aug26 ? 00:11:18 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid
haproxy 37135 1 0 Aug26 ? 00:01:17 /usr/sbin/haproxy -D -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid

below is the error i am getting in the tcollector logs:-

2016-09-17 22:36:16,331 tcollector[28582] INFO: haproxy.py (interval=0) needs to be spawned
2016-09-17 22:36:16,336 tcollector[28582] INFO: spawned haproxy.py (pid=14582)
2016-09-17 22:36:16,660 tcollector[28582] WARNING: haproxy.py: ERROR: Process ID list syntax error.
2016-09-17 22:36:16,660 tcollector[28582] WARNING: haproxy.py: ********* simple selection ********* ********* selection by list *********
2016-09-17 22:36:16,660 tcollector[28582] WARNING: haproxy.py: -A all processes -C by command name
2016-09-17 22:36:16,661 tcollector[28582] WARNING: haproxy.py: -N negate selection -G by real group ID (supports names)
2016-09-17 22:36:16,661 tcollector[28582] WARNING: haproxy.py: -a all w/ tty except session leaders -U by real user ID (supports names)
2016-09-17 22:36:16,661 tcollector[28582] WARNING: haproxy.py: -d all except session leaders -g by session OR by effective group name
2016-09-17 22:36:16,661 tcollector[28582] WARNING: haproxy.py: -e all processes -p by process ID
2016-09-17 22:36:16,661 tcollector[28582] WARNING: haproxy.py: -q by process ID (unsorted & quick)
2016-09-17 22:36:16,661 tcollector[28582] WARNING: haproxy.py: T all processes on this terminal -s processes in the sessions given
2016-09-17 22:36:16,661 tcollector[28582] WARNING: haproxy.py: a all w/ tty, including other users -t by tty
2016-09-17 22:36:16,661 tcollector[28582] WARNING: haproxy.py: g OBSOLETE -- DO NOT USE -u by effective user ID (supports names)
2016-09-17 22:36:16,661 tcollector[28582] WARNING: haproxy.py: r only running processes U processes for specified users
2016-09-17 22:36:16,661 tcollector[28582] WARNING: haproxy.py: x processes w/o controlling ttys t by tty
2016-09-17 22:36:16,662 tcollector[28582] WARNING: haproxy.py: *********** output format ********** *********** long options ***********
2016-09-17 22:36:16,662 tcollector[28582] WARNING: haproxy.py: -o,o user-defined -f full --Group --User --pid --cols --ppid
2016-09-17 22:36:16,662 tcollector[28582] WARNING: haproxy.py: -j,j job control s signal --group --user --sid --rows --info
2016-09-17 22:36:16,662 tcollector[28582] WARNING: haproxy.py: -O,O preloaded -o v virtual memory --cumulative --format --deselect
2016-09-17 22:36:16,662 tcollector[28582] WARNING: haproxy.py: -l,l long u user-oriented --sort --tty --forest --version
2016-09-17 22:36:16,662 tcollector[28582] WARNING: haproxy.py: -F extra full X registers --heading --no-heading --context
2016-09-17 22:36:16,662 tcollector[28582] WARNING: haproxy.py: --quick-pid
2016-09-17 22:36:16,662 tcollector[28582] WARNING: haproxy.py: ********* misc options *********
2016-09-17 22:36:16,662 tcollector[28582] WARNING: haproxy.py: -V,V show version L list format codes f ASCII art forest
2016-09-17 22:36:16,662 tcollector[28582] WARNING: haproxy.py: -m,m,-L,-T,H threads S children in sum -y change -l format
2016-09-17 22:36:16,663 tcollector[28582] WARNING: haproxy.py: -M,Z security data c true command name -c scheduling class
2016-09-17 22:36:16,663 tcollector[28582] WARNING: haproxy.py: -w,w wide output n numeric WCHAN,UID -H process hierarchy
2016-09-17 22:36:16,663 tcollector[28582] WARNING: haproxy.py: HAProxy (pid 37135 37118 37117 37116 37115 37114) went away? Command '['ps', '--no-headers', '-o', 'cmd', '37135 37118 37117 37116 37115 37114']' returned non-zero exit status 1

Thanks,
Neeraj

[johann8384]

I probably should modify it to only detect the PID if there is not a socket file defined in the configuration. It's trying to be clever and detect the HAProxy process and detect the socket file.

[vitaliyf]

If you are using "nbproc" option in haproxy with a value >1 beware that there are caveats for it, specifically about stats.

There is stats bind-process directive that says:

stats bind-process [ all | odd | even | <number 1-64>[-<number 1-64>] ] ...

Limits the stats socket to a certain set of processes numbers. By default the
stats socket is bound to all processes, causing a warning to be emitted when
nbproc is greater than 1 because there is no way to select the target process
when connecting. However, by using this setting, it becomes possible to pin
the stats socket to a specific set of processes, typically the first one.

We ended up not using nbproc (due to documented caveats that did surface actual problems) and are running multiple full instances of haproxy with separate .pid and .sock files (and binding to multiple IPs). We modified haproxy.py tcollector to iterate over /var/run/haproxy-**.sock files and emit stats from each one (tagged with the value of .sock filename's "suffix").

Our solution is a bit specific to us so wouldn't be a great contribution, but I wanted to chime in so that you are beware. A generic solution (perhaps have a config value for .sock file or file pattern instead of trying to detect it) should be easy to implement.

[Boolman]

I had the same issue, I went ahead and modified the script to support multiple procs in haproxy, http://ix.io/vLR

[johann8384]

@Boolman can you submit a PR for this against the current haproxy collector?