Cannot start mysql_statsd, get

[DutchessNicole]

Hi,

I'm trying to get mysql_statsd to run on an ansible-deployed server.

In order to do so I've had to mess around with users and installing the system through pip, since it complains when I try to install it as root.
So what I've done now is run the pip installer as a dedicated user 'statsd', and set up a service file that calls the pip managed mysql_statsd file in the user's home dir.

The service is called like so:

[Unit]
Name = statsd
Description = mysql statsd

[Service]
ExecStart=/home/statsd/.local/bin/mysql_statsd --config /etc/mysql_statsd.conf
Restart=always
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=mysql_statsd
WorkingDirectory=/home/statsd
User=statsd
Group=statsd
RestartSec=10
# Environment=NODE_ENV=production PORT=5000

[Install]
WantedBy=multi-user.target

with the configuration file (in /etc/, that does seem to work)

[daemon]
logfile = /home/statsd/mysql_statsd/log/daemon.log
pidfile = /home/statsd/mysql_statsd.pid

[statsd]
host = {{ mariadb_statsd_host }}
port = {{ mariadb_statsd_port }}
prefix = mysql
include_hostname = true

[mysql]
; host = localhost
socket = /tmp/mysql.sock
username = {{ mariadb_statsd_user }}
password = {{ mariadb_statsd_pass }}
stats_types = status,variables,innodb,slave
query_variables = SHOW GLOBAL VARIABLES
interval_variables = 1000
query_status = SHOW GLOBAL STATUS
interval_status = 1000
query_innodb = SHOW ENGINE INNODB STATUS
interval_innodb = 1000
query_slave = SHOW SLAVE STATUS
interval_slave = 1000
query_commit = COMMIT
interval_commit = 1000
sleep_interval = 0


[metrics]
; g = gauge, c = counter (increment), t = timer, r = raw value, d = delta
variables.max_connections = g
status.max_used_connections = g
status.connections = d
status.aborted_connects = d
status.open_tables = g
status.open_files = g
status.open_streams = g
status.opened_tables = d
status.queries = d
status.empty_queries = d
status.slow_queries = d
status.questions = d
status.com_select = d
status.com_insert = d
status.com_update = d
status.com_delete = d
status.com_insert_select = d
status.qcache_queries_in_cache = g
status.qcache_inserts = d
status.qcache_hits = d
status.qcache_prunes = d
status.qcache_not_cached = d
status.qcache_lowmem_prunes = d
status.qcache_free_memory = g
status.qcache_free_blocks = g
status.qcache_total_blocks = g
status.flush_commands = g
status.created_tmp_disk_tables = d
status.created_tmp_tables = d
status.threads_running = g
status.threads_created = d
status.threads_connected = g
status.threads_cached = g
status.wsrep_flow_control_sent = g
status.wsrep_flow_control_recv = g
status.wsrep_local_sent_queue = g
status.wsrep_local_recv_queue = g
status.wsrep_cert_deps_distance = g
status.wsrep_local_cert_failures = d
status.rep_local_bf_aborts = d
status.wsrep_last_committed = d
status.wsrep_flow_control_paused = g
innodb.spin_waits = d
innodb.spin.rounds = d
innodb.os_waits = d
innodb.spin_rounds = d
innodb.os_waits = d
innodb.pending_normal_aio_reads = g
innodb.pending_normal_aio_writes = g
innodb.pending_ibuf_aio_reads = g
innodb.pending_aio_log_ios = g
innodb.pending_aio_sync_ios = g
innodb.pending_log_flushes = g
innodb.pending_buf_pool_flushes = g
innodb.pending_log_writes = g
innodb.pending_chkp_writes = g
innodb.file_reads = d
innodb.file_writes = d
innodb.file_fsyncs = d
innodb.ibuf_inserts = d
innodb.ibuf_merged = d
innodb.ibuf_merges = d
innodb.log_bytes_written = d
innodb.unflushed_log = g
innodb.log_bytes_flushed = d
innodb.log_writes = d
innodb.pool_size = g
innodb.free_pages = g
innodb.database_pages = g
innodb.modified_pages = g
innodb.pages_read = d
innodb.pages_created = d
innodb.pages_written = d
innodb.queries_inside = d
innodb.queries_queued = d
innodb.read_views = d
innodb.rows_inserted = d
innodb.rows_updated = d
innodb.rows_deleted = d
innodb.rows_read = d
innodb.innodb_transactions = d
innodb.unpurged_txns = d
innodb.history_list = g
innodb.current_transactions = g
innodb.active_transactions = g
innodb.locked_transactions = g
innodb.innodb_locked_tables = g
innodb.innodb_tables_in_use = g
innodb.read_views = g
innodb.hash_index_cells_total = g
innodb.hash_index_cells_used = g
innodb.total_mem_alloc = d
innodb.additional_pool_alloc = d
innodb.last_checkpoint = d
innodb.uncheckpointed_bytes = g
innodb.ibuf_used_cells = g
innodb.ibuf_free_cells = g
innodb.ibuf_cell_count = g
innodb.adaptive_hash_memory = g

slave.seconds_behind_master = g

; innodb.bufferpool_*.<metric> will whitelist these metrics for all bufferpool instances
; If you don't have multiple bufferpools it won't do anything
innodb.bufferpool_*.pool_size = g
innodb.bufferpool_*.pool_size_bytes = g
innodb.bufferpool_*.free_pages = g
innodb.bufferpool_*.database_pages = g
innodb.bufferpool_*.old_database_pages = g
innodb.bufferpool_*.modified_pages = g
innodb.bufferpool_*.pending_reads = g
innodb.bufferpool_*.pending_writes_lru = g
innodb.bufferpool_*.pending_writes_flush_list = g
innodb.bufferpool_*.pending_writes_single_page = g
innodb.bufferpool_*.pages_made_young = d
innodb.bufferpool_*.pages_not_young = d
innodb.bufferpool_*.pages_made_young_ps = g
innodb.bufferpool_*.pages_not_young_ps = g
innodb.bufferpool_*.pages_read = d
innodb.bufferpool_*.pages_created = d
innodb.bufferpool_*.pages_written = d
innodb.bufferpool_*.pages_read_ps = g
innodb.bufferpool_*.pages_created_ps = g
innodb.bufferpool_*.pages_written_ps = g
innodb.bufferpool_*.buffer_pool_hit_total = g
innodb.bufferpool_*.buffer_pool_hits = g
innodb.bufferpool_*.buffer_pool_young = g
innodb.bufferpool_*.buffer_pool_not_young = g
innodb.bufferpool_*.pages_read_ahead = g
innodb.bufferpool_*.pages_read_evicted = g
innodb.bufferpool_*.pages_read_random = g
innodb.bufferpool_*.lru_len = g
innodb.bufferpool_*.lru_unzip = g
innodb.bufferpool_*.io_sum = d
innodb.bufferpool_*.io_sum_cur = g
innodb.bufferpool_*.io_unzip = d
innodb.bufferpool_*.io_unzip_cur = g

Now, the daemon does start and the mysql_statsd script seems to pick up the configuration just fine, but it continually generates errors for all the threads it spawns. Then they all die, systemd restarts the entire service 10 seconds later and the same thing happens all over again.

I have no idea how to debug this as it's all python code that I'm not good at.

These are the errors that are put in the daemon.log file (specified in the config, that's how I know it's being read):

Caught CTRL+C / SIGKILL
Stopping threads
Waiting for 3 threads
All threads stopped
<mysql_statsd.mysql_statsd.MysqlStatsd instance at 0x7f7ab181acf8>
Caught CTRL+C / SIGKILL
Stopping threads
Waiting for 2 threads
All threads stopped
<mysql_statsd.mysql_statsd.MysqlStatsd instance at 0x7fc25eba7cf8>
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
TypeError: 'bool' object is not callable

Caught CTRL+C / SIGKILL
Stopping threads
Waiting for 2 threads
All threads stopped
<mysql_statsd.mysql_statsd.MysqlStatsd instance at 0x7f809f130cf8>
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
TypeError: 'bool' object is not callable

Caught CTRL+C / SIGKILL
Stopping threads
Waiting for 3 threads
All threads stopped
<mysql_statsd.mysql_statsd.MysqlStatsd instance at 0x7f43503f0cf8>
Caught CTRL+C / SIGKILL
Stopping threads
Waiting for 3 threads
All threads stopped
<mysql_statsd.mysql_statsd.MysqlStatsd instance at 0x7fae44e33cf8>
Caught CTRL+C / SIGKILL
Stopping threads
Waiting for 3 threads
All threads stopped
<mysql_statsd.mysql_statsd.MysqlStatsd instance at 0x7f9296f52cf8>

and finally the (I believe) relevant sections of the ansible script:

- name: Add statsd group
  group:
    name: statsd
    system: yes

- name: Add statsd user
  user:
    name: statsd
    shell: /bin/bash
    group: statsd
    home: /home/statsd
    system: yes

- name: install pymysql dependencies
  apt:
    name: default-libmysqlclient-dev
    state: present

- name: copy statsd configuration file
  template:
    src: mysql-statsd.conf.j2
    dest: /etc/mysql_statsd.conf
    owner: root
    group: root
    mode: 0744

- name: Install statsd
  become: yes
  become_user: statsd
  pip:
    name: ["mysql_statsd"]
    state: present

- name: install statsd service file
  copy:
    src: statsd.service
    dest: /etc/systemd/system/statsd.service
    mode: 0644
    owner: root
    group: root

- name: create log directory
  file:
    path: /home/statsd/mysql_statsd/log/
    state: directory
    owner: statsd
    group: statsd
    mode: 0755

Is there any way to figure out where things are going wrong?

If you need any more information I'd be happy to provide it.
Thanks!