Occasional 'Mysql2::Error: Malformed packet' errors on simple selects

[cdekker]

Since upgrading to 0.3.20 about a month ago on my production environment I started getting these errors randomly.

An ActiveRecord::StatementInvalid occurred in templates#test_invoice:

  Mysql2::Error: Malformed packet: SELECT `permissions`.`pkey` FROM `permissions`  WHERE `permissions`.`user_id` = 121 AND `permissions`.`client_id` = 37
  activerecord (3.2.22) lib/active_record/connection_adapters/abstract_mysql_adapter.rb:245:in `query'

Sometimes I get these errors on similar queries:
Mysql2::Error: Lost connection to MySQL server during query:

They happen infrequent enough to be unable to reproduce them. At first I thought this was related to server RAM running out, but this doesn't seem to be the case after increasing the RAM and the errors still occuring.

Similarly, sometimes around those errors, also Rails errors start to creep up in core parts that normally work fine.

A NoMethodError occurred in invoice_batches#show:

  undefined method `fields' for nil:NilClass
  activerecord (3.2.22) lib/active_record/connection_adapters/mysql2_adapter.rb:218:in `exec_query'

This is my database.yml:

production:
  adapter: mysql2
  encoding: utf8
  reconnect: false
  database: (...)
  pool: 5
  username: (...)
  password: (...)
  host: localhost
  flags: <%= 65536 | 131072 %>

[developingchris]

What version of mysql are you using, I think I'm running into the same thing, only I can consistently reproduce it on version 5.1.
I'm trying to use mysql2 to assist me in an upgrade to 5.1 and getting malformed packet errors quite a lot.

[cdekker]

I had it both occuring in 5.5 (default package on Ubuntu 14.04) and 5.6 Percona, but very sporadically and was not able to consistently reproduce it.

Just moved to a new server with CentOS 7 and MariaDB.

Also moved from Sidekiq to Delayed_Job as a background worker with it's dedicated non-stop running process(es). Had the idea that that caused some issues as well with Passenger and processes being abandoned or terminated too quickly.

[will-in-wi]

IIRC, we found that having an older libmysql client with a newer server could cause this issue. Something to look into.