Binlog file size rapidly expanding

[upats]

Hi all, I apologize this is a bit of a shot in the dark. My Moodle DB server went down after running out of disk space. Investigating I found that the binlog files used for MySQL primary/backup replication were being created very rapidly, up to 300 MB every minute or two while the DB tables were staying more or less the same size. I looked at running queries and noticed some queries related to saml like INSERT INTO mdl_auth_saml2_kvstore (type,k,value,expire) VALUES with very long strings for the token value.

I disabled the SAML plugin and things immediately went back to normal (or what I assume is normal; bin logs growing at a rate of a few MB per minute). So my thought was that very long strings of SAML keys being refreshed at once might be causing the issue. I waited awhile and turned SAML back on and didn't see the issue again for a few months. The first time this happened was in May, and then it just happened again yesterday. Again I waited a day or so, turned SAML back on and it seems fine.

Still investigating and I'm considering seeing if I can exlude the auth_saml tables from replication, but I wanted to reach out the devs to see if you have have any thoughts.

[brendanheywood]

The amount of data stored in the db KV store is minimal, and should be some fraction of the normal moodle user session data. It sounds like there was some crazy sort of redirect loop in someones browser or even an attack of some kinda happening which was churning the data and not normal saml logins. Turning it off would have broken the redirect loop with an exception for the user(s) affected.

If it happens again you'll want to tail the access logs and see if there is one anon user on a set ip smashing the same urls over and over, and then debug them to see what's going wrong.

[iv-enovation]

This seems to be a real issue, we recently experienced the same problem of the disk completely filled up with binlog files

From the binlog, the query that caused the situation is:
UPDATE mdl_auth_saml2_kvstore SET type = 'session',k = 'a91d5c8906ed52d0983b2725d3adec9e',value = 'xxx' - where "value" is about 5.4M of plain text

The query is issued by: extlib/simplesamlphp/lib/SimpleSAML/SessionHandlerStore.php: $this->store->set('session', $sessionId, $session, $expire);

The stored session contains multiple (counted ~75K) SimpleSAMLphp's states, each containing data with URLs to SCORM files. Each URL has 'mobile' in it so the issue may be related to mobile SCORM usage

None of the recorded session are expired, the oldest entry has the expiry 1 sec after the binlog entry's timestamp

Attaching a sample 'data' value, urldecoded and unserialized in order to replace server URLs. sphost.example.net is Moodle, idphost.example.net is and IdP
update_statement_data_value_unserialized.txt.gz

[JF10R]

Also encountered this issue. The logs generated by the mdl_auth_saml2_kvstore queries inside log_bin files filled dozens of gigabytes rather quickly when confronted to massive traffic.