MySQL logs disk full error

[chensun]

We noticed on some KFP cluster listing run failed consistently with 524 timeout error.

Looking at the pod logs, metadata-writer and persistence-agent are in crash loop with error failed to connect to mysql.
Checking mysql pod logs, it's flooded with

disk is full writing './binlog.~rec~'

Did some research, looks like MySQL 8.0 changed the default log expiration from 10 days to 30 days:
https://dba.stackexchange.com/questions/233048/my-disk-is-full-of-binlog-files

[chensun]

/cc @jlyaoyuli