Enable backups for existing cluster

[micronax]

Is there any way to enable backups for a existing cluster or do I need to recreate the cluster with the backupSchedule specs?

At least the README is unclear at this point.

[HBO2]

@AMecea I am not sure, but I think the scheduled backup it still not working right? I am using the latest operator. and confirmed the backup it self is working, but scheduling in the cluster seems not to work because the cron trigger is doing noting:

....
spec:
  backupSchedule: 0 0 */20 * * *
  backupScheduleJobsHistoryLimit: 1
  backupSecretName: my-cluster-backup-secret
  backupUri: s3://backup/
....

[AMecea]

Indeed this seems to be a bug, This part is not tested enough. I will fix this issue ASAP. Thanks for reporting it!

[micronax]

OK, one more thing: How are backups supposed to work? I just followed all the instructions. A cluster-backup-bjob is created but does not run. Also, I cant see any output from any sidecar etc.

I setup a 5-min-schedule for testing: */5 * * * *

[AMecea]

The schedule is formed from six fields (so you should have something like 0 */5 * * * *), check godoc of the cron library that we use.

And make sure that you are using the version v0.2.2. Please let me know if it works.

[micronax]

OK, updated the schedule. Now waiting.

Do I need to upgrade the clusters as well? I upgraded the helm chart which triggered a successful recreation of the operator but the cluster itself stayed untouched.

[AMecea]

Just the operator should be updated the cluster is fine. Maybe update the scheduler of the cluster if needed.

[micronax]

It seems like it is not yet working :(

cluster.yaml

apiVersion: mysql.presslabs.org/v1alpha1
kind: MysqlCluster
metadata:
  name: cluster
spec:
  replicas: 2
  secretName: cluster-secret

  backupSchedule: '0 */5 * * * *'
  backupSecretName: cluster-backup-secret
  backupUri: gs://*****/
  backupScheduleJobsHistoryLimit: 10

  volumeSpec:
    accessModes: [ "ReadWriteOnce" ]
    resources:
      requests:
        storage: 20Gi

cluster-backup.yaml

apiVersion: mysql.presslabs.org/v1alpha1
kind: MysqlBackup
metadata:
  name: cluster-backup

spec:
  clusterName: cluster
  backupUri: gs://*****/
  backupSecretName: cluster-backup-secret

The secrets are also created how described in the README.

A few more debugging:

$ kubectl get mysqlbackup --namespace mysql
NAME             AGE
cluster-backup   8m

$ kubectl describe backup cluster-backup --namespace mysql
error: the server doesn't have a resource type "backup"

$ kubectl describe mysqlbackup --namespace mysql
Name:         cluster-backup
Namespace:    mysql
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"mysql.presslabs.org/v1alpha1","kind":"MysqlBackup","metadata":{"annotations":{},"name":"cluster-backup","namespace":"mysql"},"spec":{"ba...
API Version:  mysql.presslabs.org/v1alpha1
Kind:         MysqlBackup
Metadata:
  Creation Timestamp:  2018-12-19T15:15:33Z
  Generation:          1
  Resource Version:    4407591
  Self Link:           /apis/mysql.presslabs.org/v1alpha1/namespaces/mysql/mysqlbackups/cluster-backup
  UID:                 ee55435c-03a0-11e9-9623-901b0ed5b6c1
Spec:
  Backup Secret Name:  cluster-backup-secret
  Backup URL:          gs://*******/cluster-2018-12-19T15:15:33.xbackup.gz
  Backup Uri:          gs://*******/
  Cluster Name:        cluster
Status:
Events:
  Type    Reason              Age   From                    Message
  ----    ------              ----  ----                    -------
  Normal  JobSyncSuccessfull  9m    mysqlbackup-controller  *v1.Job mysql/cluster-backup-bjob created successfully

Misconfiguration or bug?

[AMecea]

The config is fine. Can you show me some logs from the operator container, related to this? Does it work recurrent backups for a newly created cluster?

[AMecea]

Also while testing on my end I found a bug the cleanup does not delete the right backups, it's deleting the newly created backups instead of the old ones. So don't use backupScheduleJobsHistoryLimit: 1 but instead use backupScheduleJobsHistoryLimit: null.

I will reopen this issue while this is solved.

[micronax]

Just found out that after the upgrade the whole setup is broken:

Orchestrator keeps throwing error messages like:

[martini] Started GET /api/cluster/cluster.mysql for 10.42.3.59:35542
2018-12-20 15:03:09 ERROR Unable to determine cluster name. clusterHint=cluster.mysql
[martini] Completed 500 Internal Server Error in 8.181376ms
[martini] Started GET /api/discover/cluster-mysql-0.cluster-mysql-nodes.mysql/3306 for 10.42.3.59:35542
[martini] Completed 500 Internal Server Error in 14.348477ms
[martini] Started GET /api/discover/cluster-mysql-1.cluster-mysql-nodes.mysql/3306 for 10.42.3.59:35542
2018-12-20 15:03:09 ERROR x509: certificate is valid for MySQL_Server_5.7.23-24_Auto_Generated_Server_Certificate, not cluster-mysql-0.cluster-mysql-nodes.mysql
2018-12-20 15:03:09 ERROR x509: certificate is valid for MySQL_Server_5.7.23-24_Auto_Generated_Server_Certificate, not cluster-mysql-1.cluster-mysql-nodes.mysql
[martini] Completed 500 Internal Server Error in 18.537356ms

Also the cluster lost its connection to the operator / orchestrator with:

[Note] Access denied for user 'orchestrator'@'10.42.4.139' (using password: YES)

while I never changed any passwords or secrets...

[AMecea]

I know this issue, you have to use --reuse-values (in helm command) to keep the orchestrator credentials unchanged when doing an upgrade.

To fix this you can scale the cluster to 0 (replicas) then back to normal. This will reupdate the orchestrator credentials.

Please let me know if this works or you encounter any problems,

[micronax]

OK thank you.. I already re-installed the mysql-operator. But now I get the following error during cluster init 🙈

--initialize specified but the data directory has files in it. Aborting.

I deleted the whole namespace, including volumes, secrets and config maps.