strange fuzzy results, deletes what should be preserved

[epcim]

Well can't get the point why, but it's happening.

[root@db4 galera_backup]# cat /etc/rotate-backups.ini 
[/galera_backup]
hourly = 3
daily = 1
weekly = 0
monthly = 0
yearly = 0
ionice = idle

Mind the "Deleting directory" in the second example. The only change is in "hour" in "timestamp".

[root@db4 galera_backup]# /bin/rotate-backups -nv -x '*'$(date  --date="1 days ago" "+%Y-%m-%d")'*'
2016-03-17 12:54:35 db4.sl.example.lab rotate_backups.cli[36454] INFO Performing a dry run (because of -n option) ..
2016-03-17 12:54:35 db4.sl.example.lab rotate_backups[36454] DEBUG Using configuration file /etc/rotate-backups.ini ..
2016-03-17 12:54:35 db4.sl.example.lab rotate_backups[36454] DEBUG Using configuration file /etc/rotate-backups.ini ..
2016-03-17 12:54:35 db4.sl.example.lab rotate_backups[36454] INFO Scanning directory for backups: /galera_backup
2016-03-17 12:54:35 db4.sl.example.lab rotate_backups[36454] DEBUG Excluded 'galera_backup_db4.sl.example.lab_2016-03-16_12-00' (it matched the exclude list).
2016-03-17 12:54:35 db4.sl.example.lab rotate_backups[36454] DEBUG Excluded 'galera_backup_db4.sl.example.lab_2016-03-16_22-00' (it matched the exclude list).
2016-03-17 12:54:35 db4.sl.example.lab rotate_backups[36454] INFO Found 3 timestamped backups in /galera_backup.
2016-03-17 12:54:35 db4.sl.example.lab rotate_backups[36454] INFO Preserving /galera_backup/galera_backup_db4.sl.example.lab_2016-03-17_12-00 (matches 'daily' retention period) ..
2016-03-17 12:54:35 db4.sl.example.lab rotate_backups[36454] INFO Preserving /galera_backup/galera_backup_db4.sl.example.lab_2016-03-17_16-00 (matches 'hourly' retention period) ..
2016-03-17 12:54:35 db4.sl.example.lab rotate_backups[36454] INFO Preserving /galera_backup/galera_backup_db4.sl.example.lab_2016-03-17_18-00 (matches 'hourly' retention period) ..
2016-03-17 12:54:35 db4.sl.example.lab rotate_backups[36454] INFO Nothing to do! (all backups preserved)
[root@db4 galera_backup]# 

[root@db4 galera_backup]# date
Thu Mar 17 12:57:34 CET 2016

[root@db4 galera_backup]# mv /galera_backup/galera_backup_db4.sl.example.lab_2016-03-17_18-00 /galera_backup/galera_backup_db4.sl.example.lab_2016-03-17_10-00


[root@db4 galera_backup]# /bin/rotate-backups -nv -x '*'$(date  --date="1 days ago" "+%Y-%m-%d")'*'
2016-03-17 12:58:01 db4.sl.example.lab rotate_backups.cli[37843] INFO Performing a dry run (because of -n option) ..
2016-03-17 12:58:01 db4.sl.example.lab rotate_backups[37843] DEBUG Using configuration file /etc/rotate-backups.ini ..
2016-03-17 12:58:01 db4.sl.example.lab rotate_backups[37843] DEBUG Using configuration file /etc/rotate-backups.ini ..
2016-03-17 12:58:01 db4.sl.example.lab rotate_backups[37843] INFO Scanning directory for backups: /galera_backup
2016-03-17 12:58:01 db4.sl.example.lab rotate_backups[37843] DEBUG Excluded 'galera_backup_db4.sl.example.lab_2016-03-16_12-00' (it matched the exclude list).
2016-03-17 12:58:01 db4.sl.example.lab rotate_backups[37843] DEBUG Excluded 'galera_backup_db4.sl.example.lab_2016-03-16_22-00' (it matched the exclude list).
2016-03-17 12:58:01 db4.sl.example.lab rotate_backups[37843] INFO Found 3 timestamped backups in /galera_backup.
2016-03-17 12:58:01 db4.sl.example.lab rotate_backups[37843] INFO Preserving /galera_backup/galera_backup_db4.sl.example.lab_2016-03-17_10-00 (matches 'daily' retention period) ..
2016-03-17 12:58:01 db4.sl.example.lab rotate_backups[37843] INFO Deleting directory /galera_backup/galera_backup_db4.sl.example.lab_2016-03-17_12-00 ..
2016-03-17 12:58:01 db4.sl.example.lab rotate_backups[37843] INFO Preserving /galera_backup/galera_backup_db4.sl.example.lab_2016-03-17_16-00 (matches 'hourly' retention period) ..


# root owned are created manually, for testing
# the 11-00-00 was added later
#
[root@db4 galera_backup]# ll
total 12
drwxr-xr-x 6 mysql mysql 4096 Mar 16 00:08 galera_backup_db4.sl.example.lab_2016-03-16_12-00
drwxr-xr-x 6 mysql mysql 4096 Mar 16 20:33 galera_backup_db4.sl.example.lab_2016-03-16_22-00
drwxr-xr-x 2 root  root     6 Mar 17 12:54 galera_backup_db4.sl.example.lab_2016-03-17_10-00
drwxr-xr-x 2 root  root     6 Mar 17 12:59 galera_backup_db4.sl.example.lab_2016-03-17_11-00-00
drwxr-xr-x 6 mysql mysql 4096 Mar 16 20:33 galera_backup_db4.sl.example.lab_2016-03-17_12-00
drwxr-xr-x 2 root  root     6 Mar 17 12:54 galera_backup_db4.sl.example.lab_2016-03-17_16-00

[xolox]

Hi Petr, thanks for the feedback and sorry if rotate-backups unexpectedly rotated away more backups than you expected it to. This behavior can be explained (given the current algorithm for deciding which backups remain and which are rotated away) but that doesn't necessarily mean the behavior is satisfactory for you and/or other users.

Conceptual explanation

When you configure hourly=3 it means that the first backup in each of the past three hours (relative to the most recent backup) is preserved. Backups that are more than three hours old will be deleted unless they are matched by another rotation frequency.

Applied to the first example

In your first example the following three timestamps are used:

1. 2016-03-17 12:00
2. 2016-03-17 16:00
3. 2016-03-17 18:00

18:00 minus three hours is 15:00 which matches backups 2 and 3 but not 1, however backup 1 fits in the slot for the daily backup and so is preserved as well.

Applied to the second example

In your second example the following three timestamps are used:

1. 2016-03-17 10:00
2. 2016-03-17 12:00
3. 2016-03-17 16:00

16:00 minus three hours is 13:00 which matches only backup 3, however backup 1 fits in the slot for the daily backup and so is preserved as well.

[xolox]

I only just realized that you came to the same conclusion when you opened issue #3. I'll see if I can add an option to change this behavior as suggested there.

[epcim]

Exactly but anyway, thanks for explanation. Your code is rock solid. I understood the documentation wrong way. I vote for make the example above a part of the documentation.

The major difference is that I do not create backups "hourly" or so.. but at specified timers. Thus it doesn't do what expected - (as you wrote) the "default algorithm" do not match all user needs. #3 address that to make your piece of code even more usable. Thanks.