barman-cloud-restore "seeking backwards is not allowed" #351
Comments
|
@cescobarresi could you please provide the complete backtrace? |
|
@amenonsen here the full Traceback. If it can be usefull I can paste the full DEBUG log, but it basically does the GET of When it has finished the download I get the error in Thanks. |
|
@cescobarresi thanks for the details. I have not yet fully understood the problem. I tried test.py from the page you linked to, and I actually get the same error even with Python 3.9.0 and 3.8.6: The other thing I'm a bit puzzled about is that the error seems dependent on the tarfile containing symlinks, but the tar files uploaded by barman-cloud-backup should not contain symlinks (neither pg_wal nor the symlinks under pg_tblspc are included in the tar files). Since you tested a restore by hand, perhaps you could shed some light on this? Are there some other symlinks? (We've tried to reproduce this using barman-cloud-restore directly, but have not been able to do so.) |
|
I was using a symlik as destination of Following your suggestion I manually downloaded the backup a searched for symlinks inside. The only one I could find i the root "." which in the source is a symlink. barman@srvpsqlutils:/mnt/temp_disk/test_restore$ tar -vtf data.tar.gz | grep lr
lrwxrwxrwx postgres/postgres 0 2019-09-19 11:45 . -> /media/data"dot" is the root of the tarball, here the first few files of the tar: barman@srvpsqlutils:/mnt/temp_disk/test_restore$ tar -vtf data.tar.gz | head
lrwxrwxrwx postgres/postgres 0 2019-09-19 11:45 . -> /media/data
-rw------- postgres/postgres 3 2019-09-19 11:29 ./PG_VERSION
-rw------- postgres/postgres 88 2019-09-19 11:29 ./postgresql.auto.conf
-rw-r----- postgres/postgres 30 2021-07-17 00:00 ./current_logfiles
drwx------ postgres/postgres 0 2019-09-19 11:27 lost+found/
drwx------ postgres/postgres 0 2019-09-19 11:29 pg_tblspc/
drwx------ postgres/postgres 0 2021-07-17 03:46 global/
-rw------- postgres/postgres 16384 2021-04-12 02:53 global/2695
-rw------- postgres/postgres 16384 2020-12-14 16:57 global/2397
-rw------- postgres/postgres 24576 2021-04-12 02:51 global/1261_fsmIn the source server, where root@srvpsql02:/var/lib/postgresql/11# ls -la
total 8
drwxr-xr-x 2 postgres postgres 4096 Sep 19 2019 .
drwxr-xr-x 10 postgres postgres 4096 Jul 5 10:45 ..
lrwxrwxrwx 1 postgres postgres 11 Sep 19 2019 main -> /media/dataI don't find it wrong to have With regard to you having the python bug in you system: at the end of the issue bpo-12800 you can see that the fix has been merged in python master, 3.9 and 3.8. Depending on the release you are running you may have the bug in The thing that is confusing me is that the python I have installed doesn't have the barman@srvpsqlutils:~$ /usr/bin/python3 --version
Python 3.8.10
barman@srvpsqlutils:~$ /usr/bin/python3 test.py
barman@srvpsqlutils:~$ tar -tvf test.tar
-rw-rw-r-- barman/barman 12 2021-07-23 14:28 message.txt
lrwxrwxrwx barman/barman 0 2021-07-23 14:28 symlink.txt -> message.txt
barman@srvpsqlutils:~$ cat /usr/bin/barman-cloud-restore
#!/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'barman==2.12','console_scripts','barman-cloud-restore'
__requires__ = 'barman==2.12'
import re
import sys
from pkg_resources import load_entry_point
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
sys.exit(
load_entry_point('barman==2.12', 'console_scripts', 'barman-cloud-restore')()
)
$ python3 /usr/bin/barman-cloud-restore -P barman-cloud s3://backup-bucket srvpsql02 20210718T040002 /var/lib/postgresql/11/main -v
2021-07-23 14:17:39,666 [986373] INFO: Found credentials in shared credentials file: ~/.aws/credentials
2021-07-23 14:17:40,191 [986373] INFO: Found file from backup '20210718T040002' of server 'srvpsql02': srvpsql02/base/20210718T040002/data.tar.gz
2021-07-23 14:25:47,467 [986373] ERROR: Barman cloud restore exception: seeking backwards is not allowed |
It's perfectly valid to have PGDATA, say /var/lib/postgresql/13/main, be a symlink to /mnt/somedir. But there were two significant problems with how barman-cloud-backup handled this case. 1. If you run `barman-cloud-restore … /some/dir`, you would expect that directory to BE your data directory, not a symlink to somewhere else. Even if you were OK with the symlink, there would be no way to allow you to specify a different directory, the way it is now possible to do for tablespaces. 2. As reported in #351, there are Python-version-dependent problems with extracting from a tarfile that contains symlinks. Using Python 3.8.10 or newer may fix the problem or some part of it, but barman-cloud-restore can still trigger the "seeking backwards is not allowed" error under 3.8.10 with a backup from a server where PGDATA is a symlink. The existing behaviour being unreasonable and problematic—and clearly not anticipated by the code—we change _backup_copy to check if PGDATA is a symlink, and if so, just backup whatever it points to. Closes #351
|
@cescobarresi Thanks! Your mention of PGDATA itself being a symlink helped me to reproduce the problem. Here's a quick update.
By fixing the problem at the source, in barman-cloud-backup, we avoid the tarfile version-dependent behaviour altogether. I have verified this fix, and it works OK. If you have a chance, please try it in your environment. Of course, you will have to take a new backup, you will not be able to restore the old one. |
Hello,
when running
barman-cloud-restoreI get the following error:ERROR: Barman cloud restore exception: seeking backwards is not allowedI researched online this error and found that it is actually a python error in the tarfile module. There is a bug report detailing this error and also a "test.py" to verify the bug.
Using the provided "test.py" I verified that I had a faulty python version. I have upgraded to Ubuntu 20.04 with Python 3.8.10. Now running the "test.py" doesn't get me the error. But barman-cloud-restore keeps returning the same error.
The archive command we are using is
archive_command = 'barman-cloud-wal-archive -P barman-cloud -z s3://bucket-name/ server-name %p'I tried manually running a restore, without the help of barman-cloud-restore, and found that the backups are correct.
Has any one using
barman-cloud-restoreexperienced this same issue?Thanks
Francesco
The text was updated successfully, but these errors were encountered: