dd Backup only for the first two partitions #686

bismosa · 2023-10-08T13:16:10Z

bismosa
Oct 8, 2023

I just switched from an SD card to an SSD.
So far I have always carried out my backups on the PC as an image backup. I like this type of backup because a restore can be done in just a few minutes.

However, with a 500GB SSD this is no longer so easy. You can shrink the image with pishrink, but you have to save the entire image first. With a slightly weaker WiFi connection, this takes many hours ore days.

With the option
DEFAULT_DD_BACKUP_SAVE_USED_PARTITIONS_ONLY
https://www.linux-tips-and-tricks.de/en/faq/#a16
I could shrink the partition and create smaller images. It's also great for home automation on my Raspberry.

However, I still have a Raspberry running as a surveillance camera. Here I would like to create 3 partitions:
boot
root (8GB)
data (240GB)

I don't need an extra backup of the data partition.

Is it possible to back up only the first two partitions with dd?
Did I perhaps miss an option?

Many thanks for your great work on this project!
Best Regards!

framps · 2023-10-13T03:55:55Z

framps
Oct 13, 2023
Maintainer

You have to use either tar or rsync. Then you can use option IGNORE_ADDITIONAL_PARTITIONS to backup the first two partitions only

0 replies

bismosa · 2023-10-14T14:46:24Z

bismosa
Oct 14, 2023
Author

I already use an rsync backup. This also works very well.
I would also like to create a dd backup so that I can create a new SSD or memory card on the Windows PC as quickly as possible if necessary.
I was hoping that it might not only work with unpartitioned storage space.

So I'm expanding my question into a feature request.

I just tried out whether this could be done with dd.
First I checked the partition information

root@pi:~# fdisk -lu /dev/sda
Disk /dev/sda: 238,47 GiB, 256060514304 bytes, 500118192 sectors
Disk model: SSD 840 PRO Seri
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x1919906e

Device     Boot    Start       End   Sectors   Size Id Type
/dev/sda1           8192    532479    524288   256M  c W95 FAT32 (LBA)
/dev/sda2         532480  26511359  25978880  12,4G 83 Linux
/dev/sda3       26511360 500117503 473606144 225,8G  7 HPFS/NTFS/exFAT

So i get the End-Sector from the second partition: 26511359
Now i can make a backup with count (i have to add 1):
dd if=/dev/sda bs=512 count=26511360 of=/backup/backup.img

/backup was mounted to /dev/sda3

I have now restored the backup to an other SSD and it works. Only the /dev/sda3 had to be formatted again.

2 replies

framps Oct 15, 2023
Maintainer

Please check whether https://github.com/framps/raspiBackup/blob/master/helper/raspiBackupRestore2Image.sh works for you ;-)

bismosa Oct 20, 2023
Author

Very interesting script.
Will be even better than creating a new DD backup.
Unfortunately I haven't managed to use the script yet. There is always an error:

root@server:/media/HDD1/raspiBackup#  bash raspiBackupRestore2Image.sh server/server-rsync-backup-20231016-001001
raspiBackupRestore2Image.sh v0.1.8, $/$ -
===> Backup source disk size: 15193 (MiB)
0+0 Datensätze ein
0+0 Datensätze aus
0 bytes copied, 0,000543853 s, 0,0 kB/s
Checking that no-one is using this disk right now ... OK

Disk /dev/loop0: 14,9 GiB, 15931015168 bytes, 31115264 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

>>> Script header accepted.
>>> Script header accepted.
>>> Script header accepted.
>>> Script header accepted.
>>> Created a new DOS disklabel with disk identifier 0x0a3ee2b5.
/dev/loop0p1: Created a new partition 1 of type 'W95 FAT32 (LBA)' and of size 14,9 GiB.
/dev/loop0p2: Done.

New situation:
Disklabel type: dos
Disk identifier: 0x0a3ee2b5

Device       Boot Start      End  Sectors  Size Id Type
/dev/loop0p1       2048 31115263 31113216 14,9G  c W95 FAT32 (LBA)

The partition table has been altered.
Calling ioctl() to re-read partition table.
Re-reading the partition table failed.: Das Argument ist ungültig
The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or   kpartx(8).
Syncing disks.
===> Reloading new partition table
===> Restoring backup into server/server-rsync-backup-20231016-001001/server-backup.dd
--- RBK0009I: server: raspiBackup.sh V0.6.8 - 2023-08-22 (7c4a713) Fr 20. Okt 16:32:40 CEST 2023 gestartet.
!!! RBK0143W: Ziel /dev/loop0 passt nicht zu dem Backup. Partitionierung wird trotzdem vorgenommen.
!!! RBK0065W: Gerät /dev/loop0 wird repartitioniert und die gesamten Daten werden gelöscht.
--- RBK0067I: Momentane Partitionen auf /dev/loop0:
Number  Start   End      Size     Type     File system  Flags
 1      1,05MB  15931MB  15930MB  primary               lba
--- RBK0066I: Gerät /dev/loop0 wird überschrieben mit der gesicherten Boot- und Rootpartition.
--- RBK0069I: Bootpartition /dev/loop0p1 wird formatiert und erhält die zurückgespielte Bootpartition.
--- RBK0070I: Rootpartition /dev/loop0p2 wird formatiert und erhält die zurückgespielte Rootpartition.
--- RBK0038I: Bist Du sicher? j/N
--- RBK0050I: Backup wird von /media/HDD1/raspiBackup/server/server-rsync-backup-20231016-001001 zurückgespielt.
!!! RBK0139W: Partitionserstellung ignoriert Fehler.
--- RBK0053I: Erste Partition (Bootpartition) wird auf /dev/loop0p1 zurückgespielt.
??? RBK0185E: Rootpartitionserstellung endet fehlerhaft mit RC 1.
--- RBK0033I: Bitte warten bis aufgeräumt wurde.
??? RBK0077E: Restore wurde fehlerhaft beendet. Siehe vorhergehende Fehlermeldungen.
--- RBK0010I: server: raspiBackup.sh V0.6.8 - 2023-08-22 (7c4a713) Fr 20. Okt 16:56:21 CEST 2023 beendet mit Returncode 117.
--- RBK0026I: Debug Logdatei wurde in /root/raspiBackup.logr gesichert.
mount: /mnt1: special device /dev/loop0p2 does not exist.
grep: /mnt1/etc/fstab: Datei oder Verzeichnis nicht gefunden
umount: /mnt1: not mounted.
??? Unrecoverable error. Unable to find PARTUUID of / in image
??? Error 1 received
--- Cleaning up

Here is the Full Log:
raspiBackup.logr.log

framps · 2023-10-22T14:40:50Z

framps
Oct 22, 2023
Maintainer

Are you sure your rsync backup is correct? According the debug log you have only the boot partition but no root partition in the backup (/dev/mmcblk0p2 is missing).

20231020-170904 DBG 3879:                  *** cat /media/HDD1/raspiBackup/@HOSTNAME@/@HOSTNAME@-rsync-backup-20231016-001001/@HOSTNAME@-backup.sfdisk
20231020-170904 DBG 3879:                      label: dos
20231020-170904 DBG 3879:                      label-id: 0x0a3ee2b5
20231020-170904 DBG 3879:                      device: /dev/mmcblk0
20231020-170904 DBG 3879:                      unit: sectors
20231020-170904 DBG 3879:                      
20231020-170904 DBG 3879:                      /dev/mmcblk0p1 : start=        2048, size=    31113216, type=c

1 reply

bismosa Oct 22, 2023
Author

I think so. However, it is an rsync backup.
Is this only possible with a dd backup?

framps · 2023-10-22T16:20:05Z

framps
Oct 22, 2023
Maintainer

The script can use a rsync or tar backup as input and creates a dd backup. A normal backup hast two partitions defined in the sfdisk file. You only have the boot partition defined. That's why I think the backup is somewhere incorrect and caues the issue. Do you have the backup creation debug log handy?

0 replies

bismosa · 2023-10-24T14:11:45Z

bismosa
Oct 24, 2023
Author

Thank you for the support and pointing me in the right direction!
The backup was always created completely. However, I had a general configuration error!
I use a raspberry pi 3 b+ and just recently switched to using an SSD.
For this I created an SD card with the bootcode.bin file.

The raspberry pi 3 b+ does not need this SD card. Because this was present, the SD card was mounted under /boot.
Only the memory card with one partition was then included in the sfdisk file.

Now that I have removed the SD card and created a new backup, it now works with this script. The sfdisk file now also contains 2 partitions.

The only drawback with the script... there must be as much memory (temporarily) free on the backup path as the size of the entire SSD.

Thank you.

0 replies

framps · 2023-10-24T14:33:03Z

framps
Oct 24, 2023
Maintainer

The only drawback with the script... there must be as much memory (temporarily) free on the backup path as the size of the entire SSD.

Yes, That's right. You're the first one who uses the script with more than 2 partitions. The size of all partitions is accumulated right now. Just update the script accordingly and accumulate the first two partitions only 😉

0 replies

framps · 2023-10-24T15:52:41Z

framps
Oct 24, 2023
Maintainer

I just created the required update for the script for your convenience. But please note: It's untested.

        local partitionregex="/dev/.*[p]?([0-9]+)[^=]+=[^0-9]*([0-9]+)[^=]+=[^0-9]*([0-9]+)[^=]+=[^0-9a-z]*([0-9a-z]+)"
	local partitionNo=0
	local sumSize=0

	while IFS="" read line; do
		if [[ -z $line ]]; then
			continue
		fi

		if [[ $line =~ $partitionregex ]]; then
			local p=${BASH_REMATCH[1]}
			local start=${BASH_REMATCH[2]}
			local size=${BASH_REMATCH[3]}
			local id=${BASH_REMATCH[4]}

			if [[ $id == 85 || $id == 5 ]]; then
				continue
			fi

			if [[ $sumSize == 0 ]]; then
				sumSize=$((start+size))
			else
				(( sumSize+=size ))
			fi
	   		(( partitionNo++ ))
		fi
  	       if (( partitionNo >= 2 )); then
 		      break;
	      fi

       done < "$file"

Please let me know whether it works for you.

1 reply

bismosa Oct 24, 2023
Author

Thank you.
I probably expressed myself incorrectly. Excuse me!
I think the change is not needed at all?
If I have the option
DEFAULT_IGNORE_ADDITIONAL_PARTITIONS=1
(and I think I even have to use that with an rsync backup) then the additional partition doesn't even appear in the sfdisk file?

So only the storage space of the first two partitions needs to be available temporarily.

I have now tried to restore the created dd file with win32diskimager on a fresh SD scard. Runs perfectly.

framps · 2023-10-24T19:45:14Z

framps
Oct 24, 2023
Maintainer

The sfdisk file created by raspiBackup includes all partitions of the system which was saved. The unmodified script calculates the required dd image size thus with all partitions and allocates the dd file with this size. Later on pishrink reduces the size. The fix I provided will allocate the dd file only by the size of the first two partitions and ignores any other following partitions.

It's up to you to apply my fix. The script is provided as is and not supported. I had some spare time and thought you may be interested in the fix. Just use it or forget the fix 😃

0 replies

bismosa · 2023-10-25T07:23:24Z

bismosa
Oct 25, 2023
Author

The sfdisk file created by raspiBackup includes all partitions of the system which was saved

But that's not the case for me?
I have:

root@camTEST:~# ls /dev/sd*
/dev/sda  /dev/sda1  /dev/sda2  /dev/sda3
root@camTEST:~# fdisk -lu /dev/sda
Disk /dev/sda: 238,47 GiB, 256060514304 bytes, 500118192 sectors
Disk model: SSD 840 PRO Seri
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x1919906e

Device     Boot    Start       End   Sectors   Size Id Type
/dev/sda1           8192    532479    524288   256M  c W95 FAT32 (LBA)
/dev/sda2         532480  26511359  25978880  12,4G 83 Linux
/dev/sda3       26511360 500117503 473606144 225,8G 83 Linux

The sfdisk file::

label: dos
label-id: 0x1919906e
device: /dev/sda
unit: sectors
sector-size: 512

/dev/sda1 : start=        8192, size=      524288, type=c
/dev/sda2 : start=      532480, size=    25978880, type=83

And it doesn't matter which file system I have as the third partition. I tested it with both NTFS and ext4

It's up to you to apply my fix. The script is provided as is and not supported. I had some spare time and thought you may be interested in the fix. Just use it or forget the fix

I am very interested in a good and, above all, generic solution that also works on multiple systems.
I wouldn't have been able to manage this script like that.
I would also like to test this, but I only have the 2 entries on 3 partitions. I can't add the missing one manually, then I get the error:

/dev/loop0p3: Start sector 26511360 out of range.
Failed to add #3 partition: Das numerische Ergebnis ist außerhalb des gültigen Bereiches

This is also logical for me, since loop0:

Disk /dev/loop0: 12,64 GiB, 13573816320 bytes, 26511360 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

The unmodified script works for me without any problems 😃

0 replies

framps · 2023-10-25T08:15:22Z

framps
Oct 25, 2023
Maintainer

And it doesn't matter which file system I have as the third partition

Hm ... that's interesting. I thought you will get all partitions in the sfdisk file.

The unmodified script works for me without any problems 😃

Ok. Then enjoy the script 😃

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dd Backup only for the first two partitions #686

{{title}}

Replies: 10 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

dd Backup only for the first two partitions #686

bismosa Oct 8, 2023

Replies: 10 comments · 4 replies

framps Oct 13, 2023 Maintainer

bismosa Oct 14, 2023 Author

framps Oct 15, 2023 Maintainer

bismosa Oct 20, 2023 Author

framps Oct 22, 2023 Maintainer

bismosa Oct 22, 2023 Author

framps Oct 22, 2023 Maintainer

bismosa Oct 24, 2023 Author

framps Oct 24, 2023 Maintainer

framps Oct 24, 2023 Maintainer

bismosa Oct 24, 2023 Author

framps Oct 24, 2023 Maintainer

bismosa Oct 25, 2023 Author

framps Oct 25, 2023 Maintainer

bismosa
Oct 8, 2023

Replies: 10 comments 4 replies

framps
Oct 13, 2023
Maintainer

bismosa
Oct 14, 2023
Author

framps Oct 15, 2023
Maintainer

bismosa Oct 20, 2023
Author

framps
Oct 22, 2023
Maintainer

bismosa Oct 22, 2023
Author

framps
Oct 22, 2023
Maintainer

bismosa
Oct 24, 2023
Author

framps
Oct 24, 2023
Maintainer

framps
Oct 24, 2023
Maintainer

bismosa Oct 24, 2023
Author

framps
Oct 24, 2023
Maintainer

bismosa
Oct 25, 2023
Author

framps
Oct 25, 2023
Maintainer