Best way of backing up whole DietPi system (SSD)

[LaurenceGough]

Hi guys,

Unfortunately many months of work has disappeared after running the command rpi-update (yes beginners mistake) it soft bricked my Pi and nothing worked so I had to start fresh with a new image.

Sadly my DietPi backup I made didn't work as it warned of a UUID mismatch "warning uuids of the backup and current system differ", I tried anyway and yes it didn't work.

Going forward I do not wish to repeat this...

Sadly even my Portainer backup was completely useless. So I having to setup everything from complete scratch again.

What method are you guys using for a full SSD backup or "clone"?

[Joulinar]

I guess the question is going into some direction as this one on our forum https://dietpi.com/forum/t/create-a-backup-img-x86/14014

You could use dd to create an image out of the entire SSD

[MichaIng]

On another DietPi system, e.g. also from a VM or so, the DietPi-Imager can be used to create a compressed image from an attached drive. It shrinks filesystem and partition to create a minimal raw image file, before further compressing it as 7z archive:

bash -c "$(curl -sSfL 'https://raw.githubusercontent.com/MichaIng/DietPi/master/.build/images/dietpi-imager')"

From next release on (this Sunday), you can also skip the 7z-compression:

export SKIP_ARCHIVE=1
bash -c "$(curl -sSfL 'https://raw.githubusercontent.com/MichaIng/DietPi/master/.build/images/dietpi-imager')"

or have the contained partition and filesystem shrunk only, so that a cloning tool can be used to clone the drive to a smaller one:

export SHRINK_ONLY=1
bash -c "$(curl -sSfL 'https://raw.githubusercontent.com/MichaIng/DietPi/master/.build/images/dietpi-imager')"

Also from next release on, the dietpi-fs_partition_resize.service is enabled automatically on original and new image/system, so that partition and filesystem are expended automatically on next boot.

[jalsco]

@MichaIng Are there examples showing how to RESTORE from an image copy of system made using dietpi-imager ?

[Joulinar]

usually, the whole image needs to be flashed back to SD card, usb pen stick or SSD. It's a full image of the entire system.

[jalsco]

Also @MichaIng - I run dietpi-sync to manually copy my backup partition to usb stick... The dry-run looks good so I let it run ( about 60gb to copy ) , when I drop back and look at that terminal session I see page after page of chown and mknod commands that give failed: Operation not permitted (1) .... Should I be worried about that ? Is it a problem ?

note: I am logged in as root

[MichaIng]

Looks like the filesystem on the USB stick does not support UNIX permission? All FAT variants do not, NTFS only in an emulated way. So yes, those backups cannot be restored for a functional system anymore with all UNIX permissions lost.

[jalsco]

https://forums.raspberrypi.com/viewtopic.php?t=255134

Ah.....

[MichaIng]

Jep exactly. Whenever you sync more than just some data files like media files, always use a proper filesystem like ext4, so all permissions and ownership, often required for things to work properly, to be kept. Other than noted on the link, NTFS does not really support UNIX permissions. With the ntfs-3g driver, it's emulated, but I'm not sure how reliable this works, and whether it works as well when restoring things on a different system. At least I wouldn't count on it. Downside of course is that Windows can't read the filesystem natively, and I had a hard time finding out that even with recent WSL2 --mount option it isn't trivial to mount USB devices and it isn't possible to mount SD cards :(.

[jalsco]

thanks - yes - I changed it to ext4 and restarted the dietpi-sync and now is proceeding with no errors.
💪😉

[jalsco]

Does dietpi-imager can a copy of the entire disk ?
I only want it to take copy of one partition (the boot partition) - is it possible

So I boot from an SD then

I just want take image copy of nvme0n1p1 - can I do that ? I do not need copies of p5 and p6

I trying to store the output ( img ) onto /dev/sda1 - seems like it always wants to store it on the same partition that it boot from , is that true ?

[jalsco]

I am not sure how to answer. My inexperience on linux is showing through again 😉....
I have already some versions of all the files backed up, courtesy of dietpi-backup (which uses rsync).

So, as well as those, I wanted an img. .. of the content of boot partition (14gb)

My thinking (which may be invalid) goes like this: Assume one day the nvme disk dies - eg. fried unexpectedly by an electrical surge or lightning strike. To recover, some steps like these:

1. buy some new hardware (whatever was fried) - including a new NVMe disk.
2. format the nvme disk it into partitions,
3. then "restore a saved image" so I can boot up "my dietpi system (albeit a bit of an older version)",
4. then restore the latest files from a dietpi-backup to bring it up to date (eg - max of 1 day old ).

That's all I am trying to do - create the image to use in step 3 - in the case of my actual disk getting fried. In that case, I first have to make a bootable Dietpi system. My thinking is that I would restore "an image of my system" - then it would already have all my customization, my ssh keys etc, and my software configs and everything I have added or changed (as at the point in time that I created that image) .

Seems like not easily done - to make an IMG from the used 14gb on that 160gb partition... 🤷🏻‍♂️

[MichaIng]

A file/rsync backup indeed means that the system needs to be still bootable or the partitioning in takt at least. Although, dietpi-backup has the ability to:

1. Restore automatically on first boot of a fresh system via AUTO_SETUP_BACKUP_RESTORE in dietpi.txt.
2. It adjust the boot configs and /etc/fstab automatically to match the new UUIDs/PARTUUIDs, which is the missing but important meta data that is missing in a pure file/rsync backup.

What is the NVMe actually used for? You said "boot partition", but I do not see any of it's partitions being mounted to /boot? For now it looks like all is on a single partition on SD card and NVMe contains so system-relevant files? Or was it mounted to an external system?

[jalsco]

If I have the SD card inserted - it boots from that. Can then - easily mount and manipulate the NVMe partitions (eg make an image backup) . It was booted from the SD when I ran the lsblk above.

Usually I do not have the SD card inserted. I keep it stored with my backups.

Usual case - day to day operations - is no SD inserted. In that case it boots Dietpi from the NVMe. The boot partition is : /dev/nvme0n1p1

[MichaIng]

/dev/nvme0n1p1 contains as well the root filesystem, i.e. boot partition and root partition are the same, right? In case of using dietpi-imager, it would be easier if the NVMe contained a single partition only. The files from the other two partitions could be well stored somewhere into the root filesystem as well. Then dietpi-imager can automatically shrink this filesystem + partition to a minimum before the image file is created.

[jalsco]

I do not want to bring p5 and p6 into the DietPi partition (p1) because they are NFS partitions used elsewhere and can be very full at times, and also they do not need to be backed up. For now I will just avoid using DietPi-Imager.

Does p1 contain all those things ? " as well the root filesystem, i.e. boot partition and root partition are the same " ?? I guess so... It does boot from it. And there are no other disks attached to the system. It obviously contains whatever needed to boot... Here is parted :

[jalsco]

Ah, and the DietPi-Imager stores the resulting disk image+archive always into the current working directory/where you cded before you executed the script.

Thanks - good to know how to set the destination !

I think that I am starting to understand the problem: If I were to use:

dd if=/dev/nvme0n1p1 of=/path/to/image.img bs=1M status=progress

It will still need 160gb of space to put the file somewhere - even if later the zip process will crunch that down to something under 14gb. So without that amount of space at the beginning step, it is just never going to work....

Is that a fair summary of the problem I am facing ?

[MichaIng]

Yes that is the difference between a bit-by-bit disk/partition backup which dd does (including all empty space, but also all filesystem and in case partition table meta data and everything outside of partitions/filesystems), to rsync/cp etc which copies actual files.

What you could do to avoid high disk usage:

dd if=/dev/nvme0n1p1 bs=1M status=progress | xz > nvme0n1p1.img.xz

[jalsco]

Would that mean I can avoid the gparted steps described below (i.e. shrink from 160gb -> 16 then do DD and then resize 16 -> 160) ? That would be totally awesome.... I will try it next time I take an image backup.

[MichaIng]

Yes exactly. The backup will take notably longer, but usually that doesn't matter much.

[jalsco]

I guess I could gparted the 160gb down to 16gb...
Then do the dd if=/dev/nvme0n1p1 of=/path/to/image.img bs=1M status=progress
Then gparted it back up to full size

update: ok - I did all that and it worked 😁👍

• I added in a step of using zerofree , to zero out all the unused space on the partition, actually even before I shrunk the partition.
• Using status=progress on the dd command is essential - as it takes a very long time.
• 7-zip absolutely crushed the image file . It reduced it to about 25-30% of its original size.
  • The partition contained: 15.6gb of data
  • I shrunk the partition to new size of: 19,646 mb - with gparted
  • dd command produced image file of size: 20600324096, ie 20gb
  • the zip file is only 4.8gb

-rw-r--r-- 1 root   root 20600324096 Dec 18 08:57 nvme0n1p1.img
[Sun Dec 18] root@batopi: /media/sandisk$ 7zr a archive.7z nvme0n1p1.img

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=C.UTF-8,Utf16=on,HugeFiles=on,64 bits,6 CPUs LE)

Scanning the drive:
1 file, 20600324096 bytes (20 GiB)

Creating archive: archive.7z

Items to compress: 1


Files read from disk: 1
Archive size: 5016239250 bytes (4784 MiB)
Everything is Ok
[Sun Dec 18] root@batopi: /media/sandisk$