From 7883f7ac6e8142af0dac89b98e4ac8c5f2709543 Mon Sep 17 00:00:00 2001 From: gdha Date: Tue, 23 Apr 2024 14:49:37 +0000 Subject: [PATCH] add new issues to the ReaR User Guide Signed-off-by: gdha --- docs/issues/2021-07-05.2648.issue.closed.md | 446 ++++++ docs/issues/2021-10-18.2698.issue.closed.md | 265 ++++ docs/issues/2023-04-21.2971.issue.closed.md | 427 ++++++ docs/issues/2023-11-01.3063.issue.closed.md | 109 ++ docs/issues/2023-11-17.3086.issue.closed.md | 53 + docs/issues/2023-11-29.3095.issue.closed.md | 86 ++ docs/issues/2023-12-21.3115.issue.closed.md | 192 +++ docs/issues/2023-12-21.3116.issue.closed.md | 388 +++++ docs/issues/2023-12-27.3117.issue.closed.md | 198 +++ docs/issues/2024-01-06.3121.issue.closed.md | 140 ++ docs/issues/2024-01-26.3139.issue.closed.md | 152 ++ docs/issues/2024-02-07.3146.issue.closed.md | 340 +++++ docs/issues/2024-02-08.3148.issue.closed.md | 128 ++ docs/issues/2024-02-08.3150.pr.merged.md | 60 + docs/issues/2024-02-12.3152.issue.closed.md | 151 ++ docs/issues/2024-02-14.3153.pr.merged.md | 64 + docs/issues/2024-02-16.3155.pr.merged.md | 111 ++ docs/issues/2024-02-19.3157.pr.merged.md | 183 +++ docs/issues/2024-02-19.3158.pr.merged.md | 312 ++++ docs/issues/2024-02-19.3159.pr.merged.md | 67 + docs/issues/2024-02-20.3160.pr.merged.md | 139 ++ docs/issues/2024-02-21.3161.issue.open.md | 85 ++ docs/issues/2024-02-22.3162.issue.open.md | 92 ++ docs/issues/2024-02-23.3163.pr.merged.md | 306 ++++ docs/issues/2024-02-27.3164.issue.open.md | 94 ++ docs/issues/2024-02-27.3165.pr.merged.md | 47 + docs/issues/2024-02-27.3166.pr.open.md | 93 ++ docs/issues/2024-02-28.3167.issue.closed.md | 190 +++ docs/issues/2024-02-28.3168.pr.merged.md | 1427 +++++++++++++++++++ docs/issues/2024-02-29.3169.issue.open.md | 112 ++ docs/issues/2024-02-29.3170.issue.closed.md | 596 ++++++++ docs/issues/2024-02-29.3171.pr.open.md | 915 ++++++++++++ docs/issues/2024-03-01.3172.pr.merged.md | 160 +++ docs/issues/2024-03-01.3173.pr.merged.md | 79 + docs/issues/2024-03-06.3174.pr.merged.md | 41 + docs/issues/2024-03-07.3175.pr.open.md | 443 ++++++ docs/issues/2024-03-07.3176.pr.merged.md | 160 +++ docs/issues/2024-03-08.3177.pr.merged.md | 198 +++ docs/issues/2024-03-12.3178.issue.open.md | 327 +++++ docs/issues/2024-03-14.3179.pr.open.md | 149 ++ docs/issues/2024-03-18.3180.issue.closed.md | 190 +++ docs/issues/2024-03-18.3181.pr.closed.md | 68 + docs/issues/2024-03-18.3182.pr.merged.md | 49 + docs/issues/2024-03-18.3183.pr.merged.md | 38 + docs/issues/2024-03-27.3185.issue.open.md | 327 +++++ docs/issues/2024-03-27.3186.issue.open.md | 67 + docs/issues/2024-03-28.3187.issue.open.md | 130 ++ docs/issues/2024-04-01.3188.pr.merged.md | 92 ++ docs/issues/2024-04-02.3189.issue.open.md | 634 ++++++++ docs/issues/2024-04-03.3190.issue.closed.md | 330 +++++ docs/issues/2024-04-03.3191.issue.closed.md | 186 +++ docs/issues/2024-04-04.3192.pr.merged.md | 72 + docs/issues/2024-04-04.3193.issue.open.md | 93 ++ docs/issues/2024-04-04.3194.issue.open.md | 885 ++++++++++++ docs/issues/2024-04-04.3195.issue.closed.md | 206 +++ docs/issues/2024-04-04.3196.pr.closed.md | 41 + docs/issues/2024-04-05.3197.issue.closed.md | 80 ++ docs/issues/2024-04-05.3198.issue.closed.md | 92 ++ docs/issues/2024-04-05.3199.issue.open.md | 183 +++ docs/issues/2024-04-06.3200.issue.open.md | 230 +++ docs/issues/2024-04-07.3201.issue.open.md | 176 +++ docs/issues/2024-04-08.3202.issue.open.md | 240 ++++ docs/issues/2024-04-09.3203.pr.open.md | 825 +++++++++++ docs/issues/2024-04-09.3204.pr.merged.md | 81 ++ docs/issues/2024-04-09.3205.pr.merged.md | 30 + docs/issues/2024-04-09.3206.pr.merged.md | 115 ++ docs/issues/2024-04-10.3207.issue.open.md | 82 ++ docs/issues/2024-04-19.3208.issue.open.md | 78 + 68 files changed, 15145 insertions(+) create mode 100644 docs/issues/2021-07-05.2648.issue.closed.md create mode 100644 docs/issues/2021-10-18.2698.issue.closed.md create mode 100644 docs/issues/2023-04-21.2971.issue.closed.md create mode 100644 docs/issues/2023-11-01.3063.issue.closed.md create mode 100644 docs/issues/2023-11-17.3086.issue.closed.md create mode 100644 docs/issues/2023-11-29.3095.issue.closed.md create mode 100644 docs/issues/2023-12-21.3115.issue.closed.md create mode 100644 docs/issues/2023-12-21.3116.issue.closed.md create mode 100644 docs/issues/2023-12-27.3117.issue.closed.md create mode 100644 docs/issues/2024-01-06.3121.issue.closed.md create mode 100644 docs/issues/2024-01-26.3139.issue.closed.md create mode 100644 docs/issues/2024-02-07.3146.issue.closed.md create mode 100644 docs/issues/2024-02-08.3148.issue.closed.md create mode 100644 docs/issues/2024-02-08.3150.pr.merged.md create mode 100644 docs/issues/2024-02-12.3152.issue.closed.md create mode 100644 docs/issues/2024-02-14.3153.pr.merged.md create mode 100644 docs/issues/2024-02-16.3155.pr.merged.md create mode 100644 docs/issues/2024-02-19.3157.pr.merged.md create mode 100644 docs/issues/2024-02-19.3158.pr.merged.md create mode 100644 docs/issues/2024-02-19.3159.pr.merged.md create mode 100644 docs/issues/2024-02-20.3160.pr.merged.md create mode 100644 docs/issues/2024-02-21.3161.issue.open.md create mode 100644 docs/issues/2024-02-22.3162.issue.open.md create mode 100644 docs/issues/2024-02-23.3163.pr.merged.md create mode 100644 docs/issues/2024-02-27.3164.issue.open.md create mode 100644 docs/issues/2024-02-27.3165.pr.merged.md create mode 100644 docs/issues/2024-02-27.3166.pr.open.md create mode 100644 docs/issues/2024-02-28.3167.issue.closed.md create mode 100644 docs/issues/2024-02-28.3168.pr.merged.md create mode 100644 docs/issues/2024-02-29.3169.issue.open.md create mode 100644 docs/issues/2024-02-29.3170.issue.closed.md create mode 100644 docs/issues/2024-02-29.3171.pr.open.md create mode 100644 docs/issues/2024-03-01.3172.pr.merged.md create mode 100644 docs/issues/2024-03-01.3173.pr.merged.md create mode 100644 docs/issues/2024-03-06.3174.pr.merged.md create mode 100644 docs/issues/2024-03-07.3175.pr.open.md create mode 100644 docs/issues/2024-03-07.3176.pr.merged.md create mode 100644 docs/issues/2024-03-08.3177.pr.merged.md create mode 100644 docs/issues/2024-03-12.3178.issue.open.md create mode 100644 docs/issues/2024-03-14.3179.pr.open.md create mode 100644 docs/issues/2024-03-18.3180.issue.closed.md create mode 100644 docs/issues/2024-03-18.3181.pr.closed.md create mode 100644 docs/issues/2024-03-18.3182.pr.merged.md create mode 100644 docs/issues/2024-03-18.3183.pr.merged.md create mode 100644 docs/issues/2024-03-27.3185.issue.open.md create mode 100644 docs/issues/2024-03-27.3186.issue.open.md create mode 100644 docs/issues/2024-03-28.3187.issue.open.md create mode 100644 docs/issues/2024-04-01.3188.pr.merged.md create mode 100644 docs/issues/2024-04-02.3189.issue.open.md create mode 100644 docs/issues/2024-04-03.3190.issue.closed.md create mode 100644 docs/issues/2024-04-03.3191.issue.closed.md create mode 100644 docs/issues/2024-04-04.3192.pr.merged.md create mode 100644 docs/issues/2024-04-04.3193.issue.open.md create mode 100644 docs/issues/2024-04-04.3194.issue.open.md create mode 100644 docs/issues/2024-04-04.3195.issue.closed.md create mode 100644 docs/issues/2024-04-04.3196.pr.closed.md create mode 100644 docs/issues/2024-04-05.3197.issue.closed.md create mode 100644 docs/issues/2024-04-05.3198.issue.closed.md create mode 100644 docs/issues/2024-04-05.3199.issue.open.md create mode 100644 docs/issues/2024-04-06.3200.issue.open.md create mode 100644 docs/issues/2024-04-07.3201.issue.open.md create mode 100644 docs/issues/2024-04-08.3202.issue.open.md create mode 100644 docs/issues/2024-04-09.3203.pr.open.md create mode 100644 docs/issues/2024-04-09.3204.pr.merged.md create mode 100644 docs/issues/2024-04-09.3205.pr.merged.md create mode 100644 docs/issues/2024-04-09.3206.pr.merged.md create mode 100644 docs/issues/2024-04-10.3207.issue.open.md create mode 100644 docs/issues/2024-04-19.3208.issue.open.md diff --git a/docs/issues/2021-07-05.2648.issue.closed.md b/docs/issues/2021-07-05.2648.issue.closed.md new file mode 100644 index 00000000..c976851e --- /dev/null +++ b/docs/issues/2021-07-05.2648.issue.closed.md @@ -0,0 +1,446 @@ +[\#2648 Issue](https://github.com/rear/rear/issues/2648) `closed`: OUTPUT=USB bug summary +========================================================================================= + +**Labels**: `enhancement`, `bug`, `cleanup`, `no-issue-activity` + +#### [DEvil0000](https://github.com/DEvil0000) opened issue at [2021-07-05 16:38](https://github.com/rear/rear/issues/2648): + +- ReaR version ("/usr/sbin/rear -V"): + Relax-and-Recover 2.6 / 2020-06-17 + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): + Ubuntu 20.04.2, kernel 5.4.0-77-generic + +- Hardware (PC or PowerNV BareMetal or ARM) or virtual machine (KVM + guest or PoverVM LPAR): + apu + +- System architecture (x86 compatible or PPC64/PPC64LE or what exact + ARM device): + x86\_64 + +- Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or + ELILO or Petitboot): + coreboot + +- Description of the issue (ideally so that others can reproduce + it): + I did start to dig a bit deeper in output usb mode and found more + then one bug with this method. To reduce effort of tracking I think + a summary ticket does make sense. So this ticket is meant to provide + this summary. + +The only output=usb settings working fine for me are: + +- UEFI + GPT (if uefi is supported by the system) + +General improvements possible for output=usb: + +- add grub bootloader option +- reuse output=rawdisk code (when grub is not used this may make + sense) +- reuse output=iso code (grub can boot from iso) +- to me it is not clear what should be done by format and what by + mkrescue but I would consider moving the bootloader writing to + format command +- hybrid gpt/mbr could be used if wanted +- hybrid efi/legacy could be used if wanted + +bugs with the current version: + +- in case of EFI it always creates a gpt partition - msdos is just not + handled (300\_format\_usb\_disk.sh:34) +- gpt partitions are not created correctly when **NO** EFI is used. + \*\* the bootloader must be within a gpt partition with type *ef02 + BIOS boot partition* (this partition must have *bios\_grub* flag + set) + \*\* it would make perfect sense to create a separate boot partition + containing the system image (like it is done for efi) +- the flag "legacy\_boot" must be configurable since some systems may + want it and some don't like it (from my understanding it is normally + not used but if it is used then for the /boot partition) +- the extlinux/syslinux boot config is not correct for my APU board + with coreboot. Not sure what is wrong there but since output=rawdisk + creates a working one it must be something there. (no matter if + msdos or gpt is used) \#2644 + +#### [jsmeix](https://github.com/jsmeix) commented at [2021-07-06 12:27](https://github.com/rear/rear/issues/2648#issuecomment-874716146): + +We had already +[https://github.com/rear/rear/issues/764](https://github.com/rear/rear/issues/764) +`Switch to grub2 (grub-mkrescue) for creating the recovery ISO image` +but noone contributed the needed code to ReaR +so it got closed because of `no-issue-activity`. + +@pcahyna +if time permits could you have a look here +and provide some general comment? +Thank you in advance! + +#### [jsmeix](https://github.com/jsmeix) commented at [2021-07-27 09:18](https://github.com/rear/rear/issues/2648#issuecomment-887352421): + +With also +[https://github.com/rear/rear/pull/2650](https://github.com/rear/rear/pull/2650) +and +[https://github.com/rear/rear/pull/2661](https://github.com/rear/rear/pull/2661) +merged +at least some of the issues mentioned here should be (hopefully) fixed. + +I can neither test coreboot nor serial console +because I do not have matching hardware +but I will test how far OUTPUT=USB works for me +on my older x86\_64 laptop with traditional BIOS. + +#### [jsmeix](https://github.com/jsmeix) commented at [2021-07-27 09:49](https://github.com/rear/rear/issues/2648#issuecomment-887373048): + +It still works for me with my setup as in +[https://github.com/rear/rear/pull/2659\#issuecomment-886692737](https://github.com/rear/rear/pull/2659#issuecomment-886692737) +and now even booting the existing system from the built-in local +harddisk +works via the GRUB2 menu entry `Boot next disk` +because now the created boot/grub2/grub.cfg contains + + menuentry "Boot next disk" { + insmod chain + set root=(hd1) + chainloader +1 + } + +#### [jsmeix](https://github.com/jsmeix) commented at [2021-07-27 10:05](https://github.com/rear/rear/issues/2648#issuecomment-887383494): + +I think the function create\_grub2\_cfg() is still somewhat messy +because my created boot/grub2/grub.cfg contains + + search --no-floppy --file /boot/efiboot.img --set + search --no-floppy --set=root --label MY-BOOT + +I think it does not make sense to have several GRUB2 `search` commands +at the same place (I guess only the last one matters in this case) +in contrast to having several GRUB2 `search` commands +each one in its own separated `menuentry` as in +the "Multi-boot manual config" section of the "GNU GRUB Manual" +[https://www.gnu.org/software/grub/manual/grub/grub.html\#Multi\_002dboot-manual-config](https://www.gnu.org/software/grub/manual/grub/grub.html#Multi_002dboot-manual-config) + +I think I have to clean up our various methods how to set up +the GRUB environment variable `root` via different variables +`grub2_set_root` and `GRUB2_SET_USB_ROOT` for the same thing +plus a hardcoded `search --no-floppy --file /boot/efiboot.img --set` + +#### [DEvil0000](https://github.com/DEvil0000) commented at [2021-07-27 16:49](https://github.com/rear/rear/issues/2648#issuecomment-887670657): + +To give an update on the current state from my point of view: +All bugs and shot commings should be fixed now. Maybe there is still one +left around the serial configuration. + +There are still some improvements possible without any priority to me: + +- reuse output=iso code and optionally store a iso in the boot + partition instead of the initrd (grub can boot from iso) +- hybrid gpt/mbr could be used (see gdisk) +- hybrid efi/legacy could be used + +I will re-test master as soon as possible what may be next week. +Thank you very much for your help with those changes + +#### [pcahyna](https://github.com/pcahyna) commented at [2021-07-27 17:10](https://github.com/rear/rear/issues/2648#issuecomment-887683889): + +> * grub can boot from iso + +I am interested how to do it - the only procedure I found is to let Grub +use the kernel and initrd contained in the ISO ( like +[http://www.panticz.de/MultiBootUSB](http://www.panticz.de/MultiBootUSB) +) but not to chainload the bootloader found on the ISO. + +#### [jsmeix](https://github.com/jsmeix) commented at [2021-07-28 06:39](https://github.com/rear/rear/issues/2648#issuecomment-888052817): + +@pcahyna +thank you that you joined here because I need your GRUB2 knowledge: + +Could you tell whether or not my assumption in +[https://github.com/rear/rear/issues/2648\#issuecomment-887383494](https://github.com/rear/rear/issues/2648#issuecomment-887383494) +is true - i.e. whether or not it makes sense to have more than one +`search` command in the general section (i.e. outside of a +`menuentry`) +in grub.cfg? + +Could you also have a look at +[https://github.com/rear/rear/pull/2662](https://github.com/rear/rear/pull/2662) +there in particulat at my `FIXME:` comment +in usr/share/rear/lib/bootloader-functions.sh +about whether or not `--set=esp` makes sense? + +#### [jsmeix](https://github.com/jsmeix) commented at [2021-07-28 06:48](https://github.com/rear/rear/issues/2648#issuecomment-888057391): + +My goal behind adding GRUB2 support for OUTPUT=USB is +that ReaR uses only GRUB2 as recovery system bootloader +so that we can drop using syslinux +cf. +[https://github.com/rear/rear/issues/764](https://github.com/rear/rear/issues/764) + +#### [pcahyna](https://github.com/pcahyna) commented at [2021-07-28 07:41](https://github.com/rear/rear/issues/2648#issuecomment-888088849): + +@jsmeix sorry that I haven't replied earlier. I suppose your question is +whether the second `search` will overwrite the result of the first +`search` so the first one becomes useless? I don't know the answer +off-hand, I will try to find out. + +#### [jsmeix](https://github.com/jsmeix) commented at [2021-07-28 09:19](https://github.com/rear/rear/issues/2648#issuecomment-888153162): + +@pcahyna no need to be sorry - we all have various day job things to do. + +Yes, that is my question. +When my assumption is true I would like to consolidate our current +various methods how to set up the GRUB environment variable `root` +into one single method and use only that method everywhere +so that we get one single `search` command in our grub.cfg +that is the right one for the intended purpose of that particular +grub.cfg + +In particular the current hardcoded + + search --no-floppy --file /boot/efiboot.img --set + +causes pain in my eyes and even more in my brain +because that cannot be right for BIOS and +it is also not right for OUTPUT=USB because +as far as I see boot/efiboot.img is created by +output/ISO/Linux-i386/700\_create\_efibootimg.sh +but that is only run for OUTPUT=ISO and it actually creates it +as isofs/boot/efiboot.img only if USING\_UEFI\_BOOTLOADER is true. + +#### [DEvil0000](https://github.com/DEvil0000) commented at [2021-07-28 09:59](https://github.com/rear/rear/issues/2648#issuecomment-888178276): + +@pcahyna for grub boot from iso you basically configure a loop device as +source for the kernel and initrd (including path within the ISO). +Maybe you can also chainload to it but I am not sure about that. + +see: +[https://wiki.ubuntuusers.de/GRUB\_2/Skripte/](https://wiki.ubuntuusers.de/GRUB_2/Skripte/) +[https://help.ubuntu.com/community/Grub2/ISOBoot](https://help.ubuntu.com/community/Grub2/ISOBoot) +[https://www.linux-community.de/blog/iso-images-mit-grub-2-booten/](https://www.linux-community.de/blog/iso-images-mit-grub-2-booten/) + +edit: +the idea behind this is that the booting source would always be the same +iso. No matter if you boot via ipmi or usb stick recover media. + +#### [pcahyna](https://github.com/pcahyna) commented at [2021-07-28 13:05](https://github.com/rear/rear/issues/2648#issuecomment-888292461): + +@DEvil0000 thanks, that's what I thought. I would then say that doing it +this way is needlessly complicated (a change of naming convention for +the kernel and initrd would break the process) and does not have big +advantage compared to putting the kernel+initrd in the filesystem +directly. (You would not be reusing the bootloader and its configuration +files from the ISO, so there would not be much to reuse.) + +#### [jsmeix](https://github.com/jsmeix) commented at [2021-07-28 13:47](https://github.com/rear/rear/issues/2648#issuecomment-888323386): + +I have a generic question regarding "grub can boot from iso": + +Does "grub can boot from iso" mean something different +than using GRUB2 as bootloader on an ISO as described +in the section "Making a GRUB bootable CD-ROM" in +[https://www.gnu.org/software/grub/manual/grub/grub.html](https://www.gnu.org/software/grub/manual/grub/grub.html) +? + +#### [pcahyna](https://github.com/pcahyna) commented at [2021-07-28 16:36](https://github.com/rear/rear/issues/2648#issuecomment-888454280): + +@jsmeix no, we mean chainloading a bootable ISO image (image as a file). +The "Making a GRUB bootable CD-ROM" deals with creating an ISO image +that will be then used to burn a CD-ROM and one will boot from the +CD-ROM. We are instead interested in having only an ISO image file +accessible to GRUB loaded from the harddisk (or USB disk) and booting +off that image file, if possible as if BIOS had booted a CD-ROM created +from this image. + +#### [DEvil0000](https://github.com/DEvil0000) commented at [2021-07-28 17:14](https://github.com/rear/rear/issues/2648#issuecomment-888479647): + +@jsmeix I think it gets more clear with an example grub menu entry: + + menuentry "Ubuntu 20.04 ISO" { + set isofile="/boot/ubuntu-20.04-desktop-amd64.iso" + rmmod tpm + loopback loop (hd0,5)$isofile + linux (loop)/casper/vmlinuz boot=casper iso-scan/filename=$isofile noprompt noeject + initrd (loop)/casper/initrd + } + +I think it may also be possible to chainload from the iso instead and +reuse the bootloader from there with something like: +(not tested and did not find documentation about this) + + menuentry "Ubuntu 20.04 ISO" { + set isofile="/boot/ubuntu-20.04-desktop-amd64.iso" + rmmod tpm + loopback loop (hd0,5)$isofile + chainloader (loop) + # or chainloader (loop)$efipath + } + +So everything recovery system related would be one single iso file which +is the same for ipmi inserted or on a usb drive. +I could also imagine to write a grub mod for wget of an iso so one would +not need to mess with PXE boot - which is just not possible in our +customer networks. +Maybe this explains my usecase better. + +edit: there is a tftp module since PXE needs it - this is maybe also a +option for getting a iso without a full PXE boot (no dhcp, no dns or +such). +`linux (tftp,a.b.c.d)/tftpboot/bzImage` or similar may work + +#### [pcahyna](https://github.com/pcahyna) commented at [2021-07-28 17:20](https://github.com/rear/rear/issues/2648#issuecomment-888483014): + +> I think it may also be possible to chainload from the iso instead and +> reuse the bootloader from there with something like: + +I don't think it is possible. I searched and found docs to do it only +for grub4dos (a separate GRUB fork) and Syslinux ( +[https://wiki.syslinux.org/wiki/index.php?title=Boot\_an\_Iso\_image](https://wiki.syslinux.org/wiki/index.php?title=Boot_an_Iso_image) +and +[https://wiki.syslinux.org/wiki/index.php?title=MEMDISK\#ISO\_images](https://wiki.syslinux.org/wiki/index.php?title=MEMDISK#ISO_images) +) - but the goal was to get rid of Syslinux, not to add another +dependency on it. (@lzaoral, this may be of interest for you for testing +purposes though). + +#### [jsmeix](https://github.com/jsmeix) commented at [2021-07-29 10:09](https://github.com/rear/rear/issues/2648#issuecomment-888988194): + +Regarding more than one 'search' command in grub.cfg +[https://github.com/rear/rear/issues/2648\#issuecomment-887383494](https://github.com/rear/rear/issues/2648#issuecomment-887383494) +I asked our SUSE GRUB maintainer and he replied (excerpts) + + > Currently our generated grub.cfg contains (excerpt) + > -------------------------------------------------------------- + > search --no-floppy --file /boot/efiboot.img --set + > search --no-floppy --set=root --label MY-BOOT + > -------------------------------------------------------------- + > + > The first 'search' may not set 'root' because /boot/efiboot.img + > may not exist. + + Yes. + + > The second 'search' may not set 'root' because nothing with + > filesystem label 'MY-BOOT' may exist. + + Yes. + + > I assume the second 'search' overwrites 'root' if it was set + > by the first 'search'. + + Yes. + + > I assume when /boot/efiboot.img exists but no 'MY-BOOT' exists + > then 'root' is still set to what the first 'search' had set. + > Or will 'root' be re-set to some default/fallback value when the + > second 'search' cannot find a filesystem with label 'MY-BOOT'? + + No it won't be re-set if the second search fails, so the value is + retained. That basically will do the work if you want the last found + wins. + +#### [jsmeix](https://github.com/jsmeix) commented at [2021-07-29 12:52](https://github.com/rear/rear/issues/2648#issuecomment-889117293): + +With +[https://github.com/rear/rear/pull/2662](https://github.com/rear/rear/pull/2662) +merged +we (hopefully) moved one more step forward with +GRUB2 as bootloader for OUTPUT=USB. + +#### [jsmeix](https://github.com/jsmeix) commented at [2021-07-29 12:54](https://github.com/rear/rear/issues/2648#issuecomment-889118703): + +@DEvil0000 @pcahyna +I am not in the office until next Tuesday so +I wish you a relaxed and recovering weekend! + +#### [github-actions](https://github.com/apps/github-actions) commented at [2021-09-28 02:11](https://github.com/rear/rear/issues/2648#issuecomment-928624604): + +Stale issue message + +#### [jsmeix](https://github.com/jsmeix) commented at [2021-10-15 10:39](https://github.com/rear/rear/issues/2648#issuecomment-944196488): + +This needs to be done before ReaR 2.7 release. +I think it is already mostly done. +I think only some config variable names need to be clarified. +See also +[https://github.com/rear/rear/issues/2666](https://github.com/rear/rear/issues/2666) + +#### [github-actions](https://github.com/apps/github-actions) commented at [2021-12-15 02:14](https://github.com/rear/rear/issues/2648#issuecomment-994225335): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2022-02-21 02:09](https://github.com/rear/rear/issues/2648#issuecomment-1046402947): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2022-04-23 02:49](https://github.com/rear/rear/issues/2648#issuecomment-1107323765): + +Stale issue message + +#### [jsmeix](https://github.com/jsmeix) commented at [2022-06-01 11:31](https://github.com/rear/rear/issues/2648#issuecomment-1143485252): + +I postponed +[https://github.com/rear/rear/issues/2666](https://github.com/rear/rear/issues/2666) +to ReaR 2.8 + +Via +[https://github.com/rear/rear/pull/2815](https://github.com/rear/rear/pull/2815) +I describe in default.conf the currently +implemented user config variable names +USB\_BOOT\_PART\_SIZE and USB\_DEVICE\_BOOT\_LABEL + +I think the current user config variable names +USB\_BOOT\_PART\_SIZE and USB\_DEVICE\_BOOT\_LABEL +are OK - at least I don't know really better names. + +This issue is no longer a blocker for ReaR 2.8 +but +[https://github.com/rear/rear/pull/2815](https://github.com/rear/rear/pull/2815) +is. + +I postpone this issue to ReaR 2.8. + +#### [github-actions](https://github.com/apps/github-actions) commented at [2022-08-01 03:52](https://github.com/rear/rear/issues/2648#issuecomment-1200664186): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2022-10-09 03:52](https://github.com/rear/rear/issues/2648#issuecomment-1272448165): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2022-12-10 02:29](https://github.com/rear/rear/issues/2648#issuecomment-1344985157): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2023-02-11 02:30](https://github.com/rear/rear/issues/2648#issuecomment-1426579840): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2023-04-15 02:19](https://github.com/rear/rear/issues/2648#issuecomment-1509470487): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2023-06-17 02:26](https://github.com/rear/rear/issues/2648#issuecomment-1595583934): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2023-08-19 01:57](https://github.com/rear/rear/issues/2648#issuecomment-1684669652): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2023-10-21 02:01](https://github.com/rear/rear/issues/2648#issuecomment-1773602441): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2023-12-23 02:03](https://github.com/rear/rear/issues/2648#issuecomment-1868172689): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2024-03-03 02:04](https://github.com/rear/rear/issues/2648#issuecomment-1974979352): + +Stale issue message + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2021-10-18.2698.issue.closed.md b/docs/issues/2021-10-18.2698.issue.closed.md new file mode 100644 index 00000000..b785e7e6 --- /dev/null +++ b/docs/issues/2021-10-18.2698.issue.closed.md @@ -0,0 +1,265 @@ +[\#2698 Issue](https://github.com/rear/rear/issues/2698) `closed`: OUTPUT=USB support for UEFI and BIOS dual boot (cf. OUTPUT=RAWDISK) +====================================================================================================================================== + +**Labels**: `enhancement`, `no-issue-activity` + +#### [DEvil0000](https://github.com/DEvil0000) opened issue at [2021-10-18 12:33](https://github.com/rear/rear/issues/2698): + +At the moment creating a boot media does only support legacy BIOS boot +or UEFI boot but not both from within the same image. I think this is +true in general but definitely for USB media. +It should be possible to create boot media supporting both modes in the +same image. +The format command does have a --efi option and default/without --efi +goes the legacy BIOS boot route. + +Proposed changes: + +- add a --bios switch which only does legacy bios installs. +- change format without --efi/--bios to create a hybrid stick +- make grub the default bootloader (since I know this one best and + know how to do it with that) +- small partitoning changes may be needed +- bootloader auto detect in + rescue/default/850\_save\_sysfs\_uefi\_vars.sh needs to get changed + (must work if booted in legacy and no boot/efi, efivars and + efibootmgr is availabe) e.g. + UEFI\_BOOTLOADER=/usr/lib/grub/x86\_64-efi/monolithic/grubx64.efi + +If you agree I will start the next days to prepare a PR. + +#### [jsmeix](https://github.com/jsmeix) commented at [2021-10-18 13:28](https://github.com/rear/rear/issues/2698#issuecomment-945768169): + +@DEvil0000 +of course I agree when you do a PR. +I appreciate it and I look forward to it. + +In the end if we could come to one single generic bootloader setup +for the bootloader of the ReaR recovery system that is by default +`Hybrid UEFI and BIOS with GRUB2 as bootloader` +it could be a great step forward to get that part of ReaR simple and +straightforward. + +#### [pcahyna](https://github.com/pcahyna) commented at [2021-10-18 13:33](https://github.com/rear/rear/issues/2698#issuecomment-945773580): + +Is it really possible to create an image which supports both BIOS and +UEFI? In particular, would a disk with an EFI partition table (I suppose +it is needed for UEFI support?) be bootable by BIOS? + +#### [jsmeix](https://github.com/jsmeix) commented at [2021-10-18 13:42](https://github.com/rear/rear/issues/2698#issuecomment-945781117): + +I am not at all a bootloader expert but +[https://wiki.syslinux.org/wiki/index.php?title=Isohybrid](https://wiki.syslinux.org/wiki/index.php?title=Isohybrid) +reads in the "UEFI" section + + The ISO 9660 filesystem is then supposed to boot from optical media + and from disk storage via BIOS and via UEFI. Unfortunately... + +#### [DEvil0000](https://github.com/DEvil0000) commented at [2021-10-18 13:43](https://github.com/rear/rear/issues/2698#issuecomment-945781903): + +As far as I know *setup iso images* of e.g. *ubuntu* have this hybrid +mode but of course they do it somewhat differently due to their *iso* +nature. +I just tested USB media with GPT and GRUB and it seems to work in both +worlds. +My understanding is, that BIOS looks for bootloader code at a specific +disk address (MBR) while UEFI is looking for a EFI partition. So they +should not conflict. The difference from MSDOS partition table to GPT +should basically be where the second chunk of bootloader gets written +to. + +#### [jsmeix](https://github.com/jsmeix) commented at [2021-10-18 13:47](https://github.com/rear/rear/issues/2698#issuecomment-945785294): + +Only as a side note FYI: +At SUSE we had (or perhaps even still have?) some SUSE specific +parted enhacement called `gpt_sync_mbr` partitioning scheme, cf. + + # find usr/sbin/rear usr/share/rear/ -type f | xargs grep -l 'gpt_sync_mbr' + usr/share/rear/layout/prepare/default/420_autoresize_last_partitions.sh + usr/share/rear/layout/prepare/GNU/Linux/100_include_partition_code.sh + usr/share/rear/layout/save/default/950_verify_disklayout_file.sh + usr/share/rear/layout/save/GNU/Linux/200_partition_layout.sh + +Again: I am not an expert in this area. + +#### [pcahyna](https://github.com/pcahyna) commented at [2021-10-18 13:49](https://github.com/rear/rear/issues/2698#issuecomment-945787676): + +Ok, so it should be possible for ISOs, as there are working hybrid +Ubuntu ISOs. My question was more about hard disk (OUTPUT=USB) +partitioning. + +#### [jsmeix](https://github.com/jsmeix) commented at [2021-10-18 13:56](https://github.com/rear/rear/issues/2698#issuecomment-945795570): + +@pcahyna +even if such a "Hybrid UEFI and BIOS with GRUB2 as bootloader" setup +does not work in general it would not be worse than what we have now +where our recovery system bootloader setup also does not work in +general. + +#### [jsmeix](https://github.com/jsmeix) commented at [2021-10-18 14:23](https://github.com/rear/rear/issues/2698#issuecomment-945827400): + +I guess one could have both a bios\_grub partition and a ESP +and have their partitioning data both in MBR and in a GPT +so a disk with a ReaR recovery system could have a partitioning like + +- free space for GPT (e.g. up to USB\_PARTITION\_ALIGN\_BLOCK\_SIZE) +- 8 MiB bios\_grub partition +- 512 MiB ESP +- /boot partition with USB\_BOOT\_PART\_SIZE +- ReaR data partition up to USB\_DEVICE\_FILESYSTEM\_PERCENTAGE +- free space for secondary GPT at the end of the disk + +This way the four partitions could be both in MBR and in the GPT and +kernel, initrd, and GRUB2 could be hopefully installed (e.g. two times +as neded) +so that this disk can boot both with BIOS firmware and UEFI firmware. +I would hope that BIOS firmware ignores GPT and ESP and +that UEFI firmware ignores MBR and bios\_grub partition. + +#### [OliverO2](https://github.com/OliverO2) commented at [2021-10-18 17:32](https://github.com/rear/rear/issues/2698#issuecomment-946002099): + +It's already there in ReaR. To quote from +[rear/03-configuration.adoc](https://github.com/rear/rear/blob/master/doc/user-guide/03-configuration.adoc#rescue-media-output) +(emphasis added): + +> OUTPUT=RAWDISK +> Create a bootable raw disk image on as rear-$(hostname).raw.gz. +> Supports UEFI boot if syslinux/EFI or Grub 2/EFI is installed. +> Supports Legacy BIOS boot if syslinux is installed. **Supports +> UEFI/Legacy BIOS dual boot** if syslinux and one of the supported EFI +> bootloaders are installed. + +#### [jsmeix](https://github.com/jsmeix) commented at [2021-10-19 06:24](https://github.com/rear/rear/issues/2698#issuecomment-946403594): + +@OliverO2 +thank you so much to remind us on what we already have in ReaR! +In this case we have it because you had contributed it to ReaR. +I am sorry that I had not sufficiently checked what we already have in +ReaR. +My excuse is that I personaly do not use OUTPUT=RAWDISK +but I should have known what OUTPUT=RAWDISK provides. + +@DEvil0000 @pcahyna +so the good news is that for OUTPUT=USB we do not need +to implement things from scratch. +Instead we should try to re-use the matching code from OUTPUT=RAWDISK +or at least learn from the OUTPUT=RAWDISK code how to implement it. + +#### [DEvil0000](https://github.com/DEvil0000) commented at [2021-10-19 08:30](https://github.com/rear/rear/issues/2698#issuecomment-946483055): + +@OliverO2 I was not aware that this works with syslinux. Thanks for the +hint will have a look there, too. + +#### [DEvil0000](https://github.com/DEvil0000) commented at [2021-10-19 12:27](https://github.com/rear/rear/issues/2698#issuecomment-946671655): + +@OliverO2 I am a bit confused about the raw output mode. You completely +skip the format workflow and have your own format/partitioning and +bootloader config code instead. correct? +Due to this the script looks a bit cleaner then the lib code but does +lack quite some features if I get this correct. + +#### [OliverO2](https://github.com/OliverO2) commented at [2021-10-19 14:41](https://github.com/rear/rear/issues/2698#issuecomment-946795259): + +Yes, there is no need for a format workflow. You get a disk image and +copy it, that's all. The `OUTPUT=RAWDISK` section in `default.conf` +explains how to use it. + +Regarding features: `RAWDISK` is for modern scenarios, so there are no +fancy grub configuration variants, for example. (With UEFI, as the +firmware offers to select boot options, there is less need for a boot +loader to replicate that feature.) + +#### [DEvil0000](https://github.com/DEvil0000) commented at [2021-10-19 17:31](https://github.com/rear/rear/issues/2698#issuecomment-946945265): + +sadly modern UEFI is quite rare for the hardware pool at our customers +sice they tend to have special hardware and keep it for at least +10years. I need all the legacy features for some machines while for some +I need the modern ones. +Having it basically duplicated instead of using the lib functions is +maybe not the best idea in any case. + +#### [OliverO2](https://github.com/OliverO2) commented at [2021-10-21 10:30](https://github.com/rear/rear/issues/2698#issuecomment-948475628): + +The creation of `RAWDISK` output was guided by a number of factors: + +1. the defined goal of creating a PBA for TCG OPAL-2-compliant + self-encrypting disks, +2. the quality, comprehensibility, and fitness for re-use of existing + code, +3. the volume of tests required to support all the options available, +4. the risk of inadvertently breaking existing functionality, +5. last not least: finite resources. + +Not using the "lib" functions was a deliberate decision based on careful +research. + +#### [jsmeix](https://github.com/jsmeix) commented at [2021-11-03 12:25](https://github.com/rear/rear/issues/2698#issuecomment-958982051): + +With +[https://github.com/rear/rear/pull/2705](https://github.com/rear/rear/pull/2705) +merged +"rear format" has now in addition to the --efi switch a --bios switch. +If none is given (i.e. by default) it will now do hybrid formatting +with a BIOS boot partition (on GPT) and an EFI system partition. +This is a precondition for this issue here. + +#### [github-actions](https://github.com/apps/github-actions) commented at [2022-01-03 02:21](https://github.com/rear/rear/issues/2698#issuecomment-1003837059): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2022-03-05 02:23](https://github.com/rear/rear/issues/2698#issuecomment-1059654805): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2022-05-07 02:50](https://github.com/rear/rear/issues/2698#issuecomment-1120117636): + +Stale issue message + +#### [jsmeix](https://github.com/jsmeix) commented at [2022-06-02 07:29](https://github.com/rear/rear/issues/2698#issuecomment-1144540284): + +I postponed this issue to ReaR 2.8 + +#### [github-actions](https://github.com/apps/github-actions) commented at [2022-08-02 03:53](https://github.com/rear/rear/issues/2698#issuecomment-1201984965): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2022-10-10 04:00](https://github.com/rear/rear/issues/2698#issuecomment-1272756111): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2022-12-10 02:29](https://github.com/rear/rear/issues/2698#issuecomment-1344985138): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2023-02-11 02:30](https://github.com/rear/rear/issues/2698#issuecomment-1426579820): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2023-04-15 02:19](https://github.com/rear/rear/issues/2698#issuecomment-1509470472): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2023-06-17 02:26](https://github.com/rear/rear/issues/2698#issuecomment-1595583925): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2023-08-19 01:57](https://github.com/rear/rear/issues/2698#issuecomment-1684669642): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2023-10-21 02:01](https://github.com/rear/rear/issues/2698#issuecomment-1773602427): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2023-12-23 02:03](https://github.com/rear/rear/issues/2698#issuecomment-1868172684): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2024-03-03 02:04](https://github.com/rear/rear/issues/2698#issuecomment-1974979348): + +Stale issue message + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2023-04-21.2971.issue.closed.md b/docs/issues/2023-04-21.2971.issue.closed.md new file mode 100644 index 00000000..71b30705 --- /dev/null +++ b/docs/issues/2023-04-21.2971.issue.closed.md @@ -0,0 +1,427 @@ +[\#2971 Issue](https://github.com/rear/rear/issues/2971) `closed`: Fedora: GRUB2 can't find command 'echo' 'linux' 'initrd' 'linuxefi' 'initrdefi' 'search' 'chainloader' 'reboot' 'exit' +========================================================================================================================================================================================= + +**Labels**: `enhancement`, `bug`, `fixed / solved / done`, +`external tool`, `not ReaR / invalid` + +#### [Jerry2840](https://github.com/Jerry2840) opened issue at [2023-04-21 03:38](https://github.com/rear/rear/issues/2971): + + + +- ReaR version ("/usr/sbin/rear -V"): + Relax-and-Recover 2.6 / 2020-06-17 + +- If your ReaR version is not the current version, explain why you + can't upgrade: + Using Fedora 38 current package: rear-2.6-9.fc38.x86\_64 + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): + + + + OS_VENDOR=RedHatEnterpriseServer + OS_VERSION=VERSION_ID=38 + +- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat + /etc/rear/local.conf"): + local.conf is all comments. + site.conf: + + + + OUTPUT=USB + BACKUP=NETFS + BACKUP_URL=usb:///dev/disk/by-label/REAR-000 + BACKUP_PROG_EXCLUDE=( "${BACKUP_PROG_EXCLUDE[@]}" '/tmp/*' '/jerry/*' ) + USB_UEFI_PART_SIZE="512" + TIMESYNC=CHRONY + +- Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM + guest or PowerVM LPAR): + ASUS C202SA Chrome book + dmidecode shows: + System Information + Manufacturer: GOOGLE + Product Name: Terra + +- System architecture (x86 compatible or PPC64/PPC64LE or what exact + ARM device): + Intel(R) Celeron(R) CPU N3060 @ 1.60GHz + x86\_64 + +- Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or + ELILO or Petitboot): + UEFI GRUB + +- Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or + multipath (DM or NVMe): + 16G eMMC + +- Storage layout ("lsblk -ipo + NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"): + The storage layout is the default that Fedora Workstation 38 + creates. + + + + NAME KNAME PKNAME TRAN TYPE FSTYPE LABEL SIZE MOUNTPOINT + /dev/sda /dev/sda usb disk udf RELAXRECOVER 28.6G + |-/dev/sda1 /dev/sda1 /dev/sda part vfat REAR-EFI 512M + `-/dev/sda2 /dev/sda2 /dev/sda part ext3 REAR-000 28.1G /tmp/rear.TUzMKqWPJoXOn8X/outputfs + /dev/mmcblk0 /dev/mmcblk0 disk 14.7G + |-/dev/mmcblk0p1 /dev/mmcblk0p1 /dev/mmcblk0 part vfat 600M /boot/efi + |-/dev/mmcblk0p2 /dev/mmcblk0p2 /dev/mmcblk0 part ext4 1G /boot + `-/dev/mmcblk0p3 /dev/mmcblk0p3 /dev/mmcblk0 part btrfs fedora_localhost-live 13.1G /home + /dev/mmcblk0boot0 /dev/mmcblk0boot0 disk 4M + /dev/mmcblk0boot1 /dev/mmcblk0boot1 disk 4M + /dev/zram0 /dev/zram0 disk 3.8G [SWAP] + +- Description of the issue (ideally so that others can reproduce + it): + I just see the error in the log file /var/log/rear/rear-c202s.log + about the integer expression expected. + I do have the problem that the USB flash drive boots and I get the + menu, but I get all of these messages for each menu item. + I am listing the visible menu item that you see on the screen and + then error lines are what I see when I hit enter to select the menu + item. + + + + GRUB version 2.06 + + Relax-and-Recover (no Secure Boot) + error: ../../grub-core/script/function.c:119:can't find command `echo'. + error: ../../grub-core/script/function.c:119:can't find command `linux'. + error: ../../grub-core/script/function.c:119:can't find command `echo'. + error: ../../grub-core/script/function.c:119:can't find command `initrd'. + + Press any key to continue... + + Relax-and-Recover (no Secure Boot) + error: ../../grub-core/script/function.c:119:can't find command `echo'. + error: ../../grub-core/script/function.c:119:can't find command `linuxefi'. + error: ../../grub-core/script/function.c:119:can't find command `echo'. + error: ../../grub-core/script/function.c:119:can't find command `initrdefi'. + + Press any key to continue... + + Boot original system + error: ../../grub-core/script/function.c:119:can't find command `search'. + error: ../../grub-core/script/function.c:119:can't find command `chainloader'. + + Press any key to continue... + + Reboot + error: ../../grub-core/script/function.c:119:can't find command `reboot'. + + Press any key to continue... + + Exit to EFI Shell + error: ../../grub-core/script/function.c:119:can't find command `exit'. + + Press any key to continue... + +- Workaround, if any: + +- Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" + debug log files): + +You can drag-drop log files into this editor to create an attachment +or paste verbatim text like command output or file content +by including it between a leading and a closing line of +three backticks like this: + + https://github.com/rear/rear/blob/master/usr/share/rear/layout/save/default/950_verify_disklayout_file.sh#L82 + + I added the Log line so I could see what it was comparing and saw the p1 + highest_used_part_num=0 + for partition in "${partitions[@]}" ; do + # We test only partitions of the form /dev/sdX1 /dev/sdX2 /dev/sdX3 (i.e. of the form $disk_dev$part_num). + part_num=${partition#$disk_dev} + Log "part_num = $part_num highest_used_part_num = $highest_used_part_num" + test $part_num -gt $highest_used_part_num && highest_used_part_num=$part_num + done + + 2023-04-20 20:45:28.338509014 part_num = p1 highest_used_part_num = 0 + /usr/share/rear/layout/save/default/950_verify_disklayout_file.sh: line 83: test: p1: integer expression expected + 2023-04-20 20:45:28.345388057 part_num = p2 highest_used_part_num = 0 + /usr/share/rear/layout/save/default/950_verify_disklayout_file.sh: line 83: test: p2: integer expression expected + 2023-04-20 20:45:28.351797496 part_num = p3 highest_used_part_num = 0 + +[rear-c202s.log](https://github.com/rear/rear/files/11291981/rear-c202s.log) + +#### [jsmeix](https://github.com/jsmeix) commented at [2023-04-21 06:51](https://github.com/rear/rear/issues/2971#issuecomment-1517351303): + +The messages in +[https://github.com/rear/rear/files/11291981/rear-c202s.log](https://github.com/rear/rear/files/11291981/rear-c202s.log) + + /usr/share/rear/layout/save/default/950_verify_disklayout_file.sh: line 83: test: p1: integer expression expected + /usr/share/rear/layout/save/default/950_verify_disklayout_file.sh: line 83: test: p2: integer expression expected + /usr/share/rear/layout/save/default/950_verify_disklayout_file.sh: line 83: test: p3: integer expression expected + +are unrelated to the GRUB2 error messages. + +Regarding the 'integer expression expected' messages: + +In general what layout/save/default/950\_verify\_disklayout\_file.sh +does +is not required functionality to make ReaR work because it is only +there +as a safeguard that the entries in disklayout.conf are syntactically +OK +which they should normally be. + +The part in layout/save/default/950\_verify\_disklayout\_file.sh +that results those 'integer expression expected' messages +only checks for non consecutive partitions and only when +partitions of the form /dev/sdX1 /dev/sdX2 /dev/sdX3 +(i.e. of the form $disk\_dev$part\_num) are used, +see the comments in that code. +When partition device nodes like /dev/mmcblk0p1 /dev/mmcblk0p2 +/dev/mmcblk0p3 /dev/mmcblk0boot0 /dev/mmcblk0boot1 are used +(i.e. of the form ${disk\_dev}p$part\_num) +that test is skipped. + +So the 'integer expression expected' messages +are no errors but expected failures of the 'test' +to also find out if the form $disk\_dev$part\_num is used. + +@Jerry2840 +in general when you run ReaR (or any other rogram) in debug mode +you may get various kind of messages that indicate this or that +failures +but you would need to understand what is actually meant to happen +in each case to decide if a failure is expected and handled +or if a failure is unexpected and indicates a real error. + +For example on my currently used computer: + + # dmesg | egrep -i 'fail|error' + + [ 0.000000] tsc: Fast TSC calibration failed + [ 0.000000] DMAR: Parse DMAR table failure. + [ 3.400876] RAS: Correctable Errors collector initialized. + [ 46.184357] ACPI BIOS Error (bug): Could not resolve symbol [\_TZ.PSL.CFGD], AE_NOT_FOUND (20210730/psargs-330) + [ 46.185935] ACPI Error: Aborting method \_TZ.PSL due to previous error (AE_NOT_FOUND) (20210730/psparse-531) + ... + +My computer works perfectly well for me as far as I can tell +and I don't have sufficient low-level knowledge to decide +if those messages mean or indicate some real problem +so I just ignore them and go on bona fide. + +#### [jsmeix](https://github.com/jsmeix) commented at [2023-04-21 06:57](https://github.com/rear/rear/issues/2971#issuecomment-1517356981): + +Regarding the GRUB2 error messages +I am afraid I cannot really help here +because I am not a sufficient GRUB2 expert +to imagine what their reason could be. + +Only a blind guess +in particular because you use a Red Hat system +(but I am not a Red Hat user): + +As far as I know on Red Hat based systems +the whole GRUB2 software is split into several RPM packages +and by default not all GRUB2 software is installed - in particular +some GRUB2 modules are not installed by default - so you may +need to additionally install certain GRUB2 RPM packages +to get GRUB2 working for ReaR. + +See +[https://github.com/rear/rear/issues/2783\#issuecomment-1085678921](https://github.com/rear/rear/issues/2783#issuecomment-1085678921) + +#### [jsmeix](https://github.com/jsmeix) commented at [2023-04-21 07:01](https://github.com/rear/rear/issues/2971#issuecomment-1517361684): + +@pcahyna +could you have a look here (as time permits) +regarding the GRUB2 error messages? + +#### [jsmeix](https://github.com/jsmeix) commented at [2023-04-21 07:32](https://github.com/rear/rear/issues/2971#issuecomment-1517397127): + +Via +[https://github.com/rear/rear/commit/c8f5e3f41c970a89304cd2b1bc5d04a2d2a7fdb4](https://github.com/rear/rear/commit/c8f5e3f41c970a89304cd2b1bc5d04a2d2a7fdb4) +in layout/save/default/950\_verify\_disklayout\_file.sh +it now suppresses unhelpful stderr messages like +"test: p1: integer expression expected" +that appear for partitions of the form like +/dev/mmcblk0p1 (i.e. of the form ${disk\_dev}p$part\_num). + +Later if needed and as time permits +layout/save/default/950\_verify\_disklayout\_file.sh +may be enhanced to check for non consecutive partitions +also for partitions of the form like +/dev/mmcblk0p1 (i.e. of the form ${disk\_dev}p$part\_num). +Probably this is not needed in practice because +the parted versions that are nowadays used +should be sufficiently new so that they +support FEATURE\_PARTED\_RESIZEPART or FEATURE\_PARTED\_RESIZE +(i.e. the 'resizepart' or 'resize' command), +cf. lib/layout-functions.sh + +#### [schlomo](https://github.com/schlomo) commented at [2023-04-23 09:19](https://github.com/rear/rear/issues/2971#issuecomment-1519007503): + +First of all, @Jerry2840, big congratulations of probably being the +first ReaR user on a *Chromebook* 😄 (I also use an old Chromebook as a +"kitchen dashboard"...) + +Your GRUB2 error looks like the Grub modules are missing or not found, +please take a look at your rescue media and check the location of the +`echo.mod` file on it. In your logfile I found this line + + grub2-mkstandalone: info: copying `/usr/lib/grub/x86_64-efi/echo.mod' -> `/tmp/grub.XXpN8c/boot/grub/x86_64-efi/echo.mod'. + +which suggests that the module was copied, but maybe it got copied to +the wrong place. + +#### [github-actions](https://github.com/apps/github-actions) commented at [2023-06-23 02:48](https://github.com/rear/rear/issues/2971#issuecomment-1603613408): + +Stale issue message + +#### [pcahyna](https://github.com/pcahyna) commented at [2023-06-23 14:27](https://github.com/rear/rear/issues/2971#issuecomment-1604362746): + +Hello, sorry for the late reply. If you still can reproduce the issue, I +wonder what is in the GRUB image if you can get to the GRUB shell. Do +you have the (memdisk) device? grub2-mkstandalone creates this. To find +out what modules you have, use + + ls (memdisk)/boot/grub/x86_64-efi/ + +If (memdisk) is not there, I suspect you have booted a different GRUB +image than the one produced by ReaR. + +#### [pcahyna](https://github.com/pcahyna) commented at [2023-06-27 14:13](https://github.com/rear/rear/issues/2971#issuecomment-1609593994): + +I found more details. I can reproduce the problem starting with Fedora +37 (Fedora 36 was fine). In the GRUB image produced on Fedora 37 we have + + prefix=(hd1,gpt1)/EFI/BOOT + root=memdisk + +(hd1 is the USB disk where ReaR has been booted from). This does not +work, as /EFI/BOOT on the ReaR image does not contain any GRUB +modules. +If I execute + + set prefix=(memdisk)/boot/grub + set root=hd1,gpt + +then I can boot the rescue system. +In Fedora 36 I have + + prefix=(memdisk)/boot/grub + root=hd1,gpt1 + +and the image works out of the box. So, something has changed with +variables in the image produced by grub2-mkstandalone between Fedora 36 +and Fedora 37. + +#### [jsmeix](https://github.com/jsmeix) commented at [2023-08-08 13:00](https://github.com/rear/rear/issues/2971#issuecomment-1669569718): + +Since +[https://github.com/rear/rear/pull/3031](https://github.com/rear/rear/pull/3031) +is merged +there is Secure Boot support for OUTPUT=USB +in our ReaR upstream master code. + +The basic idea for Secure Boot booting of the ReaR recovery system +is to "just copy" the (signed) EFI binaries of the Linux distribution +(shim\*.efi and grub\*.efi as first and second stage UEFI bootloaders) +instead of let ReaR make its own EFI binary via build\_bootx86\_efi() +and @pcahyna shows in his +[https://github.com/rear/rear/pull/3031\#issuecomment-1669336117](https://github.com/rear/rear/pull/3031#issuecomment-1669336117) +that this also works for normal UEFI boot +(i.e. UEFI boot with Secure Boot disabled) +at least on RHEL 9. + +So it could be a possible workaround for now +to "misuse" the new Secure Boot support for OUTPUT=USB +via something like (in local.conf or site.conf) + + SECURE_BOOT_BOOTLOADER=/boot/efi/EFI/redhat/shimx64.efi + +(specify the actual 'shim\*.efi' file on your system) +for normal UEFI boot (i.e. with Secure Boot disabled). + +@Jerry2840 +could you try out if this workaround +actually makes things work for you? + +See the section +"Testing current ReaR upstream GitHub master code" in +[https://en.opensuse.org/SDB:Disaster\_Recovery](https://en.opensuse.org/SDB:Disaster_Recovery) +how you could try out the current ReaR upstream GitHub master code +from within a separated directory as a test to find out +if things work better with current ReaR upstream master code +compared to your installed ReaR version. + +#### [github-actions](https://github.com/apps/github-actions) commented at [2023-10-10 02:03](https://github.com/rear/rear/issues/2971#issuecomment-1754204227): + +Stale issue message + +#### [pcahyna](https://github.com/pcahyna) commented at [2023-10-10 13:56](https://github.com/rear/rear/issues/2971#issuecomment-1755480737): + +> So it could be a possible workaround for now to "misuse" the new +> Secure Boot support for OUTPUT=USB via something like (in local.conf +> or site.conf) +> +> SECURE_BOOT_BOOTLOADER=/boot/efi/EFI/redhat/shimx64.efi +> +> (specify the actual 'shim\*.efi' file on your system) for normal UEFI +> boot (i.e. with Secure Boot disabled). + +That should indeed work and we should actually make this the default, +i.e. always use the OS-provided GRUB image (and the Secure Boot shim if +present), without having to set `SECURE_BOOT_BOOTLOADER` manually. + +#### [github-actions](https://github.com/apps/github-actions) commented at [2023-12-10 02:11](https://github.com/rear/rear/issues/2971#issuecomment-1848835096): + +Stale issue message + +#### [github-actions](https://github.com/apps/github-actions) commented at [2024-02-11 02:05](https://github.com/rear/rear/issues/2971#issuecomment-1937393475): + +Stale issue message + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-02-15 12:53](https://github.com/rear/rear/issues/2971#issuecomment-1946042578): + +@Jerry2840 is it still a problem? If so, have you tried the workaround +suggested above? + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-02-15 14:45](https://github.com/rear/rear/issues/2971#issuecomment-1946241649): + +@Jerry2840 Could you please try the following Fedora 38 update? +[https://bodhi.fedoraproject.org/updates/FEDORA-2024-49ddbf447d](https://bodhi.fedoraproject.org/updates/FEDORA-2024-49ddbf447d) + +ReaR in Fedora has been unmaintained for a year at least so the update +above should fix a lot of problems you may encounter. + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-02-16 09:39](https://github.com/rear/rear/issues/2971#issuecomment-1948048702): + +The issue is actually caused by a regression in the `grub2` package +which was fixed by the following commit +[https://src.fedoraproject.org/rpms/grub2/c/8dde55b253cc29289794efe91e70f70f23c79a83](https://src.fedoraproject.org/rpms/grub2/c/8dde55b253cc29289794efe91e70f70f23c79a83) +in `grub2-2.06-110.fc38` on Fedora 38. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-04 18:49](https://github.com/rear/rear/issues/2971#issuecomment-1977239522): + +The GRUB bug in Fedora was fixed about a month ago, +[https://bugzilla.redhat.com/show\_bug.cgi?id=2209435](https://bugzilla.redhat.com/show_bug.cgi?id=2209435). +(The bug description mentions PXE, but the mkstandalone image problem is +caused by the same issue.) Closing. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2023-11-01.3063.issue.closed.md b/docs/issues/2023-11-01.3063.issue.closed.md new file mode 100644 index 00000000..ef9bfca0 --- /dev/null +++ b/docs/issues/2023-11-01.3063.issue.closed.md @@ -0,0 +1,109 @@ +[\#3063 Issue](https://github.com/rear/rear/issues/3063) `closed`: Can rear handle multiple backups on just one USB stick? +========================================================================================================================== + +**Labels**: `enhancement`, `support / question`, `needs sponsorship`, +`no-issue-activity` + +#### [R-Sommer](https://github.com/R-Sommer) opened issue at [2023-11-01 13:05](https://github.com/rear/rear/issues/3063): + +It looks like the mailing list is no longer active. Hope it’s ok to ask +a question here. + +rear can create a bootable USB stick for recovering ONE system but I +would like to have just one USB stick for recovering multiple systems. +Unfortunately, I’m not able to figure out reading the documentation and +hundreds of articles in the internet. + +For me it would be perfect when each backup would automatically be added +to USB sticks’ grub. + +Can someone please shed some light e.g.: Is it necessary to install rear +on each system or can/have this one USB stick (to) be used to create the +backups? + +#### [jsmeix](https://github.com/jsmeix) commented at [2023-11-10 14:46](https://github.com/rear/rear/issues/3063#issuecomment-1805874286): + +Yes, the mailing list is no longer active. +It is perfectly right to ask questions here. + +In general I recommend to +Keep Separated Issues Separated ("KSIS"), +cf. RFC 1925 item (5). + +In this case use a separated USB device for each system +and even use a separated USB device for each new backup. + +See +[https://github.com/rear/rear/discussions/2948\#discussioncomment-5139092](https://github.com/rear/rear/discussions/2948#discussioncomment-5139092) +(excerpt) + + As far as I see it is (at least in practice) impossible + to use one USB disk to recover two different systems. + + I think there is only one ReaR recovery system possible + for one USB disk because it is the data that is stored + inside the ReaR recovery system which is specific for the + original system where "rear mkrescue/mkbackup" was run + in particular the disklayout.conf file where information + about partitions, filesystems and mountpoints is stored. + +See also +[https://github.com/rear/rear/issues/3048\#issuecomment-1715415036](https://github.com/rear/rear/issues/3048#issuecomment-1715415036) +(excerpt) + + in particular with flash memory based storage + like USB sticks or USB SSDs: + + Never use one same medium for more than one backup. + Never use a medium with a recent backup that you may need + to store one more backup on it. + For each backup use a separated clean medium. + +#### [R-Sommer](https://github.com/R-Sommer) commented at [2023-11-16 17:01](https://github.com/rear/rear/issues/3063#issuecomment-1814858749): + +What about storing several ISO files on a USB stick and configuring grub +to allow selecting any of these for recovering? + +I'm using a similar approach but with a product called Macrium Reflect. +I can boot from my stick and backup to the free area on a second +partition or recover any of those backups. This gives me the oppertunity +to take the stick everytime with me and recover any of my systems even +on a new hardware (in case of a desaster). + +To mitigate the risk of losing everything I connect the stick with my +NAS and sync new backups. And I clone the stick on a other stick and put +it in an other safe place. + +As one downside of this product I cannot exclude directories and that's +why I'm interested in rear. + +#### [jsmeix](https://github.com/jsmeix) commented at [2023-11-17 08:17](https://github.com/rear/rear/issues/3063#issuecomment-1815918244): + +@R-Sommer +the crucial point is +neither whether or not it can be implemented +nor how to implement it. +The crucial point is +who will actually initially implement it and +then further support and maintain the code +(for about at least one year or longer) +until it works sufficiently well and reliable. + +#### [we-sell-bags](https://github.com/we-sell-bags) commented at [2024-01-03 07:34](https://github.com/rear/rear/issues/3063#issuecomment-1874953697): + +maybe look here: +grub-n-iso\_multiboot + +seems to be a script that adds bootable .iso images to allow them to +appear in the grub bootloader. +it also works with a usb stick as the main boot device. +the ISO are just file images stored in a normal directory + +#### [github-actions](https://github.com/apps/github-actions) commented at [2024-03-04 02:46](https://github.com/rear/rear/issues/3063#issuecomment-1975558272): + +Stale issue message + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2023-11-17.3086.issue.closed.md b/docs/issues/2023-11-17.3086.issue.closed.md new file mode 100644 index 00000000..3b9789f0 --- /dev/null +++ b/docs/issues/2023-11-17.3086.issue.closed.md @@ -0,0 +1,53 @@ +[\#3086 Issue](https://github.com/rear/rear/issues/3086) `closed`: Support for Amazon Linux 2 and 2023 +====================================================================================================== + +**Labels**: `support / question`, `no-issue-activity` + +#### [ramzcode](https://github.com/ramzcode) opened issue at [2023-11-17 14:47](https://github.com/rear/rear/issues/3086): + +I am using ReaR version 2.7 on Amazon Linux2 and trying to test a +recovery case. + +Backup works but the Recovery AMI wont come up. Anyone tried this ? Any +suggestions. + +Thanks in advance + +#### [ramzcode](https://github.com/ramzcode) commented at [2023-11-22 18:28](https://github.com/rear/rear/issues/3086#issuecomment-1823271291): + +Yes we found solution, make sure you are not using the default syslinux +verion, use the latest 6.4.x version of syslinux. after testing the +recovery process, will keep posted and close the thread. + +#### [jsmeix](https://github.com/jsmeix) commented at [2023-11-23 07:37](https://github.com/rear/rear/issues/3086#issuecomment-1823923124): + +It seems +[https://github.com/rear/rear/issues/3090](https://github.com/rear/rear/issues/3090) +is a related/dependant issue which in particular shows +some more details about the used Amazon Linux2 system + +#### [schlomo](https://github.com/schlomo) commented at [2024-01-13 18:53](https://github.com/rear/rear/issues/3086#issuecomment-1890705493): + +See \#3095, I think that Amazon Linux and AWS EC2 is still a bit of +uncharted terrain for ReaR + +#### [github-actions](https://github.com/apps/github-actions) commented at [2024-03-14 02:01](https://github.com/rear/rear/issues/3086#issuecomment-1996256531): + +Stale issue message + +#### [ramzcode](https://github.com/ramzcode) commented at [2024-04-19 07:00](https://github.com/rear/rear/issues/3086#issuecomment-2065884391): + +@schlomo @jsmeix We have successfully completed Amazon Linux 2 and 2023 +testing 100% and it works with ReaR, there were OS specific dependencies +and ReaR config level changes as well. I did the PoC on AWS between two +different regions. + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-19 17:07](https://github.com/rear/rear/issues/3086#issuecomment-2066963960): + +Can you please share a guide for AL? Du tust others can benefit from +your experience? + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2023-11-29.3095.issue.closed.md b/docs/issues/2023-11-29.3095.issue.closed.md new file mode 100644 index 00000000..e6ed8714 --- /dev/null +++ b/docs/issues/2023-11-29.3095.issue.closed.md @@ -0,0 +1,86 @@ +[\#3095 Issue](https://github.com/rear/rear/issues/3095) `closed`: Amazon Linux 2023 : DHCP is enabled but no DHCP client binary (dhcpcd dhclient dhcp6c dhclient6) was found - systemd-networkd - ReaR 2.7 +=========================================================================================================================================================================================================== + +**Labels**: `support / question`, `no-issue-activity` + +#### [ramzcode](https://github.com/ramzcode) opened issue at [2023-11-29 07:46](https://github.com/rear/rear/issues/3095): + +Using ReaR version 2.7 on Amazon Linux 2023. + +systemd-networkd is responsible for DHCP and network setup. + +ReaR throws error that no binary resalted to DHCP client is found and +yes it is not needed on AL2023. The Array does not contain binary for +AL2023 builds. + +systemd-networkd.service and /usr/lib/systemd/systemd-networkd are core +components responsible in this case it seems. + +Please suggest if there is a fix or workaround to this. +Thanks in advance + +#### [ramzcode](https://github.com/ramzcode) commented at [2023-11-30 13:49](https://github.com/rear/rear/issues/3095#issuecomment-1833818971): + +UPDATE: For AL2023, Using EPEL Repo RPM +[https://dl.fedoraproject.org/pub/epel/8/Everything/x86\_64/Packages/d/](https://dl.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/d/) +install the DHCPCD agent on the source VM before building the rescue +system and before running the ReaR backups. Once the First restore is +done make sure we remove this RPM for sure. + +It worked and i could access the Rescue instance via SSH with no +problem. Yet to do the Full restore. Will update + +#### [jsmeix](https://github.com/jsmeix) commented at [2023-11-30 14:11](https://github.com/rear/rear/issues/3095#issuecomment-1833855355): + +I think +usr/share/rear/prep/GNU/Linux/210\_include\_dhclient.sh +[https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/prep/GNU/Linux/210\_include\_dhclient.sh](https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/prep/GNU/Linux/210_include_dhclient.sh) +could be improved to show better messages +(e.g. DebugPrint, LogPrintError, Error) +when USE\_DHCLIENT is 'yes' +but none of the + + dhcp_clients=(dhcpcd dhclient dhcp6c dhclient6) + +can be found. + +#### [ramzcode](https://github.com/ramzcode) commented at [2023-12-01 04:38](https://github.com/rear/rear/issues/3095#issuecomment-1835446196): + +@jsmeix It did have a clear error message but the distro won't have any +of the listed dhcp clients by default cause it handles everything using +the systems-networkd service and it's based components. + +As a workaround and to make rear support DHCP on the rescue system I +need to install an extra DHCPCD binary using EPEL repo based binary. + +I think rear does its job correctly. + +#### [schlomo](https://github.com/schlomo) commented at [2024-01-13 18:52](https://github.com/rear/rear/issues/3095#issuecomment-1890703811): + +@ramzcode if you'd ask me I'd say that Amazon Linux 2 is probably not +yet supported by ReaR, sorry. Maybe there isn't much needed for that - I +simply don't know. + +About DHCP indeed we don't support `systemd-networkd` properly, and we +probably also don't really support all the other stuff that it does for +networking. + +Do you feel up for a contribution by submitting a PR with the missing +Amazon Linux 2 support? Alternatively please consider our [commercial +support options](https://relax-and-recover.org/support/) + +#### [github-actions](https://github.com/apps/github-actions) commented at [2024-03-14 02:01](https://github.com/rear/rear/issues/3095#issuecomment-1996256515): + +Stale issue message + +#### [ramzcode](https://github.com/ramzcode) commented at [2024-04-19 07:02](https://github.com/rear/rear/issues/3095#issuecomment-2065887455): + +@schlomo @jsmeix Along with the systemd-network service, i did install +installed the dhchcd client with caution and making sure it won't start +on boot. Laster used ReaR and it easily handled the DHCP problem on the +recovery image and we had no trouble moving forward. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2023-12-21.3115.issue.closed.md b/docs/issues/2023-12-21.3115.issue.closed.md new file mode 100644 index 00000000..6f9f5878 --- /dev/null +++ b/docs/issues/2023-12-21.3115.issue.closed.md @@ -0,0 +1,192 @@ +[\#3115 Issue](https://github.com/rear/rear/issues/3115) `closed`: ReaR 2.7 - SLES 15.5 - EFI variables are not supported on this system - Recovery +=================================================================================================================================================== + +**Labels**: `support / question`, `no-issue-activity` + +#### [ramzcode](https://github.com/ramzcode) opened issue at [2023-12-21 08:59](https://github.com/rear/rear/issues/3115): + +- ReaR version ("/usr/sbin/rear -V"): + 2.7 + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): + SLES 15.5 + +- Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM + guest or PowerVM LPAR): + AWS EC2 VM / KVM + +- System architecture (x86 compatible or PPC64/PPC64LE or what exact + ARM device): + x86 + +- Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or + ELILO or Petitboot): + UEFI + +- Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or + multipath (DM or NVMe): + NVMe + +- Description of the issue (ideally so that others can reproduce it): + +While Running "rear recover" in the rescue environment, Bootloader +installation fails. + +- Workaround, if any: None as of now. + +Issue Specific Log: + + ++ bootloader='EFI\BOOT\grub.efi' + ++ for efipart in $boot_efi_parts + +++ get_device_name /dev/nvme0n1p2 + +++ local name=nvme0n1p2 + +++ name=nvme0n1p2 + +++ contains_visible_char nvme0n1p2 + ++++ tr -d -c '[:graph:]' + +++ test nvme0n1p2 + +++ [[ nvme0n1p2 =~ ^mapper/ ]] + +++ [[ -L /dev/nvme0n1p2 ]] + +++ [[ nvme0n1p2 =~ ^dm- ]] + +++ name=nvme0n1p2 + +++ echo /dev/nvme0n1p2 + +++ [[ -r /dev/nvme0n1p2 ]] + +++ return 0 + ++ partition_block_device=/dev/nvme0n1p2 + +++ get_partition_number /dev/nvme0n1p2 + +++ local partition_block_device=/dev/nvme0n1p2 + ++++ echo /dev/nvme0n1p2 + ++++ grep -o -E '[0-9]+$' + +++ local partition_number=2 + +++ test 2 -gt 0 + +++ test 2 -le 128 + +++ echo 2 + ++ partition_number=2 + +++ get_device_from_partition /dev/nvme0n1p2 2 + +++ local partition_block_device + +++ local device + +++ local partition_number + +++ partition_block_device=/dev/nvme0n1p2 + +++ test -b /dev/nvme0n1p2 + +++ partition_number=2 + +++ device=/dev/nvme0n1p + +++ [[ /dev/nvme0n1p2 != /dev/nvme0n1p2 ]] + +++ [[ /dev/nvme0n1p = *\/\m\m\c\b\l\k+([0-9])p ]] + +++ [[ /dev/nvme0n1p = *\/\n\v\m\e+([0-9])n+([0-9])p ]] + +++ device=/dev/nvme0n1 + +++ test -b /dev/nvme0n1 + +++ echo /dev/nvme0n1 + ++ disk=/dev/nvme0n1 + ++ LogPrint 'Creating EFI Boot Manager entry '\''SUSE_LINUX 15.5'\'' for '\''EFI\BOOT\grub.efi'\'' (UEFI_BOOTLOADER='\''/boot/efi/EFI/BOOT/grub.efi'\'') ' + ++ Log 'Creating EFI Boot Manager entry '\''SUSE_LINUX 15.5'\'' for '\''EFI\BOOT\grub.efi'\'' (UEFI_BOOTLOADER='\''/boot/efi/EFI/BOOT/grub.efi'\'') ' + ++ test -w /var/log/brutils/brutils-ip-172-31-7-113.log + ++ echo '2023-12-21 07:01:38.265015842 Creating EFI Boot Manager entry '\''SUSE_LINUX 15.5'\'' for '\''EFI\BOOT\grub.efi'\'' (UEFI_BOOTLOADER='\''/boot/efi/EFI/BOOT/grub.efi'\'') ' + 2023-12-21 07:01:38.265015842 Creating EFI Boot Manager entry 'SUSE_LINUX 15.5' for 'EFI\BOOT\grub.efi' (UEFI_BOOTLOADER='/boot/efi/EFI/BOOT/grub.efi') + ++ Print 'Creating EFI Boot Manager entry '\''SUSE_LINUX 15.5'\'' for '\''EFI\BOOT\grub.efi'\'' (UEFI_BOOTLOADER='\''/boot/efi/EFI/BOOT/grub.efi'\'') ' + ++ Log efibootmgr --create --gpt --disk /dev/nvme0n1 --part 2 --write-signature --label '"SUSE_LINUX' '15.5"' --loader '"\EFI\BOOT\grub.efi"' + ++ test -w /var/log/brutils/brutils-ip-172-31-7-113.log + ++ echo '2023-12-21 07:01:38.271230092 efibootmgr --create --gpt --disk /dev/nvme0n1 --part 2 --write-signature --label "SUSE_LINUX 15.5" --loader "\EFI\BOOT\grub.efi"' + 2023-12-21 07:01:38.271230092 efibootmgr --create --gpt --disk /dev/nvme0n1 --part 2 --write-signature --label "SUSE_LINUX 15.5" --loader "\EFI\BOOT\grub.efi" + ++ efibootmgr --create --gpt --disk /dev/nvme0n1 --part 2 --write-signature --label 'SUSE_LINUX 15.5' --loader '\EFI\BOOT\grub.efi' + EFI variables are not supported on this system. + ++ LogPrintError 'efibootmgr failed to create EFI Boot Manager entry on /dev/nvme0n1 partition 2 (ESP /dev/nvme0n1p2 )' + ++ Log 'efibootmgr failed to create EFI Boot Manager entry on /dev/nvme0n1 partition 2 (ESP /dev/nvme0n1p2 )' + ++ test -w /var/log/brutils/brutils-ip-172-31-7-113.log + ++ echo '2023-12-21 07:01:38.279364912 efibootmgr failed to create EFI Boot Manager entry on /dev/nvme0n1 partition 2 (ESP /dev/nvme0n1p2 )' + 2023-12-21 07:01:38.279364912 efibootmgr failed to create EFI Boot Manager entry on /dev/nvme0n1 partition 2 (ESP /dev/nvme0n1p2 ) + ++ PrintError 'efibootmgr failed to create EFI Boot Manager entry on /dev/nvme0n1 partition 2 (ESP /dev/nvme0n1p2 )' + ++ is_true 1 + ++ case "$1" in + ++ return 0 + ++ LogPrintError 'efibootmgr failed to create EFI Boot Manager entry for '\''EFI\BOOT\grub.efi'\'' (UEFI_BOOTLOADER='\''/boot/efi/EFI/BOOT/grub.efi'\'')' + ++ Log 'efibootmgr failed to create EFI Boot Manager entry for '\''EFI\BOOT\grub.efi'\'' (UEFI_BOOTLOADER='\''/boot/efi/EFI/BOOT/grub.efi'\'')' + ++ test -w /var/log/brutils/brutils-ip-172-31-7-113.log + ++ echo '2023-12-21 07:01:38.285741702 efibootmgr failed to create EFI Boot Manager entry for '\''EFI\BOOT\grub.efi'\'' (UEFI_BOOTLOADER='\''/boot/efi/EFI/BOOT/grub.efi'\'')' + 2023-12-21 07:01:38.285741702 efibootmgr failed to create EFI Boot Manager entry for 'EFI\BOOT\grub.efi' (UEFI_BOOTLOADER='/boot/efi/EFI/BOOT/grub.efi') + ++ PrintError 'efibootmgr failed to create EFI Boot Manager entry for '\''EFI\BOOT\grub.efi'\'' (UEFI_BOOTLOADER='\''/boot/efi/EFI/BOOT/grub.efi'\'')' + ++ return 1 + +#### [jsmeix](https://github.com/jsmeix) commented at [2023-12-21 12:57](https://github.com/rear/rear/issues/3115#issuecomment-1866200558): + +Works for me on my SLES15-SP4 test VM (QEMU/KVM) +even with Secure Boot: + + RESCUE localhost:~ # rear -D recover + ... + Running mkinitrd... + Recreated initrd (/sbin/mkinitrd). + Creating EFI Boot Manager entries... + Creating EFI Boot Manager entry 'SUSE_LINUX 15.4' for 'EFI\sles\shim.efi' (UEFI_BOOTLOADER='/boot/efi/EFI/sles/shim.efi') + Installing secure boot loader (shim)... + Running 'wrapup' stage ====================== + Finished 'recover'. The target system is mounted at '/mnt/local'. + +Except from what +usr/share/rear/finalize/Linux-i386/670\_run\_efibootmgr.sh +does in the log file: + + ++ disk=/dev/vda + ++ LogPrint 'Creating EFI Boot Manager entry '\''SUSE_LINUX 15.4'\'' for '\''EFI\sles\shim.efi'\'' (UEFI_BOOTLOADER='\''/boot/efi/EFI/sles/shim.efi'\' + ') ' + ++ Log 'Creating EFI Boot Manager entry '\''SUSE_LINUX 15.4'\'' for '\''EFI\sles\shim.efi'\'' (UEFI_BOOTLOADER='\''/boot/efi/EFI/sles/shim.efi'\'') ' + ++ test -w /var/log/rear/rear-localhost.log + ++ echo '2023-12-21 13:53:30.021779451 Creating EFI Boot Manager entry '\''SUSE_LINUX 15.4'\'' for '\''EFI\sles\shim.efi'\'' (UEFI_BOOTLOADER='\''/boo + t/efi/EFI/sles/shim.efi'\'') ' + 2023-12-21 13:53:30.021779451 Creating EFI Boot Manager entry 'SUSE_LINUX 15.4' for 'EFI\sles\shim.efi' (UEFI_BOOTLOADER='/boot/efi/EFI/sles/shim.efi' + ) + ++ Print 'Creating EFI Boot Manager entry '\''SUSE_LINUX 15.4'\'' for '\''EFI\sles\shim.efi'\'' (UEFI_BOOTLOADER='\''/boot/efi/EFI/sles/shim.efi'\'') + ' + ++ Log efibootmgr --create --gpt --disk /dev/vda --part 1 --write-signature --label '"SUSE_LINUX' '15.4"' --loader '"\EFI\sles\shim.efi"' + ++ test -w /var/log/rear/rear-localhost.log + ++ echo '2023-12-21 13:53:30.027151905 efibootmgr --create --gpt --disk /dev/vda --part 1 --write-signature --label "SUSE_LINUX 15.4" --loader "\EFI\sl + es\shim.efi"' + 2023-12-21 13:53:30.027151905 efibootmgr --create --gpt --disk /dev/vda --part 1 --write-signature --label "SUSE_LINUX 15.4" --loader "\EFI\sles\shim.e + fi" + ++ efibootmgr --create --gpt --disk /dev/vda --part 1 --write-signature --label 'SUSE_LINUX 15.4' --loader '\EFI\sles\shim.efi' + efibootmgr: ** Warning ** : Boot000B has same label SUSE_LINUX 15.4 + BootCurrent: 0005 + Timeout: 3 seconds + BootOrder: 000C,0001,000B,0002,0000,0003,0004,0005,0006,0007,0008,0009,000A + Boot0000* UiApp + Boot0001* sles-secureboot + Boot0002* UEFI Misc Device + Boot0003* EFI Internal Shell + Boot0004* UEFI QEMU DVD-ROM QM00001 + Boot0005* UEFI QEMU DVD-ROM QM00003 + Boot0006* UEFI Misc Device 2 + Boot0007* UEFI PXEv4 (MAC:5254001BF414) + Boot0008* UEFI PXEv6 (MAC:5254001BF414) + Boot0009* UEFI HTTPv4 (MAC:5254001BF414) + Boot000A* UEFI HTTPv6 (MAC:5254001BF414) + Boot000B* SUSE_LINUX 15.4 + Boot000C* SUSE_LINUX 15.4 + ++ NOBOOTLOADER= + +#### [abbbi](https://github.com/abbbi) commented at [2024-01-10 21:15](https://github.com/rear/rear/issues/3115#issuecomment-1885745899): + +booted via regular bios during recovery? missing efivarfs (modprobe) and +as result sys/firmware/efi/efivars not correctly mounted? @ramzcode + +#### [schlomo](https://github.com/schlomo) commented at [2024-01-13 18:34](https://github.com/rear/rear/issues/3115#issuecomment-1890678969): + +@ramzcode you write that you use an AWS EC2 instance, did you configure +your AMI to use `uefi` as described in +[https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ami-boot.html](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ami-boot.html)? +Can you please check with `efibootmgr -v` (and share us the result) how +it looks on your source machine? Or is this a P2V scenario where you try +to restore a system **onto** AWS EC2 that was previously running +somewhere else? + +#### [github-actions](https://github.com/apps/github-actions) commented at [2024-03-14 02:01](https://github.com/rear/rear/issues/3115#issuecomment-1996256496): + +Stale issue message + +#### [ramzcode](https://github.com/ramzcode) commented at [2024-04-19 07:06](https://github.com/rear/rear/issues/3115#issuecomment-2065892132): + +@schlomo Dropped 15.5 and tested with 15.2 and it worked, did not get +time to test 15.5 further. Also moved was between AWS EC2 + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2023-12-21.3116.issue.closed.md b/docs/issues/2023-12-21.3116.issue.closed.md new file mode 100644 index 00000000..e8045a09 --- /dev/null +++ b/docs/issues/2023-12-21.3116.issue.closed.md @@ -0,0 +1,388 @@ +[\#3116 Issue](https://github.com/rear/rear/issues/3116) `closed`: Changing TMPDIR in local.conf not working +============================================================================================================ + +**Labels**: `support / question`, `no-issue-activity` + +#### [nnachtegael](https://github.com/nnachtegael) opened issue at [2023-12-21 17:01](https://github.com/rear/rear/issues/3116): + + + +- ReaR version ("/usr/sbin/rear -V"): + + + + from ~/rear + sudo usr/sbin/rear -V + Relax-and-Recover 2.7 / Git + +- If your ReaR version is not the current version, explain why you + can't upgrade: + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): + + + + No LSB modules are available. + Distributor ID: Ubuntu + Description: Ubuntu 22.04.3 LTS + Release: 22.04 + Codename: jammy + +- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat + /etc/rear/local.conf"): + + + + OUTPUT=USB + BACKUP=NETFS + BACKUP_PROG=rsync + BACKUP_URL="usb:///dev/disk/by-label/REAR-ORICO4TB" + USB_DEVICE_FILESYSTEM=ext4 + USB_RETAIN_BACKUP_NR=1 + #export TMPDIR="/mnt/sde2/reartmp" + export TMPDIR="${TMPDIR-/mnt/sde2/reartmp}" + +- Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM + guest or PowerVM LPAR): + Home build PC + +- System architecture (x86 compatible or PPC64/PPC64LE or what exact + ARM device): + x86 + +- Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or + ELILO or Petitboot): + +- Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or + multipath (DM or NVMe): + + + + sudo fdisk -l + Disk /dev/nvme0n1: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors + Disk model: Seagate FireCuda 510 SSD ZP1000GM30001 + Units: sectors of 1 * 512 = 512 bytes + Sector size (logical/physical): 512 bytes / 512 bytes + I/O size (minimum/optimal): 512 bytes / 512 bytes + Disklabel type: gpt + Disk identifier: CB1BC206-CD11-40A8-95D5-EBDAE3FFFA5D + + Device Start End Sectors Size Type + /dev/nvme0n1p1 2048 999423 997376 487M EFI System + /dev/nvme0n1p2 999424 1953523711 1952524288 931G Linux RAID + + + Disk /dev/nvme1n1: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors + Disk model: Seagate FireCuda 510 SSD ZP1000GM30001 + Units: sectors of 1 * 512 = 512 bytes + Sector size (logical/physical): 512 bytes / 512 bytes + I/O size (minimum/optimal): 512 bytes / 512 bytes + Disklabel type: gpt + Disk identifier: 1E6695CA-F1E5-40EF-9BE4-972661123072 + + Device Start End Sectors Size Type + /dev/nvme1n1p1 2048 999423 997376 487M EFI System + /dev/nvme1n1p2 999424 1953523711 1952524288 931G Linux RAID + + + Disk /dev/sda: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors + Disk model: WDC WD2002FFSX-6 + Units: sectors of 1 * 512 = 512 bytes + Sector size (logical/physical): 512 bytes / 4096 bytes + I/O size (minimum/optimal): 4096 bytes / 4096 bytes + + + Disk /dev/sdb: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors + Disk model: WDC WD2002FFSX-6 + Units: sectors of 1 * 512 = 512 bytes + Sector size (logical/physical): 512 bytes / 4096 bytes + I/O size (minimum/optimal): 4096 bytes / 4096 bytes + + + Disk /dev/sdc: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors + Disk model: WDC WD2002FFSX-6 + Units: sectors of 1 * 512 = 512 bytes + Sector size (logical/physical): 512 bytes / 4096 bytes + I/O size (minimum/optimal): 4096 bytes / 4096 bytes + + + Disk /dev/md0: 2.73 TiB, 3000395366400 bytes, 5860147200 sectors + Units: sectors of 1 * 512 = 512 bytes + Sector size (logical/physical): 512 bytes / 4096 bytes + I/O size (minimum/optimal): 524288 bytes / 1572864 bytes + + + Disk /dev/md128: 930.91 GiB, 999557169152 bytes, 1952260096 sectors + Units: sectors of 1 * 512 = 512 bytes + Sector size (logical/physical): 512 bytes / 512 bytes + I/O size (minimum/optimal): 512 bytes / 512 bytes + + + Disk /dev/sdd: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors + Disk model: ASM236X NVME + Units: sectors of 1 * 512 = 512 bytes + Sector size (logical/physical): 512 bytes / 512 bytes + I/O size (minimum/optimal): 512 bytes / 33553920 bytes + Disklabel type: gpt + Disk identifier: C8A7D900-2F8F-44C9-84CA-5AAC269EC59D + + Device Start End Sectors Size Type + /dev/sdd1 16384 2113535 2097152 1G EFI System + /dev/sdd2 2113536 7814037134 7811923599 3.6T Linux filesystem + + + Disk /dev/sde: 4.55 TiB, 5000947302400 bytes, 9767475200 sectors + Disk model: Elements 2620 + Units: sectors of 1 * 512 = 512 bytes + Sector size (logical/physical): 512 bytes / 4096 bytes + I/O size (minimum/optimal): 4096 bytes / 4096 bytes + Disklabel type: gpt + Disk identifier: 096E1D0E-60BE-4FB0-A184-A78C80E1F24A + + Device Start End Sectors Size Type + /dev/sde1 16384 2113535 2097152 1G EFI System + /dev/sde2 2113536 9767473151 9765359616 4.5T Microsoft basic data + +- Storage layout ("lsblk -ipo + NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"): + + + + NAME KNAME PKNAME TRAN TYPE FSTYPE LABEL SIZE MOUNTPOINT + /dev/sda /dev/sda sata disk linux_raid_member ubuntu-host:0 1.8T + `-/dev/md0 /dev/md0 /dev/sda raid10 ext4 2.7T /mnt/md0 + /dev/sdb /dev/sdb sata disk linux_raid_member ubuntu-host:0 1.8T + `-/dev/md0 /dev/md0 /dev/sdb raid10 ext4 2.7T /mnt/md0 + /dev/sdc /dev/sdc sata disk linux_raid_member ubuntu-host:0 1.8T + `-/dev/md0 /dev/md0 /dev/sdc raid10 ext4 2.7T /mnt/md0 + /dev/sdd /dev/sdd usb disk 3.6T + |-/dev/sdd1 /dev/sdd1 /dev/sdd part vfat REAR-EFI 1G + `-/dev/sdd2 /dev/sdd2 /dev/sdd part ext4 REAR-ORICO4TB 3.6T + /dev/sde /dev/sde usb disk 4.5T + |-/dev/sde1 /dev/sde1 /dev/sde part vfat REAR-EFI 1G + `-/dev/sde2 /dev/sde2 /dev/sde part ntfs wd5tb 4.5T /mnt/sde2 + /dev/sr0 /dev/sr0 sata rom 1024M + /dev/nvme0n1 /dev/nvme0n1 nvme disk 931.5G + |-/dev/nvme0n1p1 /dev/nvme0n1p1 /dev/nvme0n1 nvme part vfat 487M /boot/efi + `-/dev/nvme0n1p2 /dev/nvme0n1p2 /dev/nvme0n1 nvme part linux_raid_member kvm-host:128 931G + `-/dev/md128 /dev/md128 /dev/nvme0n1p2 raid1 ext4 930.9G / + /dev/nvme1n1 /dev/nvme1n1 nvme disk 931.5G + |-/dev/nvme1n1p1 /dev/nvme1n1p1 /dev/nvme1n1 nvme part vfat 487M + `-/dev/nvme1n1p2 /dev/nvme1n1p2 /dev/nvme1n1 nvme part linux_raid_member kvm-host:128 931G + `-/dev/md128 /dev/md128 /dev/nvme1n1p2 raid1 ext4 930.9G / + +- Description of the issue (ideally so that others can reproduce it): + +I try to change the TEMPDIR for my rsync backup to an attached USB +/dev/sde2/reartmp directory. +The directory /reartmp exists on the USB disk. + +I did an export TMPDIR="/mnt/sde2/reartmp" +before running the usr/sbin/rear -v mkbackup + +TEMPDIR stays at /var/tmp even after the +export TMPDIR="${TMPDIR-/mnt/sde2/reartmp}" +in my /home/nnoel/rear/etc/rear/local.conf + +see in my attached log file + + 2023-12-21 16:57:16.200891182 Creating rsync archive '/var/tmp/rear.dO7flE5aeOqUj3H/outputfs/rear/kvm-host/20231221.1655/backup' + +if I try export TMPDIR="/mnt/sde2/reartmp" in local.conf +it also doesn't work: same situation TMPDIR stays at /var/tmp + +- Workaround, if any: + +- - Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" + debug log files): + ~/rear$ sudo usr/sbin/rear -v mkbackup log file + +[rear-kvm-host.log](https://github.com/rear/rear/files/13744404/rear-kvm-host.log) + +You can drag-drop log files into this editor to create an attachment +or paste verbatim text like command output or file content +by including it between a leading and a closing line of +three backticks like this: + + verbatim content + +#### [jsmeix](https://github.com/jsmeix) commented at [2023-12-22 08:22](https://github.com/rear/rear/issues/3116#issuecomment-1867380472): + +@nnachtegael +usr/share/rear/conf/default.conf reads + + # To have a specific working area directory prefix for Relax-and-Recover call + # export TMPDIR="/prefix/for/rear/working/directory" + # before calling 'rear' (/prefix/for/rear/working/directory must already exist). + +see +[https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/default.conf\#L55](https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/default.conf#L55) + +#### [jsmeix](https://github.com/jsmeix) commented at [2023-12-22 08:32](https://github.com/rear/rear/issues/3116#issuecomment-1867389860): + +For some reasoning and details behind see also +[https://github.com/rear/rear/issues/2654](https://github.com/rear/rear/issues/2654) + +#### [nnachtegael](https://github.com/nnachtegael) commented at [2023-12-22 10:58](https://github.com/rear/rear/issues/3116#issuecomment-1867546374): + +I restarted from beginning + +- $ sudo -i +- \#export TMPDIR="/mnt/sde2/reartmp" +- set permissions 777 for /mnt/sde2/reartmp +- \#git clone + [https://github.com/rear/rear.git](https://github.com/rear/rear.git) +- edit /root/rear/etc/rear/local.conf + +OUTPUT=USB +BACKUP=NETFS +BACKUP\_PROG=rsync +BACKUP\_URL="usb:///dev/disk/by-label/REAR-000" +USB\_DEVICE\_FILESYSTEM=ext4 +USB\_RETAIN\_BACKUP\_NR=1 +export TMPDIR="/mnt/sde2/reartmp" + +- \#usr/sbin/rear -v format -- --efi /dev/sdd + +- message is "Could not remove build area + /mnt/sde2/reartmp/rear.1GDQ69LwxnIc1SU" + [rear-format-efi.log](https://github.com/rear/rear/files/13751623/rear-format-efi.log) + +- manually cleaned /mnt/sde2/reartmp + +- \#usr/sbin/rear -v mkbackup + +- messages "/bin/ldd: line 158: /dev/null: Permission denied" + +- BUG in + /root/rear/usr/share/rear/build/default/990\_verify\_rootfs.sh line + 57: + ReaR recovery system in + '/mnt/sde2/reartmp/rear.6UbnO5EbBKxcYHs/rootfs' is broken: 'ldd + /bin/bash' failed + [rear-mkbackup.log](https://github.com/rear/rear/files/13751684/rear-mkbackup.log) + +I'm now stopped by this ldd error message +Searching for this ldd error did not gave me an answer... + +#### [jsmeix](https://github.com/jsmeix) commented at [2023-12-22 12:31](https://github.com/rear/rear/issues/3116#issuecomment-1867633280): + +It "just works" for me with + + OUTPUT=USB + BACKUP=NETFS + BACKUP_URL=usb:///dev/disk/by-label/REAR-000 + +I.e. I do the backup with 'tar' not with 'rsync' +because I am not a 'rsync' user. +And I do not use UEFI but BIOS. + +As TMPDIR I use a mounted partiton +with ext2 filesystem on my /dev/sda harddisk: + + # mount -v /dev/sda6 /other + mount: /dev/sda6 mounted on /other. + + # export TMPDIR="/other" + + # ls -ld /other + drwxr-xr-x 5 root root 4096 Dec 15 12:04 /other + + # usr/sbin/rear -D format /dev/sdb + ... + Using build area: /other/rear.D6fkLoxK8bHihSv + ... + Exiting rear format (PID 13329) and its descendant processes ... + Running exit tasks + To remove the build area you may use (with caution): rm -Rf --one-file-system /other/rear.D6fkLoxK8bHihSv + + # usr/sbin/rear -D mkbackup + ... + Using build area: /other/rear.R0pgPzGqxtkAYha + ... + Exiting rear mkbackup (PID 15867) and its descendant processes ... + Running exit tasks + To remove the build area you may use (with caution): rm -Rf --one-file-system /other/rear.R0pgPzGqxtkAYha + +Also with automated removal of the build area + + # usr/sbin/rear -v format /dev/sdb + ... + Exiting rear format (PID 11371) and its descendant processes ... + Running exit tasks + + # usr/sbin/rear -v mkbackup + ... + Exiting rear mkbackup (PID 12418) and its descendant processes ... + Running exit tasks + +I checked that during "rear -v format" and "rear -v mkbackup" +build areas appeared under /other/ and got automatically removed. + +#### [jsmeix](https://github.com/jsmeix) commented at [2023-12-22 12:43](https://github.com/rear/rear/issues/3116#issuecomment-1867645120): + +I cannot help with 'rsync' specific issues +because I am not a 'rsync' user. +I assume it does not matter regarding TMPDIR +if 'tar' or 'rsync' is used. + +I don't know if 'rsync' with OUTPUT=USB works with + + OUTPUT=USB + BACKUP=NETFS + BACKUP_PROG=rsync + BACKUP_URL="usb:///dev/disk/by-label/REAR-000" + USB_DEVICE_FILESYSTEM=ext4 + USB_RETAIN_BACKUP_NR=1 + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-01-02 11:24](https://github.com/rear/rear/issues/3116#issuecomment-1873905770): + + NAME KNAME PKNAME TRAN TYPE FSTYPE LABEL SIZE MOUNTPOINT + /dev/sde /dev/sde usb disk 4.5T + |-/dev/sde1 /dev/sde1 /dev/sde part vfat REAR-EFI 1G + `-/dev/sde2 /dev/sde2 /dev/sde part ntfs wd5tb 4.5T /mnt/sde2 + + #export TMPDIR="/mnt/sde2/reartmp" + +I doubt that putting TMPDIR on a NTFS filesystem can work properly. Most +likely NTFS does not support all the needed attributes, like the +executable bit (that would explain the ldd error). + +#### [schlomo](https://github.com/schlomo) commented at [2024-01-13 18:28](https://github.com/rear/rear/issues/3116#issuecomment-1890670683): + +@nnachtegael can you please share with us the actual problem you are +trying to solve? Can it be that your system is so full that there is not +enough disk space on any hard disk for the ReaR build area? Or do you +want to do a ReaR backup without touching the local disks? + +In any case, as @pcahyna mentioned the `TMPDIR` for sure needs to be a +regular **Linux** filesystem. + +The same holds true for the target of an rsync backup, but looking at +your disks it seems like that is already the case. + +For backup to USB storage I'd also recommend using an archive instead of +rsync as you would then benefit from the compression, which both reduces +the required disk space and probably increases the backup performance. + +So if you can share more of the problem you want to solve (like big +picture) or more of your context, then we can surely give you some ideas +how to solve it. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-02-08 13:41](https://github.com/rear/rear/issues/3116#issuecomment-1934147881): + +It seems that TMPDIR in local.conf will not work in any case: \#2654 + +#### [github-actions](https://github.com/apps/github-actions) commented at [2024-04-09 02:03](https://github.com/rear/rear/issues/3116#issuecomment-2044008798): + +Stale issue message + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2023-12-27.3117.issue.closed.md b/docs/issues/2023-12-27.3117.issue.closed.md new file mode 100644 index 00000000..21753335 --- /dev/null +++ b/docs/issues/2023-12-27.3117.issue.closed.md @@ -0,0 +1,198 @@ +[\#3117 Issue](https://github.com/rear/rear/issues/3117) `closed`: BACKUP=RSYNC progress not showing with '--progress' in BACKUP\_RSYNC\_OPTIONS +================================================================================================================================================ + +**Labels**: `support / question`, `no-issue-activity` + +#### [DheerajSS2000](https://github.com/DheerajSS2000) opened issue at [2023-12-27 12:31](https://github.com/rear/rear/issues/3117): + + + +- ReaR version ("/usr/sbin/rear -V"): + Relax-and-Recover 2.7 / Git + +- If your ReaR version is not the current version, explain why you + can't upgrade: + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): + + + + NAME="Fedora Linux" + VERSION="35 (Workstation Edition)" + ID=fedora + VERSION_ID=35 + VERSION_CODENAME="" + PLATFORM_ID="platform:f35" + PRETTY_NAME="Fedora Linux 35 (Workstation Edition)" + ANSI_COLOR="0;38;2;60;110;180" + LOGO=fedora-logo-icon + CPE_NAME="cpe:/o:fedoraproject:fedora:35" + HOME_URL="https://fedoraproject.org/" + DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f35/system-administrators-guide/" + SUPPORT_URL="https://ask.fedoraproject.org/" + BUG_REPORT_URL="https://bugzilla.redhat.com/" + REDHAT_BUGZILLA_PRODUCT="Fedora" + REDHAT_BUGZILLA_PRODUCT_VERSION=35 + REDHAT_SUPPORT_PRODUCT="Fedora" + REDHAT_SUPPORT_PRODUCT_VERSION=35 + PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy" + VARIANT="Workstation Edition" + VARIANT_ID=workstation + +- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat + /etc/rear/local.conf"): + + + + OUTPUT=ISO + OUTPUT_URL="rsync://dheeraj@::backup/" + BACKUP=RSYNC + BACKUP_PROG=rsync + BACKUP_URL="rsync://dheeraj@::backup/" + BACKUP_RSYNC_OPTIONS+=(-z -r -v --progress --password-file=/etc/rear/rsync_pass) + +- Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM + guest or PowerVM LPAR): + Virtual machine + +- System architecture (x86 compatible or PPC64/PPC64LE or what exact + ARM device): + x86 compatible + +- Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or + ELILO or Petitboot): + BIOS and GRUB + +- Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or + multipath (DM or NVMe): + local disk + +- Storage layout ("lsblk -ipo + NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"): + + + + [fedora@fedora35 rear]$ lsblk + + NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS + sda 8:0 0 512G 0 disk + ├─sda1 8:1 0 1G 0 part /boot + └─sda2 8:2 0 511G 0 part /home + / + sr0 11:0 1 1024M 0 rom + zram0 252:0 0 3.8G 0 disk [SWAP] + +- Description of the issue (ideally so that others can reproduce it): + +While trying to backup and restore using rsync it is not showing +progress , it will be showing only " Backed up 0 MiB \[avg 0 KiB/sec\]", +util entire backup is completed. I tried this with fedora34,35 and 36, +the same issue i was facing for all these versions + +- Workaround, if any: + +- Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" + debug log files): + [rear-fedora35.log](https://github.com/rear/rear/files/13778558/rear-fedora35.log) + +You can drag-drop log files into this editor to create an attachment +or paste verbatim text like command output or file content +by including it between a leading and a closing line of +three backticks like this: + + verbatim content + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-01-02 11:52](https://github.com/rear/rear/issues/3117#issuecomment-1873932303): + +I believe this is due to the redirection of stdout from the backup +program. The output, including the progress info, can be found in +${TMP\_DIR}/${BACKUP\_PROG\_ARCHIVE}.log : + +[https://github.com/rear/rear/blob/0bd84e259c7c61612a1d8eb296ee1e81a2cbc87b/usr/share/rear/backup/RSYNC/default/500\_make\_rsync\_backup.sh\#L55](https://github.com/rear/rear/blob/0bd84e259c7c61612a1d8eb296ee1e81a2cbc87b/usr/share/rear/backup/RSYNC/default/500_make_rsync_backup.sh#L55) + +I understand that this behavior is not very useful in the case of +`--progress`. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-01-05 10:54](https://github.com/rear/rear/issues/3117#issuecomment-1878479578): + +There are two different kind of progress things here. + +ReaR's own progress subsystem via +usr/share/rear/lib/progresssubsystem.nosh +that shows ProgressInfo "Backed up XXX MiB \[avg YYY KiB/sec\]" +messages which seems to not work as intended because it is + + showing only " Backed up 0 MiB [avg 0 KiB/sec]", + util entire backup is completed + +instead of increasing numbers for "XXX MiB" +and current numbers for "YYY KiB/sec". + +Additionally in this case here there is +the `rsync -v --progress` info. +It is questionable whether or not that info +should appear on the user's terminal. +It is also questionable whether or not that info +should appear in the backup log file because: + +I am not a BACKUP=RSYNC user but I guess that perhaps +when there is '--progress' or '-v' in BACKUP\_RSYNC\_OPTIONS +then the code to generate the numbers for the +ProgressInfo message in +backup/RSYNC/default/500\_make\_rsync\_backup.sh +may no longer work, in particular note the line + + nfile="$(tail -1 "${TMP_DIR}/${BACKUP_PROG_ARCHIVE}.log")" + +[https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/backup/RSYNC/default/500\_make\_rsync\_backup.sh\#L98](https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/backup/RSYNC/default/500_make_rsync_backup.sh#L98) + +I guess because of '--progress' or because of '-v' +the last line in ${TMP\_DIR}/${BACKUP\_PROG\_ARCHIVE}.log +may be no longer what is needed for the code +to generate the numbers for the ProgressInfo messages. + +#### [schlomo](https://github.com/schlomo) commented at [2024-01-13 18:20](https://github.com/rear/rear/issues/3117#issuecomment-1890660220): + +I'd **guess** that the root cause of the problem is that you use the +`rsync` protocol and not the `ssh` protocol. The way how we "calculate" +the progress of the backup is actually via `df` and `du` with those two +functions: +[https://github.com/rear/rear/blob/d0496cc751b7f667eb0c8ab0cc5116c1ec05007a/usr/share/rear/backup/RSYNC/default/500\_make\_rsync\_backup.sh\#L65-L73](https://github.com/rear/rear/blob/d0496cc751b7f667eb0c8ab0cc5116c1ec05007a/usr/share/rear/backup/RSYNC/default/500_make_rsync_backup.sh#L65-L73) + +But since you use `rsync://` this probably doesn't work so that the +numbers stay `0` all the time. TBH, I cannot think of a clear way how to +provide truthful info on rsync progress without parsing the output. +There are multiple open source tools out there that can do this trick, +and provide a progress bar for rsync. Also, newer rsync versions support +`--info=progress2` that provides transfer statistics for the entire job. + +I guess that it would be a nice contribution to ReaR to rework the rsync +progress info to use that feature instead of checking the free disk +space via SSH. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-01-15 12:47](https://github.com/rear/rear/issues/3117#issuecomment-1892113801): + +> Also, newer rsync versions support `--info=progress2` that provides +> transfer statistics for the entire job. + +Redirection of output will hide it though, no? Just as in the case of +`--progress`. + +#### [schlomo](https://github.com/schlomo) commented at [2024-01-15 12:49](https://github.com/rear/rear/issues/3117#issuecomment-1892118263): + +Yes, but we can pass the rsync output directly to the user via `2>&8` or +something like that. + +#### [github-actions](https://github.com/apps/github-actions) commented at [2024-04-07 02:06](https://github.com/rear/rear/issues/3117#issuecomment-2041276243): + +Stale issue message + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-01-06.3121.issue.closed.md b/docs/issues/2024-01-06.3121.issue.closed.md new file mode 100644 index 00000000..588c2f4d --- /dev/null +++ b/docs/issues/2024-01-06.3121.issue.closed.md @@ -0,0 +1,140 @@ +[\#3121 Issue](https://github.com/rear/rear/issues/3121) `closed`: BACKUP=BORG --prefix deprecated --glob-archives could be used instead +======================================================================================================================================== + +**Labels**: `enhancement`, `external tool`, `no-issue-activity` + +#### [llucps](https://github.com/llucps) opened issue at [2024-01-06 16:19](https://github.com/rear/rear/issues/3121): + + + +- ReaR version ("/usr/sbin/rear -V"): + +Relax-and-Recover 2.7 / Git + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): + +Debian GNU/Linux 12 (bookworm) + +- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat + /etc/rear/local.conf"): + + + + ### Add library path + LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/systemd/ + + ### Rescue image will be the default (ISO) ### + OUTPUT=ISO + OUTPUT_URL="sshfs://xxx@xxx.xx/." + COPY_AS_IS=("/etc/ssl/private" "/usr/share/file/magic") + + ### Borg stuff ### + BACKUP=BORG + BORGBACKUP_HOST="xxxx.xxx.xx" + BORGBACKUP_PORT=23 + BORGBACKUP_USERNAME="xxxxx" + BORGBACKUP_REPO="/./xxxx-backup" + BORGBACKUP_ARCHIVE_PREFIX="$HOSTNAME" + BORGBACKUP_PRUNE_KEEP_DAILY=14 + BORGBACKUP_PRUNE_KEEP_WEEKLY=2 + BORGBACKUP_PRUNE_KEEP_MONTHLY=3 + BORGBACKUP_COMPRESSION="zlib,5" + COPY_AS_IS_EXCLUDE=("/tmp" "/dev" "/proc" "/sys" "/lost+found" "/home/lost+found/" "/run/" "/var/backup/server/" "/var/backup/websites/") + BORGBACKUP_SHOW_STATS=YES + export BORG_PASSPHRASE="xxxxxxxxxxxxx" + +- Description of the issue (ideally so that others can reproduce it): + +According to Borg documentation the `--prefix` parameter is deprecated +and should not be used anymore. + +[https://borgbackup.readthedocs.io/en/stable/usage/prune.html](https://borgbackup.readthedocs.io/en/stable/usage/prune.html) +[https://github.com/borgbackup/borg/issues/7031](https://github.com/borgbackup/borg/issues/7031) + +In my logs I get this warning: + + Pruning old backup archives in Borg repository /./xxxx-backup on xxx.xxx.xx + Warning: "--prefix" has been deprecated. Use "--glob-archives 'yourprefix*'" (-a) instead. + +I'm assuming that although the BORGBACKUP\_ARCHIVE\_PREFIX="$HOSTNAME" +works for now it should be replaced for something like this? + +`BORGBACKUP_ARCHIVE_GLOB="$HOSTNAME-*"` + +What do you think? +Thanks. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-01-10 14:13](https://github.com/rear/rear/issues/3121#issuecomment-1884928044): + +I am not a BACKUP=BORG user +so I cannot actually help with BACKUP=BORG specific issues. + +@llucps +do you know since when --glob-archives is supported by Borg? +I.e. since which Borg version is --glob-archives supported? + +I ask because in ReaR we maintain backward compatibility, see +[https://github.com/rear/rear/wiki/Coding-Style\#maintain-backward-compatibility](https://github.com/rear/rear/wiki/Coding-Style#maintain-backward-compatibility) + +So we cannot replace --prefix with --glob-archives +instead we must find some reasonable solution +that works for older and newer Borg versions. + +But we do not need to maintain backward compatibility +for obsoleted Borg versions that are no longer supported +by upstream Borg. + +On +[https://www.borgbackup.org/releases/](https://www.borgbackup.org/releases/) +I found (excerpts) + + Borg 2.0 is currently in testing - do not use it for production. + + Borg 1.4 is currently in testing - do not use it for production. + + Borg 1.2 is the current stable series of Borg. + The current release is 1.2.7, released on 2023-12-02. + + Borg 1.1 is not supported any more, please use Borg 1.2.x. + The current release is 1.1.18, released on 2022-06-05. + + Borg 1.0 is not supported any more, please use Borg 1.2.x. + The last release in this series was 1.0.13, released on 2019-02-15. + +#### [llucps](https://github.com/llucps) commented at [2024-01-10 16:43](https://github.com/rear/rear/issues/3121#issuecomment-1885212680): + +@jsmeix According the the release notes the --glob-archives option was +introduced in borg 1.2.2 (2022-08-20) + +[https://github.com/borgbackup/borg/blob/1.2.7/docs/changes.rst\#version-122-2022-08-20](https://github.com/borgbackup/borg/blob/1.2.7/docs/changes.rst#version-122-2022-08-20) + +And more info here +[https://github.com/borgbackup/borg/issues/6806](https://github.com/borgbackup/borg/issues/6806) + +#### [schlomo](https://github.com/schlomo) commented at [2024-01-13 17:59](https://github.com/rear/rear/issues/3121#issuecomment-1890631300): + +[https://github.com/borgbackup/borg/issues/6806\#issuecomment-1170319393](https://github.com/borgbackup/borg/issues/6806#issuecomment-1170319393) +actually states that Borg 2.0 will not support the `--prefix` option any +more, so the 1.x series presumably will continue to support it. + +@llucps can you tell us how much our ReaR code will have to change in +order to support Borg 2.x? If this is the only change, then we should +just adjust it. But if there are more incompatible changes then we +should maybe consider adding a `BORG2` backup method that is fully +optimised for the Borg 2.x series. We have also other backup methods +(most notably `GALAXY`) where we have multiple backup methods to support +multiple versions of the same backup software. + +#### [github-actions](https://github.com/apps/github-actions) commented at [2024-03-14 02:01](https://github.com/rear/rear/issues/3121#issuecomment-1996256474): + +Stale issue message + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-01-26.3139.issue.closed.md b/docs/issues/2024-01-26.3139.issue.closed.md new file mode 100644 index 00000000..912c8db2 --- /dev/null +++ b/docs/issues/2024-01-26.3139.issue.closed.md @@ -0,0 +1,152 @@ +[\#3139 Issue](https://github.com/rear/rear/issues/3139) `closed`: Syslog and journald in rescue system not showing service logs +================================================================================================================================ + +**Labels**: `bug`, `critical / security / legal`, `blocker` + +#### [schlomo](https://github.com/schlomo) opened issue at [2024-01-26 13:52](https://github.com/rear/rear/issues/3139): + +Current ReaR from master shows a strange problem with regard to logging +in the rescue system, for example: + + RESCUE rear-u2204:~ # journalctl -u ssh --no-pager --boot + -- No entries -- + +(I'm connected via SSH) + +whereas the original system shows me the log: + + root@rear-u2204:/src/rear# journalctl -u ssh --no-pager --boot + Jan 26 13:04:26 rear-u2204 systemd[1]: Starting OpenBSD Secure Shell server... + Jan 26 13:04:26 rear-u2204 sshd[634]: Server listening on 0.0.0.0 port 22. + Jan 26 13:04:26 rear-u2204 sshd[634]: Server listening on :: port 22. + Jan 26 13:04:26 rear-u2204 systemd[1]: Started OpenBSD Secure Shell server. + Jan 26 13:04:34 rear-u2204 sshd[873]: Accepted publickey for root from 192.168.11.6 port 44082 ssh2: RSA SHA256:seEU/B1WR98Ck/27Mh9MARstsMqMyMnv63t30+0w1K8 + Jan 26 13:04:34 rear-u2204 sshd[873]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0) + Jan 26 13:07:38 rear-u2204 sshd[2141]: Accepted publickey for root from 192.168.11.6 port 45826 ssh2: RSA SHA256:seEU/B1WR98Ck/27Mh9MARstsMqMyMnv63t30+0w1K8 + Jan 26 13:07:38 rear-u2204 sshd[2141]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0) + +Also syslog doesn't work as expected, again on the rescue system: + + RESCUE rear-u2204:~ # logger reartest + RESCUE rear-u2204:~ # journalctl --boot -g reartest + -- No entries -- + +and on the source system it works as expected: + + root@rear-u2204:/src/rear# logger reartest + root@rear-u2204:/src/rear# journalctl --boot -g reartest + Jan 26 13:40:34 rear-u2204 root[53389]: reartest + +I actually observed this while working on \#3138 but didn't want to add +this problem into that PR. The problem was **identical in appearance** +on all systems (RHEL 7,8,9 and SLES 12,15 and Ubuntu 20,22). + +Especially the syslog problem is very strange as it seems that +`/dev/log` is configured correctly: + + RESCUE rear-u2204:~ # systemctl list-sockets + LISTEN UNIT ACTIVATES + /dev/log systemd-journald.socket systemd-journald.service + /run/systemd/journal/socket systemd-journald.socket systemd-journald.service + /run/systemd/journal/stdout systemd-journald.socket systemd-journald.service + /run/udev/control systemd-udevd-control.socket systemd-udevd.service + kobject-uevent 1 systemd-udevd-kernel.socket systemd-udevd.service + + 5 sockets listed. + Pass --all to see loaded but inactive sockets, too. + RESCUE rear-u2204:~ # systemctl status systemd-journald + ● systemd-journald.service - Journal Service + Loaded: loaded (/usr/lib/systemd/system/systemd-journald.service; static) + Active: active (running) since Fri 2024-01-26 13:31:15 UTC; 12min ago + TriggeredBy: ● systemd-journald.socket + Docs: man:systemd-journald.service(8) + man:systemd-journald.conf(5) + Main PID: 75 (systemd-journal) + Status: "Processing requests..." + Tasks: 1 (limit: 4563) + Memory: 9.1M + CPU: 35ms + CGroup: /system.slice/systemd-journald.service + └─75 /usr/lib/systemd/systemd-journald + + Jan 26 13:31:15 rear-u2204 systemd-journald[75]: Journal started + Jan 26 13:31:15 rear-u2204 systemd-journald[75]: Runtime Journal (/run/log/journal/3a64d7109b544834b661896de0e54ded) is 8.0M, max 77.8M, 69.8M free. + Notice: journal has been rotated since unit was started, output may be incomplete. + +Starting Rsyslog also doesn't work because some rsyslog modules are +missing: + + RESCUE rear-u2204:~ # /etc/scripts/run-syslog + rsyslog internal message (3,-2066): could not load module 'lmnet', errors: trying to load module /usr/lib/x86_64-linux-gnu/rsyslog/lmnet.so: /usr/lib/x86_64-linux-gnu/rsyslog/lmnet.so: cannot open shared object file: No such file or directory [v8.2112.0 try https://www.rsyslog.com/e/2066 ] + Error during class init for object 'conf' - failing... + rsyslogd initializiation failed - global classes could not be initialized. + Did you do a "make install"? + Suggested action: run rsyslogd with -d -n options to see what exactly fails. + rsyslogd: run failed with error -2066 (see rsyslog.h or try https://www.rsyslog.com/e/2066 to learn what that number means) + +In general the JournalD log looks very different from the one on the +source machine, for example it reports all lines as if they belong to +systemd, for example + + RESCUE rear-u2204:~ # journalctl -ef --no-pager -n 20 + Jan 26 13:45:33 rear-u2204 systemd[1]: Registering bus object implementation for path=/org/freedesktop/systemd1/unit iface=org.freedesktop.systemd1.Service + Jan 26 13:45:33 rear-u2204 systemd[1]: Registering bus object implementation for path=/org/freedesktop/systemd1/unit iface=org.freedesktop.systemd1.Slice + Jan 26 13:45:33 rear-u2204 systemd[1]: Registering bus object implementation for path=/org/freedesktop/systemd1/unit iface=org.freedesktop.systemd1.Socket + Jan 26 13:45:33 rear-u2204 systemd[1]: Registering bus object implementation for path=/org/freedesktop/systemd1/unit iface=org.freedesktop.systemd1.Swap + Jan 26 13:45:33 rear-u2204 systemd[1]: Registering bus object implementation for path=/org/freedesktop/systemd1/unit iface=org.freedesktop.systemd1.Target + Jan 26 13:45:33 rear-u2204 systemd[1]: Registering bus object implementation for path=/org/freedesktop/systemd1/unit iface=org.freedesktop.systemd1.Timer + Jan 26 13:45:33 rear-u2204 systemd[1]: Registering bus object implementation for path=/org/freedesktop/LogControl1 iface=org.freedesktop.LogControl1 + Jan 26 13:45:33 rear-u2204 systemd[1]: Accepted new private connection. + Jan 26 13:45:33 rear-u2204 systemd[1]: Bus private-bus-connection: changing state AUTHENTICATING → RUNNING + Jan 26 13:45:33 rear-u2204 systemd[1]: Got message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1/unit/rsyslog_2eservice interface=org.freedesktop.DBus.Properties member=GetAll cookie=1 reply_cookie=0 signature=s error-name=n/a error-message=n/a + Jan 26 13:45:33 rear-u2204 systemd[1]: Failed to read pids.max attribute of cgroup root, ignoring: No data available + Jan 26 13:45:33 rear-u2204 systemd[1]: Sent message type=method_return sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a cookie=1 reply_cookie=1 signature=a{sv} error-name=n/a error-message=n/a + Jan 26 13:45:33 rear-u2204 systemd[1]: Got message type=method_call sender=n/a destination=org.freedesktop.systemd1 path=/org/freedesktop/systemd1/unit/syslog_2esocket interface=org.freedesktop.DBus.Properties member=Get cookie=2 reply_cookie=0 signature=ss error-name=n/a error-message=n/a + Jan 26 13:45:33 rear-u2204 systemd[1]: Sent message type=method_return sender=org.freedesktop.systemd1 destination=n/a path=n/a interface=n/a member=n/a cookie=2 reply_cookie=2 signature=v error-name=n/a error-message=n/a + Jan 26 13:45:33 rear-u2204 systemd[1]: Bus private-bus-connection: changing state RUNNING → CLOSING + Jan 26 13:45:33 rear-u2204 systemd[1]: Bus private-bus-connection: changing state CLOSING → CLOSED + Jan 26 13:45:33 rear-u2204 systemd[1]: Got disconnect on private connection. + Jan 26 13:47:17 rear-u2204 systemd[1]: Received SIGCHLD from PID 632 (less). + Jan 26 13:47:17 rear-u2204 systemd[1]: Child 632 (less) died (code=exited, status=0/SUCCESS) + Jan 26 13:47:17 rear-u2204 systemd[1]: sshd.service: Child 632 belongs to sshd.service. + +(while running this I connected via SSH to this box and no new log lines +appeared) + +This problem makes debugging stuff in the rescue system very difficult +and I'd consider it a critical bug. + +@rear/contributors do you experience the same? Any ideas what could be +the problem here? + +#### [schlomo](https://github.com/schlomo) commented at [2024-01-26 13:55](https://github.com/rear/rear/issues/3139#issuecomment-1912107171): + +A quick test with ReaR version `2.6` shows unfortunately exactly the +same behaviour 😢 + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-01-27 13:33](https://github.com/rear/rear/issues/3139#issuecomment-1913157701): + +> The problem was identical in appearance on all systems (RHEL 7,8,9 and +> SLES 12,15 and Ubuntu 20,22). + +strange, I saw the problem on RHEL 9 (this was the motivation for fixing +the rsyslog startup in PR \#3041 , which is clearly not a working +solution on Ubuntu) but not RHEL 8. syslog messages went to journal +there. + +#### [schlomo](https://github.com/schlomo) commented at [2024-01-27 17:06](https://github.com/rear/rear/issues/3139#issuecomment-1913261630): + +@pcahyna TBH, I didn't do the `logger` test on all systems - so that it +indeed might have not been a problem on all of them. But I did check for +service logs and that was as described above (service logs missing from +journal) on all systems. The reason for checking was that the PPDM +service fails to start on boot, so I had to check for that. + +#### [github-actions](https://github.com/apps/github-actions) commented at [2024-03-28 02:04](https://github.com/rear/rear/issues/3139#issuecomment-2024269568): + +Stale issue message + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-07.3146.issue.closed.md b/docs/issues/2024-02-07.3146.issue.closed.md new file mode 100644 index 00000000..e392db43 --- /dev/null +++ b/docs/issues/2024-02-07.3146.issue.closed.md @@ -0,0 +1,340 @@ +[\#3146 Issue](https://github.com/rear/rear/issues/3146) `closed`: mkopalpba fails on Fedora 39 +=============================================================================================== + +**Labels**: `support / question`, `no-issue-activity` + +#### [kolaqsq](https://github.com/kolaqsq) opened issue at [2024-02-07 09:59](https://github.com/rear/rear/issues/3146): + +- ReaR version ("/usr/sbin/rear -V"): + + + + $ /usr/sbin/rear -V + Relax-and-Recover 2.7-git.5364.096bfde5.master / 2024-02-06 + +- If your ReaR version is not the current version, explain why you + can't upgrade: + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): + + + + $ cat /etc/os-release + NAME="Fedora Linux" + VERSION="39 (Workstation Edition)" + ID=fedora + VERSION_ID=39 + VERSION_CODENAME="" + PLATFORM_ID="platform:f39" + PRETTY_NAME="Fedora Linux 39 (Workstation Edition)" + ANSI_COLOR="0;38;2;60;110;180" + LOGO=fedora-logo-icon + CPE_NAME="cpe:/o:fedoraproject:fedora:39" + DEFAULT_HOSTNAME="fedora" + HOME_URL="https://fedoraproject.org/" + DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f39/system-administrators-guide/" + SUPPORT_URL="https://ask.fedoraproject.org/" + BUG_REPORT_URL="https://bugzilla.redhat.com/" + REDHAT_BUGZILLA_PRODUCT="Fedora" + REDHAT_BUGZILLA_PRODUCT_VERSION=39 + REDHAT_SUPPORT_PRODUCT="Fedora" + REDHAT_SUPPORT_PRODUCT_VERSION=39 + SUPPORT_END=2024-11-12 + VARIANT="Workstation Edition" + VARIANT_ID=workstation + +- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat + /etc/rear/local.conf"): + + + + $ sudo cat /etc/rear/local.conf + # This file etc/rear/local.conf is intended for the user's + # manual configuration of Relax-and-Recover (ReaR). + # For configuration through packages and other automated means + # we recommend a separated file named site.conf next to this file + # and leave local.conf as is (ReaR upstream will never ship a site.conf). + # The default OUTPUT=ISO creates the ReaR rescue medium as ISO image. + # You need to specify your particular backup and restore method for your data + # as the default BACKUP=REQUESTRESTORE does not really do that (see "man rear"). + # Configuration variables are documented in /usr/share/rear/conf/default.conf + # and the examples in /usr/share/rear/conf/examples/ can be used as templates. + # ReaR reads the configuration files via the bash builtin command 'source' + # so bash syntax like VARIABLE="value" (no spaces at '=') is mandatory. + # Because 'source' executes the content as bash scripts you can run commands + # within your configuration files, in particular commands to set different + # configuration values depending on certain conditions as you need like + # CONDITION_COMMAND && VARIABLE="special_value" || VARIABLE="usual_value" + # but that means CONDITION_COMMAND gets always executed when 'rear' is run + # so ensure nothing can go wrong if you run commands in configuration files. + # Some variables are for secret values (like passwords or encryption keys) + # which must be set to a secret value in a confidential way via + # { VARIABLE='secret_value' ; } 2>/dev/null + # even for a single command to discard STDERR also for 'set -x'. + # See /usr/share/rear/conf/default.conf for details and further information. + OUTPUT=RAWDISK + OUTPUT_URL="file:///var/lib/rear/output" + SECURE_BOOT_BOOTLOADER="/boot/efi/EFI/fedora/shimx64.efi" + + $ sudo cat /etc/rear/site.conf + cat: /etc/rear/site.conf: No such file or directory + +- Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM + guest or PowerVM LPAR): **PC** + +- System architecture (x86 compatible or PPC64/PPC64LE or what exact + ARM device): **X86\_64** + +- Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or + ELILO or Petitboot): **UEFI + GRUB (or whatever fedora uses)** + +- Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or + multipath (DM or NVMe): **2x NVMe SSD + 1x SATA SSD** + +- Storage layout ("lsblk -ipo + NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"): + + + + $ lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT + NAME KNAME PKNAME TRAN TYPE FSTYPE LABEL SIZE MOUNTPOINT + /dev/sda /dev/sda sata disk 111,8G + |-/dev/sda1 /dev/sda1 /dev/sda part 128M + `-/dev/sda2 /dev/sda2 /dev/sda part 111,7G + /dev/sdb /dev/sdb usb disk 0B + /dev/zram0 /dev/zram0 disk 35,2G [SWAP] + /dev/nvme1n1 /dev/nvme1n1 nvme disk 476,9G + |-/dev/nvme1n1p1 + | /dev/nvme1n1p1 /dev/nvme1n1 nvme part vfat 600M /boot/efi + |-/dev/nvme1n1p2 + | /dev/nvme1n1p2 /dev/nvme1n1 nvme part ext4 1G /boot + `-/dev/nvme1n1p3 + /dev/nvme1n1p3 /dev/nvme1n1 nvme part crypto 475,4G + `-/dev/mapper/luks-bcae9f97-dd49-4e00-b50c-7c7e255dd7e9 + /dev/dm-0 /dev/nvme1n1p3 crypt btrfs fedora_localhost-live 475,3G /home + /dev/nvme0n1 /dev/nvme0n1 nvme disk 931,5G + +- Description of the issue (ideally so that others can reproduce + it): + Following of TCG OPAL + [guide](https://github.com/rear/rear/blob/master/doc/user-guide/13-tcg-opal-support.adoc) + results in failure on step 2.2: + *Run `sudo rear mkopalpba` (ignore messages about keyboard + mappings)* + This step result in Error: `ERROR: Failed to copy ...` + +- Workaround, if any: **Do not know any** + +- Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" + debug log files): + Full log of command with `-d` flag: + + + + $ sudo rear -d mkopalpba + Relax-and-Recover 2.7-git.5364.096bfde5.master / 2024-02-06 + Running rear mkopalpba (PID 44778 date 2024-02-07 12:39:49) + Command line options: /usr/sbin/rear -d mkopalpba + Using log file: /var/log/rear/rear-kotelok-deluxe.log + Using build area: /var/tmp/rear.RbLjm4rUxijy5AN + Running 'init' stage ====================== + Running workflow mkopalpba on the normal/original system + Running 'prep' stage ====================== + Re-configuring Relax-and-Recover to create a TCG Opal pre-boot authentication (PBA) image + TIP: Your system seems to include a Plymouth graphical boot animation. You can achieve a nicer user + interface for the PBA by setting OPAL_PBA_{PROGS,COPY_AS_IS,LIBS} to include Plymouth components. + No 'console=...' setting for recovery system kernel (none in /proc/cmdline) + Found EFI system partition /dev/nvme1n1p1 on /boot/efi type vfat + Using UEFI Boot Loader for Linux (USING_UEFI_BOOTLOADER=1) + Using autodetected kernel '/boot/vmlinuz-6.7.3-200.fc39.x86_64' as kernel in the recovery system + Modified ReaR recovery system area after 'prep' stage (/var/tmp/rear.RbLjm4rUxijy5AN/rootfs not empty) + Running 'rescue' stage ====================== + Creating recovery system root filesystem skeleton layout + Handling network interface 'enp7s0' + enp7s0 is a physical device + Handled network interface 'enp7s0' + Skipping 'lo': not bound to any physical interface. + Handling network interface 'wlp8s0' + wlp8s0 is a physical device + Handled network interface 'wlp8s0' + Included current keyboard mapping (via 'dumpkeys -f') + Included default US keyboard mapping /lib/kbd/keymaps/legacy/i386/qwerty/defkeymap.map.gz + Included other keyboard mappings in /lib/kbd/keymaps + Using '/boot/efi/EFI/fedora/shimx64.efi' as UEFI Secure Boot bootloader file + Running 'build' stage ====================== + Copying files and directories + Copying binaries and libraries + A list of executables with their dependencies has been stored in /var/tmp/rear.RbLjm4rUxijy5AN/tmp/executable-dependencies + Copying only currently loaded kernel modules (MODULES contains 'loaded_modules') and those in MODULES_LOAD + ERROR: Failed to copy '/lib/modules/6.7.3-200.fc39.x86_64/kernel/arch/x86/crypto/crc32-pclmul.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/arch/x86/crypto/crc32c-intel.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/arch/x86/crypto/crct10dif-pclmul.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/arch/x86/crypto/ghash-clmulni-intel.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/arch/x86/crypto/polyval-clmulni.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/arch/x86/crypto/sha1-ssse3.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/arch/x86/crypto/sha256-ssse3.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/arch/x86/crypto/sha512-ssse3.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/arch/x86/events/intel/intel-cstate.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/arch/x86/events/intel/intel-uncore.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/arch/x86/kvm/kvm-intel.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/arch/x86/kvm/kvm.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/crypto/polyval-generic.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/acpi/acpi_pad.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/acpi/acpi_tad.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/acpi/video.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/block/loop.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/block/zram/zram.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/bluetooth/btbcm.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/bluetooth/btintel.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/bluetooth/btmtk.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/bluetooth/btrtl.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/bluetooth/btusb.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/gpu/drm/display/drm_display_helper.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/gpu/drm/drm_buddy.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/gpu/drm/i915/i915.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/gpu/drm/ttm/ttm.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/hid/hid-logitech-dj.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/hid/hid-logitech-hidpp.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/hwmon/coretemp.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/i2c/algos/i2c-algo-bit.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/i2c/busses/i2c-i801.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/i2c/i2c-smbus.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/input/joydev.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/input/misc/pcspkr.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/input/misc/uinput.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/leds/trigger/ledtrig-audio.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/md/dm-crypt.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/media/cec/core/cec.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/mfd/intel_pmc_bxt.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/misc/eeprom/ee1004.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/misc/mei/hdcp/mei_hdcp.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/misc/mei/mei-gsc.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/misc/mei/mei-me.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/misc/mei/mei.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/misc/mei/pxp/mei_pxp.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/mtd/mtd.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/mtd/spi-nor/spi-nor.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/net/ethernet/realtek/r8169.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/net/phy/realtek.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/net/wireless/intel/iwlwifi/iwlwifi.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/net/wireless/intel/iwlwifi/mvm/iwlmvm.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/nvme/common/nvme-auth.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/nvme/host/nvme-core.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/nvme/host/nvme.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/pinctrl/intel/pinctrl-tigerlake.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/platform/x86/gigabyte-wmi.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/platform/x86/intel/wmi/intel-wmi-thunderbolt.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/platform/x86/wmi-bmof.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/platform/x86/wmi.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/powercap/intel_rapl_common.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/powercap/intel_rapl_msr.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/soundwire/soundwire-bus.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/soundwire/soundwire-cadence.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/soundwire/soundwire-generic-allocation.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/soundwire/soundwire-intel.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/spi/spi-intel-pci.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/spi/spi-intel.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/thermal/intel/intel_powerclamp.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/usb/storage/uas.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/usb/storage/usb-storage.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/watchdog/iTCO_vendor_support.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/drivers/watchdog/iTCO_wdt.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/fs/binfmt_misc.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/fs/fat/fat.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/fs/fat/vfat.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/fs/fuse/fuse.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/lib/crypto/libarc4.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/bluetooth/bluetooth.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/bluetooth/bnep/bnep.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/bluetooth/rfcomm/rfcomm.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/ipv4/netfilter/ip_tables.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/ipv4/netfilter/nf_reject_ipv4.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/ipv6/netfilter/ip6_tables.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/mac80211/mac80211.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/netfilter/ipset/ip_set.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/netfilter/nf_conntrack.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/netfilter/nf_conntrack_broadcast.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/netfilter/nf_conntrack_netbios_ns.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/netfilter/nf_nat.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/netfilter/nf_tables.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/netfilter/nfnetlink.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/netfilter/nft_chain_nat.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/netfilter/nft_ct.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/netfilter/nft_fib.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/netfilter/nft_fib_inet.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/netfilter/nft_reject.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/netfilter/nft_reject_inet.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/qrtr/qrtr.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/rfkill/rfkill.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/sunrpc/sunrpc.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/net/wireless/cfg80211.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/ac97_bus.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/core/seq/snd-seq-dummy.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/core/seq/snd-seq.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/core/snd-compress.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/core/snd-hrtimer.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/core/snd-hwdep.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/core/snd-pcm-dmaengine.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/core/snd-pcm.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/core/snd-seq-device.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/core/snd-timer.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/core/snd.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/hda/ext/snd-hda-ext-core.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/hda/snd-hda-core.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/hda/snd-intel-dspcfg.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/hda/snd-intel-sdw-acpi.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/pci/hda/snd-hda-codec-generic.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/pci/hda/snd-hda-codec-hdmi.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/pci/hda/snd-hda-codec-realtek.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/pci/hda/snd-hda-codec.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/pci/hda/snd-hda-intel.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soc/codecs/snd-soc-hdac-hda.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soc/intel/common/snd-soc-acpi-intel-match.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soc/snd-soc-acpi.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soc/snd-soc-core.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soc/sof/intel/snd-sof-intel-hda-common.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soc/sof/intel/snd-sof-intel-hda-mlink.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soc/sof/intel/snd-sof-intel-hda.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soc/sof/intel/snd-sof-pci-intel-tgl.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soc/sof/snd-sof-pci.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soc/sof/snd-sof-utils.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soc/sof/snd-sof.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soc/sof/xtensa/snd-sof-xtensa-dsp.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soundcore.ko.xz + /lib/modules/6.7.3-200.fc39.x86_64/kernel/virt/lib/irqbypass.ko.xz' + Some latest log messages since the last called script 400_copy_modules.sh: + /lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soc/sof/xtensa -> /var/tmp/rear.RbLjm4rUxijy5AN/rootfs/lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soc/sof/xtensa + '/lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soc/sof/xtensa/snd-sof-xtensa-dsp.ko.xz' -> '/var/tmp/rear.RbLjm4rUxijy5AN/rootfs/lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soc/sof/xtensa/snd-s + cp: preserving permissions for '/var/tmp/rear.RbLjm4rUxijy5AN/rootfs/lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soc/sof/xtensa': No such file or directory + '/lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soundcore.ko.xz' -> '/var/tmp/rear.RbLjm4rUxijy5AN/rootfs/lib/modules/6.7.3-200.fc39.x86_64/kernel/sound/soundcore.ko.xz' + /lib/modules/6.7.3-200.fc39.x86_64/kernel/virt -> /var/tmp/rear.RbLjm4rUxijy5AN/rootfs/lib/modules/6.7.3-200.fc39.x86_64/kernel/virt + /lib/modules/6.7.3-200.fc39.x86_64/kernel/virt/lib -> /var/tmp/rear.RbLjm4rUxijy5AN/rootfs/lib/modules/6.7.3-200.fc39.x86_64/kernel/virt/lib + '/lib/modules/6.7.3-200.fc39.x86_64/kernel/virt/lib/irqbypass.ko.xz' -> '/var/tmp/rear.RbLjm4rUxijy5AN/rootfs/lib/modules/6.7.3-200.fc39.x86_64/kernel/virt/lib/irqbypass.ko.xz' + cp: preserving permissions for '/var/tmp/rear.RbLjm4rUxijy5AN/rootfs/lib/modules/6.7.3-200.fc39.x86_64/kernel/virt/lib': No such file or directory + You may use debugscript mode '-D' for full debug messages with 'set -x' output + Aborting due to an error, check /var/log/rear/rear-kotelok-deluxe.log for details + Exiting rear mkopalpba (PID 44778) and its descendant processes ... + Running exit tasks + To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.RbLjm4rUxijy5AN + Terminated + +#### [kolaqsq](https://github.com/kolaqsq) commented at [2024-02-13 12:36](https://github.com/rear/rear/issues/3146#issuecomment-1941428420): + +Error is present on fresh install of Debian 12 +Error is NOT present on fresh install of Ubuntu 22.04 + +#### [github-actions](https://github.com/apps/github-actions) commented at [2024-04-14 03:08](https://github.com/rear/rear/issues/3146#issuecomment-2053876796): + +Stale issue message + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-08.3148.issue.closed.md b/docs/issues/2024-02-08.3148.issue.closed.md new file mode 100644 index 00000000..20db8784 --- /dev/null +++ b/docs/issues/2024-02-08.3148.issue.closed.md @@ -0,0 +1,128 @@ +[\#3148 Issue](https://github.com/rear/rear/issues/3148) `closed`: ErrorIfDeprecated when 'gpt\_sync\_mbr' is used +================================================================================================================== + +**Labels**: `cleanup`, `fixed / solved / done` + +#### [jsmeix](https://github.com/jsmeix) opened issue at [2024-02-08 08:29](https://github.com/rear/rear/issues/3148): + +Support for the SUSE specific 'gpt\_sync\_mbr' partitioning scheme +(see +[https://github.com/rear/rear/issues/544](https://github.com/rear/rear/issues/544) +what that is) +should be dropped in ReaR because +the SUSE specific patches for 'gpt\_sync\_mbr' in parted were +removed in March 2016 for SLES12 SP2 so 'gpt\_sync\_mbr' +is no longer supported by SUSE since 2016. + +This issue is triggered by +[https://github.com/rear/rear/pull/3145](https://github.com/rear/rear/pull/3145) +therein in particular +[https://github.com/rear/rear/pull/3145\#discussion\_r1481388431](https://github.com/rear/rear/pull/3145#discussion_r1481388431) +that reads (excerpt) + + I think you can safely ignore the 'gpt_sync_mbr' partitioning. + + Details: + + I found https://build.opensuse.org/request/show/519107 + where the SUSE specific patches to parted were removed + that implemented the 'gpt_sync_mbr' partitioning. + The comment therein (excerpt) + --------------------------------------------------------------- + clean-up regarding pMBR handling to sync with SLE12-SP* + --------------------------------------------------------------- + indicates that the 'gpt_sync_mbr' partitioning + was no longer supported since some SLES12 service pack. + + Further investigation with the openSUSE build service + tool 'osc' shows + --------------------------------------------------------------- + # osc search parted | grep '^SUSE:SLE-12' + SUSE:SLE-12-SP1:GA parted + SUSE:SLE-12-SP1:Update parted + SUSE:SLE-12-SP2:GA parted + SUSE:SLE-12-SP2:Update parted + SUSE:SLE-12-SP3:GA parted + SUSE:SLE-12-SP3:Update parted + SUSE:SLE-12-SP4:GA parted + SUSE:SLE-12-SP4:Update parted + SUSE:SLE-12:GA parted + + # osc cat SUSE:SLE-12-SP2:GA parted parted.changes + ... + Thu Mar 24 11:52:51 UTC 2016 - puzel@suse.com + + - Drop (SUSE specific) support for hybrid pMBR (gpt_sync_mbr + label) (fate#317849) + - remove: parted-gpt-mbr-sync.patch + - remove: libparted-ppc-prepboot-in-syncmbr.patch + - refresh patches + --------------------------------------------------------------- + + SUSE:SLE-12-SP1:GA parted parted.changes does not contain that. + + So the SUSE specific patches for 'gpt_sync_mbr' were removed + for SLES12 SP2. + + This means a SLES12 system that was installed with SLES12 + before SLES12 SP2 could have 'gpt_sync_mbr' partitioning. + Subsequent system upgrades to further service packs only + upgrade RPM packages but leave the partitioning unchanged. + I think that one can also upgrade from SLES12 to SLES15 + without new partitioning (i.e. without installing from scratch) + so there could be even SLES15 systems with old inherited + 'gpt_sync_mbr' partitioning. + + If issues appear in ReaR because of an old inherited + 'gpt_sync_mbr' partitioning, it is SUSE's problem + +Because there could be recent SLES15 systems +with old inherited 'gpt\_sync\_mbr' partitioning +users with such systems should get an obvious information +that 'gpt\_sync\_mbr' support is deprecated in ReaR +so we at ReaR upstream cas see if there are really +users with old inherited 'gpt\_sync\_mbr' partitioning. + +In particular I never again tested a system +with 'gpt\_sync\_mbr' partitioning since the time +when I had implemented basic support for it +in November 2015 via +[https://github.com/rear/rear/pull/681](https://github.com/rear/rear/pull/681) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-08 08:58](https://github.com/rear/rear/issues/3148#issuecomment-1933624297): + +FYI +and for the fun of it some "forensics": + +According to +[https://github.com/rear/rear/issues/544\#issuecomment-120913214](https://github.com/rear/rear/issues/544#issuecomment-120913214) +the root is +[https://bugzilla.opensuse.org/show\_bug.cgi?id=220839](https://bugzilla.opensuse.org/show_bug.cgi?id=220839) +and according to its +[https://bugzilla.opensuse.org/show\_bug.cgi?id=220839\#c42](https://bugzilla.opensuse.org/show_bug.cgi?id=220839#c42) +the SUSE specific 'gpt\_sync\_mbr' partitioning scheme +was introduced in July 2008. + +In November 2015 I implemented basic support for it in ReaR. + +In March 2016 the SUSE patches for 'gpt\_sync\_mbr' were removed +for SLES12 SP2 which was released in November 2016, cf. +[https://en.wikipedia.org/wiki/SUSE\_Linux\_Enterprise](https://en.wikipedia.org/wiki/SUSE_Linux_Enterprise) + +Now in February 2024 support for 'gpt\_sync\_mbr' in ReaR +will get shown as deprecated via ErrorIfDeprecated. + +Let's wait and see how long it will take until the last system +with old inherited 'gpt\_sync\_mbr' partitioning has gone ;-) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-21 13:28](https://github.com/rear/rear/issues/3148#issuecomment-1956653556): + +With +[https://github.com/rear/rear/pull/3159](https://github.com/rear/rear/pull/3159) +merged +this issue should be fixed. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-08.3150.pr.merged.md b/docs/issues/2024-02-08.3150.pr.merged.md new file mode 100644 index 00000000..c959756c --- /dev/null +++ b/docs/issues/2024-02-08.3150.pr.merged.md @@ -0,0 +1,60 @@ +[\#3150 PR](https://github.com/rear/rear/pull/3150) `merged`: Add veeam support +=============================================================================== + +#### [idna38](https://github.com/idna38) opened issue at [2024-02-08 22:22](https://github.com/rear/rear/pull/3150): + +#### Relax-and-Recover (ReaR) Pull Request Template + +Please fill in the following items before submitting a new pull request: + +##### Pull Request Details: + +- Type: **New Feature** + +- Impact: **Normal** + +- Reference to related issue (URL): \#3075 + +- How was this pull request tested? Baremetal, VMware, KVM, QEMU, + RHEL7, RHEL8, RHEL9, Veeam V11 & Veeam V12 + +- Description of the changes in this pull request: + Add support for Veeam Agent for Linux Bare Metal Recovery + +#### [schlomo](https://github.com/schlomo) commented at [2024-03-27 07:59](https://github.com/rear/rear/pull/3150#issuecomment-2022156446): + +Update: @idna38 will be running some more tests with different distros, +probably next week. + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-05 07:32](https://github.com/rear/rear/pull/3150#issuecomment-2039145168): + +@idna38 great finding! Is this a bug in the Veeam systemd unit file on +Debian/Ubuntu? + +In any case, the right place for this fix is not in the `default` +directory but in a `Debian` or such subdirectory. Please run `rear dump` +to look for possible values all call me. + +#### [idna38](https://github.com/idna38) commented at [2024-04-15 19:35](https://github.com/rear/rear/pull/3150#issuecomment-2057663689): + +> Update: @idna38 will be running some more tests with different +> distros, probably next week. + +Update: The following official supported Linux distributions by Veeam, +were tested successful: + +RHEL 8.x +RHEL 9.x +Debian 10.13 +Debian 11.9 +Debian 12.5 +Ubuntu 18.04 +Ubuntu 20.04 +Ubuntu 22.04 +SLES 12 SP5 +SLES 15 SP5 + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-12.3152.issue.closed.md b/docs/issues/2024-02-12.3152.issue.closed.md new file mode 100644 index 00000000..9babdc6e --- /dev/null +++ b/docs/issues/2024-02-12.3152.issue.closed.md @@ -0,0 +1,151 @@ +[\#3152 Issue](https://github.com/rear/rear/issues/3152) `closed`: SLES15 SP5: Cannot create initrd (found no mkinitrd in the recreated system) +=============================================================================================================================================== + +**Labels**: `enhancement`, `bug`, `fixed / solved / done` + +#### [abbbi](https://github.com/abbbi) opened issue at [2024-02-12 13:30](https://github.com/rear/rear/issues/3152): + +Rear 2.7 +-------- + +hi, + +it appears that + +`finalize/SUSE_LINUX/i386/550_rebuild_initramfs.sh` + +makes use of the "mkinitrd" executable. This executable has been moved +to a special package on SLES15 SP5 called +`dracut-mkinitrd-deprecated`, which seems to be optional and may not be +existent on the system backed up. + +In reality it seems this "mkinitrd" exectuable is just an wrapper +shellscript to call dracut, see the following +discussion: + +[https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/WKZFPUPW3BQ4GYLI4HIWLJDWANUOBLIT/](https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/WKZFPUPW3BQ4GYLI4HIWLJDWANUOBLIT/) + +However, as the executable may be missing, this can result in recovery +to spoil: + + 2024-01-10 13:38:11.294431793 Running mkinitrd... + 2024-01-10 13:38:11.298370613 WARNING: + Cannot create initrd (found no mkinitrd in the recreated system). + Check the recreated system (mounted at /mnt/local) + and decide yourself, whether the system will boot or not. + +maybe it would make sense to implement an fall-back to dracut commands +if mkinitrd is not available: + +`dracut -f --regenerate-all ` + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-02-12 13:37](https://github.com/rear/rear/issues/3152#issuecomment-1938695394): + +See +[https://github.com/rear/rear/pull/2825](https://github.com/rear/rear/pull/2825) + +#### [abbbi](https://github.com/abbbi) commented at [2024-02-12 19:52](https://github.com/rear/rear/issues/3152#issuecomment-1939450254): + +@pcahyna thanks, it may make sense to either port these changes to the +SUSE related parts too, or use common code base for this? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-13 13:24](https://github.com/rear/rear/issues/3152#issuecomment-1941510047): + +I can reproduce it. + + Running mkinitrd... + WARNING: + Cannot create initrd (found no mkinitrd in the recreated system). + Check the recreated system (mounted at /mnt/local) + and decide yourself, whether the system will boot or not. + + Installing GRUB2 boot loader... + Determining where to install GRUB2 (no GRUB2_INSTALL_DEVICES specified) + Found possible boot disk /dev/vda - installing GRUB2 there + +Nevertheless the recreated system boots well for me +as expected because I recreated on 100% compatible +replacement "hardware" (on a same VM) so the same initrd +from the original system should still work. + +I will have a look how to solve it properly. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-16 11:59](https://github.com/rear/rear/issues/3152#issuecomment-1948261363): + +@abbbi +could you please test if +[https://github.com/rear/rear/pull/3155](https://github.com/rear/rear/pull/3155) +also works for you? + +In particular check your "rear -D recover" log file +for possible issues when dracut is run +and during reboot of the recreated system +for issues that could be related to the initrd. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-22 10:07](https://github.com/rear/rear/issues/3152#issuecomment-1959107099): + +This issue happens only since SLES15-SP5. + +This is the reason why I did not notice it +because I falsely assumed that SLE service packs +do not break backward compatibility without serious reason. + +I do not re-test ReaR for new SLE service packs. See +[https://github.com/rear/rear/wiki/Test-Matrix-ReaR-2.7](https://github.com/rear/rear/wiki/Test-Matrix-ReaR-2.7) +what and where I tested for ReaR 2.7. + +In this case it seems backward compatibility got broken +by accident because a single RPM requirement in one +RPM package (mdadm) was no longer needed and that +single RPM requirement was the only reason why +dracut-mkinitrd-deprecated got installed on SLES15-SP4. + +Details: + +I installed SLES15-SP4 and there +dracut-mkinitrd-deprecated is installed by default. + +In contrast on SLES15-SP5 dracut-mkinitrd-deprecated +is no longer installed by default. + +Both on SLES15-SP4 and SLES15-SP5 dracut-mkinitrd-deprecated +belongs to the sle-module-basesystem repository + + # zypper search -v dracut-mkinitrd-deprecated + ... + Name ... Repository + dracut-mkinitrd-deprecated ... sle-module-basesystem + +so on SLES15-SP5 dracut-mkinitrd-deprecated +should be always available to be "just installed". + +On SLES15-SP4 dracut-mkinitrd-deprecated +is installed because mdadm requires /sbin/mkinitrd + + SLES15-SP4: # rpm -e --test dracut-mkinitrd-deprecated + error: Failed dependencies: + /sbin/mkinitrd is needed by (installed) mdadm-4.1-... + +On SLES15-SP5 mdadm does no longer require /sbin/mkinitrd + + SLES15-SP5: # rpm -q --changelog mdadm + ... + * Thu Nov 24 2022 colyli@suse.com + - mdadm.spec: remove "PreReq: %{_sbindir}/mkinitrd" as it is + unnecessary now. (bsc#1202352) + +where `bsc#1202352` is +[https://bugzilla.opensuse.org/show\_bug.cgi?id=1202352](https://bugzilla.opensuse.org/show_bug.cgi?id=1202352) + +Therein another openSUSE mail thread +on `factory@lists.opensuse.org` is referenced +[https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/GDAZ7TVXS2BAXENVBAMHVXVHLSR7D7NQ/](https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/GDAZ7TVXS2BAXENVBAMHVXVHLSR7D7NQ/) + +#### [abbbi](https://github.com/abbbi) commented at [2024-03-04 18:29](https://github.com/rear/rear/issues/3152#issuecomment-1977206593): + +Tested and can Confirm its working for me, thanks! + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-14.3153.pr.merged.md b/docs/issues/2024-02-14.3153.pr.merged.md new file mode 100644 index 00000000..a58bc482 --- /dev/null +++ b/docs/issues/2024-02-14.3153.pr.merged.md @@ -0,0 +1,64 @@ +[\#3153 PR](https://github.com/rear/rear/pull/3153) `merged`: Replace the `OUTPUT=IPL` with equivalent `OUTPUT=RAMDISK` +======================================================================================================================= + +**Labels**: `cleanup`, `fixed / solved / done` + +#### [lzaoral](https://github.com/lzaoral) opened issue at [2024-02-14 12:17](https://github.com/rear/rear/pull/3153): + +#### Relax-and-Recover (ReaR) Pull Request Template + +Please fill in the following items before submitting a new pull request: + +##### Pull Request Details: + +- Type: **Enhancement** / **Clean-up** + +- Impact: **Normal** + +- Reference to related issue (URL): + [https://github.com/rear/rear/issues/3149\#issuecomment-1941620475](https://github.com/rear/rear/issues/3149#issuecomment-1941620475) + +- How was this pull request tested? I was able to successfully boot + the generated kernel and initrd on s390x Fedora Rawhide machine with + both `OUTPUT=IPL` and `OUTPUT=RAMDISK` options. + +- Description of the changes in this pull request: + +The initial PR with s390 support in ReaR introduced an s390-only +`OUTPUT=IPL` undocumented option. However, the `OUTPUT=IPL` option is +completely redundant because it does the exact same thing as the already +existing and documented OUTPUT=RAMDISK option. + +This commit removes the whole `IPL` directory sub-tree and introduces a +fallback that replaces `OUTPUT=IPL` with `OUTPUT=RAMDISK` during the +`prep` phase with a deprecation warning to still be backwards compatible +with existing `local.conf` files. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-19 11:59](https://github.com/rear/rear/pull/3153#issuecomment-1952301713): + +Only as a side note regarding +shouting at our users with "WARNING: ..." +instead of normal user information with "Warning: ..." + +Currently we have about 42 code places with "WARNING" +and only 3 code places with "Warning" :-( +Sigh! + +When time permits I will have a look at those code places and +try to improve things as far as possible with reasonable effort. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-28 08:49](https://github.com/rear/rear/pull/3153#issuecomment-1968502896): + +@lzaoral @pcahyna @rear/contributors +I would like to merge it today afternoon +unless there are objections + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-28 16:07](https://github.com/rear/rear/pull/3153#issuecomment-1969316131): + +@lzaoral +thank you for this cleanup! + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-16.3155.pr.merged.md b/docs/issues/2024-02-16.3155.pr.merged.md new file mode 100644 index 00000000..e86ad90c --- /dev/null +++ b/docs/issues/2024-02-16.3155.pr.merged.md @@ -0,0 +1,111 @@ +[\#3155 PR](https://github.com/rear/rear/pull/3155) `merged`: Update finalize/SUSE\_LINUX/i386/550\_rebuild\_initramfs.sh +========================================================================================================================= + +**Labels**: `enhancement`, `bug`, `cleanup`, `fixed / solved / done` + +#### [jsmeix](https://github.com/jsmeix) opened issue at [2024-02-16 11:56](https://github.com/rear/rear/pull/3155): + +- Type: **Bug Fix** / **Enhancement** / **Cleanup** + +- Impact: **High** + +- Reference to related issue (URL): + [https://github.com/rear/rear/issues/3152](https://github.com/rear/rear/issues/3152) + +- How was this pull request tested? + Works well for me with SLES15-SP5 + +- Description of the changes in this pull request: + +Overhauled finalize/SUSE\_LINUX/i386/550\_rebuild\_initramfs.sh + +Now it uses dracut by default and mkinitrd as fallback +which fixes +[https://github.com/rear/rear/issues/3152](https://github.com/rear/rear/issues/3152) +at least for me with SLES15-SP5. + +Additionally improved the user messages +(in particular the warning messages) +to make it more clear that the point is +to decide if the recreated system will boot +with the initrd 'as is' from the backup restore. + +Furthermore removed the whole INITRD\_MODULES code +because INITRD\_MODULES is not used and +/etc/sysconfig/kernel does no longer exist since SLES12 so the +INITRD\_MODULES code is dead code. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-21 11:07](https://github.com/rear/rear/pull/3155#issuecomment-1956413751): + +@abbbi +could you please test if the overhauled +finalize/SUSE\_LINUX/i386/550\_rebuild\_initramfs.sh +in this +[https://github.com/rear/rear/pull/3155](https://github.com/rear/rear/pull/3155) +also works for you? + +At least please respond when you cannot test it +or if you could not test it right now but later +so that I could better decide how to proceed here. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-21 11:09](https://github.com/rear/rear/pull/3155#issuecomment-1956417391): + +@rear/contributors +when no comments or feedback appears here, +in particular when there are no objections, +I would like to merge it tomorrow afternoon. + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-02-21 11:47](https://github.com/rear/rear/pull/3155#issuecomment-1956478419): + +Hmm, thinking about it. The current initrd code for Suse has the same +problem which was fixed for Fedora in +[https://github.com/rear/rear/pull/2873](https://github.com/rear/rear/pull/2873). +At least on `s390`, the initrd will not be regenerated. + +Also, if the code for SuSe and Fedora is very similar, it could make +sense to just merge it? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-21 13:22](https://github.com/rear/rear/pull/3155#issuecomment-1956642123): + +@lzaoral +merging finalize/Fedora/550\_rebuild\_initramfs.sh +and finalize/SUSE\_LINUX/i386/550\_rebuild\_initramfs.sh +into one same code is something that could be done later +via a separated pull request if feasible in practice. + +Currently finalize/Fedora/550\_rebuild\_initramfs.sh +is rather different because it deals with INITRD\_MODULES +which I completely removed for SUSE here because that code +in finalize/SUSE\_LINUX/i386/550\_rebuild\_initramfs.sh +was "very dead" code. + +With this pull request I like to switch from +mkinitrd to dracut only for SUSE\_LINUX/i386 +as a first step. + +In +[https://github.com/rear/rear/issues/3152](https://github.com/rear/rear/issues/3152) +the discussion +[https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/WKZFPUPW3BQ4GYLI4HIWLJDWANUOBLIT/](https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/WKZFPUPW3BQ4GYLI4HIWLJDWANUOBLIT/) +is mentioned which indicates that calling plain 'dracut' +is not always the same as calling plain 'mkinitrd' +regardless what some say about it's "just simple easy". + +So I prefer to change such things in small steps +and wait some reasonable time for user feedback +because when my simplified and overhauled code +"just works" for me on my QEMU/KVM test system +it does not mean it also works for others, in particular +perhaps not on whatever rather special server hardware. + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-02-22 15:03](https://github.com/rear/rear/pull/3155#issuecomment-1959637997): + +@jsmeix Sure, the merging with Fedora is a topic for a separate PR. +However, the issue described in +[https://github.com/rear/rear/pull/2873](https://github.com/rear/rear/pull/2873) +is still valid for initrd regeneration on SuSe. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-19.3157.pr.merged.md b/docs/issues/2024-02-19.3157.pr.merged.md new file mode 100644 index 00000000..785b59ed --- /dev/null +++ b/docs/issues/2024-02-19.3157.pr.merged.md @@ -0,0 +1,183 @@ +[\#3157 PR](https://github.com/rear/rear/pull/3157) `merged`: Remove hardcoded architecture-dependend strings from EFI code +=========================================================================================================================== + +**Labels**: `enhancement`, `cleanup`, `fixed / solved / done` + +#### [pcahyna](https://github.com/pcahyna) opened issue at [2024-02-19 13:12](https://github.com/rear/rear/pull/3157): + +- Type: **Enhancement** + +- Impact: **Normal** + +- Reference to related issue (URL): + +- How was this pull request tested? + +- Description of the changes in this pull request: + +Introduce variables EFI\_ARCH{\_UPPER} and GRUB2\_IMAGE\_FORMAT. Use +them for various EFI bootloader suffixes, parameters and paths instead +of hardcoding strings like BOOTX64.EFI. EFI\_ARCH is "x64" and +EFI\_ARCH\_UPPER is "X64" and GRUB2\_IMAGE\_FORMAT is "x86\_64-efi" on +x86\_64 architecture. + +Should make it easier to port the code to e.g. Arm. + +See e.g. +[https://github.com/rhboot/shim/blob/main/Make.defaults](https://github.com/rhboot/shim/blob/main/Make.defaults) +for possible values. + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-02-19 14:17](https://github.com/rear/rear/pull/3157#issuecomment-1952544921): + +Thank you a lot, @pcahyna! This will make the EFI support on `aarch64` I +have implemented locally. much easier to upstream! + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-02-28 09:21](https://github.com/rear/rear/pull/3157#issuecomment-1968557201): + +@pcahyna Could you generalise the `build_bootx86_efi` function as well, +please? + +[https://github.com/rear/rear/blob/8564a7b952481e0fdde562207ff88ff1f4b4df9d/usr/share/rear/lib/uefi-functions.sh\#L39-L126](https://github.com/rear/rear/blob/8564a7b952481e0fdde562207ff88ff1f4b4df9d/usr/share/rear/lib/uefi-functions.sh#L39-L126) + +It hardcodes the `x86_64` architecture and the name itself is quite +misleading because `x86` is not the same thing as `x86_64`. Note that +GRUB2 uses the `arm64-efi` directory on `aarch64` Fedora so plain +`uname -m` won't work. + +edit: typos + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-02-28 12:50](https://github.com/rear/rear/pull/3157#issuecomment-1968918505): + +There is still one instance of `BOOTX64.efi` left: +[https://github.com/rear/rear/blob/48ebbf974b852e6e2c1c9c92646d7036f5a19674/usr/share/rear/output/ISO/Linux-i386/260\_EFISTUB\_populate.sh\#L36](https://github.com/rear/rear/blob/48ebbf974b852e6e2c1c9c92646d7036f5a19674/usr/share/rear/output/ISO/Linux-i386/260_EFISTUB_populate.sh#L36) + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-04 15:22](https://github.com/rear/rear/pull/3157#issuecomment-1976831552): + +> @pcahyna Could you generalise the `build_bootx86_efi` function as +> well, please? +> +> [https://github.com/rear/rear/blob/8564a7b952481e0fdde562207ff88ff1f4b4df9d/usr/share/rear/lib/uefi-functions.sh\#L39-L126](https://github.com/rear/rear/blob/8564a7b952481e0fdde562207ff88ff1f4b4df9d/usr/share/rear/lib/uefi-functions.sh#L39-L126) +> +> It hardcodes the `x86_64` architecture and the name itself is quite +> misleading because `x86` is not the same thing as `x86_64`. Note that +> GRUB2 uses the `arm64-efi` directory on `aarch64` Fedora so plain +> `uname -m` won't work. + +@lzaoral done in +[3db2724](https://github.com/rear/rear/pull/3157/commits/3db2724c7860e38fad96ba4d35c8b174616c1496) + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-04 15:34](https://github.com/rear/rear/pull/3157#issuecomment-1976858445): + +> > @pcahyna Could you generalise the `build_bootx86_efi` function as +> > well, please? +> > [https://github.com/rear/rear/blob/8564a7b952481e0fdde562207ff88ff1f4b4df9d/usr/share/rear/lib/uefi-functions.sh\#L39-L126](https://github.com/rear/rear/blob/8564a7b952481e0fdde562207ff88ff1f4b4df9d/usr/share/rear/lib/uefi-functions.sh#L39-L126) +> > +> > It hardcodes the `x86_64` architecture and the name itself is quite +> > misleading because `x86` is not the same thing as `x86_64`. Note +> > that GRUB2 uses the `arm64-efi` directory on `aarch64` Fedora so +> > plain `uname -m` won't work. +> +> @lzaoral done in +> [3db2724](https://github.com/rear/rear/pull/3157/commits/3db2724c7860e38fad96ba4d35c8b174616c1496) + +I have not touched the name though + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-04 15:46](https://github.com/rear/rear/pull/3157#issuecomment-1976885201): + +I changed also the name now + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-04 16:55](https://github.com/rear/rear/pull/3157#issuecomment-1977043502): + +According to +[https://github.com/rear/rear/pull/3157\#discussion\_r1511419803](https://github.com/rear/rear/pull/3157#discussion_r1511419803) +and +[https://github.com/rear/rear/pull/3157\#pullrequestreview-1914753506](https://github.com/rear/rear/pull/3157#pullrequestreview-1914753506) +I adapted +[https://github.com/rear/rear/wiki/Coding-Style\#variables](https://github.com/rear/rear/wiki/Coding-Style#variables) +so that it now reads: + + Variables + + Curly braces only where really needed: + $FOO instead of ${FOO}, but for example ${FOO:-default_value} + except ${FOO} aids readability compared to $FOO for example as in + PREFIX${FOO}.SUFFIX versus PREFIX$FOO.SUFFIX that is harder to read. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-04 17:23](https://github.com/rear/rear/pull/3157#issuecomment-1977097091): + +I see now that this "hard to read" problem is partly my fault. I am +using emacs and and syntax highlighting of strings like +PREFIX$FOO.SUFFIX works, so this is readable without curly braces, but +"PREFIX$FOO.SUFFIX" does not (the highlighting of the content of "..." +overrides the highlighting of `$FOO`), so I need curly braces here. But +in vim and perhaps other editors this is not a problem. +[https://stackoverflow.com/questions/10802864/in-emacs-how-do-i-get-variable-within-a-string-to-be-highlighted\#comment14246359\_10802864](https://stackoverflow.com/questions/10802864/in-emacs-how-do-i-get-variable-within-a-string-to-be-highlighted#comment14246359_10802864) +[https://emacs.stackexchange.com/questions/13128/highlighting-shell-variables-within-quotes](https://emacs.stackexchange.com/questions/13128/highlighting-shell-variables-within-quotes) + +@rear/contributors what do you think? What does count as readable and +what does count as unreadable for you? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-04 17:37](https://github.com/rear/rear/pull/3157#issuecomment-1977122357): + +@pcahyna + +It is your code so in general you have +(to some reasonable extent, e.g. no real bugs) +final power over your code. + +In particular when an issue is basically +about "aesthetic judgement" it is in general +a subjective judgement where you are free +to judge as you like (except exceptions), cf. +[https://en.wikipedia.org/wiki/Critique\_of\_Judgment\#Aesthetic\_Judgement](https://en.wikipedia.org/wiki/Critique_of_Judgment#Aesthetic_Judgement) + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-04 18:08](https://github.com/rear/rear/pull/3157#issuecomment-1977173060): + +@jsmeix sure, but I want to make my code readable for others. Therefore, +I am genuinely curious what is the aesthetic judgement of other +contributors. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-05 14:37](https://github.com/rear/rear/pull/3157#issuecomment-1978918601): + +yeah I have thought that the comments will be better readable if they +contain concrete examples, even if this makes them incorrect for non-x64 +architectures + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-06 12:59](https://github.com/rear/rear/pull/3157#issuecomment-1980816967): + +I also prefer explicit real-world examples in my comments +in particular because I like to show what there really is +on my own system while I implement something so that +there is an actually true example (at least true on +my specific system at the time when I made the comment) +which has the advantage that later others can understand +how it had been when it was implemented and how it may +differ on their systems and/or at some later time. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-06 13:00](https://github.com/rear/rear/pull/3157#issuecomment-1980819424): + +@pcahyna +because it is approved by @lzaoral and me +feel free to merge it as you like. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-12 15:44](https://github.com/rear/rear/pull/3157#issuecomment-1991968920): + +@rear/contributors I am still curious how readable longer quoted strings +with embedded variable expansions and curly braces vs. no curly braces +are for you - most likely it will depend on the syntax highlighting +capabilities of your favorite editor. +If there are no opinions that contradict what I did here (it is somewhat +tuned for Emacs) I will merge the current state tomorrow. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-14 22:18](https://github.com/rear/rear/pull/3157#issuecomment-1998574603): + +thank you all for the reviews - merging! + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-14 22:19](https://github.com/rear/rear/pull/3157#issuecomment-1998576025): + +my bad, I forgot to squash all the fixups ... + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-19.3158.pr.merged.md b/docs/issues/2024-02-19.3158.pr.merged.md new file mode 100644 index 00000000..3f552cd3 --- /dev/null +++ b/docs/issues/2024-02-19.3158.pr.merged.md @@ -0,0 +1,312 @@ +[\#3158 PR](https://github.com/rear/rear/pull/3158) `merged`: docs: document booting of ReaR rescue initramfs on z/VM +===================================================================================================================== + +**Labels**: `enhancement`, `documentation`, `fixed / solved / done` + +#### [lzaoral](https://github.com/lzaoral) opened issue at [2024-02-19 13:30](https://github.com/rear/rear/pull/3158): + +##### Pull Request Details: + +- Type: **Enhancement** + +- Impact: **High** + +- Reference to related issue (URL): + [https://github.com/rear/rear/issues/3149\#issuecomment-1943375764](https://github.com/rear/rear/issues/3149#issuecomment-1943375764) + +- How was this pull request tested? + +All steps described in the new documentation were tried on a Fedora +Rawhide s390x VM running under the z/VM hypervisor. + +- Description of the changes in this pull request: + +Document booting of ReaR rescue initramfs on s390/s390x. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-19 14:53](https://github.com/rear/rear/pull/3158#issuecomment-1952615954): + +@lzaoral +Wow! +Looks "rather interesting" to me as a total ZIPL noob ;-) +Now I have a starting point :-) +Thank you for it! + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-02-19 15:13](https://github.com/rear/rear/pull/3158#issuecomment-1952659298): + +Glad to hear that! 😄 BTW, if you accidentally overwrite the `zipl` +configuration of your host (e.g. when using the same device for storing +the kernel and initrd) when experimenting, you can just call `zipl` +without any argument to restore it to the default values (at least on +Fedora/RHEL). + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-21 12:40](https://github.com/rear/rear/pull/3158#issuecomment-1956565746): + +@lzaoral +I would like to suggest to add a bit more context description +to make the "Booting ReaR rescue initramfs on s390/s390x" +section somewhat self-contained (to a reasonable extent) +so that when someone reads only this section +it is comprehensible on its own. + +Perhaps as a total ZIPL noob I am not an intended reader +of this section so my following suggestions could go too far. +Nevertheless I just make them and you can then decide +if and what context description you may add +as you think what could be best for our users. + +(1) "Bare metal" recovery (i.e. recovery on a "bare VM"): +As far as I see it seems the described method using zipl +makes the ReaR recovery system boot on the same system +where the zipl command is run. +Accordingly it seems the zipl method has the same drawback +as the kexec method, that a running Linux system is +already needed where the ReaR recovery system should boot +(to run kexec or zipl) so neither the kexec method +nor the zipl method support recovery on a "bare VM" +(i.e. on a VM where not yet a Linux system must be running). +Is this the case or do I misunderstand something? +If my understanding is right, it should be clearly described +that a running Linux system is already needed +where the ReaR recovery system should be booted. +If I misunderstand things, it should be clearly described +where "rear mkbackup" was run +versus where "rear recover" should be run +and how to make the ReaR recovery system boot +where "rear recover" should be run. + +(2) REAR\_OUTPUT usage: +It confuses me that REAR\_OUTPUT is used +both for zipl "output" via --target and +for zipl "input" via --image and --ramdisk. +I read +[https://manpages.debian.org/jessie/s390-tools/zipl.8](https://manpages.debian.org/jessie/s390-tools/zipl.8) +and I am wondering if it is really required that one same +REAR\_OUTPUT directory must be used for the zipl options + + --target=REAR_OUTPUT + --image=REAR_OUTPUT/... + --ramdisk=REAR_OUTPUT/... + +or if all could be different directories like + + --target=BOOTMAP_INSTALL_DIRECTORY + --image=/SOME/PATH/TO/KERNEL + --ramdisk=/ANOTHER/PATH/TO/INITRD + +If all could be different directories it should be clearly +described what the reason is why it is usually OK +to use one same directory for all three options. +I think the same directory for KERNEL and INITRD is natural +but perhaps a differnt --target directory could be required, +see the next item (3): + +(3) Bootloader install device: +It was inexplicable to me why + + The bootloader ... was installed on the dasda device + +regardless that no bootloader install device was specified. +I think +[https://manpages.debian.org/jessie/s390-tools/zipl.8](https://manpages.debian.org/jessie/s390-tools/zipl.8) +explains it: + + -t or --target= + Use the specified . + zipl uses this directory to store the bootmap, + i.e. a file containing boot data. + The actual boot loader is installed onto the device + containing the target directory. + Supported devices are DASD and SCSI disks. + +So it should be made more clear that via --target +the bootloader install device is implicitly specified +to avoid possibly "surprising" results when a user +specifies some --target=... that happens +to be on whatever unexpected device +where then a bootloader gets installed. +In particular when REAR\_OUTPUT is used for --target +(i.e. the ReaR recovery system kernel and initrd +may be stored anywhere where its underlying device +is not meant to be booted from). + +(4) Only for DASD disks: +According to +[https://manpages.debian.org/jessie/s390-tools/zipl.8](https://manpages.debian.org/jessie/s390-tools/zipl.8) +the zipl --target method is only supported for +DASD and SCSI disks. +If this is right, it may be clearly described. +Probably - as far as I remember the code - +currently only DASD disks are supported by ReaR +so currently the zipl --target method is supported on +all IBM Z disks that are currently supported by ReaR. +The crucial word is "currently" - i.e. if ReaR gets +enhanced to also support other IBM Z disks, the +described zipl --target method may no longer work +so to be future-proof this restriction of the +zipl --target method should be mentioned +because in practice nobody updates documentation ;-) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-26 07:39](https://github.com/rear/rear/pull/3158#issuecomment-1963487340): + +To avoid misunderstanding: + +In my above +[https://github.com/rear/rear/pull/3158\#issuecomment-1956565746](https://github.com/rear/rear/pull/3158#issuecomment-1956565746) +I wrote about "bare metal recovery", +I even wrote "true bare metal recovery". + +This was very misleading because on IBM Z systems +"true bare metal recovery" is not wanted. +On IBM Z there is always some virtualization in between. +What I actually meant was that the ReaR recovery system +can be booted on a "bare VM" i.e. on a VM where not yet +another Linux operating system must be running. + +I changed my misleading wording in my above +[https://github.com/rear/rear/pull/3158\#issuecomment-1956565746](https://github.com/rear/rear/pull/3158#issuecomment-1956565746) +from "bare metal" to "bare VM". + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-02-26 13:56](https://github.com/rear/rear/pull/3158#issuecomment-1964207577): + +Thank you a lot for the thorough review, @jsmeix! + +I've specified that the new section is only for virtual machines running +under z/VM. While it should be possible to also use ReaR on IBM Z LPARs, +I have never tried that personally (maybe @pcahyna has some +experience?). + +(1) The `zipl` method can be used independently of the running OS which +is an information I forgot include by using z/VM Control Program +directly. Actually, it should be quite straightforward to use `zipl` to +implement `OUTPUT=USB` just like you would use GRUB2 or EXTLINUX to +prepare a bootable device on `x86_64`. + +(2) Correct, the value of the `--target=` option can be arbitrary as +long as that directory is on the same device. (Actually, I was able to +create the bootmap file (even without `--force`) on a completely +different device but unsurprisingly, the machine then failed to boot.) + +(3) Hopefully done. + +(4) RHEL only supports IBM Z machines with DASD or SCSI disks so I have +zero experience with other disk types. However, `zipl(8)` on Fedora +Rawhide also mentions NVMe as a supported `--target=`. Also, it seems +that `--tape=` is the right option for tape devices. So the only option +that cannot be covered by `zipl` is the z/VM reader (a virtual punch +card reader) which cannot be used with plain kernel and initramfs anyway +(I have never tried booting RHEL installer from this device but, +apparently, you also need some additional files to do so.). + +I've updated the `zipl` section to include this information as well. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-26 14:55](https://github.com/rear/rear/pull/3158#issuecomment-1964340849): + +@lzaoral +thank you so much for your so much improved documentation! + +In particular thank you for spelling out +certain IBM Z specific terms like + + z/VM Control Program (CP) + +which helps to avoid confusion with things +like the traditional Unix `cp` program ;-) + +I think I got now (for the very first time - hooray!) +some basic understanding how all that IBM Z booting stuff +belongs together (in particular zipl and chreipl versus CP). + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-26 15:12](https://github.com/rear/rear/pull/3158#issuecomment-1964378480): + +@lzaoral +some years ago (at the time when I did my first tests +with ReaR on IBM Z) we had an internal issue at SUSE +whether or not to officially support ReaR on IBM Z. +At that time only virtual machines running under z/VM +with only DASD disks were considered. +ReaR on IBM Z LPARs was explicitly not considered. +So at least for now we do not need to worry about +support for ReaR on IBM Z LPARs or non-DASD disks. +Nevertheless it is very good to make clear that +support for ReaR on IBM Z is currently limited to +virtual machines running under z/VM with DASD disks +and perhaps also under z/VM with SCSI disks. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-26 15:23](https://github.com/rear/rear/pull/3158#issuecomment-1964410603): + +Now this + + The machine will not boot if `BOOTMAP_INSTALL_DIRECTORY` + and `REAR_DIRECTORY` are on different devices! + +makes things clear and it explains why in practice +it is probably simplest and best to use one same +directory for bootmap, kernel, and initrd. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-26 15:28](https://github.com/rear/rear/pull/3158#issuecomment-1964420668): + +@pcahyna +because in +[https://github.com/rear/rear/pull/3158\#issuecomment-1964207577](https://github.com/rear/rear/pull/3158#issuecomment-1964207577) +@lzaoral "advertised" you as a possible "IBM Z expert" ;-) +I would appreciate it if you could have a look here +(as time permits). + +@rear/contributors +I would like to merge it on Friday afternoon +unless there are objections. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-01 16:31](https://github.com/rear/rear/pull/3158#issuecomment-1973498560): + +@jsmeix sorry for being late, looking now ... + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-01 16:31](https://github.com/rear/rear/pull/3158#issuecomment-1973499523): + +can you please delay merging a bit? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-01 16:59](https://github.com/rear/rear/pull/3158#issuecomment-1973547275): + +@pcahyna +don't worry and no need to be sorry! +Of course I delay merging when you need time for review. +No rush, take your time, and +have a relaxed and recovering weekend! + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-04 09:26](https://github.com/rear/rear/pull/3158#issuecomment-1976106887): + +@pcahyna +thank you for your review! +As always your precise view reveals problematic things. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-06 12:51](https://github.com/rear/rear/pull/3158#issuecomment-1980801965): + +As far as I see now all looks well +so I would like to merge it tomorrow afternoon +unless objections appear. + +@pcahyna +please approve this pull request when it is OK for you. + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-03-07 15:04](https://github.com/rear/rear/pull/3158#issuecomment-1983701667): + +I've rebased the branch against `master` and squashed all fixups. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-08 06:30](https://github.com/rear/rear/pull/3158#issuecomment-1985112662): + +@rear/contributors +I would like to merge it today afternoon +unless there are objections. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-08 16:03](https://github.com/rear/rear/pull/3158#issuecomment-1985959444): + +@lzaoral +thank you for all your work to get that done properly and +for your patience to answer all my understanding questions! + +@pcahyna +thank you for your careful review! + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-19.3159.pr.merged.md b/docs/issues/2024-02-19.3159.pr.merged.md new file mode 100644 index 00000000..5b2fdb15 --- /dev/null +++ b/docs/issues/2024-02-19.3159.pr.merged.md @@ -0,0 +1,67 @@ +[\#3159 PR](https://github.com/rear/rear/pull/3159) `merged`: ErrorIfDeprecated when 'gpt\_sync\_mbr' is used +============================================================================================================= + +**Labels**: `cleanup`, `fixed / solved / done` + +#### [jsmeix](https://github.com/jsmeix) opened issue at [2024-02-19 14:13](https://github.com/rear/rear/pull/3159): + +- Type: **Cleanup** + +- Impact: **Unknown** + It is unknown how many user will file issues + because 'gpt\_sync\_mbr' is indispensable for them + (i.e. they have no alternative to it). + +- Reference to related issue (URL): + [https://github.com/rear/rear/issues/3148](https://github.com/rear/rear/issues/3148) + +- How was this pull request tested? + +Works well for me. +I don't have a gpt\_sync\_mbr test VM so for the test I changed +in layout/save/default/950\_verify\_disklayout\_file.sh + + if test "gpt_sync_mbr" = "$parted_mklabel" ; then + +to + + if test "gpt" = "$parted_mklabel" ; then + +to trigger it artificially with normal 'gpt'. + +- Description of the changes in this pull request: + +In layout/save/default/950\_verify\_disklayout\_file.sh +ErrorIfDeprecated when 'gpt\_sync\_mbr' is used. + +I used layout/save/default/950\_verify\_disklayout\_file.sh +and not layout/save/GNU/Linux/200\_partition\_layout.sh +because in 950\_verify\_disklayout\_file.sh +there is only one place with 'gpt\_sync\_mbr' +while in 200\_partition\_layout.sh it appears more than once +and 950\_verify\_disklayout\_file.sh runs after disklayout.conf +was created so we check the actual final result there. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-19 14:23](https://github.com/rear/rear/pull/3159#issuecomment-1952555066): + +My +[https://github.com/rear/rear/commit/732612111ece899b354917535eb35e31cb48c4c9](https://github.com/rear/rear/commit/732612111ece899b354917535eb35e31cb48c4c9) +is needed for my multi-line reason message here + + ErrorIfDeprecated gpt_sync_mbr "The 'gpt_sync_mbr' partitioning is no longer supported by SUSE since 2016 + see https://github.com/rear/rear/issues/3148" + +that avoids a too long line for the whole reason +which could be needlessly hard to read, perhaps even with +the issue URL "out of sight" beyond the right screen border. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-20 12:56](https://github.com/rear/rear/pull/3159#issuecomment-1954161114): + +@rear/contributors +unless there are objections I would like +to merge it tomorrow afternoon. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-20.3160.pr.merged.md b/docs/issues/2024-02-20.3160.pr.merged.md new file mode 100644 index 00000000..9689db15 --- /dev/null +++ b/docs/issues/2024-02-20.3160.pr.merged.md @@ -0,0 +1,139 @@ +[\#3160 PR](https://github.com/rear/rear/pull/3160) `merged`: New TextPrefix function +===================================================================================== + +**Labels**: `enhancement`, `fixed / solved / done` + +#### [jsmeix](https://github.com/jsmeix) opened issue at [2024-02-20 13:41](https://github.com/rear/rear/pull/3160): + +- Type: **Enhancement** + +- Impact: **Normal** + +- Description of the changes in this pull request: + +New TextPrefix function to +remove leading space and tab characters from input lines +and prefix each line with the argument string if specified. + +The intent is to be able to properly indent multi-line message strings +in the code and output the message without the code indentation +but prefixed as needed for example like + + message="first line + second line + last line" + LogPrint "Message text:$LF$( TextPrefix ' | ' <<<"$message" )" + +which results the following output + + Message text: + | first line + | second line + | last line + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-20 13:46](https://github.com/rear/rear/pull/3160#issuecomment-1954251341): + +I tested it with an adapted ErrorIfDeprecated() + + function ErrorIfDeprecated () { + { (( $# >= 2 )) || BugError "Must call ErrorIfDeprecated with at least 2 arguments - feature and reason" + local feature="$1" ; shift + local reason="$*" + + if IsInArray "$feature" "${DISABLE_DEPRECATION_ERRORS[@]}" ; then + LogPrint "Disabled deprecation error for '$feature'" + return 0 + fi + + local text="Reason: + $reason + + This feature is phased out in ReaR and will be eventually removed. + If it is indispensable, go to https://github.com/rear/rear/issues + and create an issue that explains why there is no alternative to it. + + To disable this error and continue using this feature for now, set + DISABLE_DEPRECATION_ERRORS+=( $feature ) + " + } 2>>/dev/$DISPENSABLE_OUTPUT_DEV + Error "Deprecation of '$feature'$LF$( TextPrefix '' <<<"$text" )" + } + +which results for me this output + + # usr/sbin/rear mkrescue + ERROR: Deprecation of 'gpt_sync_mbr' + Reason: + The 'gpt_sync_mbr' partitioning is no longer supported by SUSE since 2016 + see https://github.com/rear/rear/issues/3148 + + This feature is phased out in ReaR and will be eventually removed. + If it is indispensable, go to https://github.com/rear/rear/issues + and create an issue that explains why there is no alternative to it. + + To disable this error and continue using this feature for now, set + DISABLE_DEPRECATION_ERRORS+=( gpt_sync_mbr ) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-20 13:51](https://github.com/rear/rear/pull/3160#issuecomment-1954258836): + +As another test I replaced in lib/\_input-output-functions.sh +some + + ... | sed -e 's/^/ /' | ... + +with + + ... | TextPrefix | ... + +(plain 'TextPrefix' adds two spaces indentation). + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-20 13:53](https://github.com/rear/rear/pull/3160#issuecomment-1954263564): + +@rear/contributors +please have a look here and provide feedback +what you think about such a TextPrefix function. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-22 13:51](https://github.com/rear/rear/pull/3160#issuecomment-1959492306): + +@rear/contributors +as no news is good news +I will merge it tomorrow afternoon +unless there are severe objections. +This is only an internal helper function +so we can change it as we like if needed. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-27 12:59](https://github.com/rear/rear/pull/3160#issuecomment-1966495212): + +The new TextPrefix needs to be fixed +because currently: + + # function TextPrefix () { local prefix="${1- }" ; sed -e "s/^[ \t]*/$prefix/" ; } + + # (set -x ; echo foo | TextPrefix '/prefix/dir/' ) + + echo foo + + TextPrefix /prefix/dir/ + + local prefix=/prefix/dir/ + + sed -e 's/^[ \t]*//prefix/dir//' + sed: -e expression #1, char 13: unknown option to `s' + +The fix is to escape all / with / in prefix + + # function TextPrefix () { local prefix="${1- }" ; sed -e "s/^[ \t]*/${prefix//\//\\/}/" ; } + + # (set -x ; echo foo | TextPrefix '/prefix/dir/' ) + + echo foo + + TextPrefix /prefix/dir/ + + local prefix=/prefix/dir/ + + sed -e 's/^[ \t]*/\/prefix\/dir\//' + /prefix/dir/foo + +The fix will be done "by the way" +in the related pull request +[https://github.com/rear/rear/pull/3166](https://github.com/rear/rear/pull/3166) +via +[https://github.com/rear/rear/pull/3166/commits/1c047d93115a99023b064e8519a7d51ad2e5c36e](https://github.com/rear/rear/pull/3166/commits/1c047d93115a99023b064e8519a7d51ad2e5c36e) + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-21.3161.issue.open.md b/docs/issues/2024-02-21.3161.issue.open.md new file mode 100644 index 00000000..20055cc6 --- /dev/null +++ b/docs/issues/2024-02-21.3161.issue.open.md @@ -0,0 +1,85 @@ +[\#3161 Issue](https://github.com/rear/rear/issues/3161) `open`: Multiple disks recovery & external disk backup/recovery +======================================================================================================================== + +**Labels**: `support / question`, `no-issue-activity` + +#### [Spadille1337](https://github.com/Spadille1337) opened issue at [2024-02-21 11:41](https://github.com/rear/rear/issues/3161): + +- ReaR version ("/usr/sbin/rear -V"): + Relax-and-Recover 2.5 / Git + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): + NAME="Ubuntu" + VERSION="20.04 LTS (Focal Fossa)" + ID=ubuntu + ID\_LIKE=debian + PRETTY\_NAME="Ubuntu 20.04 LTS" + VERSION\_ID="20.04" + HOME\_URL="[https://www.ubuntu.com/](https://www.ubuntu.com/)" + SUPPORT\_URL="[https://help.ubuntu.com/](https://help.ubuntu.com/)" + BUG\_REPORT\_URL="[https://bugs.launchpad.net/ubuntu/](https://bugs.launchpad.net/ubuntu/)" + PRIVACY\_POLICY\_URL="[https://www.ubuntu.com/legal/terms-and-policies/privacy-policy](https://www.ubuntu.com/legal/terms-and-policies/privacy-policy)" + VERSION\_CODENAME=focal + UBUNTU\_CODENAME=focal + + + + lsblk + NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT + loop0 7:0 0 4K 1 loop /snap/bare/5 + loop1 7:1 0 63.9M 1 loop /snap/core20/2105 + loop2 7:2 0 55.7M 1 loop /snap/core18/2812 + loop3 7:3 0 55.7M 1 loop /snap/core18/2796 + loop4 7:4 0 74.1M 1 loop /snap/core22/1033 + loop5 7:5 0 63.9M 1 loop /snap/core20/2182 + loop6 7:6 0 497M 1 loop /snap/gnome-42-2204/141 + loop7 7:7 0 218.4M 1 loop /snap/gnome-3-34-1804/93 + loop8 7:8 0 240.8M 1 loop /snap/gnome-3-34-1804/24 + loop9 7:9 0 74.2M 1 loop /snap/core22/1122 + loop10 7:10 0 62.1M 1 loop /snap/gtk-common-themes/1506 + loop11 7:11 0 91.7M 1 loop /snap/gtk-common-themes/1535 + loop12 7:12 0 40.9M 1 loop /snap/snapd/20290 + loop13 7:13 0 12.3M 1 loop /snap/snap-store/959 + loop14 7:14 0 49.8M 1 loop /snap/snap-store/433 + loop15 7:15 0 40.4M 1 loop /snap/snapd/20671 + loop16 7:16 0 64M 1 loop /snap/sublime-text/156 + loop17 7:17 0 64M 1 loop /snap/sublime-text/137 + sda 8:0 0 100G 0 disk + ├─sda1 8:1 0 976M 0 part /boot + ├─sda2 8:2 0 1K 0 part + ├─sda5 8:5 0 3.8G 0 part [SWAP] + ├─sda6 8:6 0 9.5G 0 part /home + └─sda7 8:7 0 85.7G 0 part / + sdb 8:16 0 25G 0 disk + ├─sdb1 8:17 0 487.3M 0 part /home/ubuntu/Documents/mnt1 + ├─sdb2 8:18 0 488.3M 0 part /home/ubuntu/Documents/mnt2 + └─sdb3 8:19 0 24G 0 part /home/ubuntu/Documents/mnt3 + sr0 11:0 1 412.7M 0 rom /media/ubuntu/RELAXRECOVER + +- Issue 1 : I want to restore both the disks with same partition + mapping sda and sdb using rear recovery. +- Issue 2 : I want to take backup of sbb only and restore sdb only + with same partition mapping for sdb using rear. + +Kindly help me with the above Issues. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-21 13:42](https://github.com/rear/rear/issues/3161#issuecomment-1956680819): + +@Spadille1337 +do you have separated fully compatible replacement hardware +(or a replacement VM if you currently use a VM) available +where you could risklessly try out step by step how far +"rear recover" works for your specific intended use case? + +Cf. +[https://en.opensuse.org/SDB:Disaster\_Recovery](https://en.opensuse.org/SDB:Disaster_Recovery) + +#### [github-actions](https://github.com/apps/github-actions) commented at [2024-04-22 02:07](https://github.com/rear/rear/issues/3161#issuecomment-2068358210): + +Stale issue message + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-22.3162.issue.open.md b/docs/issues/2024-02-22.3162.issue.open.md new file mode 100644 index 00000000..a8933635 --- /dev/null +++ b/docs/issues/2024-02-22.3162.issue.open.md @@ -0,0 +1,92 @@ +[\#3162 Issue](https://github.com/rear/rear/issues/3162) `open`: RFC: Why does "rear recover" regenerate the original's system initrd in any case? +================================================================================================================================================== + +**Labels**: `enhancement`, `cleanup`, `discuss / RFC` + +#### [jsmeix](https://github.com/jsmeix) opened issue at [2024-02-22 10:58](https://github.com/rear/rear/issues/3162): + +Triggered by +[https://github.com/rear/rear/issues/3152](https://github.com/rear/rear/issues/3152) +and +[https://github.com/rear/rear/pull/3155](https://github.com/rear/rear/pull/3155) +I am wondering +why "rear recover" regenerates +the original's system initrd in any case? + +In particular why I am wondering see in my +[https://github.com/rear/rear/issues/3152\#issuecomment-1941510047](https://github.com/rear/rear/issues/3152#issuecomment-1941510047) + + Cannot create initrd (found no mkinitrd in the recreated system). + ... + Nevertheless the recreated system boots well for me + as expected because I recreated on 100% compatible + replacement "hardware" (on a same VM) so the same initrd + from the original system should still work. + +and see in my +[https://github.com/rear/rear/pull/3155\#issue-2138456399](https://github.com/rear/rear/pull/3155#issue-2138456399) + + Additionally improved the user messages + (in particular the warning messages) + to make it more clear that the point is + to decide if the recreated system will boot + with the initrd 'as is' from the backup restore. + +The default use case of "rear recover" is to recreate +on "Fully compatible replacement hardware" +(where "hardware" could be also virtual hardware), cf. +[https://en.opensuse.org/SDB:Disaster\_Recovery\#Fully\_compatible\_replacement\_hardware\_is\_needed](https://en.opensuse.org/SDB:Disaster_Recovery#Fully_compatible_replacement_hardware_is_needed) + +On "fully compatible replacement hardware" +the initrd 'as is' from the original system +together with the kernel 'as is' from the original system +(that both together are restored 'as is' from the backup) +must still work +(otherwise it is no "fully compatible replacement hardware"). + +So for the default use case of "rear recover" +there is no need to regenerate the initrd. + +But regenerating the initrd needs relatively long time +so "rear recover" is slower than actually needed +AND +regenerating the original's system initrd +during "rear recover" may generate the new initrd +with (possibly subtle but severe) differences +compared to the original's system initrd +so regenerating the initrd in any case +may do more harm than good in practice +for the default use case of "rear recover". + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-02-22 15:19](https://github.com/rear/rear/issues/3162#issuecomment-1959671814): + +I would check with SW RAID, I suspect some details of the recreated +array may differ from the original (like UUIDs) and regenerating the +initrd may be needed for picking up this. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-02-22 15:21](https://github.com/rear/rear/issues/3162#issuecomment-1959675328): + +This may be the case also for other IDs in the system config files that +may change, if the files are embedded in the ramdisk. We have a script +running at the end that updates the IDs in known config files. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-23 14:24](https://github.com/rear/rear/issues/3162#issuecomment-1961416228): + +@pcahyna +thank you for the reasons why regenerating the initrd +could be needed in any case to be more on the safe side +against (possibly subtle but severe) differences +between the original system and the recreated system. + +#### [abbbi](https://github.com/abbbi) commented at [2024-03-14 08:07](https://github.com/rear/rear/issues/3162#issuecomment-1996785565): + +just as side note, i also experiencing this mostly on systems with +multipath setups. +Im using qemu to simulate a multipath san environment and noted in my +automated tests that initrd is allways recreated, even if system +recovery is done into is started with equal qemu options. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-23.3163.pr.merged.md b/docs/issues/2024-02-23.3163.pr.merged.md new file mode 100644 index 00000000..3e3a798d --- /dev/null +++ b/docs/issues/2024-02-23.3163.pr.merged.md @@ -0,0 +1,306 @@ +[\#3163 PR](https://github.com/rear/rear/pull/3163) `merged`: Error out if TMPDIR is set in user config +======================================================================================================= + +**Labels**: `enhancement`, `fixed / solved / done` + +#### [pcahyna](https://github.com/pcahyna) opened issue at [2024-02-23 11:01](https://github.com/rear/rear/pull/3163): + +#### Relax-and-Recover (ReaR) Pull Request Template + +##### Pull Request Details: + +- Type: **Enhancement** + +- Impact: **Low** + +- Reference to related issue (URL): \#2654 + +- How was this pull request tested? + `export TMPDIR=/tmp` in `etc/rear/local.conf` and calling + `rear help` (the error appears even with `help`). + +- Description of the changes in this pull request: + Error out early if TMPDIR is set in user config - setting TMPDIR in + {site,local}.conf has not worked since 0022063 (PR \#2633), as + discussed in \#2654. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-02-23 12:29](https://github.com/rear/rear/pull/3163#issuecomment-1961242537): + +@rear/contributors please check if you agree with this change. I believe +that it is better to abort early than to behave in unexpected ways. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-23 13:11](https://github.com/rear/rear/pull/3163#issuecomment-1961304528): + +@pcahyna +thank you for this improvement. + +I fully agree that in particular for "rear mkrescue/mkbackup" +(i.e. for all workflows that run on the original system) +it is always better to abort early than to (silently) +behave in unexpected ways. + +For "rear recover" it is different. +Here it is in general better to abort early +than to (silently) behave in unexpected ways. +But for "rear recover" there are certain cases +(e.g. recreation of the initrd) +where it is better to proceed "bona fide" and +try out of something will actually fail later +than to check things in advance and abort in advance +without ever trying the real thing. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-23 13:14](https://github.com/rear/rear/pull/3163#issuecomment-1961308748): + +@pcahyna +I do not understand why you set + + saved_tmpdir="${TMPDIR-}" + +two times but you do not use the saved\_tmpdir value +between when it is set first and when it is set for the second time. +So I do not understand why you set it for the first time. +In particular I fail to understand what you like to tell with + + # Can legitimately change in internal defaults, so we will set it again + # after reading them. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-02-23 13:24](https://github.com/rear/rear/pull/3163#issuecomment-1961324195): + +> For "rear recover" it is different. +> Here it is in general better to abort early +> than to (silently) behave in unexpected ways. +> But for "rear recover" there are certain cases +> (e.g. recreation of the initrd) +> where it is better to proceed "bona fide" + +Good point, but I think that this case falls in the first category: even +with "recover", we should abort early and let user fix their local.conf, +if needed. "recover" can behave in strange ways when TMPDIR is not set +properly. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-02-23 13:26](https://github.com/rear/rear/pull/3163#issuecomment-1961327343): + +> I do not understand why you set +> +> saved_tmpdir="${TMPDIR-}" +> +> two times but you do not use the saved\_tmpdir value +> between when it is set first and when it is set for the second time. + +The first initialization can be deleted, I just did not want to leave +the value unset, but there was no good reason for it. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-23 14:06](https://github.com/rear/rear/pull/3163#issuecomment-1961388695): + +I think this case falls **indirectly** in the first category +so it is OK to always error out here. + +Details: + +I think this case does not directly fall in the first category +because I think "rear recover" likely works +even with TMPDIR (uselessly) set in a user config file +so it could be better for the user to not bother him +and proceed "bona fide" and try out if something +will actually fail during "rear recover". + +Nevertheless I think in this case it is OK to always error out +because "rear recover" cannot run without "rear mkrescue" +(or "rear mkbackup") before. +Because the user config files in the ReaR recovery system +are usually same as what they were when "rear mkrescue" +(or "rear mkbackup") was run before, we can safely assume +that "rear recover" will normally not error out with TMPDIR +set in a user config file. + +In the unusual case when the user changed a config file +in the ReaR recovery system before running "rear recover" +it is OK to error out when the user had falsely set TMPDIR in +a config file because this will correctly tell the user that +his config file change in the ReaR recovery system was wrong +and how to properly achieve what he intends. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-02-23 14:22](https://github.com/rear/rear/pull/3163#issuecomment-1961412911): + +> I think this case does not directly fall in the first category +> because I think "rear recover" likely works +> even with TMPDIR (uselessly) set in a user config file +> so it could be better for the user to not bother him +> and proceed "bona fide" and try out if something +> will actually fail during "rear recover" + +It may work now, but as soon as some code uses `mktemp` after the +configuration is sourced, it will misbehave with +`mktemp: failed to create file via template '/foo/bar/tmp.XXXXXXXXXX': No such file or directory` +if `TMPDIR=/foo/bar` exists in the original system, but does not exist +in the recovery ramdisk. + +No such code exists now, but I guess it will appear sooner or later. + +EDIT: N.B. this was not an issue until my commit +2721c2743ea959631be0d30eac92984a708c1e66, because there was code to +unset TMPDIR in the recovery system. + +EDIT: I agree with the rest of what you wrote. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-26 07:13](https://github.com/rear/rear/pull/3163#issuecomment-1963454503): + +Meanwhile I think the + + Error "Setting TMPDIR in a configuration file is not supported. To specify a working area directory prefix, export TMPDIR before executing '$PROGRAM'" + +is wrong - both the message and the Error() abort. + +I think this should be only a user notification like + + LogPrintError "Setting TMPDIR in a configuration file does not work to specify ReaR's working area, see default.conf about TMPDIR" + +Reason: + +Setting TMPDIR in a ReaR config file does not work +to specify ReaR's working area BUILD\_DIR +BUT +setting or unsetting TMPDIR in a ReaR config file +is probably the only way to specify a different TMPDIR +(i.e. different than TMPDIR for ReaR's working area) +for the programs that are called by ReaR. + +For example assume a user needs to have +ReaR's BUILD\_DIR in /path/to/much/space +but he wants to keep the system's default TMPDIR +for the programs that are called by ReaR. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-02-26 17:27](https://github.com/rear/rear/pull/3163#issuecomment-1964708501): + +@jsmeix + +> ... wrong - both the message and the Error() abort. +> I think this should be only a user notification ... + +I could do this, but the motivation for aborting early is to prevent +nasty surprises when recovering, since while one takes care to ensure +that the TMPDIR exists in the original system, I am afraid that +essentially nobody realizes the implication that the same TMPDIR would +be used for recovery as well and thus needs to be included in the +ramdisk. This thus leads to frustration when the recovery is actually +needed (often under time pressure) and resulting support calls (that +dedicated support or I will have to handle, or will land in our issue +tracker). The case when one sets TMPDIR back to the default is safe, but +most other cases will not be. Not supporting it is thus the simplest, +most robust solution. + +If we really decide to downgrade it to printing and continuing despite +what I wrote above, then I perhaps should revert +2721c2743ea959631be0d30eac92984a708c1e66, since the motivation for that +was that setting TMPDIR in the middle of user configuration is +unsupported anyway, so the code is unneeded. Restoring it would help +with the concerns above. + +Note though that supporting the case where one needs to have ReaR's +BUILD\_DIR in /path/to/much/space but one wants to keep the system's +default TMPDIR for the programs that are called by ReaR is risky. What +if for some reason we move the creation of the build area below sourcing +of the user config again? Then it would break, as it would create +BUILD\_DIR in the system default location instead of in +/path/to/much/space. I don't think we should teach users to rely on such +intimate details of the implementation. Are you aware of any case where +an user wants TMPDIR for called programs to be different than for the +build area, anyway? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-27 06:42](https://github.com/rear/rear/pull/3163#issuecomment-1965886579): + +By the way +regarding where `TMPDIR=` is used in ReaR code: + + # find usr/sbin/rear usr/share/rear/ -type f | xargs grep 'TMPDIR' | grep -v ':[ ]*#' + + usr/share/rear/conf/examples/rescue-and-backup-on-same-ISO-image-example.conf: + TMPDIR=/mnt2/tmp + + usr/share/rear/conf/default.conf: + export TMPDIR="${TMPDIR-/var/tmp}" + + usr/share/rear/restore/DUPLICITY/default/200_prompt_user_to_start_restore.sh: + test $TMPDIR && old_TMPDIR=$TMPDIR + export TMPDIR=$TARGET_FS_ROOT + test $old_TMPDIR && TMPDIR=$old_TMPDIR || unset TMPDIR + + usr/share/rear/output/USB/Linux-i386/100_create_efiboot.sh: + efi_mpt=$( mktemp -d $TMPDIR/rear-efi.XXXXXXXXXX ) || Error "mktemp failed to create mount point '$TMPDIR/rear-efi.XXXXXXXXXX' for EFI partition '$efi_part'" + +So there is a wrong config example `TMPDIR=/mnt2/tmp` in +usr/share/rear/conf/examples/rescue-and-backup-on-same-ISO-image-example.conf + + # The defaul location /tmp might be too small to contain the backup (temporary location) + # use an alternative for /tmp (as backup.tar.gz size might be too big for /tmp) + TMPDIR=/mnt2/tmp + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-27 06:44](https://github.com/rear/rear/pull/3163#issuecomment-1965888455): + +Via +[https://github.com/rear/rear/commit/9793364ba85aac019f367f45f906e55e2e4d4648](https://github.com/rear/rear/commit/9793364ba85aac019f367f45f906e55e2e4d4648) +I removed in +conf/examples/rescue-and-backup-on-same-ISO-image-example.conf +the wrong and outdated TMPDIR=/mnt2/tmp +that is no longer needed since the default +is /var/tmp which has enough space. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-27 07:06](https://github.com/rear/rear/pull/3163#issuecomment-1965910450): + +@pcahyna +thank you so much for your endless work and patience +with that endless "how to properly deal with TMPDIR" stuff! + +This makes THE difference between quick hacks +versus thorougly thought-out practicable solutions +even if a solution cannot solve all possible cases +but we can only do what is possible with reasonable effort. + +Please merge it as soon as you can. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-02-27 10:36](https://github.com/rear/rear/pull/3163#issuecomment-1966257789): + +@jsmeix thank you for the thoughtful review and correction of the +obsolete example! Merging. +By the way: +you searched for `TMPDIR`, but any invocation of `mktemp` implicitely +uses `TMPDIR`, so we have some more uses beyond those that you found. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-27 10:50](https://github.com/rear/rear/pull/3163#issuecomment-1966280588): + +@pcahyna +I searched for TMPDIR because I was interested +what we do with the TMPDIR variable itself +in particular where we modify it +(not where we use its value e.g. via mktemp). + +But I missed to search our documentation: + + # find doc -type f | xargs grep 'TMPDIR' + + doc/user-guide/03-configuration.adoc: + TMPDIR=/bigdisk:: + The +TMPDIR+ is picked up by the +mktemp+ command to create the +BUILD_DIR+ under +/bigdisk/tmp/rear.XXXX+ + The default value of +TMPDIR+ is an empty string, therefore, by default +BUILD_DIR+ is +/tmp/rear.XXXX+ + +I will also fix that right now... + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-27 10:58](https://github.com/rear/rear/pull/3163#issuecomment-1966295288): + +Via +[https://github.com/rear/rear/commit/d5556a3a0a481e941a4604b137978ca0d2d6dcc6](https://github.com/rear/rear/commit/d5556a3a0a481e941a4604b137978ca0d2d6dcc6) +I removed in doc/user-guide/03-configuration.adoc +the outdated and obsolete part about TMPDIR=/bigdisk +because since the default is /var/tmp +there is enough space for things like a big ISO image + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-02-27 11:05](https://github.com/rear/rear/pull/3163#issuecomment-1966306328): + +@jsmeix good catch! I think that documenting default values separately +from default.conf is often a bad idea - the documentation becomes soon +obsolete. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-27.3164.issue.open.md b/docs/issues/2024-02-27.3164.issue.open.md new file mode 100644 index 00000000..0ec20b57 --- /dev/null +++ b/docs/issues/2024-02-27.3164.issue.open.md @@ -0,0 +1,94 @@ +[\#3164 Issue](https://github.com/rear/rear/issues/3164) `open`: Backup of only /dev/sdb +======================================================================================== + +**Labels**: `support / question` + +#### [Spadille1337](https://github.com/Spadille1337) opened issue at [2024-02-27 07:44](https://github.com/rear/rear/issues/3164): + +- ReaR version ("/usr/sbin/rear -V"): + Relax-and-Recover 2.5 / Git + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): + + + + NAME="Ubuntu" + VERSION="20.04 LTS (Focal Fossa)" + ID=ubuntu + ID_LIKE=debian + PRETTY_NAME="Ubuntu 20.04 LTS" + VERSION_ID="20.04" + HOME_URL="https://www.ubuntu.com/" + SUPPORT_URL="https://help.ubuntu.com/" + BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" + PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" + VERSION_CODENAME=focal + UBUNTU_CODENAME=focal + +- lsblk + + + + root@ubunu2004:~# lsblk + NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT + loop0 7:0 0 4K 1 loop /snap/bare/5 + loop1 7:1 0 55M 1 loop /snap/core18/1705 + loop2 7:2 0 55.7M 1 loop /snap/core18/2812 + loop3 7:3 0 74.2M 1 loop /snap/core22/1122 + loop4 7:4 0 240.8M 1 loop /snap/gnome-3-34-1804/24 + loop5 7:5 0 497M 1 loop /snap/gnome-42-2204/141 + loop6 7:6 0 62.1M 1 loop /snap/gtk-common-themes/1506 + loop7 7:7 0 218.4M 1 loop /snap/gnome-3-34-1804/93 + loop8 7:8 0 91.7M 1 loop /snap/gtk-common-themes/1535 + loop9 7:9 0 49.8M 1 loop /snap/snap-store/433 + loop10 7:10 0 12.3M 1 loop /snap/snap-store/959 + loop11 7:11 0 40.4M 1 loop /snap/snapd/20671 + loop12 7:12 0 63.9M 1 loop /snap/core20/2182 + loop13 7:13 0 64M 1 loop /snap/sublime-text/156 + sda 8:0 0 100G 0 disk + ├─sda1 8:1 0 976M 0 part /boot + ├─sda2 8:2 0 1K 0 part + ├─sda5 8:5 0 3.8G 0 part [SWAP] + ├─sda6 8:6 0 9.5G 0 part /home + └─sda7 8:7 0 85.7G 0 part / + sdb 8:16 0 12.5G 0 disk + ├─sdb1 8:17 0 4.8G 0 part /mnt/mnt1 + ├─sdb2 8:18 0 2.4G 0 part /mnt/mnt2 + └─sdb3 8:19 0 5.4G 0 part /mnt/mnt3 + sr0 11:0 1 1024M 0 rom + +- Issue + I am using virtual machine. As we can see, I mounted /dev/sdb1 over + /mnt/mnt1, /dev/sdb2 over /mnt/mnt2 and /dev/sdb3 over /mnt/mnt3. + If I include the following line in local.conf + `EXCLUDE_COMPONENTS=("${EXCLUDE_COMPONENTS[@]}" "/dev/sda7" "/dev/sda1" "/dev/sda6" "/dev/sda2" "/dev/sda5")` + , this excludes the whole system data including /dev/sdb1, + /dev/sdb2, /dev/sdb3. If I include "/dev/sda7" then this will take + backup of /dev/sdb1, /dev/sdb2, /dev/sdb3 along with that data in + "/" as "/dev/sda7 is mounted over "/". + Kindly tell me how can I take backup of only "/dev/sdb" in this + situation. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-27 11:16](https://github.com/rear/rear/issues/3164#issuecomment-1966323357): + +@Spadille1337 +I do not understand what you like to achieve +with ReaR and a "backup" of only /dev/sdb +when /dev/sda is your system disk +(where `/` `/boot` `/home` and `swap` is) +because ReaR is meant to recreate a system +so at least the system disk must be backed up +but ReaR is no backup software and is not meant to be one. + +See +[https://en.opensuse.org/SDB:Disaster\_Recovery](https://en.opensuse.org/SDB:Disaster_Recovery) + +Therein see in particular the section +"Relax-and-Recover versus backup and restore" +[https://en.opensuse.org/SDB:Disaster\_Recovery\#Relax-and-Recover\_versus\_backup\_and\_restore](https://en.opensuse.org/SDB:Disaster_Recovery#Relax-and-Recover_versus_backup_and_restore) + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-27.3165.pr.merged.md b/docs/issues/2024-02-27.3165.pr.merged.md new file mode 100644 index 00000000..52732323 --- /dev/null +++ b/docs/issues/2024-02-27.3165.pr.merged.md @@ -0,0 +1,47 @@ +[\#3165 PR](https://github.com/rear/rear/pull/3165) `merged`: Fix couple of OS detection issues +=============================================================================================== + +**Labels**: `bug`, `cleanup`, `fixed / solved / done` + +#### [lzaoral](https://github.com/lzaoral) opened issue at [2024-02-27 09:18](https://github.com/rear/rear/pull/3165): + +#### Relax-and-Recover (ReaR) Pull Request Template + +Please fill in the following items before submitting a new pull request: + +##### Pull Request Details: + +- Type: **Bug Fix** + +- Impact: **Normal** / **High** + +- Reference to related issue (URL): + [https://github.com/rear/rear/issues/3149](https://github.com/rear/rear/issues/3149) + +- How was this pull request tested? By using `rear dump`. + +- Description of the changes in this pull request: This PR fixes + problems 2. and 3. described in detail in + [https://github.com/rear/rear/issues/3149\#issuecomment-1964290116](https://github.com/rear/rear/issues/3149#issuecomment-1964290116). + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-02-27 09:20](https://github.com/rear/rear/pull/3165#issuecomment-1966113577): + +Of course, if the system already contains the `/etc/rear/os.conf` file +provided by other means, it will still be respected by ReaR. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-28 08:23](https://github.com/rear/rear/pull/3165#issuecomment-1968459005): + +@lzaoral @pcahyna @rear/contributors +I would like to "just merge" it today afternoon +unless there are objections + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-28 16:21](https://github.com/rear/rear/pull/3165#issuecomment-1969350726): + +@lzaoral +thank you for working through that rather old code +and fixing it and cleaning it up step by step! + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-27.3166.pr.open.md b/docs/issues/2024-02-27.3166.pr.open.md new file mode 100644 index 00000000..8ad2e044 --- /dev/null +++ b/docs/issues/2024-02-27.3166.pr.open.md @@ -0,0 +1,93 @@ +[\#3166 PR](https://github.com/rear/rear/pull/3166) `open`: WIP: Use the new TextPrefix function +================================================================================================ + +**Labels**: `enhancement` + +#### [jsmeix](https://github.com/jsmeix) opened issue at [2024-02-27 11:51](https://github.com/rear/rear/pull/3166): + +Use the new TextPrefix function + +- Type: **Enhancement** + +- Impact: **None** + No impact for users. + But for us: + More standardized code by a general TextPrefix function + instead of individual `sed` calls. + Easier to find the code places where text is prefixed. + Easier to properly indent multi-line texts in the code. + +- Reference to related issue (URL): + [https://github.com/rear/rear/pull/3160](https://github.com/rear/rear/pull/3160) + +- How was this pull request tested? + [https://github.com/rear/rear/pull/3160\#issuecomment-1954251341](https://github.com/rear/rear/pull/3160#issuecomment-1954251341) + and + [https://github.com/rear/rear/pull/3160\#issuecomment-1954258836](https://github.com/rear/rear/pull/3160#issuecomment-1954258836) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-27 13:50](https://github.com/rear/rear/pull/3166#issuecomment-1966593530): + +As always it is more complicated than one thinks, +cf. RFC 1925 item 8. + +Something like + + COMMAND | TextPrefix '# ' + +cannot be used in general. +It can only be used when the output of COMMAND +is not indented or when that indentation is meaningless +because TextPrefix removes (meaningful) indentation. + +For example the above +[https://github.com/rear/rear/pull/3166/commits/9688568ec2d9f7f6031216424c498f1d800822ef](https://github.com/rear/rear/pull/3166/commits/9688568ec2d9f7f6031216424c498f1d800822ef) +should be OK because plain 'lsdasd' output is not indented +see the example in +[http://ubuntu-on-big-iron.blogspot.com/2018/03/ubuntu-ccw-howto-2-of-4-DASD.html](http://ubuntu-on-big-iron.blogspot.com/2018/03/ubuntu-ccw-howto-2-of-4-DASD.html) + + ... lsdasd + + Bus-ID Status Name Device Type BlkSz Size Blocks + ======================================================================= + 0.0.2608 active dasda 94:0 ECKD 4096 7043MB 1803060 + 0.0.2619 active dasdb 94:4 ECKD 4096 21129MB 5409180 + ... + +In contrast e.g. `lsdasd -l ...` output is indented +as shown in +[https://www.ibm.com/docs/en/linux-on-z?topic=c-lsdasd-2](https://www.ibm.com/docs/en/linux-on-z?topic=c-lsdasd-2) + + lsdasd -l 0.0.4d82 + 0.0.4d82/dasdd/94:12 + status: active + type: ECKD + blksz: 4096 + ... + +Another example where COMMAND output indentation +is meaningful and needed is the +"Example 'mdadm --misc --detail $raiddevice' output" +in layout/save/GNU/Linux/210\_raid\_layout.sh + + /dev/md/raid1sdab: + Version : 1.0 + Creation Time : Wed Oct 13 13:17:13 2021 + Raid Level : raid1 + Array Size : 12582784 (12.00 GiB 12.88 GB) + ... + +where `TextPrefix '# '` would result + + # /dev/md/raid1sdab: + # Version : 1.0 + # Creation Time : Wed Oct 13 13:17:13 2021 + # Raid Level : raid1 + # Array Size : 12582784 (12.00 GiB 12.88 GB) + # ... + +Not a big fault but less readable as the indented output. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-28.3167.issue.closed.md b/docs/issues/2024-02-28.3167.issue.closed.md new file mode 100644 index 00000000..3cdc2cbc --- /dev/null +++ b/docs/issues/2024-02-28.3167.issue.closed.md @@ -0,0 +1,190 @@ +[\#3167 Issue](https://github.com/rear/rear/issues/3167) `closed`: default TMPDIR /var/tmp leaves temporary files of called programs +==================================================================================================================================== + +**Labels**: `enhancement`, `bug`, `fixed / solved / done` + +#### [jsmeix](https://github.com/jsmeix) opened issue at [2024-02-28 13:43](https://github.com/rear/rear/issues/3167): + +Since we set in default.conf TMPDIR to /var/tmp (if unset) via + + export TMPDIR="${TMPDIR-/var/tmp}" + +programs that are called by ReaR use by default /var/tmp +for their temporary files. + +Programs may not carefully clean up their temporary files +or may leave temporary files in case of program aborts +when programs (falsely) rely on the system default /tmp +that usually gets automatically cleaned up relatively fast +e.g. via something like systemd-tmpfiles-clean.service +or at least via reboot when /tmp is a tmpfs in RAM. + +In contast to /var/tmp for programs that are called by ReaR +ReaR cleanes up its own temporary stuff in BUILD\_DIR +carefully and completely via the function +cleanup\_build\_area\_and\_end\_program() +in lib/\_input-output-functions.sh + +Perhaps it would be good when /usr/sbin/rear +sets TMPDIR to $TMP\_DIR (TMP\_DIR=$BUILD\_DIR/tmp) +if TMPDIR was initially unset when 'rear' was called +so that programs that are called by ReaR use +by default $BUILD\_DIR/tmp for their temporary files? + +An addedum regarding confidentiality, +here in particular about confidentiality of temporary files +from programs that are called by ReaR: + +Currently we do in /usr/sbin/rear + + BUILD_DIR="$( mktemp -d -t rear.XXXXXXXXXXXXXXX || Error "Could not create build area '$BUILD_DIR'" )" + ROOTFS_DIR=$BUILD_DIR/rootfs + TMP_DIR=$BUILD_DIR/tmp + mkdir -p $ROOTFS_DIR || Error "Could not create $ROOTFS_DIR" + mkdir -p $TMP_DIR || Error "Could not create $TMP_DIR" + +without any special 'umask' or 'chmod' for confidentiality +so we rely on sufficiently safe 'mktemp -d' behaviour + + '-d' + ... + The directory will have read, write, and search permissions + for the current user, but no permissions for the group or others + +as described in +[https://www.gnu.org/software/coreutils/manual/html\_node/mktemp-invocation.html\#mktemp-invocation](https://www.gnu.org/software/coreutils/manual/html_node/mktemp-invocation.html#mktemp-invocation) + +So when programs that are called by ReaR use $BUILD\_DIR/tmp +their temporary files are sufficiently safe via 'mktemp -d' +against unwanted access by non-root users. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-28 13:45](https://github.com/rear/rear/issues/3167#issuecomment-1969018735): + +@pcahyna +please have a look here (as time permits). +I really appreciate your feedback here. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-02-28 13:52](https://github.com/rear/rear/issues/3167#issuecomment-1969031794): + +@jsmeix that's an interesting point. After reading it, I have only one +remark (I may find other stuff later): + +> Perhaps it would be good when /usr/sbin/rear +> sets TMPDIR to $TMP\_DIR (TMP\_DIR=$BUILD\_DIR/tmp) +> if TMPDIR was initially unset when 'rear' was called +> so that programs that are called by ReaR use +> by default $BUILD\_DIR/tmp for their temporary files? + +Why only "if TMPDIR was initially unset when 'rear' was called" ? This +condition complicates the whole logic, and is also unhelpful. If we were +to do this change, I would do it unconditionally. If an user sets and +exports `TMPDIR=/some/big/file/system` before calling ReaR, it would be +nice to do the automatic cleanup for them as well, just as we would do +for the default `/var/tmp`. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-28 14:09](https://github.com/rear/rear/issues/3167#issuecomment-1969063408): + +@pcahyna +you already helped me a lot with your first remark! + +I falsely thought when we set in default.conf TMPDIR +to /var/tmp only if TMPDIR was unset that then I also +need to set TMPDIR to $TMP\_DIR only if TMPDIR was unset +when 'rear' was called. + +When it is better to set TMPDIR to $TMP\_DIR in any case +implementation is much simpler and straightforward for me! + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-28 14:37](https://github.com/rear/rear/issues/3167#issuecomment-1969123868): + +@pcahyna +because I am looking right now at usr/sbin/rear + + # Save the current value to detect changes. + saved_tmpdir="${TMPDIR-}" + +I am wondering why you use `${TMPDIR-}` +and not simply `$TMPDIR` because +TMPDIR is set in default.conf if it was unset +and default.conf was already sourced before. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-02-28 14:42](https://github.com/rear/rear/issues/3167#issuecomment-1969135145): + +@jsmeix I have not realized this. Thank you for pointing this out, but I +would not want to rely on such long-range interactions anyway. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-28 14:56](https://github.com/rear/rear/issues/3167#issuecomment-1969162109): + +@pcahyna +"sorry" for delivering bad news but tons of things in ReaR +depend on long-and-even-longer-range interactions ;-) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-15 08:50](https://github.com/rear/rear/issues/3167#issuecomment-1999192967): + +Meanwhile while implementing it via +[https://github.com/rear/rear/pull/3168](https://github.com/rear/rear/pull/3168) +I came to the conclusion that it is better +(at least for now to be on the safe side for the initial +implementation) +to set TMPDIR to $TMP\_DIR (TMP\_DIR=$BUILD\_DIR/tmp) +only if TMPDIR was initially unset when 'rear' was called. + +See my thoughts +[https://github.com/rear/rear/pull/3168\#issuecomment-1997014841](https://github.com/rear/rear/pull/3168#issuecomment-1997014841) +and +[https://github.com/rear/rear/pull/3168\#issuecomment-1997057144](https://github.com/rear/rear/pull/3168#issuecomment-1997057144) +and my comment in the code +[https://github.com/rear/rear/pull/3168/commits/7e278f3329fe52edaa9527dfde1816e74ea5cd5a](https://github.com/rear/rear/pull/3168/commits/7e278f3329fe52edaa9527dfde1816e74ea5cd5a) + + # We set TMPDIR to ReaR's TMP_DIR only when TMPDIR was not set + # before ReaR was launched by the user via export TMPDIR="..." + # to provide final power to the user so that a specific TMPDIR can be specified + # e.g. for certain third-party backup tools that may need a specific TMPDIR + +Therein "provide final power to the user" +is the crucial part which outweighs (at least for me) +[https://github.com/rear/rear/issues/3167\#issuecomment-1969031794](https://github.com/rear/rear/issues/3167#issuecomment-1969031794) + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-15 11:27](https://github.com/rear/rear/issues/3167#issuecomment-1999460501): + +@jsmeix setting `TMPDIR` to a subdirectory of the user-specified +`TMPDIR` should not be a problem because the temporary files will still +be under a directory that the user has specified. The user should not +assume what file or directory structure under the TMPDIR that they have +specified will the tool use. After all, even now, we don't put temporary +files directly into TMPDIR, but under $TMPDIR/rear.XXXXXX and possibly +some directories in between, and it has not been a problem. + +Changing `TMPDIR` to something that would *not* be under the +user-specified `TMPDIR` would certainly be against "final power to the +user", but that's not what you are doing here. + +Your examples of third-party backup tools show that when such a tool +needs a specific `TMPDIR`, it does not obey what the user has specified +- the code specific to the tool must set `TMPDIR` to the "right" value +itself, in any case. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-15 12:52](https://github.com/rear/rear/issues/3167#issuecomment-1999602582): + +@pcahyna +thank you for your explanation! +I will think about it over the weekend. +My current gut feeling is that you are right. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-26 12:28](https://github.com/rear/rear/issues/3167#issuecomment-2020300492): + +With +[https://github.com/rear/rear/pull/3168](https://github.com/rear/rear/pull/3168) +merged +this issue should be reasonably solved. + +What is missing is proper handling of TMPDIR in RECOVERY\_MODE +in particular for the "chroot /mnt/local" case, see +[https://github.com/rear/rear/pull/3168\#issuecomment-1988974933](https://github.com/rear/rear/pull/3168#issuecomment-1988974933) +and +[https://github.com/rear/rear/pull/3168\#issuecomment-1991347031](https://github.com/rear/rear/pull/3168#issuecomment-1991347031) + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-28.3168.pr.merged.md b/docs/issues/2024-02-28.3168.pr.merged.md new file mode 100644 index 00000000..177d6f51 --- /dev/null +++ b/docs/issues/2024-02-28.3168.pr.merged.md @@ -0,0 +1,1427 @@ +[\#3168 PR](https://github.com/rear/rear/pull/3168) `merged`: Set TMPDIR to ReaR's TMP\_DIR=$BUILD\_DIR/tmp for programs that are called by ReaR +================================================================================================================================================ + +**Labels**: `enhancement`, `fixed / solved / done` + +#### [jsmeix](https://github.com/jsmeix) opened issue at [2024-02-28 15:36](https://github.com/rear/rear/pull/3168): + +- Type: **Enhancement** + +- Impact: **High** + Hopefully a high positive impact + because of improved default confidentiality and + because of improved careful and complete cleanup of temporary + leftovers + +- Reference to related issue (URL): + [https://github.com/rear/rear/issues/3167](https://github.com/rear/rear/issues/3167) + +- How was this pull request tested? + A simple "rear mkbackup" + and "rear recover" work for me. + +- Description of the changes in this pull request: + +In usr/sbin/rear set TMPDIR to TMP\_DIR (within BUILD\_DIR) +to ensure +that also for programs that are called by ReaR +ReaR will clean up all temporary stuff carefully and completely +via the function cleanup\_build\_area\_and\_end\_program() +AND +that temporary files from programs that are called by ReaR +are sufficiently safe against unwanted access by non-root users +via 'mktemp -d' where BUILD\_DIR and implicitly +TMP\_DIR=$BUILD\_DIR/tmp +has permissions only for root but none for the group or others + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-02-28 15:45](https://github.com/rear/rear/pull/3168#issuecomment-1969267649): + +My simple test of "rear mkbackup" with + + OUTPUT=ISO + BACKUP=NETFS + BACKUP_URL=file:///other/ + BACKUP_ONLY_INCLUDE="yes" + BACKUP_PROG_INCLUDE=( /home/johannes ) + + # mount | grep other + /dev/nvme0n1p4 on /other type ext4 (rw,relatime,stripe=32) + +For testing I added at the beginning of +init/default/001\_verify\_config\_arrays.sh + + tmpdir="$( mktemp -d )" + echo tmpdata >$tmpdir/tmpfile + DebugPrint "$( find $tmpdir -ls )" + +Before the changes in this pull request: + + # usr/sbin/rear -D mkrescue + ... + Running 'init' stage ====================== + 14419482 4 drwx------ 2 root root 4096 Feb 28 13:47 /var/tmp/tmp.FWlUGBmzz8 + 14419484 4 -rw-r--r-- 1 root root 8 Feb 28 13:47 /var/tmp/tmp.FWlUGBmzz8/tmpfile + ... + To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.Bw6o1RnUaVZCc5M + + # rm -Rf --one-file-system /var/tmp/rear.Bw6o1RnUaVZCc5M + # echo $? + 0 + # find /var/tmp/rear.Bw6o1RnUaVZCc5M -ls + find: ‘/var/tmp/rear.Bw6o1RnUaVZCc5M’: No such file or directory + + # find /var/tmp/tmp.FWlUGBmzz8 -ls + 14419482 4 drwx------ 2 root root 4096 Feb 28 13:47 /var/tmp/tmp.FWlUGBmzz8 + 14419484 4 -rw-r--r-- 1 root root 8 Feb 28 13:47 /var/tmp/tmp.FWlUGBmzz8/tmpfile + +With the changes in this pull request: + + # usr/sbin/rear -D mkbackup + ... + Running 'init' stage ====================== + 14811425 4 drwx------ 2 root root 4096 Feb 28 16:16 /var/tmp/rear.7rxBQsZQKFf6f5e/tmp/tmp.esJeyHDyzM + 14811426 4 -rw-r--r-- 1 root root 8 Feb 28 16:16 /var/tmp/rear.7rxBQsZQKFf6f5e/tmp/tmp.esJeyHDyzM/tmpfile + ... + To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.7rxBQsZQKFf6f5e + + # rm -Rf --one-file-system /var/tmp/rear.7rxBQsZQKFf6f5e + # echo $? + 0 + # find /var/tmp/rear.7rxBQsZQKFf6f5e -ls + find: ‘/var/tmp/rear.7rxBQsZQKFf6f5e’: No such file or directory + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-06 15:20](https://github.com/rear/rear/pull/3168#issuecomment-1981113250): + +I did "rear mkbackup" and "rear recover" with + + OUTPUT=ISO + BACKUP=NETFS + BACKUP_URL=nfs://192.168.178.66/nfs + +which "just worked" normally for me. + +I did the same artificial change at the beginning of +init/default/001\_verify\_config\_arrays.sh +as above, i.e. + + tmpdir="$( mktemp -d )" + echo tmpdata >$tmpdir/tmpfile + DebugPrint "$( find $tmpdir -ls )" + +and got + + RESCUE localhost:~ # rear -D recover + ... + Running 'init' stage ====================== + 29774 0 drwx------ 2 root root 0 Mar 6 16:14 /var/tmp/rear.PrD9Rdyf07gHSxT/tmp/tmp.N7hW9sdA75 + 29775 4 -rw-r--r-- 1 root root 8 Mar 6 16:14 /var/tmp/rear.PrD9Rdyf07gHSxT/tmp/tmp.N7hW9sdA75/tmpfile + ... + Running exit tasks + To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.PrD9Rdyf07gHSxT + +The recreated system boots normally and +works normally (as far as I see at first glance). + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-06 15:23](https://github.com/rear/rear/pull/3168#issuecomment-1981121589): + +@rear/contributors @pcahyna +could you please review it (as time permits), +perhaps you see some obvious mistake. + +If there are no objections +I would like to merge it on Friday afternoon +so users who use our GitHub master code could test it and +report if there are issues in this or that circumstances. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-07 11:00](https://github.com/rear/rear/pull/3168#issuecomment-1983267535): + +Something is wrong in the CI runs: + + 2024-02-28 16:53:54.526692213 WARNING: + Failed to create initrd for kernel version '5.14.0-419.el9.x86_64'. + Check '/var/log/rear/rear-ip-172-31-29-183.log' to see the error messages in detail + and decide yourself, whether the system will boot or not. + 2024-02-28 16:53:54.555617047 Running dracut... + 2024-02-28 16:53:54.585746511 WARNING: + Failed to create initrd for kernel version '5.14.0-425.el9.x86_64'. + Check '/var/log/rear/rear-ip-172-31-29-183.log' to see the error messages in detail + and decide yourself, whether the system will boot or not. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-07 11:23](https://github.com/rear/rear/pull/3168#issuecomment-1983303628): + +I don't know what's wrong, but I suspect that the TMPDIR setting is used +also in the rescue system and exported to programs called in the +`/mnt/local` chroot, and the `/var/tmp/rearXXXXXX` directory does not +exist in the chroot (unlike the former `/var/tmp` value that had the +advantage of existing both in the recovery ramdisk and in the recovered +system), and some programs called in the chroot do not like an invalid +TMPDIR value. +This is purely my speculation. Now confirmed by error messages in +@jsmeix's `rear -D recover` run. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-07 11:25](https://github.com/rear/rear/pull/3168#issuecomment-1983306864): + +@jsmeix do you know where to find the error messages from dracut, if +any? I don't see them in the recovery log: +[https://artifacts.dev.testing-farm.io/659057b7-cf20-4bf1-9519-3768cad95c92/work-backup-and-restoreom9c6wvr/tests/plans/backup-and-restore/execute/data/guest/default-0/make-backup-and-restore-iso-1/data/rear-recover.log](https://artifacts.dev.testing-farm.io/659057b7-cf20-4bf1-9519-3768cad95c92/work-backup-and-restoreom9c6wvr/tests/plans/backup-and-restore/execute/data/guest/default-0/make-backup-and-restore-iso-1/data/rear-recover.log) + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-07 11:27](https://github.com/rear/rear/pull/3168#issuecomment-1983309585): + +> The recreated system boots normally and +> works normally (as far as I see at first glance). + +Do you see the dracut warnings in the recovery log? Or maybe the +distribution you are testing on does not use dracut? + +#### [gdha](https://github.com/gdha) commented at [2024-03-07 12:01](https://github.com/rear/rear/pull/3168#issuecomment-1983365220): + +@pcahyna When inspecting +[https://artifacts.dev.testing-farm.io/5f65a14d-eba9-467a-b03e-24c667b17f9c/work-backup-and-restoreu9mahx67/tests/plans/backup-and-restore/execute/data/guest/default-0/make-backup-and-restore-iso-1/output.txt](https://artifacts.dev.testing-farm.io/5f65a14d-eba9-467a-b03e-24c667b17f9c/work-backup-and-restoreu9mahx67/tests/plans/backup-and-restore/execute/data/guest/default-0/make-backup-and-restore-iso-1/output.txt) +I do see the following: + + :: [ 16:48:38 ] :: [ PASS ] :: Command 'export TMPDIR='/var/tmp'' (Expected 0, got 0) + ... + 2024-02-28 16:51:39.691572435 Running dracut... + 2024-02-28 16:51:39.711368631 WARNING: + Failed to create initrd for kernel version '6.8.0-0.rc5.41.fc41.x86_64'. + Check '/var/log/rear/rear-ip-172-31-30-152.log' to see the error messages in detail + and decide yourself, whether the system will boot or not. + +Therefore, IMHO `TMPDIR` was set to `/var/tmp` +Perhaps, add a step to copy the +`/var/log/rear/rear-ip-172-31-30-152.log` as well? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-07 12:09](https://github.com/rear/rear/pull/3168#issuecomment-1983377528): + +Sigh! +I have that dracut error too. +Somehow I must have overlooked that yesterday. +Today I see it clearly: + + RESCUE localhost:~ # rear -D recover + ... + Using build area: /var/tmp/rear.bClgGxIS12HNrGx + ... + Warning: + Failed to recreate initrd with /usr/bin/dracut. + Check '/var/log/rear/rear-localhost.log' why /usr/bin/dracut failed + and decide if the recreated system will boot + with the initrd 'as is' from the backup restore. + +In ReaR debug mode my /var/log/rear/rear-localhost.log +shows why /usr/bin/dracut failed + + ++ chroot /mnt/local /bin/bash -c 'PATH=/sbin:/usr/sbin:/usr/bin:/bin /usr/bin/dracut --force' + realpath: /var/tmp/rear.bClgGxIS12HNrGx/tmp: No such file or directory + dracut: Invalid tmpdir '/var/tmp/rear.bClgGxIS12HNrGx/tmp'. + ++ LogPrintError 'Warning: + Failed to recreate initrd with /usr/bin/dracut. + Check '\''/var/log/rear/rear-localhost.log'\'' why /usr/bin/dracut failed + and decide if the recreated system will boot + with the initrd '\''as is'\'' from the backup restore. + ' + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-07 12:11](https://github.com/rear/rear/pull/3168#issuecomment-1983380163): + +@pcahyna +thank you so much for your careful review. +So often you detect things. +I do appreciate that so very much! + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-07 12:19](https://github.com/rear/rear/pull/3168#issuecomment-1983394355): + +@jsmeix Thank mainly Anton and Lukas for implementing the CI, especially +Lukas for grepping for WARNING and ERROR in the log and reporting a test +failure even if the recovery is successful! Please do not ignore test +failures especially if they occur on multiple distros. + +By the way: +the verification steps at the end of the test script contain: +`grep -C 10 -P "$log_prefix WARNING:" $path` +so it will stop working if the letter case of the messages is changed. + +@gdha + +> Perhaps, add a step to copy the +> /var/log/rear/rear-ip-172-31-30-152.log as well? + +The steps has been in ReaR by default for a long time! See +wrapup/default/990\_copy\_logfile.sh , the definition of +`copy_log_file_exit_task`. Unfortunately the log does not contain +anything more (the message I quoted is from that very same log file), so +the message +`Check '/var/log/rear/rear-ip-172-31-29-183.log' to see the error messages` +is unhelpful and misleading. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-07 12:20](https://github.com/rear/rear/pull/3168#issuecomment-1983395939): + +> I do see the following: +> `:: [ 16:48:38 ] :: [ PASS ] :: Command 'export TMPDIR='/var/tmp'' (Expected 0, got 0)` +> Therefore, IMHO TMPDIR was set to /var/tmp + +The script sets it before calling ReaR, but then ReaR changes TMPDIR +internally (which is the cause of the problem). + +And setting TMPDIR before a `rear mkrescue` run will not affect +`rear recover` anyway, and it is `rear recover` that has the problem, +not `rear mkrescue`. The test script does not alter the `rear recover` +run in any way (except for setting some `PRE_` / `POST_` stuff). + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-07 12:25](https://github.com/rear/rear/pull/3168#issuecomment-1983403646): + +I will have to think some time about it +whether or not there could be a good solution, +i.e. a solution that is reasonably simple +so that we can safely assume it works reasonably fail-safe +and that it will be reasonably future-proof. + +What I see now and what I do not like is that +all our `chroot /mnt/local COMMANDs` +could leave arbitrary temporary stuff behind +in the recreated system. + +I wished ReaR could provide some generic means +that all ReaR's temporary files get cleaned up properly +and that ReaR's temporary files are sufficiently safe +against unwanted access. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-07 12:30](https://github.com/rear/rear/pull/3168#issuecomment-1983410829): + +By the way +regarding `WARNING:` versus `Warning:` see +[https://github.com/rear/rear/pull/3153\#issuecomment-1952301713](https://github.com/rear/rear/pull/3153#issuecomment-1952301713) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-07 12:38](https://github.com/rear/rear/pull/3168#issuecomment-1983424293): + +Regarding +"Please do not ignore test failures": +I am sorry for that. +Obviously I did not have a good day yesterday. +Yesterday there were various different things in parallel +so apparently my brain had some kind of "overflow blackout". + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-07 12:50](https://github.com/rear/rear/pull/3168#issuecomment-1983442350): + +There is no dracut error message in ReaR's log file +unless ReaR runs in debug mode, cf. +[https://github.com/rear/rear/issues/2416](https://github.com/rear/rear/issues/2416) +and +[https://github.com/rear/rear/pull/2633](https://github.com/rear/rear/pull/2633) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-11 17:06](https://github.com/rear/rear/pull/3168#issuecomment-1988974933): + +Via +[https://github.com/rear/rear/pull/3168/commits/4e78ab50f9805a9f35935783646726f283b9a9d3](https://github.com/rear/rear/pull/3168/commits/4e78ab50f9805a9f35935783646726f283b9a9d3) +I set TMPDIR to TMP\_DIR only when we are not in RECOVERY\_MODE +as a currently not yet testet attempt to avoid +[https://github.com/rear/rear/pull/3168\#issuecomment-1983377528](https://github.com/rear/rear/pull/3168#issuecomment-1983377528) + +My reasoning: + +For programs that run within the ReaR recovery system +it should not matter where TMPDIR is or whether or not +some temporary files of called programs are left behind +because the ReaR recovery system is on a ramdisk that +gets completely removed by 'reboot' or 'poweroff' +so nothing of the ReaR recovery system is left behind. + +For programs that run within the recreated/restored system via + + chroot /mnt/local COMMAND + +exporting TMPDIR to ReaR's TMP\_DIR cannot work because +there cannot be a TMP\_DIR in the recreated/restored system +i.e. there is no `/mnt/local/var/tmp/rear.XXXXXXXXXXXXXXX/tmp` +because `mktemp -d -t rear.XXXXXXXXXXXXXXX` is unpredictable +(i.e. unpredictable with sufficiently high probability). + +The alternative to create (and remove it at the end) +(e.g. via some initial 'finalize' stage script) +`/mnt/local/var/tmp/rear.XXXXXXXXXXXXXXX/tmp` +with same `/var/tmp/rear.XXXXXXXXXXXXXXX/tmp` +as in the ReaR recovery system looks a too dirty hack to me +at least according to my current gut feeling. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-12 09:58](https://github.com/rear/rear/pull/3168#issuecomment-1991229533): + +> The alternative to create (and remove it at the end) +> (e.g. via some initial 'finalize' stage script) +> `/mnt/local/var/tmp/rear.XXXXXXXXXXXXXXX/tmp` +> with same `/var/tmp/rear.XXXXXXXXXXXXXXX/tmp` +> as in the ReaR recovery system looks a too dirty hack to me +> at least according to my current gut feeling. + +I actually wanted to suggest this solution, to me it does not look that +bad, and helps (the 'remove it at the end' part) with this problem as +well: + +> What I see now and what I do not like is that +> all our chroot /mnt/local COMMANDs +> could leave arbitrary temporary stuff behind +> in the recreated system. + +But your proposed solution works as well of course. CI likes it, too. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-12 10:47](https://github.com/rear/rear/pull/3168#issuecomment-1991347031): + +@pcahyna +after sleeping on it I also think that creating +/mnt/local/var/tmp/rear.XXXXXXXXXXXXXXX/tmp +and removing it at the end is not so bad +as it looked to me yesterday. + +But for now I prefer it as it is currently done because +creating /mnt/local/var/tmp/rear.XXXXXXXXXXXXXXX/tmp +could become more tricky than expected: + +One reason is that not all workflows that can +run in the recovery system source the 'finalize' stage +so creating /mnt/local/var/tmp/rear.XXXXXXXXXXXXXXX/tmp +via some initial 'finalize' stage script +would be only partially working. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-12 10:52](https://github.com/rear/rear/pull/3168#issuecomment-1991356186): + +And the nightmare continues: + + # find usr/sbin/rear usr/share/rear/ -type f | xargs grep -l 'chroot ' + +shows in particular +usr/share/rear/build/default/490\_fix\_broken\_links.sh +and +usr/share/rear/build/default/990\_verify\_rootfs.sh +which run + + chroot $ROOTFS_DIR COMMAND + +where COMMAND fails when it uses TMPDIR +because there is no + + /var/tmp/rear.XXXXXXXXXXXXXXX/tmp + +in + + /var/tmp/rear.XXXXXXXXXXXXXXX/rootfs + +i.e. there is no + + /var/tmp/rear.XXXXXXXXXXXXXXX/rootfs/var/tmp/rear.XXXXXXXXXXXXXXX/tmp + +As an artificial test I added at the beginning of +usr/share/rear/build/default/990\_verify\_rootfs.sh + + chroot $ROOTFS_DIR /bin/bash -c 'tmpdir="$( mktemp -d )"' + +and did + + # usr/sbin/rear -D mkrescue + ... + Using build area: /var/tmp/rear.ljfRf4UOfVJktrT + ... + +and got in var/log/rear/rear-localhost.log + + + source /root/rear.github.master.TMPDIR/usr/share/rear/build/default/990_verify_rootfs.sh + ++ chroot /var/tmp/rear.ljfRf4UOfVJktrT/rootfs /bin/bash -c 'tmpdir="$( mktemp -d )"' + mktemp: failed to create directory via template '/var/tmp/rear.ljfRf4UOfVJktrT/tmp/tmp.XXXXXXXXXX': No such file or directory + +:-( + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-12 12:08](https://github.com/rear/rear/pull/3168#issuecomment-1991503836): + +With +[https://github.com/rear/rear/pull/3168/commits/9b4efb2469b8cf3585206dbb10700960b480008e](https://github.com/rear/rear/pull/3168/commits/9b4efb2469b8cf3585206dbb10700960b480008e) +I create TMPDIR inside ROOTFS\_DIR +i.e. I create $ROOTFS\_DIR$TMP\_DIR which is +/var/tmp/rear.XXX/rootfs/var/tmp/rear.XXX/tmp +so that "chroot $ROOTFS\_DIR COMMAND" +will not fail when COMMAND uses TMPDIR. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-12 12:11](https://github.com/rear/rear/pull/3168#issuecomment-1991509797): + +With latest changes and +as an artificial test added at the beginning of +usr/share/rear/build/default/990\_verify\_rootfs.sh + + chroot $ROOTFS_DIR /bin/bash -c 'tmpdir="$( mktemp -d )" ; echo tmpdata >$tmpdir/tmpfile ; find $tmpdir -ls' + +I get + + # usr/sbin/rear -D mkrescue + ... + Using build area: /var/tmp/rear.vSlVJGxFx9U5HVY + ... + Exiting rear mkrescue (PID 8329) and its descendant processes ... + +and + + # find /var/tmp/rear.vSlVJGxFx9U5HVY/rootfs/var/tmp/rear.vSlVJGxFx9U5HVY/tmp/ + /var/tmp/rear.vSlVJGxFx9U5HVY/rootfs/var/tmp/rear.vSlVJGxFx9U5HVY/tmp/ + /var/tmp/rear.vSlVJGxFx9U5HVY/rootfs/var/tmp/rear.vSlVJGxFx9U5HVY/tmp/tmp.QgqgDBsfmZ + /var/tmp/rear.vSlVJGxFx9U5HVY/rootfs/var/tmp/rear.vSlVJGxFx9U5HVY/tmp/tmp.QgqgDBsfmZ/tmpfile + +and in var/log/rear/rear-localhost.log there is + + + source /root/rear.github.master.TMPDIR/usr/share/rear/build/default/990_verify_rootfs.sh + ++ chroot /var/tmp/rear.vSlVJGxFx9U5HVY/rootfs /bin/bash -c 'tmpdir="$( mktemp -d )" ; echo tmpdata >$tmpdir/tmpfile ; find $tmpdir -ls' + 15341688 4 drwx------ 2 root root 4096 Mar 12 12:58 /var/tmp/rear.vSlVJGxFx9U5HVY/tmp/tmp.QgqgDBsfmZ + 15342002 4 -rw-r--r-- 1 root root 8 Mar 12 12:58 /var/tmp/rear.vSlVJGxFx9U5HVY/tmp/tmp.QgqgDBsfmZ/tmpfile + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-12 12:40](https://github.com/rear/rear/pull/3168#issuecomment-1991561411): + +FYI +whereto we `chroot` in the ReaR scripts + + # find usr/sbin/rear usr/share/rear/ -type f -name '*.sh' \ + | xargs grep -h 'chroot [^ ]*' \ + | grep -v '^ *#' \ + | grep -o 'chroot [^ ]*' \ + | sort -u + + chroot $ROOTFS_DIR + chroot $TARGET_FS_ROOT + chroot $TARGET_FS_ROOT' + chroot $TARGET_FS_ROOT/ + chroot $TARGET_FS_ROOT\ncd + +where `chroot $TARGET_FS_ROOT'` +and `chroot $TARGET_FS_ROOT\ncd` +are false-positives: + + You should 'chroot $TARGET_FS_ROOT' and try to fix this. + +in finalize/Debian/i386/550\_rebuild\_initramfs.sh +and + + rear_shell_history="$( echo -e "chroot $TARGET_FS_ROOT\ncd $TARGET_FS_ROOT/etc/\nvi $restored_fstab\nless $restored_fstab" )" + +in finalize/default/520\_confirm\_finalize.sh + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-12 13:56](https://github.com/rear/rear/pull/3168#issuecomment-1991713382): + +On the one hand I like such details problems +because I find it interesting to experience +the truth behind RFC 1925 items 4 and 8 and the +WYSIATI (What you see is all there is) fallacy, cf. +[https://en.wikipedia.org/wiki/Thinking,\_Fast\_and\_Slow](https://en.wikipedia.org/wiki/Thinking,_Fast_and_Slow) + +On the other had I fear such details problems +because it proves that in practice +no matter how hard you try, +you can't make a program that works +(i.e. RFC 1925 item 1 is impossible). + +There is always something unrecognized left. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-13 10:24](https://github.com/rear/rear/pull/3168#issuecomment-1994048233): + +> usr/share/rear/build/default/490\_fix\_broken\_links.sh and +> usr/share/rear/build/default/990\_verify\_rootfs.sh which run +> +> chroot $ROOTFS_DIR COMMAND +> +> where COMMAND fails when it uses TMPDIR because there is no +> +> /var/tmp/rear.XXXXXXXXXXXXXXX/tmp +> +> in +> +> /var/tmp/rear.XXXXXXXXXXXXXXX/rootfs +> +> i.e. there is no +> +> /var/tmp/rear.XXXXXXXXXXXXXXX/rootfs/var/tmp/rear.XXXXXXXXXXXXXXX/tmp + +I would expect this to be also a problem when an user sets and exports +`TMPDIR` to some value like `/a/big/and/fast/filesystem` before calling +ReaR, because `/a/big/and/fast/filesystem` will not exist either in +`rootfs`, and this problem exists *regardless of your changes in this +PR*. I suspect that to fix all this consistently one needs to add +`$TMPDIR` to `COPY_AS_IS` and `$TMPDIR/*` to `COPY_AS_IS_EXCLUDE`. One +needs to do it after `TMPDIR` is set, though. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-13 10:56](https://github.com/rear/rear/pull/3168#issuecomment-1994108376): + +With +`mkdir /run/user/111240/rear; TMPDIR=/run/user/111240/rear usr/sbin/rear -D mkrescue` +and + +> an artificial test added at the beginning of +> usr/share/rear/build/default/990\_verify\_rootfs.sh +> +> chroot $ROOTFS_DIR /bin/bash -c 'tmpdir="$( mktemp -d )" ; echo tmpdata >$tmpdir/tmpfile ; find $tmpdir -ls' + +I get this in the log: + + + source /home/pcahyna/rear/rear/usr/share/rear/build/default/990_verify_rootfs.sh + ++ LogPrint 'Testing that the recovery system in /run/user/111240/rear/rear.0K0i19A75LZJHIF/rootfs contains a usable system' + 2024-03-13 11:37:29.479931931 Testing that the recovery system in /run/user/111240/rear/rear.0K0i19A75LZJHIF/rootfs contains a + usable system + ++ chroot /run/user/111240/rear/rear.0K0i19A75LZJHIF/rootfs /bin/bash -c 'tmpdir="$( mktemp -d )" ; echo tmpdata >$tmpdir/tmpf + ile ; find $tmpdir -ls' + mktemp: failed to create directory via template '/run/user/111240/rear/tmp.XXXXXXXXXX': No such file or directory + +so indeed setting TMPDIR in the environment to a directory not included +in rootfs can be a problem even without this PR. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-13 11:54](https://github.com/rear/rear/pull/3168#issuecomment-1994207942): + +@pcahyna +with the code changes of this pull request +and the artificial test added at the beginning +of usr/share/rear/build/default/990\_verify\_rootfs.sh + + chroot $ROOTFS_DIR /bin/bash -c 'tmpdir="$( mktemp -d )" ; echo tmpdata >$tmpdir/tmpfile ; find $tmpdir -ls' + +I get + + # mkdir /var/tmp/mytmp + + # export TMPDIR=/var/tmp/mytmp + + # usr/sbin/rear -D mkrescue + ... + Using build area: /var/tmp/mytmp/rear.BMiD0hqGkOXxPOt + ... + Exiting rear mkrescue (PID 5086) and its descendant processes ... + + # less var/log/rear/rear-localhost.log + ... + + source /root/rear.github.master.TMPDIR/usr/share/rear/build/default/990_verify_rootfs.sh + ++ chroot /var/tmp/mytmp/rear.BMiD0hqGkOXxPOt/rootfs /bin/bash -c 'tmpdir="$( mktemp -d )" ; echo tmpdata >$tmpdir/tmpfile ; find $tmpdir -ls' + 14811269 4 drwx------ 2 root root 4096 Mar 13 12:48 /var/tmp/mytmp/rear.BMiD0hqGkOXxPOt/tmp/tmp.MIQ2WXDfHP + 14811270 4 -rw-r--r-- 1 root root 8 Mar 13 12:48 /var/tmp/mytmp/rear.BMiD0hqGkOXxPOt/tmp/tmp.MIQ2WXDfHP/tmpfile + +So - as far as I see - the code changes of this pull request +fix by the way that setting TMPDIR to a directory +not included in 'rootfs' ($ROOTFS\_DIR) is a problem. +Or do I misunderstand something? + +#### [schlomo](https://github.com/schlomo) commented at [2024-03-13 12:00](https://github.com/rear/rear/pull/3168#issuecomment-1994217844): + +> I suspect that to fix all this consistently one needs to add `$TMPDIR` +> to `COPY_AS_IS` and `$TMPDIR/*` to `COPY_AS_IS_EXCLUDE`. One needs to +> do it after `TMPDIR` is set, though. + +I really like this idea. It seems to me to resolve this problem more on +a root-cause level by ensuring that "whatever" is going on outside the +`chroot` will also work inside. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-13 13:23](https://github.com/rear/rear/pull/3168#issuecomment-1994397454): + +I think adding $TMPDIR to COPY\_AS\_IS +and $TMPDIR/\* to COPY\_AS\_IS\_EXCLUDE +looks needlessly complicated and obscured +(I think the intent is to add an empty $TMPDIR) +compared to `mkdir -p $ROOTFS_DIR$TMP_DIR` +that is simple and straightforward +at least as far as I see. +Or do I misunderstand something? + +I think adding an empty $TMPDIR to COPY\_AS\_IS +won't help for the `chroot /mnt/local` case and +it also won't help for whatever other directories +so "whatever" is going on outside the chroot +will not necessarily also work inside +at least as far as I see. +Or do I misunderstand something? + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-14 07:48](https://github.com/rear/rear/pull/3168#issuecomment-1996758931): + +@jsmeix I think I had not realized that your change helps also with the +preexisting problem, so it is better than I thought (but see my latest +comment). + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-14 08:25](https://github.com/rear/rear/pull/3168#issuecomment-1996838338): + +Via +[https://github.com/rear/rear/pull/3168/commits/4b772e0fe0612eca257fa4eaeff405e19f1ab6ea](https://github.com/rear/rear/pull/3168/commits/4b772e0fe0612eca257fa4eaeff405e19f1ab6ea) +I check now in prep/default/990\_verify\_empty\_rootfs.sh +only for regular files in ROOTFS\_DIR so in particular +empty directories like ROOTFS\_DIR/TMP\_DIR are OK, cf. +[https://github.com/rear/rear/commit/9b4efb2469b8cf3585206dbb10700960b480008e\#r139765325](https://github.com/rear/rear/commit/9b4efb2469b8cf3585206dbb10700960b480008e#r139765325) + +FYI +how long it takes to run `find $ROOTFS_DIR` +on my homeoffice system with a AMD Ryzen 3 4300G +and a Samsung NVMe SSD 980 + + 2024-03-14 09:14:28.331693134 Entering debugscript mode via 'set -x'. + + source /root/rear.github.master.TMPDIR/usr/share/rear/prep/default/990_verify_empty_rootfs.sh + ... + 2024-03-14 09:14:28.363963485 Source function: 'source /root/rear.github.master.TMPDIR/usr/share/rear/prep/default/990_verify_empty_rootfs.sh' returns 1 + +i.e. about a third of a second +(28.363963485 - 28.331693134 = .032270351). +I think this is acceptable. + +#### [schlomo](https://github.com/schlomo) commented at [2024-03-14 08:31](https://github.com/rear/rear/pull/3168#issuecomment-1996848881): + +Another way to solve this would be to actually create the +`ROOTFS_DIR/TMP_DIR` *after* this check, e.g. at the very beginning of +the `rescue` stage. + +TBH I'd appreciate that approach because: + +1. it reduces the clutter in the `rear` main script +2. it will allow us to catch other empty directories or special files + created during `prep` +3. One way to look at the temp dir problem is to see it as a required + step to create a "good" `ROOTFS_DIR` that also supports running + `chroot mktemp` inside. Therefore the problem belongs to `rescue` or + `build` and not to the `rear` main script. + +WDYT @jsmeix @pcahyna ? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-14 08:40](https://github.com/rear/rear/pull/3168#issuecomment-1996875429): + +@schlomo +I understand but on the other hand +I do not like to spread things that belong together +so I would prefer to do all what belongs to setting TMPDIR +at one code place. +For the same reason I don't mind if sbin/rear becomes +longer as long as things belong to sbin/rear. +So we may move out the whole ROOTFS\_DIR related code +out of sbin/rear - but not within this pull request. + +On the third hand ("tertium datur" - always in practice) +my recent +[https://github.com/rear/rear/commit/4b772e0fe0612eca257fa4eaeff405e19f1ab6ea](https://github.com/rear/rear/commit/4b772e0fe0612eca257fa4eaeff405e19f1ab6ea) +already spreads things that belong to setting TMPDIR +now over two code places :-( + +I think spreading things that belong together +over several code places is worse 'clutter' +than having longer code parts that do one thing +and do it (reasonably) well (as far as possible +with reasonable effort). + +By the way: +What is the English expression for the German +"es ist zum Mäusemelken"? + +#### [schlomo](https://github.com/schlomo) commented at [2024-03-14 09:01](https://github.com/rear/rear/pull/3168#issuecomment-1996926253): + +I'm all in favour to set `ROOTFS_DIR=/dev/null` (or something like that) +in `rear` and then create it at the beginning of `rescue`where it is +actually needed. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-14 09:06](https://github.com/rear/rear/pull/3168#issuecomment-1996947940): + +@schlomo +OK - please implement it. +If you like I could first merge this pull request +or you could implement setting TMPDIR by the way. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-14 09:11](https://github.com/rear/rear/pull/3168#issuecomment-1996968900): + +By the way regarding what there is in ROOTFS\_DIR +at the end of the 'prep' stage: +With my recent +[https://github.com/rear/rear/pull/3168/commits/05feecf0ab6db086d43d8f961e8a390c8f4a86d0](https://github.com/rear/rear/pull/3168/commits/05feecf0ab6db086d43d8f961e8a390c8f4a86d0) +I get + + # usr/sbin/rear -D mkrescue + ... + Using build area: /var/tmp/rear.xQVhflLY7FL95z9 + ... + Modified ReaR recovery system area after 'prep' stage (/var/tmp/rear.xQVhflLY7FL95z9/rootfs contains regular files) + ... + Exiting rear mkrescue (PID 5705) and its descendant processes ... + + # less var/log/rear/rear-localhost.log + ... + + source /root/rear.github.master.TMPDIR/usr/share/rear/prep/default/990_verify_empty_rootfs.sh + +++ find /var/tmp/rear.xQVhflLY7FL95z9/rootfs -type f + ++ test /var/tmp/rear.xQVhflLY7FL95z9/rootfs/etc/rear/rescue.conf + ++ DebugPrint 'Modified ReaR recovery system area after '\''prep'\'' stage (/var/tmp/rear.xQVhflLY7FL95z9/rootfs contains regular files)' + 2024-03-14 10:00:41.761946593 Modified ReaR recovery system area after 'prep' stage (/var/tmp/rear.xQVhflLY7FL95z9/rootfs contains regular files) + +++ find /var/tmp/rear.xQVhflLY7FL95z9/rootfs -ls + ++ Debug ' 14949008 4 drwxr-xr-x 4 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs + 14949016 4 drwxr-xr-x 3 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/etc + 14949017 4 drwxr-xr-x 2 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/etc/rear + 14949037 4 -rw-r--r-- 1 root root 642 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/etc/rear/rescue.conf + 14949010 4 drwxr-xr-x 4 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var + 14949011 4 drwxr-xr-x 3 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/tmp + 14949012 4 drwxr-xr-x 3 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/tmp/rear.xQVhflLY7FL95z9 + 14949013 4 drwxr-xr-x 2 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/tmp/rear.xQVhflLY7FL95z9/tmp + 14949019 4 drwxr-xr-x 3 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib + 14949020 4 drwxr-xr-x 7 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs + 14949021 4 drwxr-xr-x 2 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/v4recovery + 14949022 4 drwxr-xr-x 2 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/sm.bak + 14949023 4 drwxr-xr-x 2 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/sm + 14949024 4 drwxr-xr-x 11 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs + 14949027 4 drwxr-xr-x 2 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/nfsd + 14949032 4 drwxr-xr-x 2 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/nfs + 14949034 4 drwxr-xr-x 2 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/lockd + 14949031 4 drwxr-xr-x 2 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/portmap + 14949025 4 drwxr-xr-x 3 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/gssd + 14949026 4 drwxr-xr-x 2 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/gssd/clntXX + 14949030 4 drwxr-xr-x 2 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/statd + 14949033 4 drwxr-xr-x 2 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/mount + 14949028 4 drwxr-xr-x 2 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/cache + 14949029 4 drwxr-xr-x 2 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/rpc_pipefs/nfsd4_cb + 14949035 4 drwxr-xr-x 2 root root 4096 Mar 14 10:00 /var/tmp/rear.xQVhflLY7FL95z9/rootfs/var/lib/nfs/nfsdcltrack' + +so at least for my test case +there are only empty directories in ROOTFS\_DIR +except one regular file + + /var/tmp/rear.xQVhflLY7FL95z9/rootfs/etc/rear/rescue.conf + +which contains + + # initialize our /etc/rear/rescue.conf file sourced by the rear command in recover mode + # also the configuration is sourced by system-setup script during booting our recovery image + + SHARE_DIR="/usr/share/rear" + CONFIG_DIR="/etc/rear" + VAR_DIR="/var/lib/rear" + LOG_DIR="/var/log/rear" + + BACKUP_PROG_OPTIONS=( --anchored --xattrs --xattrs-include=security.capability --xattrs-include=security.selinux --acls ) + # The following 2 lines were added by 210_include_dhclient.sh + USE_DHCLIENT=yes + DHCLIENT_BIN=dhclient + + # The following lines were added by 490_store_write_protect_settings.sh + WRITE_PROTECTED_IDS=( ) + WRITE_PROTECTED_FS_LABEL_PATTERNS=( ) + + # All set NETFS_* variables (cf. rescue/NETFS/default/600_store_NETFS_variables.sh): + NETFS_KEEP_OLD_BACKUP_COPY= + NETFS_PREFIX=localhost + NETFS_RESTORE_CAPABILITIES=([0]="No") + + USING_UEFI_BOOTLOADER=1 + UEFI_BOOTLOADER="/boot/efi/EFI/opensuse/grubx64.efi" + +#### [schlomo](https://github.com/schlomo) commented at [2024-03-14 09:15](https://github.com/rear/rear/pull/3168#issuecomment-1996983958): + +OK, I think you should go ahead @jsmeix so that you can stop milking +mice. Please merge when you feel confident that the solution is an +improvement over the status quo and we'll take it from there. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-14 09:30](https://github.com/rear/rear/pull/3168#issuecomment-1997014841): + +@pcahyna +when you think it is OK, in particular when you think +it is an improvement over the status quo, +then please approve it. + +Afterwards I will merge it and then we wait +and see how it behaves "out there in the wild" +for (venturous) users who try out our GitHub master code. + +If we learn that it causes more trouble than it helps +I will revert it and then we know at least that +ReaR should not change an already set TMPDIR. + +I think "interesting" issues may appear in particular +with certain third-party backup tools that may need their +specific TMPDIR and not something that is set by ReaR. + +For an example where a third-party backup tool needs +a specific TMPDIR see +restore/DUPLICITY/default/200\_prompt\_user\_to\_start\_restore.sh +[https://github.com/rear/rear/blob/master/usr/share/rear/restore/DUPLICITY/default/200\_prompt\_user\_to\_start\_restore.sh\#L23](https://github.com/rear/rear/blob/master/usr/share/rear/restore/DUPLICITY/default/200_prompt_user_to_start_restore.sh#L23) +Here it happens during restore i.e. inside the recovery system +where /sbin/rear does not set `TMPDIR=$BUILD_DIR/tmp` +so this pull request is not affected by it +but it shows that in particular third-party +backup tools may need a specific TMPDIR. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-14 09:53](https://github.com/rear/rear/pull/3168#issuecomment-1997057144): + +Hmm... +Regarding "ReaR should not change an already set TMPDIR". + +I think I must implement that too for this pull request. + +So /sbin/rear would set `TMPDIR=$BUILD_DIR/tmp` +only if TMPDIR was not specified by the user via +`export TMPDIR="/path/to/my/tmpdir"` +before he called 'rear'. + +Because conf/default.conf does + + export TMPDIR="${TMPDIR-/var/tmp}" + +it sets TMPDIR to '/var/tmp' only if TMPDIR is unset +so when TMPDIR is '/var/tmp' in ReaR it means +either TMPDIR was unset +or the user had specified TMPDIR to be '/var/tmp'. + +So testing if TMPDIR is '/var/tmp' does not tell +if TMPDIR was unset before the user called 'rear'. + +Therefore I would need to add something like + + test -v TMPDIR || TMPDIR_WAS_UNSET="yes" + +directly before TMPDIR is set in default.conf +to be able to test if TMPDIR was unset +before the user called 'rear'. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-14 12:00](https://github.com/rear/rear/pull/3168#issuecomment-1997285393): + +With my recent changes in /sbin/rear via +[https://github.com/rear/rear/pull/3168/commits/7e278f3329fe52edaa9527dfde1816e74ea5cd5a](https://github.com/rear/rear/pull/3168/commits/7e278f3329fe52edaa9527dfde1816e74ea5cd5a) +"rear mkrescue" works for me for both cases +with unset TMPDIR and with specified TMPDIR +when I use the same artificial test at the beginnming of +usr/share/rear/build/default/990\_verify\_rootfs.sh + + chroot $ROOTFS_DIR /bin/bash -c 'tmpdir="$( mktemp -d )" ; echo tmpdata >$tmpdir/tmpfile ; find $tmpdir -ls' + +With unset TMPDIR after "rear -D mkrescue": + + # find /var/tmp/rear.a7njJ6TRIl5fDf4/rootfs/var/tmp/rear.a7njJ6TRIl5fDf4/tmp + + /var/tmp/rear.a7njJ6TRIl5fDf4/rootfs/var/tmp/rear.a7njJ6TRIl5fDf4/tmp + /var/tmp/rear.a7njJ6TRIl5fDf4/rootfs/var/tmp/rear.a7njJ6TRIl5fDf4/tmp/tmp.SxpqUcZrE8 + /var/tmp/rear.a7njJ6TRIl5fDf4/rootfs/var/tmp/rear.a7njJ6TRIl5fDf4/tmp/tmp.SxpqUcZrE8/tmpfile + +With "export TMPDIR=/var/tmp/mytmp" after "rear -D mkrescue": + + # find /var/tmp/mytmp/rear.2i697tfj2TI0hnF/rootfs/var/tmp/mytmp + + /var/tmp/mytmp/rear.2i697tfj2TI0hnF/rootfs/var/tmp/mytmp + /var/tmp/mytmp/rear.2i697tfj2TI0hnF/rootfs/var/tmp/mytmp/tmp.Nsttoa3prg + /var/tmp/mytmp/rear.2i697tfj2TI0hnF/rootfs/var/tmp/mytmp/tmp.Nsttoa3prg/tmpfile + +I wait now that the CI "rear recover" tests pass +and if yes I will also test myself "rear recover" +with unset TMPDIR and with specified TMPDIR. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-15 16:14](https://github.com/rear/rear/pull/3168#issuecomment-1999996162): + +Only a side note for the fun of it: + +Out of curiosity I tested how current master code +(i.e. without the changes in this pull request) +behaves when TMPDIR is relative to the current working dir +wherefrom 'rear' is called: + + # mkdir QQQtmp + + # export TMPDIR=QQQtmp + + # usr/sbin/rear -D mkrescue + Relax-and-Recover 2.7 / Git + Running rear mkrescue (PID 27033 date 2024-03-15 17:08:20) + Command line options: usr/sbin/rear -D mkrescue + Using log file: /root/rear.github.master/var/log/rear/rear-localhost.log + Using build area: QQQtmp/rear.UsuNYa5CmjjrIPN + ... + Running 'rescue' stage ====================== + Creating recovery system root filesystem skeleton layout + ERROR: Failed to copy '/root/rear.github.master/usr/share/rear/skel/default' contents to QQQtmp/rear.UsuNYa5CmjjrIPN/rootfs + Some latest log messages since the last called script 010_merge_skeletons.sh: + 2024-03-15 17:08:26.075362134 Entering debugscript mode via 'set -x'. + 2024-03-15 17:08:26.086398529 Creating recovery system root filesystem skeleton layout + 2024-03-15 17:08:26.092087642 Copying '/root/rear.github.master/usr/share/rear/skel/default' contents to QQQtmp/rear.UsuNYa5CmjjrIPN/rootfs + tar: QQQtmp/rear.UsuNYa5CmjjrIPN/rootfs: Cannot open: No such file or directory + tar: Error is not recoverable: exiting now + tar: -: Cannot write: Broken pipe + tar: Error is not recoverable: exiting now + Aborting due to an error, check /root/rear.github.master/var/log/rear/rear-localhost.log for details + + # less var/log/rear/rear-localhost.log + ... + ++ LogPrint 'Creating recovery system root filesystem skeleton layout' + 2024-03-15 17:08:26.086398529 Creating recovery system root filesystem skeleton layout + ++ pushd /root/rear.github.master/usr/share/rear/skel + ++ for skel_dir in default "$ARCH" "$OS" "$OS_MASTER_VENDOR/default" "$OS_MASTER_VENDOR_ARCH" "$OS_MASTER_VENDOR_VERSION" "$OS_VENDOR/default" "$OS_VENDOR_ARCH" "$OS_VENDOR_VERSION" "$BA + CKUP" "$OUTPUT" + ++ test default + ++ test -d default -o -s default.tar.gz + ++ test -d default + ++ Log 'Copying '\''/root/rear.github.master/usr/share/rear/skel/default'\'' contents to QQQtmp/rear.UsuNYa5CmjjrIPN/rootfs' + 2024-03-15 17:08:26.092087642 Copying '/root/rear.github.master/usr/share/rear/skel/default' contents to QQQtmp/rear.UsuNYa5CmjjrIPN/rootfs + ++ tar -C default -c . + ++ tar -C QQQtmp/rear.UsuNYa5CmjjrIPN/rootfs -x + tar: QQQtmp/rear.UsuNYa5CmjjrIPN/rootfs: Cannot open: No such file or directory + +@rear/contributors +do you perhaps know if a relative TMPDIR +is forbidden by some standard? + +Regardless if a relative TMPDIR is forbidden or not: +We should simply error out in this case +at least as long as ReaR fails with relative TMPDIR. +And because nobody reported an issue with that +it seems that ReaR works with relative TMPDIR +is not needed in practice by our users. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-18 15:51](https://github.com/rear/rear/pull/3168#issuecomment-2004294531): + +Via my recent +[https://github.com/rear/rear/pull/3168/commits/b741883d1c724f14e3cbe7fd7d295411c3aef4e9](https://github.com/rear/rear/pull/3168/commits/b741883d1c724f14e3cbe7fd7d295411c3aef4e9) +I implemented: + +Again set TMPDIR to ReaR's TMP\_DIR +in any case unless in RECOVERY\_MODE, +see +[https://github.com/rear/rear/issues/3167\#issuecomment-1999460501](https://github.com/rear/rear/issues/3167#issuecomment-1999460501) + +Set TMPDIR to its resolved absolute path +so a specified relative TMPDIR gets changed +into what works for ReaR and +verify that TMPDIR is a directory. + +Remember what TMPDIR was originally set when ReaR was launched +and provide meaningful debug info +when ReaR uses a different TMPDIR. + +Added the changes from +[https://github.com/rear/rear/pull/3181](https://github.com/rear/rear/pull/3181) +to fix +[https://github.com/rear/rear/issues/3180](https://github.com/rear/rear/issues/3180) +via this pull request (to aviod a merge conflict). + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-18 16:32](https://github.com/rear/rear/pull/3168#issuecomment-2004395630): + +So far things look good to me for "rear mkrescue": + +I use this artificial test at the beginning of +build/default/990\_verify\_rootfs.sh + + DebugPrint "TMPDIR='$TMPDIR'" + + tmpdir="$( mktemp -d -t normal_tmp.XXXX )" + echo tmpdata.normal >$tmpdir/tmpfile.normal + DebugPrint "$( find $tmpdir -ls )" + + chroot $ROOTFS_DIR /bin/bash -c 'tmpdir="$( mktemp -d -t chroot_tmp.XXXX )" ; echo tmpdata.chroot >$tmpdir/tmpfile.chroot ; find $tmpdir -ls' + DebugPrint "$( find $ROOTFS_DIR$TMPDIR/chroot_tmp* -ls )" + +Some test results: + + # export TMPDIR=" " + + # usr/sbin/rear -d mkrescue + tac: failed to create temporary file in ' ': No such file or directory + tac: failed to create temporary file in ' ': No such file or directory + ERROR: TMPDIR '' is no directory + Exiting rear mkrescue (PID 18543) and its descendant processes ... + /root/rear.github.master.TMPDIR/usr/share/rear/lib/_input-output-functions.sh: line 151: kill: (18594) - No such process + Running exit tasks + rear mkrescue failed, check /root/rear.github.master.TMPDIR/var/log/rear/rear-localhost.log for details + + # export TMPDIR="QQQ" + + # ls -l QQQ + ls: cannot access 'QQQ': No such file or directory + + # usr/sbin/rear -d mkrescue + tac: failed to create temporary file in 'QQQ': No such file or directory + tac: failed to create temporary file in 'QQQ': No such file or directory + ERROR: TMPDIR '' is no directory + Exiting rear mkrescue (PID 18641) and its descendant processes ... + /root/rear.github.master.TMPDIR/usr/share/rear/lib/_input-output-functions.sh: line 151: kill: (18692) - No such process + Running exit tasks + rear mkrescue failed, check /root/rear.github.master.TMPDIR/var/log/rear/rear-localhost.log for details + + # mkdir QQQ + + # export TMPDIR="QQQ" + + # pwd + /root/rear.github.master.TMPDIR + + # usr/sbin/rear -d mkrescue + Relax-and-Recover 2.7 / Git + Running rear mkrescue (PID 18758 date 2024-03-18 17:23:36) + Command line options: usr/sbin/rear -d mkrescue + Using log file: /root/rear.github.master.TMPDIR/var/log/rear/rear-localhost.log + Using build area: /root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s + Changing TMPDIR to '/root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s/tmp' (was 'QQQ' when ReaR was launched) + ... + TMPDIR='/root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s/tmp' + 5769278 4 drwx------ 2 root root 4096 Mar 18 17:24 /root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s/tmp/normal_tmp.BTLe + 5769358 4 -rw-r--r-- 1 root root 15 Mar 18 17:24 /root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s/tmp/normal_tmp.BTLe/tmpfile.normal + 5904418 4 drwx------ 2 root root 4096 Mar 18 17:24 /root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s/rootfs/root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s/tmp/chroot_tmp.QmIb + 5904727 4 -rw-r--r-- 1 root root 15 Mar 18 17:24 /root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s/rootfs/root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s/tmp/chroot_tmp.QmIb/tmpfile.chroot + ... + To remove the build area you may use (with caution): rm -Rf --one-file-system /root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s + + # rm -Rf --one-file-system /root/rear.github.master.TMPDIR/QQQ/rear.gU6EFIFvSEjnu6s + + # ls -l QQQ + total 0 + + # unset TMPDIR + + # usr/sbin/rear -d mkrescue + Relax-and-Recover 2.7 / Git + Running rear mkrescue (PID 9862 date 2024-03-18 17:26:22) + Command line options: usr/sbin/rear -d mkrescue + Using log file: /root/rear.github.master.TMPDIR/var/log/rear/rear-localhost.log + Using build area: /var/tmp/rear.TvfORdqueenqWjK + Setting TMPDIR to '/var/tmp/rear.TvfORdqueenqWjK/tmp' (was unset when ReaR was launched) + ... + TMPDIR='/var/tmp/rear.TvfORdqueenqWjK/tmp' + 14425255 4 drwx------ 2 root root 4096 Mar 18 17:26 /var/tmp/rear.TvfORdqueenqWjK/tmp/normal_tmp.hsc8 + 14425256 4 -rw-r--r-- 1 root root 15 Mar 18 17:26 /var/tmp/rear.TvfORdqueenqWjK/tmp/normal_tmp.hsc8/tmpfile.normal + 14549201 4 drwx------ 2 root root 4096 Mar 18 17:26 /var/tmp/rear.TvfORdqueenqWjK/rootfs/var/tmp/rear.TvfORdqueenqWjK/tmp/chroot_tmp.eB82 + 14549211 4 -rw-r--r-- 1 root root 15 Mar 18 17:26 /var/tmp/rear.TvfORdqueenqWjK/rootfs/var/tmp/rear.TvfORdqueenqWjK/tmp/chroot_tmp.eB82/tmpfile.chroot + ... + To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.TvfORdqueenqWjK + + # rm -Rf --one-file-system /var/tmp/rear.TvfORdqueenqWjK + + # ls -l /var/tmp/rear.* + ls: cannot access '/var/tmp/rear.*': No such file or directory + +Tomorrow I will test "rear recover". + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-19 12:48](https://github.com/rear/rear/pull/3168#issuecomment-2007088986): + +So far things look also good to me for "rear recover". + +Some test results: + +The standard case with unset TMPDIR: + + RESCUE localhost:~ # rear -D recover + Relax-and-Recover 2.7 / Git + Running rear recover (PID 751 date 2024-03-19 13:04:10) + Command line options: /bin/rear -D recover + Using log file: /var/log/rear/rear-localhost.log + Using build area: /var/tmp/rear.7dsqzj6rDbeRHOf + Setting TMPDIR to '/var/tmp' (was unset when ReaR was launched) + ... + Recreating initrd with /usr/bin/dracut... + Recreated initrd with /usr/bin/dracut + ... + Running exit tasks + To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.7dsqzj6rDbeRHOf + +I also did another "rear mkbackup" with + + # mkdir -p /path/to/mytmpdir + + # export TMPDIR="/path/to/mytmpdir" + + # usr/sbin/rear -D mkbackup + Relax-and-Recover 2.7 / Git + Running rear mkbackup (PID 25479 date 2024-03-19 12:43:30) + Command line options: usr/sbin/rear -D mkbackup + Using log file: /root/rear.jsmeix-TMPDIR/var/log/rear/rear-localhost.log + Using build area: /path/to/mytmpdir/rear.H9NYpL0mtHhEjXb + Changing TMPDIR to '/path/to/mytmpdir/rear.H9NYpL0mtHhEjXb/tmp' (was '/path/to/mytmpdir' when ReaR was launched) + ... + +and with that "rear recover" +worked with unset TMPDIR as above +and also with + + RESCUE localhost:~ # export TMPDIR="/path/to/mytmpdir" + + RESCUE localhost:~ # rear -D recover + Relax-and-Recover 2.7 / Git + Running rear recover (PID 749 date 2024-03-19 13:06:41) + Command line options: /bin/rear -D recover + Using log file: /var/log/rear/rear-localhost.log + Using build area: /path/to/mytmpdir/rear.jZOvb8sOBMfog8u + Using TMPDIR '/path/to/mytmpdir' (was '/path/to/mytmpdir' when ReaR was launched) + ... + Recreating initrd with /usr/bin/dracut... + Recreated initrd with /usr/bin/dracut + ... + Running exit tasks + To remove the build area you may use (with caution): rm -Rf --one-file-system /path/to/mytmpdir/rear.jZOvb8sOBMfog8u + + RESCUE localhost:~ # find /path/to/mytmpdir + /path/to/mytmpdir + /path/to/mytmpdir/rear.jZOvb8sOBMfog8u + /path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp + /path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/mappings + /path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/mappings/routes + /path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/mappings/ip_addresses + /path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/mappings/mac + /path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/by-id_change + /path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/diskbyid_mappings + /path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/restore-exclude-list.txt + /path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/touch + /path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/touch/swap-swap:-dev-sda3 + ... + /path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/touch/rear-vgchange + /path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/backuparchive_size + /path/to/mytmpdir/rear.jZOvb8sOBMfog8u/tmp/storage_drivers + /path/to/mytmpdir/rear.jZOvb8sOBMfog8u/rootfs + /path/to/mytmpdir/rear.jZOvb8sOBMfog8u/rootfs/path + /path/to/mytmpdir/rear.jZOvb8sOBMfog8u/rootfs/path/to + /path/to/mytmpdir/rear.jZOvb8sOBMfog8u/rootfs/path/to/mytmpdir + /path/to/mytmpdir/rear.H9NYpL0mtHhEjXb + /path/to/mytmpdir/rear.H9NYpL0mtHhEjXb/tmp + +This works because for the "rear mkbackup" +with `export TMPDIR="/path/to/mytmpdir"` +there is + + # find /path/to/mytmpdir/rear.H9NYpL0mtHhEjXb/rootfs/path/to/mytmpdir/rear.H9NYpL0mtHhEjXb + + /path/to/mytmpdir/rear.H9NYpL0mtHhEjXb/rootfs/path/to/mytmpdir/rear.H9NYpL0mtHhEjXb + /path/to/mytmpdir/rear.H9NYpL0mtHhEjXb/rootfs/path/to/mytmpdir/rear.H9NYpL0mtHhEjXb/tmp + +so `/path/to/mytmpdir/` exists in the ReaR recovery system +and after backup restore +even `/mnt/local/path/to/mytmpdir` exists because +`/path/to/mytmpdir/` is included in the backup + + # tar -tvf backup.tar.gz | grep mytmpdir + drwxr-xr-x root/root 0 2024-03-19 12:43 path/to/mytmpdir/ + +Only BUILD\_DIR (here /path/to/mytmpdir/rear.H9NYpL0mtHhEjXb) +is excluded from the backup in sbin/rear +so `/path/to/mytmpdir` exists in `chroot /mnt/local` + +What does not work (as expected) is + + RESCUE localhost:~ # mkdir mytmpdir + + RESCUE localhost:~ # export TMPDIR="mytmpdir" + + RESCUE localhost:~ # pwd + /root + + RESCUE localhost:~ # rear -D recover + Relax-and-Recover 2.7 / Git + Running rear recover (PID 751 date 2024-03-19 13:44:22) + Command line options: /bin/rear -D recover + Using log file: /var/log/rear/rear-localhost.log + Using build area: /root/mytmpdir/rear.8AaBPltEp78LZfU + Using TMPDIR '/root/mytmpdir' (was 'mytmpdir' when ReaR was launched) + ... + Recreating initrd with /usr/bin/dracut... + Warning: + Failed to recreate initrd with /usr/bin/dracut. + Check '/var/log/rear/rear-localhost.log' why /usr/bin/dracut failed + and decide if the recreated system will boot + with the initrd 'as is' from the backup restore. + ... + Running exit tasks + To remove the build area you may use (with caution): rm -Rf --one-file-system /root/mytmpdir/rear.8AaBPltEp78LZfU + + RESCUE localhost:~ # grep -B5 '^Failed to recreate initrd with /usr/bin/dracut' /var/log/rear/rear-localhost.log + + 2024-03-19 13:45:13.817251648 Recreating initrd with /usr/bin/dracut... + ++ chroot /mnt/local /bin/bash -c 'PATH=/sbin:/usr/sbin:/usr/bin:/bin /usr/bin/dracut --force' + realpath: /root/mytmpdir: No such file or directory + dracut: Invalid tmpdir '/root/mytmpdir'. + ++ LogPrintError 'Warning: + Failed to recreate initrd with /usr/bin/dracut. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-19 13:08](https://github.com/rear/rear/pull/3168#issuecomment-2007134820): + +Perhaps `unset TMPDIR` in `chroot /mnt/local` +is a reasonable way to ensure temporary things +work inside `chroot /mnt/local`? + +I did this change in +finalize/SUSE\_LINUX/i386/550\_rebuild\_initramfs.sh + + if chroot $TARGET_FS_ROOT /bin/bash -c "unset TMPDIR ; PATH=/sbin:/usr/sbin:/usr/bin:/bin $dracut_binary --force" ; then + +i.e. I added `unset TMPDIR ; ` +and now "rear recover" also works where it had failed before: + + RESCUE localhost:~ # mkdir mytmpdir + + RESCUE localhost:~ # export TMPDIR="mytmpdir" + + RESCUE localhost:~ # rear -D recover + Relax-and-Recover 2.7 / Git + Running rear recover (PID 753 date 2024-03-19 14:03:35) + Command line options: /bin/rear -D recover + Using log file: /var/log/rear/rear-localhost.log + Using build area: /root/mytmpdir/rear.I4VnrDeRyPJImUM + Using TMPDIR '/root/mytmpdir' (was 'mytmpdir' when ReaR was launched) + ... + Recreating initrd with /usr/bin/dracut... + Recreated initrd with /usr/bin/dracut + ... + + RESCUE localhost:~ # less /var/log/rear/rear-localhost.log + ... + 2024-03-19 14:04:34.385490756 Recreating initrd with /usr/bin/dracut... + ++ chroot /mnt/local /bin/bash -c 'unset TMPDIR ; PATH=/sbin:/usr/sbin:/usr/bin:/bin /usr/bin/dracut --force' + dracut: Executing: /usr/bin/dracut --force + ... + dracut: *** Creating initramfs image file '/boot/initrd-5.14.21-150500.55.28-default' done *** + ++ LogPrint 'Recreated initrd with /usr/bin/dracut' + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-20 08:00](https://github.com/rear/rear/pull/3168#issuecomment-2008983279): + +@pcahyna +I think it works now reasonably well (at least for me) +so I would much appreciate it if you could have a look +and approve it if it looks OK to you. +When it is merged and issues appear I will of course +try to fix them or revert the whole stuff if needed. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-20 13:25](https://github.com/rear/rear/pull/3168#issuecomment-2009561792): + +> Perhaps `unset TMPDIR` in `chroot /mnt/local` +> is a reasonable way to ensure temporary things +> work inside `chroot /mnt/local`? + +@jsmeix I hope we are not going to clutter the code with explicit +`unset TMPDIR` commands for each `chroot` command ... + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-20 13:31](https://github.com/rear/rear/pull/3168#issuecomment-2009573142): + +@pcahyna +currently I don't have a good idea how to generically +ensure `chroot /mnt/local` will work even with + + RESCUE localhost:~ # mkdir mytmpdir + RESCUE localhost:~ # export TMPDIR="mytmpdir" + +I would like to postpone that part for a subsequent +separated pull request because this pull request +was already much more complicated than I had expected. + +#### [schlomo](https://github.com/schlomo) commented at [2024-03-20 13:36](https://github.com/rear/rear/pull/3168#issuecomment-2009585478): + +@jsmeix IMHO we don't need to deal with `TMPDIR` being a relative path. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-20 14:48](https://github.com/rear/rear/pull/3168#issuecomment-2009753877): + +The changes in this pull request make ReaR working +even when TMPDIR is a relative path, see +[https://github.com/rear/rear/pull/3168\#issuecomment-2007134820](https://github.com/rear/rear/pull/3168#issuecomment-2007134820) + + RESCUE localhost:~ # mkdir mytmpdir + + RESCUE localhost:~ # export TMPDIR="mytmpdir" + + RESCUE localhost:~ # rear -D recover + ... + Using TMPDIR '/root/mytmpdir' (was 'mytmpdir' when ReaR was launched) + +so `chroot /mnt/local COMMAND` will only fail +when TMPDIR is used by COMMAND and there is +no `/root/mytmpdir` in `/mnt/local`. + +But it will work when there is `/root/mytmpdir` +in `/mnt/local` for whatever reason, e.g. by luck +or when also "rear mkbackup" was run before with +the same `export TMPDIR="mytmpdir"` + + # mkdir mytmpdir + + # export TMPDIR="mytmpdir" + + # rear.jsmeix-TMPDIR/usr/sbin/rear -D mkbackup + ... + Command line options: rear.jsmeix-TMPDIR/usr/sbin/rear -D mkbackup + Using log file: /root/rear.jsmeix-TMPDIR/var/log/rear/rear-localhost.log + Using build area: /root/mytmpdir/rear.c1FApNCh9uvUzpr + Changing TMPDIR to '/root/mytmpdir/rear.c1FApNCh9uvUzpr/tmp' (was 'mytmpdir' when ReaR was launched) + ... + Running exit tasks + To remove the build area you may use (with caution): rm -Rf --one-file-system /root/mytmpdir/rear.c1FApNCh9uvUzpr + +which includes the directory /root/mytmpdir/ in the backup + + # tar -tvf backup.tar.gz | grep mytmpdir + drwxr-xr-x root/root 0 2024-03-20 15:27 root/mytmpdir/ + +So after backup restore there is +`/root/mytmpdir` in `/mnt/local` +which makes `chroot /mnt/local COMMAND` work + + RESCUE localhost:~ # find mytmpdir + mytmpdir + mytmpdir/rear.c1FApNCh9uvUzpr + mytmpdir/rear.c1FApNCh9uvUzpr/tmp + + RESCUE localhost:~ # export TMPDIR="mytmpdir" + + RESCUE localhost:~ # rear -D recover + ... + Using build area: /root/mytmpdir/rear.PTDCYBWhedi1WjN + Using TMPDIR '/root/mytmpdir' (was 'mytmpdir' when ReaR was launched) + ... + Recreating initrd with /usr/bin/dracut... + Recreated initrd with /usr/bin/dracut + ... + Running exit tasks + To remove the build area you may use (with caution): rm -Rf --one-file-system /root/mytmpdir/rear.PTDCYBWhedi1WjN + +Cf. my similar `export TMPDIR="/path/to/mytmpdir"` example in +[https://github.com/rear/rear/pull/3168\#issuecomment-2007088986](https://github.com/rear/rear/pull/3168#issuecomment-2007088986) + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-22 13:59](https://github.com/rear/rear/pull/3168#issuecomment-2015170417): + +@jsmeix sorry for the delay, looking... + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-22 14:01](https://github.com/rear/rear/pull/3168#issuecomment-2015174686): + +@rear/contributors +I would like to merge it until Thursday (28 March) next week +(Friday 29 March and Monday 01 April are public holidays) +unless there are objections and +provided no new severe problems appear +i.e. when it is an improvement over the status quo. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-22 16:02](https://github.com/rear/rear/pull/3168#issuecomment-2015411686): + +> Another way to solve this would be to actually create the +> `ROOTFS_DIR/TMP_DIR` *after* this check, e.g. at the very beginning of +> the `rescue` stage. +> +> TBH I'd appreciate that approach because: +> +> 1. it reduces the clutter in the `rear` main script +> +> 2. it will allow us to catch other empty directories or special files created during `prep` +> +> 3. One way to look at the temp dir problem is to see it as a required step to create a "good" `ROOTFS_DIR` that also supports running `chroot mktemp` inside. Therefore the problem belongs to `rescue` or `build` and not to the `rear` main script. +> +> WDYT @jsmeix @pcahyna ? + +I like the suggestion but it certainly does not belong in this PR and +also I think that there are worse offenders that touch `ROOTFS_DIR` in +`prep` that would need to be cleaned up first. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-25 10:52](https://github.com/rear/rear/pull/3168#issuecomment-2017720837): + +@jsmeix thank you for your changes and sorry, I had yet one comment +after sleeping on it, hopefully this is the last... + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-25 12:09](https://github.com/rear/rear/pull/3168#issuecomment-2017861747): + +@pcahyna +thank you for your review again! + +Never be sorry for late comments +like comments after sleeping on an issue +(sleeping on an issue always helps - at least for me) +because in particular all deliberate comments are helpful +(regardless if they are actually right or mistaken). + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-26 06:54](https://github.com/rear/rear/pull/3168#issuecomment-2019519520): + +@rear/contributors +I will merge it today afternoon +unless objections appear. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-26 16:45](https://github.com/rear/rear/pull/3168#issuecomment-2020962513): + +Thank you for the improvement @jsmeix ! + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-27 08:14](https://github.com/rear/rear/pull/3168#issuecomment-2022176624): + +Let's hope it actually improves things, +i.e. let's hope it does not cause a severe regression +for an unforeseen use case. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-29.3169.issue.open.md b/docs/issues/2024-02-29.3169.issue.open.md new file mode 100644 index 00000000..3f28f252 --- /dev/null +++ b/docs/issues/2024-02-29.3169.issue.open.md @@ -0,0 +1,112 @@ +[\#3169 Issue](https://github.com/rear/rear/issues/3169) `open`: questions to the ReaR release cycle - best current version +=========================================================================================================================== + +**Labels**: `support / question` + +#### [Sen5ation](https://github.com/Sen5ation) opened issue at [2024-02-29 10:08](https://github.com/rear/rear/issues/3169): + +Hi there, + +We are using relax and recover in a professional environment and were +wondering which release version we can use. +The current procedure is that we download the 2.7 release from the repo, +apply some patches to it (for our use cases) and then use that version +internally. + +At the moment the 2.7 release is quite old (June 2022) and we have a lot +of bugs, especially in cloud environments. We are wondering when the 2.8 +release is planned or (if it is too far away) which (snapshot) version +you suggest to use/download here as a base for our internal rear +version? Are there ans minor releases for public download? + +Best regards +Sebastian + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-01 16:01](https://github.com/rear/rear/issues/3169#issuecomment-1973447800): + +The ReaR 2.8 release date is not yet planned, see +[https://github.com/rear/rear/milestones](https://github.com/rear/rear/milestones) + + ReaR v2.8 + No due date + +In general I recommend to try out our latest GitHub master code +because the GitHub master code is the only place where we fix things +and if there are issues it helps when you use exactly the code +where we could fix things. + +See the section +"Testing current ReaR upstream GitHub master code" in +[https://en.opensuse.org/SDB:Disaster\_Recovery](https://en.opensuse.org/SDB:Disaster_Recovery) +how you can try out our current ReaR GitHub master code +without conflicts with your already installed ReaR version. + +In general we at ReaR upstream do not support older ReaR versions. +We at ReaR upstream do not plain reject issues with older ReaR +versions +(e.g. we may answer easy to solve questions also for older ReaR +versions) +but we do not spend much time on issues with older ReaR versions +because +we do not (and cannot) fix issues in released ReaR versions. +Issues in released ReaR versions are not fixed by us (by ReaR +upstream). +Issues in released ReaR versions that got fixed in current ReaR +upstream +GitHub master code might be fixed (if the fix can be backported with +reasonable effort) by the Linux distributor wherefrom you got ReaR. + +When you +"have a lot of bugs, especially in cloud environments", +but you do not report them to us here at ReaR upstream, +usually nothing can get fixed. + +Regarding "especially in cloud environments": +Usually we at ReaR upstream do not have "cloud environments" +in particular not when one has to pay for them, so usually +nothing will get fixed for special cloud environments. + +I know from various issues here at ReaR upstream that +especially cloud environments are especially problematic +because especially cloud environments are especially +strange how things behave within cloud environments. +In particular strange storage that does not behave same +as usual storage with usual physical harddisks or SSDs +or usual virtual disks of usual QEMU/KVM virtual machines. +Also strange bootloader related things that do not behave +same as usual bootloader things on physical hardware +or on usual QEMU/KVM virtual machines. + +In general see the section +"How to contribute to Relax-and-Recover" in +[https://en.opensuse.org/SDB:Disaster\_Recovery](https://en.opensuse.org/SDB:Disaster_Recovery) + +If you or your company requires a missing feature, +see the sections "Sponsoring" +and "Professional services" in +[https://relax-and-recover.org/development/](https://relax-and-recover.org/development/) + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-11 17:34](https://github.com/rear/rear/issues/3169#issuecomment-2050178965): + +Hello @Sen5ation most people either find the older versions, e.g. 2.6 or +2.7, sufficient for their use case or they import a current build +package from here. + +Helping with the implementation of ReaR in a professional context is a +big part of the consultancy engagements that we see via [ReaR +Professional +Support](https://relax-and-recover.org/support/#:~:text=means%20for%20sponsoring-,Professional%20Support,-If%20your%20company) +and I kindly ask you to reach out to us for more details on that. + +Most new features are the result of either professional users +contributing features they find useful or engaging with us to develop +the missing features. + +I agree with you, BTW, that the lack of regular releases is bad. +Unfortunately making a release is a lot of work and so far [nobody is +willing to pay for a new release](https://relax-and-recover.org/events/) + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-29.3170.issue.closed.md b/docs/issues/2024-02-29.3170.issue.closed.md new file mode 100644 index 00000000..35598a1f --- /dev/null +++ b/docs/issues/2024-02-29.3170.issue.closed.md @@ -0,0 +1,596 @@ +[\#3170 Issue](https://github.com/rear/rear/issues/3170) `closed`: Alma Linux 9.3 not able to install bootloader. +================================================================================================================= + +**Labels**: `support / question`, `fixed / solved / done` + +#### [rvdkraan](https://github.com/rvdkraan) opened issue at [2024-02-29 14:23](https://github.com/rear/rear/issues/3170): + +When recovering using a usb key a warning is shown that the bootlaoder +could not be installed. +no code for the os used to auto recover. Advising to manually install +the bootloader. + +The OS was successfully recovered however without a bootloader the +system will not start. + + Begin dumping out configuration and system information: + This is a 'Linux-x86_64' system, compatible with 'Linux-i386'. + Configuration tree: + Linux-i386.conf : OK + GNU/Linux.conf : OK + Fedora.conf : missing/empty + Fedora/i386.conf : missing/empty + Fedora/9.3.conf : missing/empty + Fedora/9.3/i386.conf : missing/empty + RedHatEnterpriseServer.conf : missing/empty + RedHatEnterpriseServer/i386.conf : missing/empty + RedHatEnterpriseServer/9.3.conf : missing/empty + RedHatEnterpriseServer/9.3/i386.conf : missing/empty + site.conf : OK + local.conf : OK + System definition: + ARCH="Linux-i386" + OS="GNU/Linux" + OS_MASTER_VENDOR="Fedora" + OS_MASTER_VERSION="9.3" + OS_MASTER_VENDOR_ARCH="Fedora/i386" + OS_MASTER_VENDOR_VERSION="Fedora/9.3" + OS_MASTER_VENDOR_VERSION_ARCH="Fedora/9.3/i386" + OS_VENDOR="RedHatEnterpriseServer" + OS_VERSION="9.3" + OS_VENDOR_ARCH="RedHatEnterpriseServer/i386" + OS_VENDOR_VERSION="RedHatEnterpriseServer/9.3" + OS_VENDOR_VERSION_ARCH="RedHatEnterpriseServer/9.3/i386" + # Backup with NETFS: + NETFS_KEEP_OLD_BACKUP_COPY="" + NETFS_PREFIX="hms-alma" + NETFS_RESTORE_CAPABILITIES=("No") + BACKUP_DUPLICITY_NAME="rear-backup" + BACKUP_INTEGRITY_CHECK="" + BACKUP_MOUNTCMD="" + BACKUP_ONLY_EXCLUDE="no" + BACKUP_ONLY_INCLUDE="no" + BACKUP_OPTIONS="" + BACKUP_RESTORE_MOVE_AWAY_DIRECTORY="/var/lib/rear/moved_away_after_backup_restore/" + BACKUP_RESTORE_MOVE_AWAY_FILES=("/boot/grub/grubenv" "/boot/grub2/grubenv") + BACKUP_RSYNC_OPTIONS=("--sparse" "--archive" "--hard-links" "--numeric-ids" "--stats") + BACKUP_SELINUX_DISABLE="1" + BACKUP_TYPE="" + BACKUP_UMOUNTCMD="" + BACKUP_URL="usb:///dev/disk/by-label/REAR-000" + # Backup program is 'tar': + BACKUP_PROG_ARCHIVE="backup" + BACKUP_PROG_COMPRESS_OPTIONS=("--gzip") + BACKUP_PROG_COMPRESS_SUFFIX=".gz" + BACKUP_PROG_CRYPT_ENABLED="false" + BACKUP_PROG_CRYPT_KEY="" + BACKUP_PROG_CRYPT_OPTIONS="/usr/bin/openssl des3 -salt -k " + BACKUP_PROG_DECRYPT_OPTIONS="/usr/bin/openssl des3 -d -k " + BACKUP_PROG_EXCLUDE=("/tmp/*" "/dev/shm/*" "/var/lib/rear/output/*" "/var/tmp/rear.p8MJE6hNidGJS4u") + BACKUP_PROG_OPTIONS=("--anchored") + BACKUP_PROG_OPTIONS_CREATE_ARCHIVE="" + BACKUP_PROG_OPTIONS_RESTORE_ARCHIVE="" + BACKUP_PROG_SUFFIX=".tar" + BACKUP_PROG_WARN_PARTIAL_TRANSFER="1" + # Output to USB: + USB_BIOS_BOOT_DEFAULT="" + USB_BOOTLOADER="" + USB_BOOT_PART_SIZE="0" + USB_DEVICE="" + USB_DEVICE_BOOT_LABEL="REARBOOT" + USB_DEVICE_FILESYSTEM="ext3" + USB_DEVICE_FILESYSTEM_LABEL="REAR-000" + USB_DEVICE_FILESYSTEM_PARAMS="" + USB_DEVICE_FILESYSTEM_PERCENTAGE="100" + USB_DEVICE_PARTED_LABEL="" + USB_PARTITION_ALIGN_BLOCK_SIZE="8" + USB_RETAIN_BACKUP_NR="2" + USB_SUFFIX="" + USB_UEFI_PART_SIZE="1024" + OUTPUT_EFISTUB_SYSTEMD_BOOTLOADER="/usr/lib/systemd/boot/efi/systemd-bootx64.efi" + OUTPUT_LFTP_PASSWORD="" + OUTPUT_LFTP_USERNAME="" + OUTPUT_MOUNTCMD="" + OUTPUT_OPTIONS="" + OUTPUT_PREFIX="hms-alma" + OUTPUT_PREFIX_PXE="" + OUTPUT_UMOUNTCMD="" + OUTPUT_URL="" + +- ReaR version ("/usr/sbin/rear -V"): + Relax-and-Recover 2.7 / Git + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): + + + + NAME="AlmaLinux" + VERSION="9.3 (Shamrock Pampas Cat)" + ID="almalinux" + ID_LIKE="rhel centos fedora" + VERSION_ID="9.3" + PLATFORM_ID="platform:el9" + PRETTY_NAME="AlmaLinux 9.3 (Shamrock Pampas Cat)" + ANSI_COLOR="0;34" + LOGO="fedora-logo-icon" + CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos" + HOME_URL="https://almalinux.org/" + DOCUMENTATION_URL="https://wiki.almalinux.org/" + BUG_REPORT_URL="https://bugs.almalinux.org/" + + ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9" + ALMALINUX_MANTISBT_PROJECT_VERSION="9.3" + REDHAT_SUPPORT_PRODUCT="AlmaLinux" + REDHAT_SUPPORT_PRODUCT_VERSION="9.3" + +- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat + /etc/rear/local.conf"): + + + + OUTPUT=USB + BACKUP=NETFS + + BACKUP_URL="usb:///dev/disk/by-label/REAR-000" + + USE_DHCLIENT=no + USE_STATIC_NETWORKING=y + +- Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM + guest or PowerVM LPAR): + Axiomsys + +- System architecture (x86 compatible or PPC64/PPC64LE or what exact + ARM device): + X86 64bit + +- Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or + ELILO or Petitboot): + + + + BIOS Information + Vendor: American Megatrends Inc. + Version: 5.12 + Release Date: 04/20/2018 + Address: 0xF0000 + Runtime Size: 64 kB + ROM Size: 16 MB + Characteristics: + PCI is supported + BIOS is upgradeable + BIOS shadowing is allowed + Boot from CD is supported + Selectable boot is supported + BIOS ROM is socketed + EDD is supported + 5.25"/1.2 MB floppy services are supported (int 13h) + 3.5"/720 kB floppy services are supported (int 13h) + 3.5"/2.88 MB floppy services are supported (int 13h) + Print screen service is supported (int 5h) + Serial services are supported (int 14h) + Printer services are supported (int 17h) + ACPI is supported + USB legacy is supported + BIOS boot specification is supported + Targeted content distribution is supported + UEFI is supported + BIOS Revision: 5.12 + +- Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or + multipath (DM or NVMe): + Dual SSD + +- Storage layout ("lsblk -ipo + NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"): + + + + NAME KNAME PKNAME TRAN TYPE FSTYPE LABEL SIZE MOUNTPOINT + /dev/sda /dev/sda sata disk 119.2G + |-/dev/sda1 /dev/sda1 /dev/sda part vfat 600M /boot/efi + |-/dev/sda2 /dev/sda2 /dev/sda part xfs 1G /boot + `-/dev/sda3 /dev/sda3 /dev/sda part LVM2_member 117.7G + |-/dev/mapper/almalinux_hms-root /dev/dm-0 /dev/sda3 lvm xfs 70G / + |-/dev/mapper/almalinux_hms-swap /dev/dm-1 /dev/sda3 lvm swap 3.9G [SWAP] + `-/dev/mapper/almalinux_hms-home /dev/dm-2 /dev/sda3 lvm xfs 43.8G /home + /dev/sdb /dev/sdb sata disk 111.8G + `-/dev/sdb1 /dev/sdb1 /dev/sdb part xfs 111.8G /database + +#### [gdha](https://github.com/gdha) commented at [2024-03-01 14:28](https://github.com/rear/rear/issues/3170#issuecomment-1973301491): + +@rvdkraan A debug log of a 'mkrescue' could be helpful. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-01 14:35](https://github.com/rear/rear/issues/3170#issuecomment-1973312486): + +@rvdkraan +plus a debug log of "rear -D recover", see in +[https://en.opensuse.org/SDB:Disaster\_Recovery](https://en.opensuse.org/SDB:Disaster_Recovery) +in the section +"Debugging issues with Relax-and-Recover" +both parts +"To analyze and debug a 'rear mkrescue/mkbackup' failure +the following information is mandatory" +and +"To analyze and debug a "rear recover" failure the +following additional information is also mandatory" + +In general: +Caution with possible secrets in a full debug log file: +When 'rear' is run via '-D' in debugscript mode +it logs executed commands via the bash command 'set -x' +that print commands and their arguments as they are executed +so in particular when arguments contain secret values +(e.g. something like a password or whatever else) +such secret values may appear in the log file. +Also secrets may be stored in some other files +like /var/lib/rear/layout/disklayout.conf +or /var/lib/rear/layout/diskrestore.sh +cf. `[password=]` in the section +"Disk layout file syntax" in +doc/user-guide/06-layout-configuration.adoc +online at +[https://github.com/rear/rear/blob/rear-2.7/doc/user-guide/06-layout-configuration.adoc](https://github.com/rear/rear/blob/rear-2.7/doc/user-guide/06-layout-configuration.adoc) +So before you attach your full debug log file and other files +here (GitHub is a public accessible place) inspect your files +and verify that they do not accidentally contain secrets. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-01 15:01](https://github.com/rear/rear/issues/3170#issuecomment-1973350469): + +> When recovering using a usb key a warning is shown that the bootlaoder +> could not be installed. +> no code for the os used to auto recover. Advising to manually install +> the bootloader. + +It would be also helpful to provide the actual warning message and +information about the firmware used to boot (BIOS or UEFI? If UEFI, is +Secure Boot used?) + +#### [rvdkraan](https://github.com/rvdkraan) commented at [2024-03-05 10:30](https://github.com/rear/rear/issues/3170#issuecomment-1978443990): + +Hi @gdha @jsmeix @pcahyna + +Thank you all for the feedback. +It really helps to have an active community to help each other. + +I have attached the log files. +My disk is not encrypted. And did not find any secrets. If you do we +need to delete this post. + +[rear-hms-alma +recover.log](https://github.com/rear/rear/files/14493832/rear-hms-alma.recover.log) +[rear-log +mkrescue.log](https://github.com/rear/rear/files/14493833/rear-log.mkrescue.log) +![EFI +boot](https://github.com/rear/rear/assets/46997988/16d71c50-3166-4bdb-82d3-650423e318a3) + +U use the BIOS boot mode. EFI gives an warning. +It is also a bit confusing that mkrescue finishes ok but the log file +shows a few errors. + +warning in console about the bootloader: + +WARNING: +For this system +RedHatEnterpriseServer/9.3 on Linux-i386 (based on Fedora/9.3/i386) +there is no code to install a boot loader on the recovered system +or the code that we have failed to install the boot loader correctly. +Please contribute appropriate code to the Relax-and-Recover project, +see +[http://relax-and-recover.org/development/](http://relax-and-recover.org/development/) +Take a look at the scripts in /usr/share/rear/finalize - for example +for PC architectures like x86 and x86\_64 see the script +/usr/share/rear/finalize/Linux-i386/660\_install\_grub2.sh +and for POWER architectures like ppc64le see the script +/usr/share/rear/finalize/Linux-ppc64le/660\_install\_grub2.sh +| IF YOU DO NOT INSTALL A BOOT LOADER MANUALLY, | +| THEN YOUR SYSTEM WILL NOT BE ABLE TO BOOT. | +You can use 'chroot /mnt/local bash --login' +to change into the recovered system and +manually install a boot loader therein. + +TIA +Ruben + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-05 13:48](https://github.com/rear/rear/issues/3170#issuecomment-1978815190): + +@rvdkraan + +I am not a bootloader expert and +I am even less an UEFI expert +but I try as good as I can: + +Fist some general info: + +The primary user config variables +that determine which kind of bootloader +will be (re)installed during "rear recover" are +BOOTLOADER +USING\_UEFI\_BOOTLOADER +UEFI\_BOOTLOADER +see their descriptions in usr/share/rear/conf/default.conf + +When there are issues with installing the bootloader +of the recreated system during "rear recover" +it should help to specify the right values as needed +for those config variables. + +What does not work automatically or cannot work is +when you boot via UEFI on the original system +where "rear mkrescue/mkbackup" was run +BUT +you want to boot via BIOS on the replacement system +where "rear recover" is run. + +In general ReaR is meant to recreate a system +as it was before. +In particular ReaR is meant to recreate a system +on fully compatible replacement hardware +(also fully compatible replacement virtual hardware), +see the section +"Fully compatible replacement hardware is needed" in +[https://en.opensuse.org/SDB:Disaster\_Recovery](https://en.opensuse.org/SDB:Disaster_Recovery) + +It might work (but I did not test it) +when you had UEFI on the original system +but now use BIOS on the replacement system +(provided you can boot the ReaR recovery system +on the replacement system with BIOS) +that you manually adapt user config variables +in /etc/rear/local.conf and /etc/rear/rescue.conf +and the var/lib/rear/recovery/bootloader value +inside the booted ReaR recovery system +before you run "rear recover". + +Excerpts where I had a look at your log files: + +Ecxerpts from your +[https://github.com/rear/rear/files/14493833/rear-log.mkrescue.log](https://github.com/rear/rear/files/14493833/rear-log.mkrescue.log) + + + source /usr/share/rear/prep/default/320_include_uefi_env.sh + ... + ++ DebugPrint 'Found EFI system partition /dev/sda1 on /boot/efi type vfat' + ++ USING_UEFI_BOOTLOADER=1 + + + source /usr/share/rear/layout/save/default/445_guess_bootloader.sh + ... + ++ Log 'No known bootloader matches the first bytes on /dev/sda' + ... + ++ Log 'No known bootloader matches the first bytes on /dev/sdb' + ... + ++ Log 'No known bootloader matches the first bytes on /dev/sdc' + ... + ++ Log 'GRUB 2 is installed (grub-probe or grub2-probe exist).' + ... + ++ echo GRUB2-EFI + +so you have `GRUB2-EFI` in var/lib/rear/recovery/bootloader +which looks right - as far as I see at first glance. + +Then excerpts from your +[https://github.com/rear/rear/files/14493832/rear-hms-alma.recover.log](https://github.com/rear/rear/files/14493832/rear-hms-alma.recover.log) + + + source /etc/rear/rescue.conf + ... + ++ USING_UEFI_BOOTLOADER=1 + ++ UEFI_BOOTLOADER=/boot/efi/EFI/almalinux/grubx64.efi + + + source /usr/share/rear/finalize/Linux-i386/660_install_grub2.sh + ... + ++ '[' GRUB2 '!=' GRUB2-EFI ']' + ++ return 0 + +so finalize/Linux-i386/660\_install\_grub2.sh does nothing +and the subsequent script + + + source /usr/share/rear/finalize/Linux-i386/670_run_efibootmgr.sh + ... + ++ Log efibootmgr --create --gpt --disk /dev/sda --part 1 --write-signature --label '"RedHatEnterpriseServer' '9.3"' --loader '"\EFI\almalinux\grubx64.efi"' + 2024-03-05 09:28:27.331724852 efibootmgr --create --gpt --disk /dev/sda --part 1 --write-signature --label "RedHatEnterpriseServer 9.3" --loader "\EFI\almalinux\grubx64.efi" + ++ efibootmgr --create --gpt --disk /dev/sda --part 1 --write-signature --label 'RedHatEnterpriseServer 9.3' --loader '\EFI\almalinux\grubx64.efi' + EFI variables are not supported on this system. + ++ LogPrintError 'efibootmgr failed to create EFI Boot Manager entry on /dev/sda partition 1 (ESP /dev/sda1 )' + +failed because "EFI variables are not supported on this system" +(I think this is because you use BIOS on the replacement system) +so in the end no bootloader (neither for BIOS nor for UEFI) +gets installed. + +Because I am not a bootloader expert +I may misunderstand things so my above analysis +how things fail in this specific case could be just wrong. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-05 14:23](https://github.com/rear/rear/issues/3170#issuecomment-1978888236): + +@rvdkraan + +> U use the BIOS boot mode. EFI gives an warning. + +I suppose you mean "I use the BIOS boot mode. EFI gives an warning." + +If so, and you are booting in BIOS mode, this is the reason for the +bootloader warning that you see. You need to recover to a matching +system, as explained by @jsmeix (i.e. backup was created on EFI - +recover to a system with EFI). Even then, with BIOS, I think that the +recovered system will work OK, if you select the restored disk in the +BIOS boot menu as the boot device, and you can thus ignore the message. + +The EFI problem in the screenshot is a known bug of GRUB, see +[https://github.com/rear/rear/issues/2971](https://github.com/rear/rear/issues/2971) +. It will be solved by a GRUB update. In the meantime, as a workaround, +set `SECURE_BOOT_BOOTLOADER=/boot/efi/EFI/almalinux/shimx64.efi` in +`/etc/rear/local.conf` and recreate the rescue USB. This should make +your USB bootable in EFI mode and the bootloader warning that you see at +the end should not appear. + +#### [rvdkraan](https://github.com/rvdkraan) commented at [2024-03-05 14:28](https://github.com/rear/rear/issues/3170#issuecomment-1978900635): + +I try to edit the config. +The hardware should be almost identical, the disk in the system I am +restoring to is bigger. +Both systems support BIOS and UEFI but the UEFI boot of rear does not +work. +I saw that this should be fixed in a new release. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-05 14:32](https://github.com/rear/rear/issues/3170#issuecomment-1978908260): + +@rvdkraan the point is, by setting +`SECURE_BOOT_BOOTLOADER=/boot/efi/EFI/almalinux/shimx64.efi` UEFI should +work already in this release + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-06 16:09](https://github.com/rear/rear/issues/3170#issuecomment-1981222992): + +Only FYI as a side note: + +Out of curiosity I was wondering how far it could work +to call existing ReaR scripts to install a bootloader +from within the ReaR recovery system. + +I use a QEMU/KVM system with BIOS and a single /dev/sda. + +In the ReaR recovery system +before running "rear recover" +I did + + RESCUE localhost:~ # export GRUB2_INSTALL_DEVICES="/dev/sdQQQ" + +to make finalize/Linux-i386/660\_install\_grub2.sh +intentionally fail +so I got + + RESCUE localhost:~ # rear -D recover + ... + Installing GRUB2 boot loader... + Installing GRUB2 on /dev/sdQQQ (specified in GRUB2_INSTALL_DEVICES) + Failed to install GRUB2 on /dev/sdQQQ + Failed to install GRUB2 on /dev/sdQQQ + WARNING: + For this system + SUSE_LINUX/15.5 on Linux-i386 (based on SUSE/15/i386) + there is no code to install a boot loader on the recovered system + or the code that we have failed to install the boot loader correctly. + Please contribute appropriate code to the Relax-and-Recover project, + see http://relax-and-recover.org/development/ + Take a look at the scripts in /usr/share/rear/finalize - for example + for PC architectures like x86 and x86_64 see the script + /usr/share/rear/finalize/Linux-i386/660_install_grub2.sh + and for POWER architectures like ppc64le see the script + /usr/share/rear/finalize/Linux-ppc64le/660_install_grub2.sh + --------------------------------------------------- + | IF YOU DO NOT INSTALL A BOOT LOADER MANUALLY, | + | THEN YOUR SYSTEM WILL NOT BE ABLE TO BOOT. | + --------------------------------------------------- + You can use 'chroot /mnt/local bash --login' + to change into the recovered system and + manually install a boot loader therein. + ... + +Then I did + + RESCUE localhost:~ # export GRUB2_INSTALL_DEVICES="/dev/sda" + + RESCUE localhost:~ # rear -d shell + ... + REAR localhost:/usr/share/rear # export NOBOOTLOADER="yes" + + REAR localhost:/usr/share/rear # Source finalize/Linux-i386/660_install_grub2.sh + ... + Installing GRUB2 boot loader... + Generating grub configuration file ... + Found theme: /boot/grub2/themes/SLE/theme.txt + Found linux image: /boot/vmlinuz-5.14.21-150500.55.28-default + Found initrd image: /boot/initrd-5.14.21-150500.55.28-default + Warning: os-prober will not be executed to detect other bootable partitions. + Systems on them will not be added to the GRUB boot configuration. + Check GRUB_DISABLE_OS_PROBER documentation entry. + done + Installing GRUB2 on /dev/sda (specified in GRUB2_INSTALL_DEVICES) + Installing for i386-pc platform. + Installation finished. No error reported. + + REAR localhost:/usr/share/rear # exit + + RESCUE localhost:~ # reboot + +The recreated system boots normally for me. + +To make `rear shell` working +within the ReaR recovery system one must adapt +usr/share/rear/init/default/050\_check\_rear\_recover\_mode.sh +as follows + + --- a/usr/share/rear/init/default/050_check_rear_recover_mode.sh + +++ b/usr/share/rear/init/default/050_check_rear_recover_mode.sh + @@ -15,7 +15,7 @@ + if test -f /etc/rear-release ; then + # We are in the ReaR rescue/recovery system: + case "$WORKFLOW" in + - (recover|layoutonly|restoreonly|finalizeonly|mountonly|opaladmin|help) + + (recover|layoutonly|restoreonly|finalizeonly|mountonly|opaladmin|shell|help) + LogPrint "Running workflow $WORKFLOW within the ReaR rescue/recovery system" + ;; + (*) + +i.e. add `shell` to the workflows that are allowed +to be run from within the ReaR recovery system. + +#### [rvdkraan](https://github.com/rvdkraan) commented at [2024-03-07 08:20](https://github.com/rear/rear/issues/3170#issuecomment-1982891769): + +@pcahyna +The workaround worked and I was able to recover. Thanks. +@jsmeix +I do not fully grasp the shell you mentioned. +However it nice to know that a failed recovery for the bootloader can be +fixt manually. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-08 15:47](https://github.com/rear/rear/issues/3170#issuecomment-1985929365): + +@jsmeix `rear shell` looks nice, TIL! But in this case we don't even +need it, if the user boots from the recovered disk, the EFI shim's +fallback mechanism will fix up the boot entry automatically. +[https://github.com/rhboot/shim/blob/main/README.fallback](https://github.com/rhboot/shim/blob/main/README.fallback) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-08 16:21](https://github.com/rear/rear/issues/3170#issuecomment-1985991370): + +Yes, this issue only triggered me to try out +calling ReaR scripts from within the ReaR recovery system. +I used BIOS because @rvdkraan tried to recover on BIOS. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-08 16:26](https://github.com/rear/rear/issues/3170#issuecomment-1986001582): + +Fore me the interesting thing to note is that restoring a UEFI system in +BIOS mode "just works" and the warning can thus be considered excessive. +I suppose the opposite is not true (the bootloader would not get +installed and the system would not boot). + +#### [rvdkraan](https://github.com/rvdkraan) commented at [2024-03-12 12:39](https://github.com/rear/rear/issues/3170#issuecomment-1991558962): + +Hi @pcahyna + +Rebooting gave me an screen showing only a white underscore like to have +on a prompt but than without blinking. +However maybe there was an other reason the recovered system did not +boot. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-12 14:31](https://github.com/rear/rear/issues/3170#issuecomment-1991789532): + +> Hi @pcahyna +> +> Rebooting gave me an screen showing only a white underscore like to +> have on a prompt but than without blinking. However maybe there was an +> other reason the recovered system did not boot. + +hi @rvdkraan , when did that happen? Was it before applying the +`SECURE_BOOT_BOOTLOADER=` workaround, i.e. after the recovery has +printed the warning + + | IF YOU DO NOT INSTALL A BOOT LOADER MANUALLY, | + | THEN YOUR SYSTEM WILL NOT BE ABLE TO BOOT. | + +? + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-02-29.3171.pr.open.md b/docs/issues/2024-02-29.3171.pr.open.md new file mode 100644 index 00000000..626bf446 --- /dev/null +++ b/docs/issues/2024-02-29.3171.pr.open.md @@ -0,0 +1,915 @@ +[\#3171 PR](https://github.com/rear/rear/pull/3171) `open`: Make OS detection from `/etc/os-release` more robust +================================================================================================================ + +**Labels**: `enhancement`, `bug`, `cleanup` + +#### [lzaoral](https://github.com/lzaoral) opened issue at [2024-02-29 16:28](https://github.com/rear/rear/pull/3171): + +#### Relax-and-Recover (ReaR) Pull Request Template + +Please fill in the following items before submitting a new pull request: + +##### Pull Request Details: + +- Type: **Bug Fix** + +- Impact: **High** + +- Reference to related issue (URL): + [https://github.com/rear/rear/issues/3149](https://github.com/rear/rear/issues/3149) + +- How was this pull request tested? `rear dump` and then rescue on + x86\_64 Fedora Rawhide and RHEL 9.4 + +- Description of the changes in this pull request - Notable changes: + +1. Before this change, ReaR would just grep `/etc/os-release` for + suitable string. This is wrong because, e.g. Fedora was classified + as RHEL because its `/etc/os-release` contains the 'redhat' string + in URL for its issue tracker. + + `os-release(5)` \[1\] specifies `ID` and `ID_LIKE` fields for + reliable identification of the distribution. First, use the + `ID_LIKE` field to deal with derivative Linux distributions (e.g. + CentOS and RHEL, or Ubuntu and Linux Mint). Then use `ID` to detect + distributions that are either not a derivative (e.g. Arch or Fedora) + or ReaR already recognized the derivate separately (e.g. Fedora vs. + RHEL or Debian vs. Ubuntu). + +2. Set `OS_MASTER_VENDOR` only for derived distributions recognized by + ReaR. + + This change ensures that `OS_VENDOR` and `OS_MASTER_VENDOR` will not + be set to equal values which subsequently caused some ReaR stages to + be sourced more than once. + +3. Use `OS_MASTER_VERSION` for major releases. + + Contrary to its name, the `OS_MASTER_VERSION` variable was already + used for this purpose for some versions, e.g. RHEL 7. This fixes + version comparison on RHEL 8 and newer. + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-03-01 09:28](https://github.com/rear/rear/pull/3171#issuecomment-1972828204): + +Thank you for the analysis, @jsmeix! I've pushed changes that rename +`Arch` to `Arch_Linux` and completely remove `RedHatEnterpriseServer`. I +still have to test these changes on an actual RHEL machine so I'll let +you know ASAP if the recovery is still ok. + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-03-01 10:04](https://github.com/rear/rear/pull/3171#issuecomment-1972887597): + +Good news is that the changes recovered the machine successfully. +However, I've just remembered that ReaR uses `OS_VENDOR` for the name of +the recreated EFI boot entry so we will not get `Fedora` for all +Fedora-like distributions. + +@pcahyna What do you think? + +#### [rvdkraan](https://github.com/rvdkraan) commented at [2024-03-01 10:35](https://github.com/rear/rear/pull/3171#issuecomment-1972941178): + +Hi +Clould this OS detection fix also solve the root problem for \#3170 ? +Which is in my opinion also related to OS detection. + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-03-01 11:48](https://github.com/rear/rear/pull/3171#issuecomment-1973046586): + +Hey @rvdkraan! Unfortunately, this PR will not fix your issues because +both the `OS_VENDOR` and `OS_MASTER_VENDOR` variables from your report +were already correct. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-06 13:02](https://github.com/rear/rear/pull/3171#issuecomment-1980825137): + +@pcahyna +could you please have a look here (as time permits), cf. +[https://github.com/rear/rear/pull/3171\#issuecomment-1972887597](https://github.com/rear/rear/pull/3171#issuecomment-1972887597) + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-07 11:18](https://github.com/rear/rear/pull/3171#issuecomment-1983295860): + +@jsmeix sorry for the delay, I will have a look. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-20 12:38](https://github.com/rear/rear/pull/3171#issuecomment-2009471753): + +Some further thoughts regarding the +get\_var\_from\_file() function: + +Because at least at first glance it looks rather confusing +what $0 $1 $2 actually is when looking at the +get\_var\_from\_file() function code, better tell in a comment +how get\_var\_from\_file() needs to be called, +e.g. like + + # get_var_from_file file_name variable_name + +I think it would be good to BugError() +when $file\_name is no readable file +and/or when $variable\_name is empty. + +Perhaps it is somehow possible to let +get\_var\_from\_file() provide a useful exit code. +Currently it always results exit code 0 because +`echo something` is (basically) always successful: + + # function get_var_from_file() { bash -c 'source "$0"; echo "${!1}"' "$1" "$2"; } + + # get_var_from_file /etc/os-release ID && echo OK || echo FAIL + opensuse-leap + OK + + # get_var_from_file qqq ID && echo OK || echo FAIL + qqq: qqq: No such file or directory + + OK + + # get_var_from_file /etc/os-release QQQ && echo OK || echo FAIL + + OK + + # cat my-invalid-assignments + if then + VAR="value" + + # get_var_from_file my-invalid-assignments VAR && echo OK || echo FAIL + my-invalid-assignments: line 1: syntax error near unexpected token `then' + my-invalid-assignments: line 1: `if then' + + OK + +Therefore I suggest a better implementation +of the actual get\_var\_from\_file() code body: + + # function get_var_from_file() { bash -c 'source "$0" && echo "${!1}" || false' "$1" "$2" ; } + + # get_var_from_file my-invalid-assignments VAR && echo OK || echo FAIL + my-invalid-assignments: line 1: syntax error near unexpected token `then' + my-invalid-assignments: line 1: `if then' + FAIL + + # get_var_from_file /etc/os-release QQQ && echo OK || echo FAIL + + OK + + # get_var_from_file qqq ID && echo OK || echo FAIL + qqq: qqq: No such file or directory + FAIL + + # get_var_from_file /etc/os-release ID && echo OK || echo FAIL + opensuse-leap + OK + +An offhanded proposal how to let get\_var\_from\_file() +return non-zero exit code when $variable\_name is not assigned: + + # function get_var_from_file() { bash -c 'source "$0" || false ; test -v "$1" && echo "${!1}" || false' "$1" "$2" ; } + + # get_var_from_file /etc/os-release ID && echo OK || echo FAIL + opensuse-leap + OK + + # get_var_from_file /etc/os-release QQQ && echo OK || echo FAIL + FAIL + + # get_var_from_file qqq ID && echo OK || echo FAIL + qqq: qqq: No such file or directory + FAIL + + # get_var_from_file my-invalid-assignments VAR && echo OK || echo FAIL + my-invalid-assignments: line 1: syntax error near unexpected token `then' + my-invalid-assignments: line 1: `if then' + FAIL + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-20 12:41](https://github.com/rear/rear/pull/3171#issuecomment-2009476719): + +@lzaoral +I am afraid - again it gets long and longer. +Hopefully you don't mind too much +when I make such kind of pedantic comments? + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-03-21 11:25](https://github.com/rear/rear/pull/3171#issuecomment-2012011314): + +> Hopefully you don't mind too much when I make such kind of pedantic +> comments? + +@jsmeix Not at all, both yours and @pcahyna's suggestions are very +constructive! + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-03-21 12:06](https://github.com/rear/rear/pull/3171#issuecomment-2012100925): + +@jsmeix A bit shorter approach is to use the `-u` shell option. Also, +`source "$0" || false ; ...` will not halt the execution immediately in +your second implementation. + +This should return `0` if the sourcing succeeded and the given variable +is set. Otherwise, it returns `1` (the final `|| return 1` is necessary +because bash returns `127` on unbound variables). + + $ function get() { bash -c 'source "$0" || exit 1; set -u; echo "${!1}"' "$1" "$2" || return 1 } + $ get /etc/os-release ID; echo $? + fedora + 0 + $ get /etc/osrelease ID; echo $? + /etc/osrelease: line 1: /etc/osrelease: No such file or directory + 1 + $ get /etc/os-release IDD; echo $? + /etc/os-release: line 1: !1: unbound variable + 1 + +On the other hand, if it would be ok to fail on any problems in the +sourced file as well, `bash -euc ...` may be enough. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-22 13:16](https://github.com/rear/rear/pull/3171#issuecomment-2015082283): + +I like `bash -euc` very much because it is simple +and it makes useful use of the "bash in between" +which was up to now more a workaround for `source` +that has a problem with `readonly` variables, +cf. our "late but helpful party guest" ;-) comment +[https://github.com/rear/rear/pull/3171\#discussion\_r1521750947](https://github.com/rear/rear/pull/3171#discussion_r1521750947) + +So I played around with `bash -euc` and +this seems to behave best at least for my tests: + + # function get_var_from_file() { bash -eu -c 'source "$0" && echo "${!1}"' "$1" "$2" ; } + +FYI: Intentionally I separated `-eu` form `-c` to make it +more clear that there is different meaning of arguments +and I used `source ... && echo` to make it more clear +that successful `source` is a condition for `echo` +(instead of having this condition as an implicit result of `-e`). + +During my tests how it behaves in error cases +I was thinking about that it could be helpful for debugging +when also within the bash inside that function +ReaR's outer DEBUGSCRIPTS setting would be supported, +so I enhanced it further to + + # function get_var_from_file() { test "$DEBUGSCRIPTS" && local debug="-x" ; bash $debug -eu -c 'source "$0" && echo "${!1}"' "$1" "$2" ; } + +That seems to behave well at least for me on command line + + # unset myvar ; myvar="$( get_var_from_file qqq VAR && echo OK 1>&2 || echo FAIL with $? 1>&2 )" ; declare -p myvar + qqq: qqq: No such file or directory + FAIL with 1 + declare -- myvar="" + + # unset myvar ; myvar="$( get_var_from_file my-invalid-assignments VAR && echo OK 1>&2 || echo FAIL with $? 1>&2 )" ; declare -p myvar + my-invalid-assignments: line 1: syntax error near unexpected token `then' + my-invalid-assignments: line 1: `if then' + FAIL with 1 + declare -- myvar="" + + # unset myvar ; myvar="$( get_var_from_file my-shell-assignments QQQ && echo OK 1>&2 || echo FAIL with $? 1>&2 )" ; declare -p myvar + my-shell-assignments: !1: unbound variable + FAIL with 1 + declare -- myvar="" + + # unset myvar ; myvar="$( get_var_from_file my-shell-assignments VAR && echo OK 1>&2 || echo FAIL with $? 1>&2 )" ; declare -p myvar + OK + declare -- myvar="value" + + # DEBUGSCRIPTS=1 + + # unset myvar ; myvar="$( get_var_from_file qqq VAR && echo OK 1>&2 || echo FAIL with $? 1>&2 )" ; declare -p myvar + + source qqq + qqq: qqq: No such file or directory + FAIL with 1 + declare -- myvar="" + + # unset myvar ; myvar="$( get_var_from_file my-invalid-assignments VAR && echo OK 1>&2 || echo FAIL with $? 1>&2 )" ; declare -p myvar + + source my-invalid-assignments + my-invalid-assignments: line 1: syntax error near unexpected token `then' + my-invalid-assignments: line 1: `if then' + FAIL with 1 + declare -- myvar="" + + # unset myvar ; myvar="$( get_var_from_file my-shell-assignments QQQ && echo OK 1>&2 || echo FAIL with $? 1>&2 )" ; declare -p myvar + + source my-shell-assignments + ++ VAR=value + ++ STRING='lvalue = rvalue' + ++ ARR=('first element = 1' 'second element = 2' 'last element = 3') + my-shell-assignments: !1: unbound variable + FAIL with 1 + declare -- myvar="" + + # unset myvar ; myvar="$( get_var_from_file my-shell-assignments VAR && echo OK 1>&2 || echo FAIL with $? 1>&2 )" ; declare -p myvar + + source my-shell-assignments + ++ VAR=value + ++ STRING='lvalue = rvalue' + ++ ARR=('first element = 1' 'second element = 2' 'last element = 3') + + echo value + OK + declare -- myvar="value" + +The last two examples (i.e. when 'source' succeeds with +DEBUGSCRIPTS=1) +show that the whole sourced file gets output via 'set -x' +which might be unwanted e.g. when that file is huge and +which might expose (sensitive) variable values in ReaR's log file +(where stderr is redirected within ReaR's runtime environment). + +So we must also test it within ReaR's runtime environment. + +Perhaps it is best to "Keep It Simple and Secure" +and omit such transitive DEBUGSCRIPTS setting? + +For completeness: +Tests with added EMPTY and BLANK variables and +with STRING (correct) and ARR (wrong) results +(with `DEBUGSCRIPTS=1`): + + # cat my-shell-assignments + VAR="value" + EMPTY= + BLANK=' ' + STRING="lvalue \ + = \ + rvalue" + ARR=( 'first element = 1' + 'second element = 2' + 'last element = 3' ) + + # unset myvar ; myvar="$( get_var_from_file my-shell-assignments EMPTY && echo OK 1>&2 || echo FAIL with $? 1>&2 )" ; declare -p myvar + + source my-shell-assignments + ++ VAR=value + ++ EMPTY= + ++ BLANK=' ' + ++ STRING='lvalue = rvalue' + ++ ARR=('first element = 1' 'second element = 2' 'last element = 3') + + echo '' + OK + declare -- myvar="" + + # unset myvar ; myvar="$( get_var_from_file my-shell-assignments BLANK && echo OK 1>&2 || echo FAIL with $? 1>&2 )" ; declare -p myvar + + source my-shell-assignments + ++ VAR=value + ++ EMPTY= + ++ BLANK=' ' + ++ STRING='lvalue = rvalue' + ++ ARR=('first element = 1' 'second element = 2' 'last element = 3') + + echo ' ' + OK + declare -- myvar=" " + + # unset myvar ; myvar="$( get_var_from_file my-shell-assignments STRING && echo OK 1>&2 || echo FAIL with $? 1>&2 )" ; declare -p myvar + + source my-shell-assignments + ++ VAR=value + ++ EMPTY= + ++ BLANK=' ' + ++ STRING='lvalue = rvalue' + ++ ARR=('first element = 1' 'second element = 2' 'last element = 3') + + echo 'lvalue = rvalue' + OK + declare -- myvar="lvalue = rvalue" + + # unset myvar ; myvar="$( get_var_from_file my-shell-assignments ARR && echo OK 1>&2 || echo FAIL with $? 1>&2 )" ; declare -p myvar + + source my-shell-assignments + ++ VAR=value + ++ EMPTY= + ++ BLANK=' ' + ++ STRING='lvalue = rvalue' + ++ ARR=('first element = 1' 'second element = 2' 'last element = 3') + + echo 'first element = 1' + OK + declare -- myvar="first element = 1" + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-22 16:54](https://github.com/rear/rear/pull/3171#issuecomment-2015507343): + +> Perhaps it is best to "Keep It Simple and Secure" +> and omit such transitive DEBUGSCRIPTS setting? + +I think so, I would not bother with it. + +More important question is whether it is ok to fail on any problems in +the sourced file as well. I think that the sourced file may use variable +assignments that are not intended to work correctly with `set -eu`. For +example FOO="$BAR" where BAR is unset. For this reason I think I would +prefer to add `set -u` or `set -eu` after the `source` instead of having +-eu as flags on the `bash` invocation. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-25 13:15](https://github.com/rear/rear/pull/3171#issuecomment-2017986309): + +Regarding +[https://github.com/rear/rear/pull/3171\#issuecomment-2015507343](https://github.com/rear/rear/pull/3171#issuecomment-2015507343) + +I do fully agree. + +Meanwhile +(after I had some "deep thought" about it over the weekend) +I consider hardcoded "transitive DEBUG options setting" +a horrible idea regarding security and privacy. + +Not only in this specific case but in general: +I.e. when ReaR scripts run with 'set -x' +it does not mean that programs which are called by ReaR +also run with whatever program-specific debug settings. + +Reasoning +with this specific case as an example: + +For example (as far as I know) some networking config files +could contain confidential data (like WLAN passwords) +so when ReaR needs some other value from such a file +the whole file (except bash comments) +would appear in the log (in DEBUGSCRIPTS mode) +so the confidential values would leak out => security issue. + +Furthermore when whole user config files (except bash comments) +appear in the log (in DEBUGSCRIPTS mode) +ReaR would cross a privacy boundary: +From what belongs to ReaR (e.g. the value that ReaR needs +from a user config file) to what belongs only to the user +i.e. his whole config file with all his values => privacy issue. + +If at all "transitive DEBUG options setting" might be enabled +only on explicit user request via '--expose-secrets'. + +But in this specific case "transitive DEBUGSCRIPTS setting" +is not needed in practice because also without 'set -x' +the stderr messages tell sufficiently what the problem is +so that the user can inspect his config file on his own +what the actual reason inside his config file is. +Also when a user reports an issue to us +we can show him by the stderr messages what the problem is +but we neither need to know all his config values +nor should we know all his config values +to help him sufficiently. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-25 13:24](https://github.com/rear/rear/pull/3171#issuecomment-2018002598): + +The current implementation + + # function get_var_from_file() { bash -c 'source "$0"; echo "${!1}"' "$1" "$2" ; } + +does not yet work correctly +when there is stdout output of the sourced file + + # cat my-assignment + if echo HELLO ; then + VAR="value" + fi + + # unset myvar ; myvar="$( get_var_from_file my-assignment VAR )" && echo OK 1>&2 || echo FAIL with $? 1>&2 ; declare -p myvar + OK + declare -- myvar="HELLO + value" + + # unset VAR ; source my-assignment && echo OK 1>&2 || echo FAIL with $? 1>&2 ; declare -p VAR + HELLO + OK + declare -- VAR="value" + +This one fixes it for me: + + # function get_var_from_file() { bash -c 'source "$0" 1>/dev/null ; echo "${!1}"' "$1" "$2" ; } + + # unset myvar ; myvar="$( get_var_from_file my-assignment VAR )" && echo OK 1>&2 || echo FAIL with $? 1>&2 ; declare -p myvar + OK + declare -- myvar="value" + +The fix here (by added `1>/dev/null` to the 'source' call) +is only meant as an offhanded proposal to fix +when there is stdout output of the sourced file. + +It is not meant as a proposal how to deal with errors +(via `set -e` or `set -eu` or `... || exit 1` or ...). + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-25 14:22](https://github.com/rear/rear/pull/3171#issuecomment-2018119545): + +Yes, you are right; nit: `1` is superfluous, `>/dev/null` is enough. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-25 14:52](https://github.com/rear/rear/pull/3171#issuecomment-2018187735): + +A 'nit' reply only for the fun of it: +I prefer `1>` over `>` because I prefer explicit code. +In particular explicit `1>` would make it easier +in theory to search the code where stdout is redirected. +In practice this does not work because also `>` is used +so in the end it does not matter if `1>` or `>` is used +and I am fine with both. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-25 14:57](https://github.com/rear/rear/pull/3171#issuecomment-2018199250): + +@jsmeix I happen to disagree here because everyone knows `> /dev/null` +and is used to it, so if I see `1> /dev/null`, I have to think about +what is the `1` doing there, while `> /dev/null` is immediately obvious. +Related: `> /dev/null` is more visually recognizable from let's say +`7> /dev/null`, and we use numbered redirections often in ReaR. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-26 07:05](https://github.com/rear/rear/pull/3171#issuecomment-2019531750): + +Sigh - endless problems - here the next one: + +We cannot fail when 'source' fails because +'source' returns the status of the last command +executed in the sourced file so + + # function get_var_from_file() { bash -c 'source "$0" >/dev/null || exit 1 ; set -u ; echo "${!1}"' "$1" "$2" || return 1 ; } + + # cat my-assignment + echo HELLO && VAR="hello" + ls QQQ && VAR="qqq" + + # unset myvar ; myvar="$( get_var_from_file my-assignment VAR )" && echo OK 1>&2 || echo FAIL with $? 1>&2 ; declare -p myvar + ls: cannot access 'QQQ': No such file or directory + FAIL with 1 + declare -- myvar="" + + # unset VAR ; source my-assignment && echo OK 1>&2 || echo FAIL with $? 1>&2 ; declare -p VAR + HELLO + ls: cannot access 'QQQ': No such file or directory + FAIL with 2 + declare -- VAR="hello" + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-26 07:13](https://github.com/rear/rear/pull/3171#issuecomment-2019541083): + +What seems to behave correctly +(at least for me with what I test here) +is + + # function get_var_from_file() { bash -c 'source "$0" >/dev/null ; set -u ; echo "${!1}"' "$1" "$2" || return 1 ; } + + # cat my-assignment + echo HELLO && VAR="hello" + ls QQQ && VAR="qqq" + + # unset myvar ; myvar="$( get_var_from_file my-assignment VAR )" && echo OK || echo FAIL with $? ; declare -p myvar + ls: cannot access 'QQQ': No such file or directory + OK + declare -- myvar="hello" + + # unset myvar ; myvar="$( get_var_from_file my-assignment qqq )" && echo OK || echo FAIL with $? ; declare -p myvar + ls: cannot access 'QQQ': No such file or directory + my-assignment: !1: unbound variable + FAIL with 1 + declare -- myvar="" + + # unset myvar ; myvar="$( get_var_from_file qqq VAR )" && echo OK || echo FAIL with $? ; declare -p myvar + qqq: qqq: No such file or directory + qqq: !1: unbound variable + FAIL with 1 + declare -- myvar="" + + # cat my-assignment + if then + echo HELLO && VAR="hello" + ls QQQ && VAR="qqq" + + # unset myvar ; myvar="$( get_var_from_file my-assignment VAR )" && echo OK || echo FAIL with $? ; declare -p myvar + my-assignment: line 1: syntax error near unexpected token `then' + my-assignment: line 1: `if then' + my-assignment: !1: unbound variable + FAIL with 1 + declare -- myvar="" + + # cat my-assignment + echo HELLO && VAR="hello" + if then + ls QQQ && VAR="qqq" + + # unset myvar ; myvar="$( get_var_from_file my-assignment VAR )" && echo OK || echo FAIL with $? ; declare -p myvar + my-assignment: line 2: syntax error near unexpected token `then' + my-assignment: line 2: `if then' + OK + declare -- myvar="hello" + + # cat my-assignment + echo HELLO && VAR="hello" + ls QQQ && VAR="qqq" + if then + + # unset myvar ; myvar="$( get_var_from_file my-assignment VAR )" && echo OK || echo FAIL with $? ; declare -p myvar + ls: cannot access 'QQQ': No such file or directory + my-assignment: line 3: syntax error near unexpected token `then' + my-assignment: line 3: `if then' + OK + declare -- myvar="hello" + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-26 13:36](https://github.com/rear/rear/pull/3171#issuecomment-2020450590): + +@jsmeix + +> We cannot fail when 'source' fails because +> 'source' returns the status of the last command +> executed in the sourced file so (...) + +... so what? Let me quote the os-release(5) manual page: + +> The format of os-release is a newline-separated list of +> environment-like shell-compatible variable assignments. It is +> possible +> to source the configuration from Bourne shell scripts, however, +> **beyond +> mere variable assignments, no shell features are supported** (this +> means +> variable expansion is explicitly not supported), allowing +> applications +> to read the file **without implementing a shell compatible execution +> engine**. + +(emphasis mine). + +So, why to worry about the exit status from `source` if the only allowed +statements are variable assignment statements? (`ls` in your example is +thus not allowed.) I consider this a non-issue. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-26 13:58](https://github.com/rear/rear/pull/3171#issuecomment-2020504167): + +@pcahyna +in your +[https://github.com/rear/rear/pull/3171\#discussion\_r1521750947](https://github.com/rear/rear/pull/3171#discussion_r1521750947) +you wrote that +"the get\_var\_from\_file function is meant to be generic" +which was the reason that I suggested in my +[https://github.com/rear/rear/pull/3171\#discussion\_r1523283636](https://github.com/rear/rear/pull/3171#discussion_r1523283636) +that "it fits perhaps better in lib/global-functions.sh" + +When we have a generic get\_var\_from\_file function +then it must work reasonably well for any file +that can be sourced, in particular for a bash script. + +Otherwise when we only want a function to get values +from os-release then we must not have such a function +as a generic function lib/global-functions.sh +but preferably as it was before only a "local" function +called explicitly get\_var\_from\_os\_release to make it clear +that this function cannot be used generically +to get a value from any file that can be sourced. + +But why not have a generic get\_var\_from\_file function +when it seems it can be implemented reasonably well, +for example as I proposed in my last +[https://github.com/rear/rear/pull/3171\#issuecomment-2019541083](https://github.com/rear/rear/pull/3171#issuecomment-2019541083) + +Or is there something severely wrong with my recent proposal? + +I mean: It is certainly possible to make a bash script +where my recent get\_var\_from\_file function fails +but in "sufficiently normal" cases it seems to work OK + + # function get_var_from_file() { bash -c 'source "$0" >/dev/null ; set -u ; echo "${!1}"' "$1" "$2" || return 1 ; } + + # cat my-assignment + echo HELLO && VAR="$( echo -n "Today is $( date )" )" + ls QQQ && VAR="qqq" + if then + + # unset myvar ; myvar="$( get_var_from_file my-assignment VAR )" && echo OK || echo FAIL with $? ; declare -p myvar ; date + ls: cannot access 'QQQ': No such file or directory + my-assignment: line 3: syntax error near unexpected token `then' + my-assignment: line 3: `if then' + OK + declare -- myvar="Today is Tue 26 Mar 2024 03:06:09 PM CET" + Tue 26 Mar 2024 03:06:09 PM CET + +For the fun of it: +I even found a simple bash script where +my recent get\_var\_from\_file function fails, +something like: + + poweroff + sleep 120 + VAR="value" + +;-) + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-26 14:21](https://github.com/rear/rear/pull/3171#issuecomment-2020565632): + +> When we have a generic get\_var\_from\_file function +> then it must work reasonably well for any file +> that can be sourced, in particular for a bash script. + +Depends on how "generic" we want it to have. We can define it as being +generic enough to be able to read files that have the same format as +os-release, i.e. variable assignments but no other shell commands. I +believe that files under `/etc/sysconfig` fall in this category. + +> But why not have a generic get\_var\_from\_file function +> when it seems it can be implemented reasonably well, +> for example as I proposed in my last +> [https://github.com/rear/rear/pull/3171\#issuecomment-2019541083](https://github.com/rear/rear/pull/3171#issuecomment-2019541083) +> Or is there something severely wrong with my recent proposal? + +My only problem with it is that we lose the ability to detect errors +when sourcing the file (including file not found or readable), compared +to the `source ... || exit 1` version. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-27 08:41](https://github.com/rear/rear/pull/3171#issuecomment-2022217821): + +It also fails for file not found or not readable + + # function get_var_from_file() { bash -c 'source "$0" >/dev/null ; set -u ; echo "${!1}"' "$1" "$2" || return 1 ; } + + # unset myvar ; myvar="$( get_var_from_file qqq VAR )" && echo OK || echo FAIL with $? ; declare -p myvar + qqq: qqq: No such file or directory + qqq: !1: unbound variable + FAIL with 1 + declare -- myvar="" + + # unset myvar ; myvar="$( get_var_from_file /etc/shadow VAR )" && echo OK || echo FAIL with $? ; declare -p myvar + /etc/shadow: /etc/shadow: Permission denied + /etc/shadow: !1: unbound variable + FAIL with 1 + declare -- myvar="" + +but it always fails at 'set -u' because of "unbound variable". + +As far as I see it is not possible to +fail at 'source' for file not found or not readable +but not fail at 'source' when the last command in file +results non-zero exit code which is usually '1' + + $ source /etc/shadow || echo $? + bash: /etc/shadow: Permission denied + 1 + + $ source qqq || echo $? + bash: qqq: No such file or directory + 1 + + $ cat my-script + grep qqq /etc/fstab + + $ source my-script || echo $? + 1 + +so we would have to add special checks +for cases like file not found or not readable. + +I think such specific checks are not needed +because stderr shows the root cause why it failed +regardless that actually it failed at 'set -u'. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-27 11:46](https://github.com/rear/rear/pull/3171#issuecomment-2022575784): + +Bottom line of what I mean is: + +As far as I see + + function get_var_from_file() { + bash -c 'source "$0" >/dev/null ; set -u ; echo "${!1}"' "$1" "$2" || return 1 + } + +provides simple "all or nothing" return code behaviour: + +Zero return code means the variable was set in the file and +stdout is the value of the variable (could be empty or blank). + +Non-zero return code in all other cases. + +So get\_var\_from\_file can be used in a simple way like + + if my_var="$( get_var_from_file FILE_NAME VAR_NAME )" ; then + # Code when VAR_NAME was set in FILE_NAME: + ... + else + # Code when the value of VAR_NAME is unknown + # (e.g. error handling code or fallback code): + ... + fi + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-27 12:15](https://github.com/rear/rear/pull/3171#issuecomment-2022625255): + +For the fun of craziness: + +My above described simple "all or nothing" return code behaviour +does not hold in any case, in particular not for shell variables: + + # cat my-assignment + VAR="value" + BASH="my bash" + BASHPID=99999999 + + # unset myvar ; if myvar="$( get_var_from_file my-assignment BASH )" ; then declare -p myvar ; else echo FAIL with $? ; fi + declare -- myvar="my bash" + + # unset myvar ; if myvar="$( get_var_from_file my-assignment BASHPID )" ; then declare -p myvar ; else echo FAIL with $? ; fi + declare -- myvar="6854" + + # unset myvar ; if myvar="$( get_var_from_file my-assignment BASH_COMMAND )" ; then declare -p myvar ; else echo FAIL with $? ; fi + declare -- myvar="echo \"\${!1}\"" + +but BASHPID cannot be set in my-assignment because it is readonly +and BASH\_COMMAND was not set in my-assignment. + +So for the fun of craziness I further enhanced it: + + # function get_var_from_file() { bash -c 'unset $1 || exit 1 ; source "$0" >/dev/null ; set -u ; echo "${!1}"' "$1" "$2" || return 1 ; } + +This seems to even behave correctly - as far as I tested: + + # unset myvar ; if myvar="$( get_var_from_file my-assignment VAR )" ; then declare -p myvar ; else echo FAIL with $? ; fi + declare -- myvar="value" + + # unset myvar ; if myvar="$( get_var_from_file my-assignment BASH )" ; then declare -p myvar ; else echo FAIL with $? ; fi + declare -- myvar="my bash" + + # unset myvar ; if myvar="$( get_var_from_file my-assignment BASHPID )" ; then declare -p myvar ; else echo FAIL with $? ; fi + my-assignment: line 0: unset: BASHPID: cannot unset: readonly variable + FAIL with 1 + + # unset myvar ; if myvar="$( get_var_from_file my-assignment BASH_COMMAND )" ; then declare -p myvar ; else echo FAIL with $? ; fi + my-assignment: !1: unbound variable + FAIL with 1 + +The idea behind to ensure the above described +simple "all or nothing" return code behaviour +is: + +When VAR\_NAME cannot be unset before 'source FILE\_NAME' +then VAR\_NAME cannot be (successfully) set in FILE\_NAME +so get\_var\_from\_file must return non-zero return code. + +@lzaoral @pcahyna +I know I am here far beyond what is reasonable +but I currently have "just fun" to play around +with even such crazy exceptional corner cases, cf. +[https://github.com/rear/rear/pull/3168\#issuecomment-1991713382](https://github.com/rear/rear/pull/3168#issuecomment-1991713382) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-27 12:44](https://github.com/rear/rear/pull/3171#issuecomment-2022676968): + +@pcahyna @lzaoral +when you agree that having a generic function + + function get_var_from_file() { + bash -c 'unset $1 || exit 1 ; source "$0" >/dev/null ; set -u ; echo "${!1}"' "$1" "$2" || return 1 + } + +is useful in ReaR - in particular when you think +the above implementation is reasonably correct, +would it then help you with this pull request here +when I make a pull request to add that function +to lib/global-functions.sh so that this part +can be excluded from this pull request? + +And in case of issues with that function +I am to "git blame" for it ;-) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-27 12:54](https://github.com/rear/rear/pull/3171#issuecomment-2022699855): + +FYI: +How it behaves for /etc/os-release for me + + # for varname in $( cut -s -d '=' -f1 /etc/os-release ) ; \ + do unset myvar ; \ + if myvar="$( get_var_from_file /etc/os-release $varname )" ; \ + then grep "^$varname=" /etc/os-release ; \ + declare -p myvar ; \ + echo ; \ + else echo FAIL with $? ; \ + fi ; \ + done + + NAME="openSUSE Leap" + declare -- myvar="openSUSE Leap" + + VERSION="15.5" + declare -- myvar="15.5" + + ID="opensuse-leap" + declare -- myvar="opensuse-leap" + + ID_LIKE="suse opensuse" + declare -- myvar="suse opensuse" + + VERSION_ID="15.5" + declare -- myvar="15.5" + + PRETTY_NAME="openSUSE Leap 15.5" + declare -- myvar="openSUSE Leap 15.5" + + ANSI_COLOR="0;32" + declare -- myvar="0;32" + + CPE_NAME="cpe:/o:opensuse:leap:15.5" + declare -- myvar="cpe:/o:opensuse:leap:15.5" + + BUG_REPORT_URL="https://bugs.opensuse.org" + declare -- myvar="https://bugs.opensuse.org" + + HOME_URL="https://www.opensuse.org/" + declare -- myvar="https://www.opensuse.org/" + + DOCUMENTATION_URL="https://en.opensuse.org/Portal:Leap" + declare -- myvar="https://en.opensuse.org/Portal:Leap" + + LOGO="distributor-logo-Leap" + declare -- myvar="distributor-logo-Leap" + +Same result when I do this inside ReaR +by adding that code at the beginning of +init/default/001\_verify\_config\_arrays.sh + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-04-09 10:12](https://github.com/rear/rear/pull/3171#issuecomment-2044642518): + +Sorry @jsmeix , we got a bit lost in the discussion - please implement +the proposed `get_var_from_file`. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-09 11:45](https://github.com/rear/rear/pull/3171#issuecomment-2044901901): + +@pcahyna +my pleasure: +[https://github.com/rear/rear/pull/3203](https://github.com/rear/rear/pull/3203) + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-03-01.3172.pr.merged.md b/docs/issues/2024-03-01.3172.pr.merged.md new file mode 100644 index 00000000..75203157 --- /dev/null +++ b/docs/issues/2024-03-01.3172.pr.merged.md @@ -0,0 +1,160 @@ +[\#3172 PR](https://github.com/rear/rear/pull/3172) `merged`: Bump redhat-plumbers-in-action/differential-shellcheck from 5.0.2 to 5.1.0 +======================================================================================================================================== + +**Labels**: `fixed / solved / done`, `ReaR Project`, `dependencies` + +#### [dependabot](https://github.com/apps/dependabot) opened issue at [2024-03-01 11:21](https://github.com/rear/rear/pull/3172): + +Bumps +[redhat-plumbers-in-action/differential-shellcheck](https://github.com/redhat-plumbers-in-action/differential-shellcheck) +from 5.0.2 to 5.1.0. + +
+Release notes +

Sourced from redhat-plumbers-in-action/differential-shellcheck's releases.

+
+

v5.1.0

+

What's Changed

+

New

+
    +
  • Improve shell script detection based on emacs file mode header :tiger: (#357) @​jamacku
    • +
+

Documentation

+
    +
  • Update markdown warning, use supported syntax :lipstick: (#341) @​jamacku
    • +
+

Other changes

+ +

Automation and CI changes

+ +

Dependency Updates

+ + + + +

Full Changelog: https://github.com/redhat-plumbers-in-action/differential-shellcheck/compare/v5.0.2...v5.1.0

+
+
+
+Commits +
    +
  • b9df2a9 v5.1.0
    • +
  • c74f4ed feat: add support for emacs file mode line with mode:
    • +
  • f23778e feat: support -*- shell script -*- script header
    • +
  • 65342fa deps: update csutils to 3.2.0
    • +
  • 5580924 build(deps): bump test/bats from 3d3f63d to 990d8e2
    • +
  • 66806ae build(deps): bump actions/upload-artifact from 4.0.0 to 4.3.0
    • +
  • c2a8e3e build(deps): bump dorny/paths-filter from 2.11.1 to 3.0.0
    • +
  • a219af7 build(deps): bump github/codeql-action from 3.22.12 to 3.23.2
    • +
  • 97d3bdd README.md: bump actions/upload-artifact from v3 to v4 (#347)
    • +
  • ae3a070 doc: remove extra spaces from example
    • +
  • Additional commits viewable in compare view
    • +
+
+
+ +[![Dependabot compatibility +score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=redhat-plumbers-in-action/differential-shellcheck&package-manager=github_actions&previous-version=5.0.2&new-version=5.1.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) + +Dependabot will resolve any conflicts with this PR as long as you don't +alter it yourself. You can also trigger a rebase manually by commenting +`@dependabot rebase`. + +------------------------------------------------------------------------ + +
+Dependabot commands and options +
+ +You can trigger Dependabot actions by commenting on this PR: + +- `@dependabot rebase` will rebase this PR +- `@dependabot recreate` will recreate this PR, overwriting any edits + that have been made to it +- `@dependabot merge` will merge this PR after your CI passes on it +- `@dependabot squash and merge` will squash and merge this PR after + your CI passes on it +- `@dependabot cancel merge` will cancel a previously requested merge + and block automerging +- `@dependabot reopen` will reopen this PR if it is closed +- `@dependabot close` will close this PR and stop Dependabot + recreating it. You can achieve the same result by closing it + manually +- `@dependabot show ignore conditions` will show all + of the ignore conditions of the specified dependency +- `@dependabot ignore this major version` will close this PR and stop + Dependabot creating any more for this major version (unless you + reopen the PR or upgrade to it yourself) +- `@dependabot ignore this minor version` will close this PR and stop + Dependabot creating any more for this minor version (unless you + reopen the PR or upgrade to it yourself) +- `@dependabot ignore this dependency` will close this PR and stop + Dependabot creating any more for this dependency (unless you reopen + the PR or upgrade to it yourself) + +
+ +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-15 08:37](https://github.com/rear/rear/pull/3172#issuecomment-1999173297): + +@pcahyna +I dared to assign it to you because this pull request +belongs to "something from Red Hat" +so perhaps you could handle it? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-25 15:13](https://github.com/rear/rear/pull/3172#issuecomment-2018240362): + +@pcahyna +thank you for handling it! + +At least currently I cannot do that because I worry about +[https://github.com/rear/rear/issues/3130](https://github.com/rear/rear/issues/3130) + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-25 15:34](https://github.com/rear/rear/pull/3172#issuecomment-2018292749): + +well an obsolete action in the workflow file is no safer than an +uptodate one. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-26 06:51](https://github.com/rear/rear/pull/3172#issuecomment-2019517152): + +I was (perhaps falsely) thinking that an updated one +might have new security/privacy issues +(caused by new/changed functionality that is unsafe) +so each update would have to be carefully examined +before it is accepted to be used in a GitHub Action? + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-26 14:49](https://github.com/rear/rear/pull/3172#issuecomment-2020643529): + +@jsmeix I am afraid that we don't have the knowledge and capacity to +review all the code that we are using, so we basically have to trust it +(reviewing changes would not be enough - one would need to review the +whole code when we start using it for the first time) and it is better +to update to latest versions as they may contain fixes for issues that +others have found. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-27 07:58](https://github.com/rear/rear/pull/3172#issuecomment-2022156095): + +Because we have to basically blindly trust +it is mandatory to limit those automatisms +by only running what we really need and +by only allowing what is really required. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-03-01.3173.pr.merged.md b/docs/issues/2024-03-01.3173.pr.merged.md new file mode 100644 index 00000000..dd79eb9e --- /dev/null +++ b/docs/issues/2024-03-01.3173.pr.merged.md @@ -0,0 +1,79 @@ +[\#3173 PR](https://github.com/rear/rear/pull/3173) `merged`: Bump codacy/codacy-analysis-cli-action from 4.3.0 to 4.4.0 +======================================================================================================================== + +**Labels**: `dependencies` + +#### [dependabot](https://github.com/apps/dependabot) opened issue at [2024-03-01 11:21](https://github.com/rear/rear/pull/3173): + +Bumps +[codacy/codacy-analysis-cli-action](https://github.com/codacy/codacy-analysis-cli-action) +from 4.3.0 to 4.4.0. + +
+Release notes +

Sourced from codacy/codacy-analysis-cli-action's releases.

+
+

Update cli and tool versions

+

Update versions of:

+
    +
  • staticheck
    • +
  • gosec
    • +
  • codacy-analysis-cli
    • +
+
+
+
+Commits + +
+
+ +[![Dependabot compatibility +score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=codacy/codacy-analysis-cli-action&package-manager=github_actions&previous-version=4.3.0&new-version=4.4.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) + +Dependabot will resolve any conflicts with this PR as long as you don't +alter it yourself. You can also trigger a rebase manually by commenting +`@dependabot rebase`. + +------------------------------------------------------------------------ + +
+Dependabot commands and options +
+ +You can trigger Dependabot actions by commenting on this PR: + +- `@dependabot rebase` will rebase this PR +- `@dependabot recreate` will recreate this PR, overwriting any edits + that have been made to it +- `@dependabot merge` will merge this PR after your CI passes on it +- `@dependabot squash and merge` will squash and merge this PR after + your CI passes on it +- `@dependabot cancel merge` will cancel a previously requested merge + and block automerging +- `@dependabot reopen` will reopen this PR if it is closed +- `@dependabot close` will close this PR and stop Dependabot + recreating it. You can achieve the same result by closing it + manually +- `@dependabot show ignore conditions` will show all + of the ignore conditions of the specified dependency +- `@dependabot ignore this major version` will close this PR and stop + Dependabot creating any more for this major version (unless you + reopen the PR or upgrade to it yourself) +- `@dependabot ignore this minor version` will close this PR and stop + Dependabot creating any more for this minor version (unless you + reopen the PR or upgrade to it yourself) +- `@dependabot ignore this dependency` will close this PR and stop + Dependabot creating any more for this dependency (unless you reopen + the PR or upgrade to it yourself) + +
+ +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-03-06.3174.pr.merged.md b/docs/issues/2024-03-06.3174.pr.merged.md new file mode 100644 index 00000000..e1938ccd --- /dev/null +++ b/docs/issues/2024-03-06.3174.pr.merged.md @@ -0,0 +1,41 @@ +[\#3174 PR](https://github.com/rear/rear/pull/3174) `merged`: Allow 'shell' workflow in recovery system +======================================================================================================= + +**Labels**: `enhancement`, `fixed / solved / done` + +#### [jsmeix](https://github.com/jsmeix) opened issue at [2024-03-06 16:18](https://github.com/rear/rear/pull/3174): + +- Type: **Enhancement** + +- Impact: **High** + High impact for special cases (i.e. when really needed). + No impact when not needed. + +- Reference to related issue (URL): + [https://github.com/rear/rear/issues/3170\#issuecomment-1981222992](https://github.com/rear/rear/issues/3170#issuecomment-1981222992) + +- How was this pull request tested? + See + [https://github.com/rear/rear/issues/3170\#issuecomment-1981222992](https://github.com/rear/rear/issues/3170#issuecomment-1981222992) + +- Description of the changes in this pull request: + +Add 'shell' to the workflows that are allowed +to be run from within the ReaR recovery system +because it can be useful for special cases + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-08 06:30](https://github.com/rear/rear/pull/3174#issuecomment-1985112718): + +@rear/contributors +I would like to merge it today afternoon +unless there are objections. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-08 15:56](https://github.com/rear/rear/pull/3174#issuecomment-1985946402): + +@gdha @schlomo +thank you for your review! + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-03-07.3175.pr.open.md b/docs/issues/2024-03-07.3175.pr.open.md new file mode 100644 index 00000000..335793e1 --- /dev/null +++ b/docs/issues/2024-03-07.3175.pr.open.md @@ -0,0 +1,443 @@ +[\#3175 PR](https://github.com/rear/rear/pull/3175) `open`: Automatically include mounted btrfs subvolumes in NETFS backups +=========================================================================================================================== + +**Labels**: `enhancement` + +#### [lzaoral](https://github.com/lzaoral) opened issue at [2024-03-07 12:00](https://github.com/rear/rear/pull/3175): + +##### Pull Request Details: + +- Type: **Enhancement** + +- Impact: **High** + +- Reference to related issue (URL): + [https://github.com/rear/rear/issues/2928](https://github.com/rear/rear/issues/2928) + +- How was this pull request tested? `rear savelayout` and manual + inspection of generated files and backup/restore of a Fedora Rawhide + machine + +- Description of the changes in this pull request: + + - automatically include mounted btrfs subvolumes in NETFS backups + - improve generation of `$RESTORE_EXCLUDE_FILE` + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-07 13:24](https://github.com/rear/rear/pull/3175#issuecomment-1983498175): + +@lzaoral +thank you for this enhancement! + +I will test how it behaves on SLES systems +with their rather complicated default btrfs structure. + +Offhandedly I think the main problem is +possibly mounted btrfs snapshot subvolumes, see +"When btrfs is used with snapshots ... +then usual backup and restore cannot work." in +[https://en.opensuse.org/SDB:Disaster\_Recovery\#btrfs](https://en.opensuse.org/SDB:Disaster_Recovery#btrfs) + +I.e. when there are mounted btrfs "thingies" +listed as 'btrfsmountedsubvol' in disklayout.conf +that are also listed as '\#btrfssnapshotsubvol'. + +For example a disklayout.conf on SLES15-SP5 (excerpts) + + btrfsdefaultsubvol /dev/sda2 / 268 @/.snapshots/1/snapshot + ... + #btrfssnapshotsubvol /dev/sda2 / 272 @/.snapshots/2/snapshot + #btrfssnapshotsubvol /dev/sda2 / 273 @/.snapshots/3/snapshot + #btrfssnapshotsubvol /dev/sda2 / 274 @/.snapshots/4/snapshot + ... + btrfsmountedsubvol /dev/sda2 / rw,relatime,space_cache,subvolid=268,subvol=/@/.snapshots/1/snapshot @/.snapshots/1/snapshot + btrfsmountedsubvol /dev/sda2 /.snapshots rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots @/.snapshots + btrfsmountedsubvol /dev/sda2 /boot/grub2/x86_64-efi rw,relatime,space_cache,subvolid=265,subvol=/@/boot/grub2/x86_64-efi @/boot/grub2/x86_64-efi + btrfsmountedsubvol /dev/sda2 /root rw,relatime,space_cache,subvolid=262,subvol=/@/root @/root + btrfsmountedsubvol /dev/sda2 /opt rw,relatime,space_cache,subvolid=263,subvol=/@/opt @/opt + btrfsmountedsubvol /dev/sda2 /home rw,relatime,space_cache,subvolid=264,subvol=/@/home @/home + btrfsmountedsubvol /dev/sda2 /boot/grub2/i386-pc rw,relatime,space_cache,subvolid=266,subvol=/@/boot/grub2/i386-pc @/boot/grub2/i386-pc + btrfsmountedsubvol /dev/sda2 /srv rw,relatime,space_cache,subvolid=261,subvol=/@/srv @/srv + btrfsmountedsubvol /dev/sda2 /tmp rw,relatime,space_cache,subvolid=260,subvol=/@/tmp @/tmp + btrfsmountedsubvol /dev/sda2 /usr/local rw,relatime,space_cache,subvolid=259,subvol=/@/usr/local @/usr/local + btrfsmountedsubvol /dev/sda2 /var rw,relatime,space_cache,subvolid=258,subvol=/@/var @/var + +For example assume in addition to @/.snapshots/1/snapshot +that is the current system snapshot which is mounted at / +also +some other snapshots of the system like +@/.snapshots/2/snapshot and @/.snapshots/4/snapshot +are mounted e.g. at /snapshot2 and /snapshot4 + +Then a 'tar' backup would contain the system files +basically three times: + +1. the files of what is mounted at / +2. the files of what is mounted at /snapshot2 +3. the files of what is mounted at /snapshot4 + +So the backup would be basically about three times +as big as if only what is mounted at / was backed up. + +But during 'tar' restore there is no deduplication +so the restore would basically need about three times +the disk space as the original system needed. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-07 13:32](https://github.com/rear/rear/pull/3175#issuecomment-1983517679): + +> I think the main problem is +> possibly mounted btrfs snapshot subvolumes + +I am not a btrfs expert at all, but is it possible to distinguish +snapshot subvolumes from "normal" (non-snapshot) subvolumes? Then we +could save this subvolume metadata (snapshot yes/no) and do something +based on the information when recreating and restoring. First we would +probably just skip snapshots, later we could do something more +intelligent if possible. +It is definitely possible to distinguish snapshots from regular +filesystems (filessytems are equivalent to btrfs subvolumes) in ZFS. It +is also possible to recognize snapshots from regular volumes in LVM. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-07 13:49](https://github.com/rear/rear/pull/3175#issuecomment-1983546230): + +Only an offhanded thought: + +I fear btrfs normal subvolumes versus btrfs snapshot subvolumes +is only one example of a very generic problem when by default +every mounted "thingy" is included in the 'tar' backup: + +I think it is in general possible that one same +"mountable thingy" can be mounted at the same time +at different mount points e.g. at '/here' and '/there'. + +When '/here' and '/there' are included in a 'tar' backup +things may get restored twice as distinct sets of files +and not as one same set of files that is mounted two times +under the mountpoint directories '/here' and '/there'. + +Tomorrow I will experiment a bit with that. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-07 13:52](https://github.com/rear/rear/pull/3175#issuecomment-1983552670): + +Perhaps we can by default have every mounted "thingy" +included in the 'tar' backup +BUT +we may need some check for duplicates in the 'tar' backup +i.e. something that detects when one same "mountable thingy" +will become included in the 'tar' backup more than once. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-08 08:49](https://github.com/rear/rear/pull/3175#issuecomment-1985290555): + +Currently I am exploring how 'tar' behaves in general +when exact same files are provided as 'tar' arguments +to be archived. + +It seems 'tar' behaves well forgiving in this case: + + # mkdir test + + # cd test + + # echo foo >foo + + # tar -cvvvf test.tar foo foo + -rw-r--r-- root/root 4 2024-03-08 09:41 foo + hrw-r--r-- root/root 0 2024-03-08 09:41 foo link to foo + + # tar -tvvvf test.tar + -rw-r--r-- root/root 4 2024-03-08 09:41 foo + hrw-r--r-- root/root 0 2024-03-08 09:41 foo link to foo + + # mkdir untartest + + # cd untartest/ + + # tar -xvvvf ../test.tar + -rw-r--r-- root/root 4 2024-03-08 09:41 foo + hrw-r--r-- root/root 0 2024-03-08 09:41 foo link to foo + + # ls -l + total 4 + -rw-r--r-- 1 root root 4 Mar 8 09:41 foo + +So perhaps only mounted btrfs snapshot subvolumes +added to 'tar' arcives cause real problems in practice. +In this case btrfs snapshot subvolumes should be excluded +by default from being added to what 'tar' should archive. + +I think it is OK if a user mounts the same stuff +at different mount points and ReaR includes those +mount points by default in the 'tar' backup +then it is up to the user to manually exclude things +as needed from his backup. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-08 08:59](https://github.com/rear/rear/pull/3175#issuecomment-1985306750): + +@jsmeix + +> I fear btrfs normal subvolumes versus btrfs snapshot subvolumes is +> only one example of a very generic problem when by default every +> mounted "thingy" is included in the 'tar' backup: +> +> I think it is in general possible that one same "mountable thingy" can +> be mounted at the same time at different mount points + +I disagree that it is an example of this problem. Snapshots are not the +same thing mounted at different places. They are different things +mounted at different places - snapshots exist because their content is +(at least in principle) different. + +One filesystem mounted at more places can occur as well, and it will +result in an explosion of backup data, but it restoring it twice then +should not result in an increase of the size of the restored system, +only in a slower restore, because you keep restoring to the same +filesystem. + +I would not try to solve these two problems in the same way (cf. RFC +1925 item 5). + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-08 09:51](https://github.com/rear/rear/pull/3175#issuecomment-1985382738): + +> It seems 'tar' behaves well forgiving in this case: +> +> # mkdir test +> +> # cd test +> +> # echo foo >foo +> +> # tar -cvvvf test.tar foo foo +> -rw-r--r-- root/root 4 2024-03-08 09:41 foo +> hrw-r--r-- root/root 0 2024-03-08 09:41 foo link to foo +> +> # tar -tvvvf test.tar +> -rw-r--r-- root/root 4 2024-03-08 09:41 foo +> hrw-r--r-- root/root 0 2024-03-08 09:41 foo link to foo +> +> # mkdir untartest +> +> # cd untartest/ +> +> # tar -xvvvf ../test.tar +> -rw-r--r-- root/root 4 2024-03-08 09:41 foo +> hrw-r--r-- root/root 0 2024-03-08 09:41 foo link to foo +> +> # ls -l +> total 4 +> -rw-r--r-- 1 root root 4 Mar 8 09:41 foo + +It thinks that the doubled file names are different names for the same +files (i.e. hardlinks), which is not entirely correct - not sure if it +can have some unwanted consequences or not. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-08 12:16](https://github.com/rear/rear/pull/3175#issuecomment-1985593096): + +Yes. + +The whole point of my experiments with 'tar' here +is to find out if my "fear" above is true or not and +in general to better understand what we have to deal with. + +If it is actually only one root problem +then this root problem should be solved +(instead of solving each of its instances). + +If it is actually several separated problems then +each separated problem should be solved separately. + +[https://github.com/rear/rear/pull/3175\#issuecomment-1985290555](https://github.com/rear/rear/pull/3175#issuecomment-1985290555) +indicates that it is several separated problems +(but this is only my very first test in this area). + +From my experiments with 'tar' in the past I know that +'tar' behaves deterministically (i.e. as programmed and +documented when reading the whole 'tar' manual carefully) +but that could appear rather often 'unexpectedly' +(i.e. different than what one may expect offhandedly), e.g. +[https://github.com/rear/rear/issues/2911\#issuecomment-1398346148](https://github.com/rear/rear/issues/2911#issuecomment-1398346148) + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-08 12:45](https://github.com/rear/rear/pull/3175#issuecomment-1985632419): + +> If it is actually only one root problem then this root problem should +> be solved (instead of solving each of its instances). +> +> If it is actually several separated problems then each separated +> problem should be solved separatedly. + +That's an interesting idea. For multiple identical arguments to `tar`, +tar duplicates the backup and considers the secondary copy as a hardlink +to the first copy. I checked that the same happens when there is a +filesystem mounted multiple times: + + # mkdir /mnt/mount1 + # mkdir /mnt/mount2 + # mount /dev/vdb /mnt/mount1 + # mount /dev/vdb /mnt/mount2 + # touch /mnt/mount1/foo + # ls -l /mnt/mount2/foo + -rwxr-xr-x. 1 root root 0 Mar 8 07:36 /mnt/mount2/foo + # tar cvvf /dev/null /mnt/mount1 /mnt/mount2 + tar: Removing leading `/' from member names + drwxr-xr-x root/root 0 1969-12-31 19:00 /mnt/mount1/ + -rwxr-xr-x root/root 0 2024-03-08 07:36 /mnt/mount1/foo + tar: Removing leading `/' from hard link targets + drwxr-xr-x root/root 0 1969-12-31 19:00 /mnt/mount2/ + hrwxr-xr-x root/root 0 2024-03-08 07:36 /mnt/mount2/foo link to mnt/mount1/foo + +Is the same happening with different btrfs snapshots mounted at +different mountpoints? I.e. does tar consider files in different +snapshots (originally same, but possibly different when they have been +modified since the snapshot was taken) as hardlinks to the same file? + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-03-08 12:48](https://github.com/rear/rear/pull/3175#issuecomment-1985635686): + +Thank you for the feedback, @jsmeix! I'll amend the code to skip backup +of all mounted btrfs snapshot subvolumes. + +The duplication of files in backup when a filesystem/btrfs subvolume is +mounted more than once is a different (though related) issue, therefore, +I suggest to resolve it separately. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-08 15:35](https://github.com/rear/rear/pull/3175#issuecomment-1985907542): + +I tested it with SLES15-SP5 +with the default btrfs structure +on a QEMU/KVM test VM: + + # lsblk -ipo NAME,TRAN,TYPE,FSTYPE,SIZE,MOUNTPOINTS + NAME TRAN TYPE FSTYPE SIZE MOUNTPOINTS + /dev/sda ata disk 15G + |-/dev/sda1 part 8M + |-/dev/sda2 part btrfs 13G /var + | /usr/local + | /root + | /tmp + | /srv + | /boot/grub2/i386-pc + | /opt + | /home + | /boot/grub2/x86_64-efi + | /.snapshots + | / + `-/dev/sda3 part swap 2G [SWAP] + +I was in particular interested how things behave +with the "well known" (to SLES users) SUSE specific + + BACKUP_PROG_INCLUDE=( $( findmnt -n -r -o TARGET -t btrfs | grep -v '^/$' | egrep -v 'snapshots|crash' ) ) + +manual setting in etc/rear/local.conf +cf. conf/examples/SLE12-SP2-btrfs-example.conf +so I have + + OUTPUT=ISO + BACKUP=NETFS + BACKUP_OPTIONS="nfsvers=3,nolock" + BACKUP_URL=nfs://192.168.178.66/nfs + REQUIRED_PROGS+=( snapper chattr ) + PROGS+=( lsattr ) + COPY_AS_IS+=( /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default ) + BACKUP_PROG_INCLUDE=( /boot/grub2/i386-pc /boot/grub2/x86_64-efi /home /opt /root /srv /tmp /usr/local /var ) + +With that I got duplicated things in the backup.tar.gz + +To make ReaR behave backward compatible for SLES users +and because it seems to be "the right thing" in general +I implemented +[https://github.com/rear/rear/pull/3177](https://github.com/rear/rear/pull/3177) + +With this additional changes I get no longer +duplicated things in the backup.tar.gz +BUT +I did not yet test "rear recover". +This will be done next week. + +@lzaoral @pcahyna @rear/contributors +I wish you a relaxed and recovering weekend! + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-08 16:19](https://github.com/rear/rear/pull/3175#issuecomment-1985986432): + +@jsmeix if you have snapshots, can you please test +[https://github.com/rear/rear/pull/3175\#issuecomment-1985632419](https://github.com/rear/rear/pull/3175#issuecomment-1985632419) +: "does tar consider files in different snapshots (originally same, but +possibly different when they have been modified since the snapshot was +taken) as hardlinks to the same file?" ? + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-10 13:34](https://github.com/rear/rear/pull/3175#issuecomment-1987233565): + +> I was in particular interested how things behave with the "well known" +> (to SLES users) SUSE specific +> +> BACKUP_PROG_INCLUDE=( $( findmnt -n -r -o TARGET -t btrfs | grep -v '^/$' | egrep -v 'snapshots|crash' ) ) +> +> manual setting in etc/rear/local.conf cf. +> conf/examples/SLE12-SP2-btrfs-example.conf so I have +> +> OUTPUT=ISO +> BACKUP=NETFS +> BACKUP_OPTIONS="nfsvers=3,nolock" +> BACKUP_URL=nfs://192.168.178.66/nfs +> REQUIRED_PROGS+=( snapper chattr ) +> PROGS+=( lsattr ) +> COPY_AS_IS+=( /usr/lib/snapper/installation-helper /etc/snapper/config-templates/default ) +> BACKUP_PROG_INCLUDE=( /boot/grub2/i386-pc /boot/grub2/x86_64-efi /home /opt /root /srv /tmp /usr/local /var ) +> +> With that I got duplicated things in the backup.tar.gz + +Is it a regression with this PR, or did you get duplicated entries in +backup.tar.gz even before? What are the duplicated entries? Aren't you +missing `BACKUP_ONLY_INCLUDE="yes"`? (but then you should probably add +`/boot` to `BACKUP_PROG_INCLUDE` unless on SLES you have `/boot` as part +of `/`). + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-11 13:23](https://github.com/rear/rear/pull/3175#issuecomment-1988433347): + +> @jsmeix if you have snapshots, can you please test [\#3175 +> (comment)](https://github.com/rear/rear/pull/3175#issuecomment-1985632419) +> : "does tar consider files in different snapshots (originally same, +> but possibly different when they have been modified since the snapshot +> was taken) as hardlinks to the same file?" ? + +I tested ZFS and the same files in a snapshot and in the original +filesystem do not show up as hardlinks to the same file in the `tar` +output. Of course, although Btrfs is in many ways analogous to ZFS, it +can behave differently in details like that. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-26 16:43](https://github.com/rear/rear/pull/3175#issuecomment-2020955508): + +Hi all, reviewing what needs to be done there. + +- \[ \] what are the duplicated entries in backup.tar.gz currently? + [https://github.com/rear/rear/pull/3175\#issuecomment-1985907542](https://github.com/rear/rear/pull/3175#issuecomment-1985907542) +- \[ \] avoid `sort -u` , to be replaced by `uniq_unsorted` in \#3177 +- \[ \] test recovery with the SUSE-specific BACKUP\_PROG\_INCLUDE + setting + [https://github.com/rear/rear/pull/3175\#issuecomment-1985907542](https://github.com/rear/rear/pull/3175#issuecomment-1985907542) +- \[ \] test snapshots: does tar consider files in different snapshots + (originally same, but possibly different when they have been + modified since the snapshot was taken) as hardlinks to the same + file? + [https://github.com/rear/rear/pull/3175\#issuecomment-1985986432](https://github.com/rear/rear/pull/3175#issuecomment-1985986432) +- \[ \] exclude snapshots from backup: + [https://github.com/rear/rear/pull/3175\#issuecomment-1985635686](https://github.com/rear/rear/pull/3175#issuecomment-1985635686) + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-04-09 10:04](https://github.com/rear/rear/pull/3175#issuecomment-2044626832): + +Hi @jsmeix can you please have a look? I believe the first four items in +the checklist above are for you (the second only partially, you +implement `uniq_unsorted` in \#3177 and then @lzaoral will use it here). +Is the task list ok? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-09 12:48](https://github.com/rear/rear/pull/3175#issuecomment-2045106512): + +@pcahyna +I will have a look. + +First I would like to implement `uniq_unsorted` +or perhaps even better named `unique_unsorted` cf. +[https://github.com/rear/rear/pull/3177\#issuecomment-2045095158](https://github.com/rear/rear/pull/3177#issuecomment-2045095158) +so that @lzaoral could use it here. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-03-07.3176.pr.merged.md b/docs/issues/2024-03-07.3176.pr.merged.md new file mode 100644 index 00000000..1103838b --- /dev/null +++ b/docs/issues/2024-03-07.3176.pr.merged.md @@ -0,0 +1,160 @@ +[\#3176 PR](https://github.com/rear/rear/pull/3176) `merged`: Skip btrfs subvolumes when detecting ESP partitions +================================================================================================================= + +**Labels**: `bug`, `fixed / solved / done` + +#### [lzaoral](https://github.com/lzaoral) opened issue at [2024-03-07 16:40](https://github.com/rear/rear/pull/3176): + +##### Pull Request Details: + +- Type: **Bug Fix** + +- Impact: **Normal** + +- Reference to related issue (URL): N/A + +- How was this pull request tested? Recovery of Fedora Rawhide UEFI + machine. + +- Description of the changes in this pull request: + +The idea is to find all direct partitions that contain the ESP mount +point and to skip all other transitive `fs:` dependencies. + +The `diskdeps.conf` file contains following entries on default Fedora +installations (the list was shortened to only the relevant ones): + + /dev/vda1 /dev/vda + /dev/vda4 /dev/vda + /dev/vda5 /dev/vda + fs:/boot/efi /dev/vda1 + fs:/boot/efi fs:/boot + fs:/boot/efi fs:/ + fs:/boot/efi btrfsmountedsubvol:/ + fs:/boot /dev/vda4 + fs:/boot fs:/ + fs:/boot btrfsmountedsubvol:/ + fs:/ /dev/vda5 + btrfsmountedsubvol:/ /dev/vda5 + +The ESP partition is only on `/dev/vda1`. However, the `find_partition` +call +was not taking into account the need to skip mounted btrfs subvolumes as +well. +Therefore, `/dev/vda5` was listed as an ESP partition as well. + +This change makes sure that only direct ESP partitions are listed and +fixes a bug where ReaR would create broken BootXXXX entries which point +to +completely unrelated partitions. + +Relevant excerpts from logs: + + ++ efibootmgr --create --gpt --disk /dev/vda --part 1 --write-signature --label 'RedHatEnterpriseServer 41' --loader '\EFI\fedora\grubx64.efi' + ... + ++ efibootmgr --create --gpt --disk /dev/vda --part 5 --write-signature --label 'RedHatEnterpriseServer 41' --loader '\EFI\fedora\grubx64.efi' + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-08 12:01](https://github.com/rear/rear/pull/3176#issuecomment-1985573300): + +> disktodo.conf contains following entries + +ITYM `diskdeps.conf`, `disktodo.conf` has a different format. Please fix +also the commit message. + +> a bug where ReaR would create broken BootXXXX entries which point to +> completely unrelated partitions. + +Can you please show an example if you have saved it somewhere? + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-03-08 12:41](https://github.com/rear/rear/pull/3176#issuecomment-1985625759): + +Thank you, @pcahyna! I've fixed the typo as I meant the `diskdeps.conf` +file and amended the commit message with the creation of those +`BootXXXX` entries. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-19 14:27](https://github.com/rear/rear/pull/3176#issuecomment-2007330664): + +thank you a lot for the fix and for providing an example of the buggy +behavior! + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-20 07:40](https://github.com/rear/rear/pull/3176#issuecomment-2008958945): + +Only a side question FYI: + +I am wondering why the ESP kernel device node +is determined so indirectly. + +Do you know if there is a reason why not +something like 'lsblk' is used to get directly +the partition kernel device node +that matches a mount point? + +E.g. on a running system +(my homeoffice workstation): + + # lsblk -nrpo PKNAME,KNAME,MOUNTPOINT | grep '/boot/efi' + /dev/nvme0n1 /dev/nvme0n1p1 /boot/efi + +I don't use the SUSE btrfs structure +so I don't have mounted btrfs subvolumes. + +I assume 'lsblk' cannot be used here because +we are in the recovery system (in 'finalize' stage) +and need the ESP partition of the recreated system +below /mnt/local where the ESP is perhaps not mounted +so we need to determine it from the diskdeps.conf file? + +Or why not directly from disklayout.conf? + +E.g. on my homeoffice workstation (excerpt): + + # Format: part /dev/ + part /dev/nvme0n1 535822336 1048576 rear-noname boot,esp /dev/nvme0n1p1 + +I.e. why indirectly via the ESP mount point +than directly the ESP 'part' entry in disklayout.conf? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-20 07:41](https://github.com/rear/rear/pull/3176#issuecomment-2008960671): + +@lzaoral @pcahyna +don't worry about my questions. +Feel free to merge as you like (I approved it "as is"). + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-03-20 08:59](https://github.com/rear/rear/pull/3176#issuecomment-2009068478): + +@jsmeix I'll add the comment explaining why the exclusions are required. + +> I.e. why indirectly via the ESP mount point than directly the ESP +> 'part' entry in disklayout.conf? + +AFAIK, it is necessary to create boot entry for every physical device +present in the RAID array which contains the ESP: + +- [https://bugzilla.redhat.com/show\_bug.cgi?id=1958222](https://bugzilla.redhat.com/show_bug.cgi?id=1958222) +- [https://github.com/rear/rear/pull/2608](https://github.com/rear/rear/pull/2608) + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-03-20 11:23](https://github.com/rear/rear/pull/3176#issuecomment-2009341775): + +@pcahyna @jsmeix I've rebased the PR and added the explanation comment. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-20 13:48](https://github.com/rear/rear/pull/3176#issuecomment-2009611532): + +@jsmeix I looked at the places that determine the ESP filesystem / +device and it seems like a mess to me (several different methods are +used), but I don't have enough motivation to fix this as the code +"mostly works" now (I suspect it might have some problems when multipath +is used though) so I am going to stop investigating and merge the code +as it is. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-20 14:57](https://github.com/rear/rear/pull/3176#issuecomment-2009775158): + +@pcahyna +yes, +feel free to merge when it is OK for you. +We can only improve things step by step +as far as possible with reasonable effort. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-03-08.3177.pr.merged.md b/docs/issues/2024-03-08.3177.pr.merged.md new file mode 100644 index 00000000..e96a08c4 --- /dev/null +++ b/docs/issues/2024-03-08.3177.pr.merged.md @@ -0,0 +1,198 @@ +[\#3177 PR](https://github.com/rear/rear/pull/3177) `merged`: New unique\_unsorted() function +============================================================================================= + +**Labels**: `enhancement`, `fixed / solved / done` + +#### [jsmeix](https://github.com/jsmeix) opened issue at [2024-03-08 15:11](https://github.com/rear/rear/pull/3177): + +- Type: **Enhancement** + +- Impact: **High** + High impact where needed. + No impact esewhere. + +- Reference to related issue (URL): + +Triggred by my experiments in +[https://github.com/rear/rear/pull/3175](https://github.com/rear/rear/pull/3175) +how 'tar' behaves when exact same files or directories +are provided as 'tar' arguments to be archived. + +- How was this pull request tested? + +See +[https://github.com/rear/rear/pull/3175\#issuecomment-1985907542](https://github.com/rear/rear/pull/3175#issuecomment-1985907542) +But I did not yet test "rear recover". + +- Description of the changes in this pull request: + +In lib/global-functions.sh added a +new unique\_unsorted() function +that outputs lines in STDIN or in a file +without subsequent duplicate lines +which keeps the ordering of the lines. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-08 15:18](https://github.com/rear/rear/pull/3177#issuecomment-1985879052): + +@jsmeix there is `sort -u` added in \#3175 , isn't that enough? Should +be, unless we require preserving the order - do we? To us (me and +@lzaoral ) it seemed that the order is quite unpredictable / random +anyway. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-08 15:46](https://github.com/rear/rear/pull/3177#issuecomment-1985926458): + +I want to preserve the ordering to be on the safe side. + +I have no real-world use-case why the ordering is important. + +It is just my gut feeling that in some special cases +the ordering could be crucial because for 'tar' +the ordering of the arguments does make a difference +(what is stored where in the archive) and in some special +cases that could be crucial. + +Only what comes to my mind offhandedly right now: + + # tar -czf fullbackup.tgz / /database + +might behave rather different during restore compared to + + # tar -czf fullbackup.tgz /database / + +(under the assumption that /database is on its own filesystem) +i.e. +first the basic system gets restored (some Gigabytes), +then the database gets restored (some Terabytes) +versus +first the the database gets restored (some Terabytes), +then the basic system gets restored (some Gigabytes) + +As an exercise for the reader: +For both cases estimate the probability +that restore of the basic system fails +;-) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-08 15:52](https://github.com/rear/rear/pull/3177#issuecomment-1985937521): + +Sigh! +I guess it is ReaR's careless code why +the order is quite unpredictable / random anyway +when one leaves such things to ReaR's automatism +and that is one reason behind why I implemented +BACKUP\_ONLY\_INCLUDE and BACKUP\_ONLY\_EXCLUDE +to provide "final power to the user". + +So + + BACKUP_ONLY_INCLUDE="yes" + BACKUP_PROG_INCLUDE=( / /database ) + +versus + + BACKUP_ONLY_INCLUDE="yes" + BACKUP_PROG_INCLUDE=( /database / ) + +(under the assumption that /database is on its own filesystem) +should behave rather different during restore +but I did not test it - something to do for next week. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-08 16:11](https://github.com/rear/rear/pull/3177#issuecomment-1985972348): + +I avoid false expectations about this pull request: +This one does not help when when a filesystem/btrfs subvolume +is mounted more than once at different mountpoint directories +because this one only skips exactly same duplicated arguments +what 'tar' and 'rsync' should archive so this pull request +is meant only to help when by accident exactly same arguments +are provided more than once to 'tar' and 'rsync'. + +I think it can never be needed that same arguments +are provided more than once to 'tar' and 'rsync' +so I think it is (hopefully) safe to always ignore +exactly same duplicated arguments. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-08 16:15](https://github.com/rear/rear/pull/3177#issuecomment-1985979251): + +@jsmeix I would characterize it as a way to avoid `sort -u` in +[https://github.com/rear/rear/pull/3175](https://github.com/rear/rear/pull/3175) +and thus preserve the order of tar arguments in cases where the order +matters ( e.g. explicit `BACKUP_PROG_INCLUDE` setting ). Correct me if I +am wrong. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-08 16:32](https://github.com/rear/rear/pull/3177#issuecomment-1986013000): + +Great @schlomo I had independently exactly the same idea for the +function's name! + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-11 06:57](https://github.com/rear/rear/pull/3177#issuecomment-1987742609): + +Thank you for the better function name! +I will use `uniq_unsorted` + +I was thinking about a shorter function name +that still clearly tells what that function does +but had no better idea than the long verbose name. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-09 12:42](https://github.com/rear/rear/pull/3177#issuecomment-2045095158): + +I am wondering if an even better function name +could be `unique_unsorted` with spelled out 'unique' +to make it more clear that this function +is something "on its own" which is in particular +not a wrapper or some enhanced reimplementation +of the `uniq` program (with all its options)? + +And in general I don't like old \*nix style +crppld wrds lk 'uniq\_usrtd' ;-) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-11 09:23](https://github.com/rear/rear/pull/3177#issuecomment-2049280190): + +For now I kept the if check for a filename argument +because in the STDIN case I like to timeout after 5 seconds +when there is nothing at STDIN or when STDIN does not get closed +to avoid that 'awk' waits endlessly for (further) input +which would make ReaR behave as if it hung up, cf. +[https://github.com/rear/rear/wiki/Coding-Style\#beware-of-the-emptiness](https://github.com/rear/rear/wiki/Coding-Style#beware-of-the-emptiness) + +@schlomo +I assume all this can be implemented in a single case +but I would need time to find out how to do that properly. +If a proper implementation in a single case +is obvious to you, please show it to me here. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-12 12:35](https://github.com/rear/rear/pull/3177#issuecomment-2051679239): + +@rear/contributors +if there are no objections I will merge it today +a bit after 15:00 CEST (13:00 UTC). + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-12 12:46](https://github.com/rear/rear/pull/3177#issuecomment-2051694960): + +Looks good to me. + +Maybe silly question: + +`awk "..." "$@"` doesn't work? It would save you the if clause + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-12 13:31](https://github.com/rear/rear/pull/3177#issuecomment-2051770541): + +@schlomo +I merged it "as is" because I had tested that at least a bit +and - at least currently - I don't find time to test a rather +different implementation. + +Furthermore I would have to think about +if a check if stdin is a tty makes sense, cf. +[https://github.com/rear/rear/pull/3177\#discussion\_r1562435947](https://github.com/rear/rear/pull/3177#discussion_r1562435947) +and if yes if then it helps to have the stdin case +well separated from the "input from file" case, +cf. RFC 1925 item (5) + + It is always possible to aglutenate [sic] multiple separate problems + into a single complex interdependent solution. In most cases + this is a bad idea. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-03-12.3178.issue.open.md b/docs/issues/2024-03-12.3178.issue.open.md new file mode 100644 index 00000000..6e292ff3 --- /dev/null +++ b/docs/issues/2024-03-12.3178.issue.open.md @@ -0,0 +1,327 @@ +[\#3178 Issue](https://github.com/rear/rear/issues/3178) `open`: rear does not recognize nvme when trying to format it with "rear format" +========================================================================================================================================= + +**Labels**: `enhancement` + +#### [xwhitebeltx](https://github.com/xwhitebeltx) opened issue at [2024-03-12 15:22](https://github.com/rear/rear/issues/3178): + + + +- ReaR version ("/usr/sbin/rear -V"): + + + + Relax-and-Recover 2.7-git.5396.573f7f01.master / 2024-03-08 + +- If your ReaR version is not the current version, explain why you + can't upgrade: + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): + + + + PRETTY_NAME="Ubuntu 22.04.4 LTS" + NAME="Ubuntu" + VERSION_ID="22.04" + VERSION="22.04.4 LTS (Jammy Jellyfish)" + VERSION_CODENAME=jammy + ID=ubuntu + ID_LIKE=debian + HOME_URL="https://www.ubuntu.com/" + SUPPORT_URL="https://help.ubuntu.com/" + BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" + PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" + UBUNTU_CODENAME=jammy + +- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat + /etc/rear/local.conf"): + + + + OUTPUT=USB + BACKUP=NETFS + BACKUP_URL=usb:///dev/disk/by-label/REAR-000 + ONLY_INCLUDE_VG=( "system" ) + USB_UEFI_PART_SIZE="1000" + KERNEL_CMDLINE="noip unattended" + USER_INPUT_TIMEOUT=15 + USB_RETAIN_BACKUP_NR=1 + COPY_AS_IS+=( /usr/local/bin ) + +- Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM + guest or PowerVM LPAR): + + + + HP ProLiant DL385 Gen11 + +- System architecture (x86 compatible or PPC64/PPC64LE or what exact + ARM device): + + + + x86 compatible + +- Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or + ELILO or Petitboot): + + + + UEFI + GRUB + +- Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or + multipath (DM or NVMe): + + + + 2xlocal Micron 960GB NVMe SSD + +- Storage layout ("lsblk -ipo + NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"): + + + + NAME KNAME PKNAME TRAN TYPE FSTYPE LABEL SIZE MOUNTPOINT + /dev/loop0 /dev/loop0 loop squashfs 63.3M /snap/core20/1879 + /dev/loop1 /dev/loop1 loop squashfs 63.9M /snap/core20/2182 + /dev/loop2 /dev/loop2 loop squashfs 111.9M /snap/lxd/24322 + /dev/loop3 /dev/loop3 loop squashfs 87M /snap/lxd/27428 + /dev/loop4 /dev/loop4 loop squashfs 70.2M /snap/powershell/264 + /dev/loop5 /dev/loop5 loop squashfs 53.2M /snap/snapd/19122 + /dev/loop6 /dev/loop6 loop squashfs 39.1M /snap/snapd/21184 + /dev/nvme1n1 /dev/nvme1n1 nvme disk 894.3G + |-/dev/nvme1n1p1 /dev/nvme1n1p1 /dev/nvme1n1 nvme part vfat 1G /boot/efi + |-/dev/nvme1n1p2 /dev/nvme1n1p2 /dev/nvme1n1 nvme part ext4 2G /boot + `-/dev/nvme1n1p3 /dev/nvme1n1p3 /dev/nvme1n1 nvme part LVM2_member 891.2G + `-/dev/mapper/system-root /dev/dm-0 /dev/nvme1n1p3 lvm ext4 891.2G / + /dev/nvme0n1 /dev/nvme0n1 nvme disk ext4 894.3G + +- Description of the issue (ideally so that others can reproduce it): + + + + executing "rear -v format -- --efi /dev/nvme0n1" returns this error: + + Relax-and-Recover 2.7-git.5396.573f7f01.master / 2024-03-08 + Running rear format (PID 24112 date 2024-03-12 15:15:12) + Using log file: /var/log/rear/rear-avpukr.log + Running workflow format on the normal/original system + ERROR: + ==================== + BUG in /usr/share/rear/format/USB/default/200_check_usb_layout.sh line 28: + Unable to determine raw device for /dev/nvme0n1 + -------------------- + Please report it at https://github.com/rear/rear/issues + and include all related parts from /var/log/rear/rear-avpukr.log + preferably the whole debug information via 'rear -D format' + ==================== + Use debug mode '-d' for some debug messages or debugscript mode '-D' for full debug messages with 'set -x' output + Aborting due to an error, check /var/log/rear/rear-avpukr.log for details + Exiting rear format (PID 24112) and its descendant processes ... + Running exit tasks + Terminated + +- Workaround, if any: + + + + nothing that i could find + +- Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" + debug log files): + + + + [rear-avpukr.log](https://github.com/rear/rear/files/14574841/rear-avpukr.log) + +You can drag-drop log files into this editor to create an attachment +or paste verbatim text like command output or file content +by including it between a leading and a closing line of +three backticks like this: + + verbatim content + +#### [gdha](https://github.com/gdha) commented at [2024-03-18 16:46](https://github.com/rear/rear/issues/3178#issuecomment-2004430078): + +@xwhitebeltx Why do you want to treat a Micron 960GB NVMe SSD as an USB +device? I guess you want to make a bootable UEFI device? Remember, a +local SSD device is **not** safe to keep ReaR archives on. +@rear/contributors This is something we haven't foreseen so far (I +think, but could be wrong \[again\])? + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-18 16:53](https://github.com/rear/rear/issues/3178#issuecomment-2004444217): + +@gdha in real use this is indeed unusual. For testing it is useful +though to treat NVMe as any other disk device - you can take a machine +with a secondary NVMe device, make a backup to it as if it were USB, and +boot from it for recovery. +(yes, it is more realistic to test on a machine with a secondary USB +disk, or at least a SCSI or SATA disk since it also presents itself as a +`sd*` block device, but a machine with NVMe may be the only thing you +have) + +#### [schlomo](https://github.com/schlomo) commented at [2024-03-18 17:07](https://github.com/rear/rear/issues/3178#issuecomment-2004477308): + +Any thoughts why we can't use `find_device` here? +[https://github.com/rear/rear/blob/2073e77f1ed213653bbf846d67bef346a9f65da9/usr/share/rear/format/USB/default/200\_check\_usb\_layout.sh\#L12-L13](https://github.com/rear/rear/blob/2073e77f1ed213653bbf846d67bef346a9f65da9/usr/share/rear/format/USB/default/200_check_usb_layout.sh#L12-L13) + +It seems to me like our code simply doesn't handle the nvme use case +yet, it assumes a sysfs device path like +`/devices/pci0000:00/0000:00:10.0/host2/target2:0:1/2:0:1:0/block/sdb/sdb1` +whereas in your example we have +`devices/pci0000:c0/0000:c0:03.3/0000:c5:00.0/nvme/nvme0/nvme0n1` + +@xwhitebeltx have you tried creating a partition and specifying that +instead of the whole disk? + +#### [xwhitebeltx](https://github.com/xwhitebeltx) commented at [2024-03-18 17:52](https://github.com/rear/rear/issues/3178#issuecomment-2004573795): + +thank you all for your responses! + +@gdha - we do as @pcahyna suggested, i have 2 local NVMes: + +1. OS +2. backup drive i can boot from and restore the OS back to its original + state + in this case the main purpose is not backup of critical data, it is + to be able to "reset" the server anytime we want with an automated + boot + restore process, in places where PXE is banned. + +@schlomo - it works fine when the SSD is SATA, first time i'm trying +that on an NVMe, +i have not tried creating a partition, i'll try tomorrow morning and +update =) + +#### [xwhitebeltx](https://github.com/xwhitebeltx) commented at [2024-03-19 07:33](https://github.com/rear/rear/issues/3178#issuecomment-2006106386): + +well, it worked but it ignored the partition(which is totally good for +me), the initial state was this: + + NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS + loop0 7:0 0 63.3M 1 loop /snap/core20/1879 + loop1 7:1 0 63.9M 1 loop /snap/core20/2182 + loop2 7:2 0 111.9M 1 loop /snap/lxd/24322 + loop3 7:3 0 87M 1 loop /snap/lxd/27428 + loop4 7:4 0 70.2M 1 loop /snap/powershell/264 + loop5 7:5 0 53.2M 1 loop /snap/snapd/19122 + loop6 7:6 0 39.1M 1 loop /snap/snapd/21184 + nvme1n1 259:0 0 894.3G 0 disk + ├─nvme1n1p1 259:2 0 1G 0 part /boot/efi + ├─nvme1n1p2 259:3 0 2G 0 part /boot + └─nvme1n1p3 259:4 0 891.2G 0 part + └─system-root 253:0 0 891.2G 0 lvm / + nvme0n1 259:1 0 894.3G 0 disk + └─nvme0n1p1 259:6 0 894.3G 0 part + +after i executed "rear -v format -- --efi /dev/nvme0n1p1" i got this +output: + + root@avpukr:~# rear -v format -- --efi /dev/nvme0n1p1 + Relax-and-Recover 2.7-git.5396.573f7f01.master / 2024-03-08 + Running rear format (PID 222220 date 2024-03-19 06:56:31) + Using log file: /var/log/rear/rear-avpukr.log + Running workflow format on the normal/original system + USB or disk device /dev/nvme0n1p1 is not formatted with ext2/3/4 or btrfs filesystem + Formatting /dev/nvme0n1p1 will remove all currently existing data on that whole device + Type exactly 'Yes' to format /dev/nvme0n1p1 with ext3 filesystem + (default 'No' timeout 15 seconds) + Yes + Repartitioning /dev/nvme0n1 + Creating partition table of type gpt on /dev/nvme0n1 + Making an EFI bootable device /dev/nvme0n1 + Creating EFI system partition /dev/nvme0n11 with size 1000 MiB aligned at 8 MiB + Setting 'esp' flag on EFI partition /dev/nvme0n11 + Creating ReaR data partition /dev/nvme0n12 up to 100% of /dev/nvme0n1 + Setting 'legacy_boot' flag on ReaR data partition /dev/nvme0n12 + Creating vfat filesystem on EFI system partition on /dev/nvme0n1p1 + Creating ext3 filesystem with label 'REAR-000' on ReaR data partition /dev/nvme0n1p2 + Adjusting filesystem parameters on ReaR data partition /dev/nvme0n1p2 + Exiting rear format (PID 222220) and its descendant processes ... + Running exit tasks + +and then the disk structure is as following: + + NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS + loop0 7:0 0 63.3M 1 loop /snap/core20/1879 + loop1 7:1 0 63.9M 1 loop /snap/core20/2182 + loop2 7:2 0 111.9M 1 loop /snap/lxd/24322 + loop3 7:3 0 87M 1 loop /snap/lxd/27428 + loop4 7:4 0 70.2M 1 loop /snap/powershell/264 + loop5 7:5 0 53.2M 1 loop /snap/snapd/19122 + loop6 7:6 0 39.1M 1 loop /snap/snapd/21184 + nvme1n1 259:0 0 894.3G 0 disk + ├─nvme1n1p1 259:2 0 1G 0 part /boot/efi + ├─nvme1n1p2 259:3 0 2G 0 part /boot + └─nvme1n1p3 259:4 0 891.2G 0 part + └─system-root 253:0 0 891.2G 0 lvm / + nvme0n1 259:1 0 894.3G 0 disk + ├─nvme0n1p1 259:5 0 1000M 0 part + └─nvme0n1p2 259:6 0 893.3G 0 part + +i've performed a backup with "rear -v mkbackup" +and i've restored it successfully! + +there's one unexpected screen which i'm not sure yet what's happening +there and if it's related to rear or not, +actions i took: + +1. reboot +2. F11 for boot menu +3. choose the REAR-000 NVMe Drive +4. this screen pops up: + ![image](https://github.com/rear/rear/assets/100077488/56120363-4939-4226-b738-9a97c98a34b1) + and then the rear grub menu, from there it all seems to be working + fine + +thanks for the workaround! is this something you would consider +implementing in future releases? treating NVMe devices as USB? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-19 08:22](https://github.com/rear/rear/issues/3178#issuecomment-2006263503): + +@xwhitebeltx +that "unexpected screen" was implemeted by me via +[https://github.com/rear/rear/commit/a0a6429119bc284fcf93775bd579bcd74d1c3b40](https://github.com/rear/rear/commit/a0a6429119bc284fcf93775bd579bcd74d1c3b40) +see also +[https://github.com/rear/rear/pull/3025\#issuecomment-1634068353](https://github.com/rear/rear/pull/3025#issuecomment-1634068353) + +Actually there is no new screen. +It is a timeout delay on the normal initial GRUB screen +(without timeout it shows up only for a fraction of a second) +before GRUB replaces it with its boot menu screen. +With the timeout delay you can see and even read +what GRUB shows (in particular possible error messages) +which helps to understand why GRUB fails when it fails +(e.g. it fails when a wrong 'root' device is used). + +#### [xwhitebeltx](https://github.com/xwhitebeltx) commented at [2024-03-19 09:32](https://github.com/rear/rear/issues/3178#issuecomment-2006502421): + +@jsmeix oh, nice =) +thank you very much, +i'm sorry for the silly question but this is my first time reporting a +bug in github, +please tell me should i close it as resolved (workaround provided) or do +you want to leave it open until it is fixed? +what is the proper conduct? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-19 09:38](https://github.com/rear/rear/issues/3178#issuecomment-2006524899): + +@xwhitebeltx +don't worry how you may close it "correctly". +Leave that "problem" to us. + +Personally I would like to fix it but unfortunately +too often I don't find the needed time for it, +too much other stuff elsewhere :-( +I would like to keep it open for some time, +hope dies last :-) + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-03-14.3179.pr.open.md b/docs/issues/2024-03-14.3179.pr.open.md new file mode 100644 index 00000000..e1af880e --- /dev/null +++ b/docs/issues/2024-03-14.3179.pr.open.md @@ -0,0 +1,149 @@ +[\#3179 PR](https://github.com/rear/rear/pull/3179) `open`: Migrate MAC addresses and interface names in NetworkManager keyfiles during network configuration migration +======================================================================================================================================================================= + +**Labels**: `enhancement` + +#### [pcahyna](https://github.com/pcahyna) opened issue at [2024-03-14 08:07](https://github.com/rear/rear/pull/3179): + +##### Pull Request Details: + +- Type: **Enhancement** + +- Impact: **Normal** + +- Reference to related issue (URL): + +- How was this pull request tested? + Migration of Fedora Linux system to different hardware (different + NICs). With this patch, the network connections are preserved and + functional after recovery, even though the NICs have different names + and different MAC adresses. + +- Description of the changes in this pull request: + Migrate NM keyfiles during network conf migration. See + [https://fedoramagazine.org/converting-networkmanager-from-ifcfg-to-keyfiles/](https://fedoramagazine.org/converting-networkmanager-from-ifcfg-to-keyfiles/) + for more details on NetworkManager keyfiles. + Only MAC addresses and interface names are migrated for now. + TODO: migrate also IP addresses and routes. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-14 22:33](https://github.com/rear/rear/pull/3179#issuecomment-1998592244): + +Regarding "TODO: migrate also IP addresses and routes." + +Is it worth supporting migration of IP addresses and routes? What is the +use case? I suppose it is for machine cloning to several not completely +identical (different network address) copies, as I don't see how it is +useful for the typical use of ReaR (disaster and recovery) where you +need an identical address after recovery. Would it be acceptable to +remove this feature (say in ReaR 3.0)? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-15 07:36](https://github.com/rear/rear/pull/3179#issuecomment-1999090980): + +@pcahyna +offhandedly I don't know what the use case is +for migration of IP addresses and routes. + +My guess is similar as yours that at some time in the past +someone (mis?)used ReaR for migration of networking setup? + +Perhaps (also only a guess) the idea behind is to use ReaR +to migrate a system from one environment into a different +environment, for example from bare metal hardware in one +network environment to a VM in another network environment +or something like that? + +On the one hand ReaR has too many (often half-baked) features +that are almost never used in practice (and bit-rotting) +so I would like to drop corner-case features that are not +sufficiently well implemented and basically unmaintained +at ReaR upstream because it is a nightmare to maintain +when customers have issues with such features. + +On the other hand such features can be a starting point for +further development to get them sufficiently well implemented +so such features could become business opportunities when +customers want to pay for further development. + +I think the solution out of this conflict is +deprecation via `ErrorIfDeprecated()`. + +This way we learn via somewhat enforced user feedback +which of those features are actually in use and +we can ask back those users what their use-case is +so we better understand those corner-case features +and then we can make an informed decision +what to do with each individual corner-case feature +(drop it, keep it as is, further develop it). + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-15 07:58](https://github.com/rear/rear/pull/3179#issuecomment-1999117329): + +@pcahyna +could you by the way show $network\_config\_file values +in user output messages with `'` to make it clear +when a single value consists of more than one word. + +As far as I see it is those two code places + + Log "Wrote new MAC addresses and network interfaces in $network_config_file" + ... + LogPrintError "Failed to rewrite MAC addresses and network interfaces in $network_config_file" + +that should be enhanced to + + Log "Wrote new MAC addresses and network interfaces in '$network_config_file'" + ... + LogPrintError "Failed to rewrite MAC addresses and network interfaces in '$network_config_file'" + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-15 13:58](https://github.com/rear/rear/pull/3179#issuecomment-1999726493): + +> My guess is similar as yours that at some time in the past someone +> (mis?)used ReaR for migration of networking setup? +> +> Perhaps (also only a guess) the idea behind is to use ReaR to migrate +> a system from one environment into a different environment, for +> example from bare metal hardware in one network environment to a VM in +> another network environment or something like that? + +I supposed the idea was to use ReaR for cloning, i.e. backup a machine +once and then restore it to multiple instances, each one will then need +a different IP address. It seems though that according to the commit +message of 844d50b75ac4b7722f4fee7a5ee3350b93f3adb7 by @schlomo where +this functionality was introduced, they had indeed rather the idea of +"migrating bare metal hardware in one network environment to a VM in +another network environment" in mind. + +#### [schlomo](https://github.com/schlomo) commented at [2024-03-15 14:11](https://github.com/rear/rear/pull/3179#issuecomment-1999755249): + +Oh yes, that was the first version of migration support developed for +P2V. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-15 16:39](https://github.com/rear/rear/pull/3179#issuecomment-2000041814): + +> @pcahyna could you by the way show $network\_config\_file values in +> user output messages with `'` to make it clear when a single value +> consists of more than one word. +> +> As far as I see it is those two code places +> +> Log "Wrote new MAC addresses and network interfaces in $network_config_file" +> ... +> LogPrintError "Failed to rewrite MAC addresses and network interfaces in $network_config_file" +> +> that should be enhanced to +> +> Log "Wrote new MAC addresses and network interfaces in '$network_config_file'" +> ... +> LogPrintError "Failed to rewrite MAC addresses and network interfaces in '$network_config_file'" + +@jsmeix sure, 5ea58a7f91d80aa2dbe4da16802ab2a8c46b28a5 + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-15 16:41](https://github.com/rear/rear/pull/3179#issuecomment-2000044704): + +I now pushed another change, 5ea58a7f91d80aa2dbe4da16802ab2a8c46b28a5, +for a proper test for an empty array. Unfortunately, the code is now +untested, as I have not tested with this change. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-03-18.3180.issue.closed.md b/docs/issues/2024-03-18.3180.issue.closed.md new file mode 100644 index 00000000..186f2778 --- /dev/null +++ b/docs/issues/2024-03-18.3180.issue.closed.md @@ -0,0 +1,190 @@ +[\#3180 Issue](https://github.com/rear/rear/issues/3180) `closed`: Scaring Error() exit in sbin/rear when unsuccessful mktmp +============================================================================================================================ + +**Labels**: `bug`, `fixed / solved / done`, `blocker` + +#### [jsmeix](https://github.com/jsmeix) opened issue at [2024-03-18 14:03](https://github.com/rear/rear/issues/3180): + +- ReaR version ("/usr/sbin/rear -V"): + current GitHub master code + +- Description of the issue (ideally so that others can reproduce it): + +One method to reporoduce it is to e.g. + + # export TMPDIR=QQQ + +where the directory QQQ does not exists. + +On a SLES15-SP5 test VM with default btrfs structure +I get the following excerpts on the terminal +(i.e. also the 'set-x' output is on the terminal +because it `Could not create build area ''`): + + # usr/sbin/rear -D mkrescue + tac: failed to create temporary file in 'QQQ': No such file or directory + tac: failed to create temporary file in 'QQQ': No such file or directory + mktemp: failed to create directory via template 'QQQ/rear.XXXXXXXXXXXXXXX': No such file or directory + ERROR: Could not create build area '' + ===== Stack trace ===== + Trace 0: usr/sbin/rear:426 main + === End stack trace === + ++ LogPrint 'Error exit of rear mkrescue (PID 1802) and its descendant processes' + ... + Running exit tasks + ... + ++ Log 'Exit task '\''cleanup_build_area_and_end_program'\''' + ... + +++ Log 'Removing build area ' + ... + +++ rm -Rf --one-file-system + ... + rmdir: removing directory, '' + rmdir: failed to remove '': No such file or directory + ... + Could not remove build area (something still exists therein) + ... + +++ mounted_in_BUILD_DIR=' proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) + sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) + devtmpfs on /dev type devtmpfs (rw,nosuid,size=4096k,nr_inodes=1048576,mode=755,inode64) + ... + /dev/sda2 on / type btrfs (rw,relatime,space_cache,subvolid=268,subvol=/@/.snapshots/1/snapshot) + ... + /dev/sda2 on /.snapshots type btrfs (rw,relatime,space_cache,subvolid=267,subvol=/@/.snapshots) + /dev/sda2 on /boot/grub2/x86_64-efi type btrfs (rw,relatime,space_cache,subvolid=265,subvol=/@/boot/grub2/x86_64-efi) + /dev/sda2 on /boot/grub2/i386-pc type btrfs (rw,relatime,space_cache,subvolid=266,subvol=/@/boot/grub2/i386-pc) + /dev/sda2 on /home type btrfs (rw,relatime,space_cache,subvolid=264,subvol=/@/home) + /dev/sda2 on /opt type btrfs (rw,relatime,space_cache,subvolid=263,subvol=/@/opt) + /dev/sda2 on /root type btrfs (rw,relatime,space_cache,subvolid=262,subvol=/@/root) + /dev/sda2 on /srv type btrfs (rw,relatime,space_cache,subvolid=261,subvol=/@/srv) + /dev/sda2 on /tmp type btrfs (rw,relatime,space_cache,subvolid=260,subvol=/@/tmp) + /dev/sda2 on /usr/local type btrfs (rw,relatime,space_cache,subvolid=259,subvol=/@/usr/local) + /dev/sda2 on /var type btrfs (rw,relatime,space_cache,subvolid=258,subvol=/@/var) + tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=201216k,nr_inodes=50304,mode=700,inode64)' + ... + Something is still mounted within the build area + ... + You must manually umount it, then you could manually remove the build area + ... + To manually remove the build area use (with caution): rm -Rf --one-file-system + ... + Terminated + +The "tac: failed to create temporary file" comes from + + function get_bash_flags_and_options_commands () { + ... + set +o | tac | tr '\n' ';' + +because 'tac' needs TMPDIR when called in a pipe + + # cat /etc/os-release | tac + tac: failed to create temporary file in 'QQQ': No such file or directory + +in contrast to when 'tac' is called with a regular file +(at least for me with /usr/bin/tac from coreutils-8.32). + +I also tried what happens on that test VM +when I do what ReaR tells to "manually remove the build area": + + # rm -Rfv --one-file-system && echo y || echo n + y + + # rm -Rfvi --one-file-system && echo y || echo n + rm: missing operand + Try 'rm --help' for more information. + n + +Puh! +Basically by pure luck 'rm' does not remove anything +when there is nothing explicitly specified to be removed: + + # rm + rm: missing operand + Try 'rm --help' for more information. + +As far as I see the problem is in sbin/rear +only between successful + + source $SHARE_DIR/lib/_input-output-functions.sh + +and unsuccessful + + BUILD_DIR="$( mktemp -d -t rear.XXXXXXXXXXXXXXX || Error "Could not create build area '$BUILD_DIR'" )" + +and the basically only thing that is in between is + + QuietAddExitTask cleanup_build_area_and_end_program + +so the whole problematic code is + + if ! source $SHARE_DIR/lib/_input-output-functions.sh ; then + echo -e "ERROR: BUG in $PRODUCT\nFailed to source $SHARE_DIR/lib/_input-output-functions.sh\nPlease report it at $BUG_REPORT_SITE" >&2 + exit 1 + fi + ... + if test "$WORKFLOW" != "help" ; then + # Prepend temporary working area (BUILD_DIR) removal exit task + # so it gets executed directly before the above listed exit tasks. + # This must be done before the first possible call of the 'Error' function + # otherwise we may error out but leave the build area behind: + QuietAddExitTask cleanup_build_area_and_end_program + # Create temporary working area: + BUILD_DIR="$( mktemp -d -t rear.XXXXXXXXXXXXXXX || Error "Could not create build area '$BUILD_DIR'" )" + +which I did via +[https://github.com/rear/rear/commit/3f4bc829ec106ecf3456e305d48a4f2423458303](https://github.com/rear/rear/commit/3f4bc829ec106ecf3456e305d48a4f2423458303) +that points to +[https://github.com/rear/rear/pull/2633](https://github.com/rear/rear/pull/2633) +therein `3f4bc82` appears only once just before +[https://github.com/rear/rear/pull/2633\#issuecomment-866666112](https://github.com/rear/rear/pull/2633#issuecomment-866666112) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-18 14:11](https://github.com/rear/rear/issues/3180#issuecomment-2004028997): + +@pcahyna +because you did much improve the +cleanup\_build\_area\_and\_end\_program() related code +I dared to also assing you here because +I would much appreciate it +if you could also have a look here. + +I think I will at least for now move the + + QuietAddExitTask cleanup_build_area_and_end_program + +directly after the + + BUILD_DIR="$( mktemp -d -t rear.XXXXXXXXXXXXXXX ... + +because if 'mktemp' fails then there is no +$TMPDIR/rear.XXXXXXXXXXXXXXX +so no build area is left behind +(and if at all only an empty $TMPDIR/rear.XXXXXXXXXXXXXXX +would be left behind). + +I wonder if it is needed to be on the safe side +to manually call + + rm -Rf --one-file-system $BUILD_DIR || true + +when 'mktemp' fails or if this is superfluous because +when 'mktemp' fails, there cannot be a $BUILD\_DIR? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-18 15:53](https://github.com/rear/rear/issues/3180#issuecomment-2004301447): + +This issue will be fixed via +[https://github.com/rear/rear/pull/3168](https://github.com/rear/rear/pull/3168) +see +[https://github.com/rear/rear/pull/3168\#issuecomment-2004294531](https://github.com/rear/rear/pull/3168#issuecomment-2004294531) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-26 12:23](https://github.com/rear/rear/issues/3180#issuecomment-2020289048): + +With +[https://github.com/rear/rear/pull/3168](https://github.com/rear/rear/pull/3168) +merged +this issue is fixed. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-03-18.3181.pr.closed.md b/docs/issues/2024-03-18.3181.pr.closed.md new file mode 100644 index 00000000..acf711c0 --- /dev/null +++ b/docs/issues/2024-03-18.3181.pr.closed.md @@ -0,0 +1,68 @@ +[\#3181 PR](https://github.com/rear/rear/pull/3181) `closed`: FYI: Fix for issue 3180 (will be done together with \#3168) +========================================================================================================================= + +**Labels**: `bug`, `won't fix / can't fix / obsolete`, `blocker` + +#### [jsmeix](https://github.com/jsmeix) opened issue at [2024-03-18 14:43](https://github.com/rear/rear/pull/3181): + +In sbin/rear do + + QuietAddExitTask cleanup_build_area_and_end_program + +after the build area was successfully created to fix +[https://github.com/rear/rear/issues/3180](https://github.com/rear/rear/issues/3180) +but still before the first possible call +of the 'Error' function to avoid to error out +but leave the build area behind, see +[https://github.com/rear/rear/pull/2633](https://github.com/rear/rear/pull/2633) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-18 14:48](https://github.com/rear/rear/pull/3181#issuecomment-2004121608): + +With the changes of this pull request I get + + # export TMPDIR=QQQ + + # ls QQQ + ls: cannot access 'QQQ': No such file or directory + + # usr/sbin/rear -D mkrescue + tac: failed to create temporary file in 'QQQ': No such file or directory + tac: failed to create temporary file in 'QQQ': No such file or directory + mktemp: failed to create directory via template 'QQQ/rear.XXXXXXXXXXXXXXX': No such file or directory + ERROR: Could not create build area '' + + LogPrint 'Exiting rear mkrescue (PID 4187) and its descendant processes ...' + + Log 'Exiting rear mkrescue (PID 4187) and its descendant processes ...' + + test -w /root/rear.github.master/var/log/rear/rear-localhost.log + + return 0 + + Print 'Exiting rear mkrescue (PID 4187) and its descendant processes ...' + Exiting rear mkrescue (PID 4187) and its descendant processes ... + ... + Running exit tasks + ... + + eval '(( EXIT_FAIL_MESSAGE )) && echo '\''rear mkrescue failed, check /root/rear.github.master/var/log/rear/rear-localhost.log for details'\'' 1>&8' + ... + rear mkrescue failed, check /root/rear.github.master/var/log/rear/rear-localhost.log for details + ... + + eval 'exec 8>&-' + ... + + eval 'exec 7>&-' + ... + + eval 'exec 6<&-' + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-18 15:54](https://github.com/rear/rear/pull/3181#issuecomment-2004304504): + +This one is obsoleted by one part of +[https://github.com/rear/rear/commit/b741883d1c724f14e3cbe7fd7d295411c3aef4e9](https://github.com/rear/rear/commit/b741883d1c724f14e3cbe7fd7d295411c3aef4e9) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-26 12:32](https://github.com/rear/rear/pull/3181#issuecomment-2020309580): + +With +[https://github.com/rear/rear/pull/3168](https://github.com/rear/rear/pull/3168) +merged +the changes of this pull request are also merged +so this pull request can be closed. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-03-18.3182.pr.merged.md b/docs/issues/2024-03-18.3182.pr.merged.md new file mode 100644 index 00000000..3588586b --- /dev/null +++ b/docs/issues/2024-03-18.3182.pr.merged.md @@ -0,0 +1,49 @@ +[\#3182 PR](https://github.com/rear/rear/pull/3182) `merged`: \#3067 use exit code of nc to check BACULA directory availability +=============================================================================================================================== + +**Labels**: `fixed / solved / done`, `minor bug`, `external tool` + +#### [gdha](https://github.com/gdha) opened issue at [2024-03-18 16:11](https://github.com/rear/rear/pull/3182): + +#### Relax-and-Recover (ReaR) Pull Request Template + +Please fill in the following items before submitting a new pull request: + +##### Pull Request Details: + +- Type: **Bug Fix** + +- Impact: **Normal** + +- Reference to related issue (URL): + \#3067 + \#3082 + +- How was this pull request tested? Not yet - requested it via issue + \#3067 + +- Description of the changes in this pull request: + `ERROR: Bacula director debian11-DR6 is not responding` - purpose is + to have a simple test to verify to Bacula connector is responsive or + not. + +#### [gdha](https://github.com/gdha) commented at [2024-04-18 12:39](https://github.com/rear/rear/pull/3182#issuecomment-2063770752): + +@rear/contributors what shall we do with this PR? Merge or drop? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-19 06:41](https://github.com/rear/rear/pull/3182#issuecomment-2065861644): + +I think we should merge it +regardless that the users who had the issues do not respond +because the changes intend to improve things. +I.e. I think better try to improve things than do nothing. + +#### [gdha](https://github.com/gdha) commented at [2024-04-19 08:25](https://github.com/rear/rear/pull/3182#issuecomment-2066083125): + +This PR may fix issues \#3067 and \#3082 - we will see what the future +brings... + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-03-18.3183.pr.merged.md b/docs/issues/2024-03-18.3183.pr.merged.md new file mode 100644 index 00000000..3a6bb79e --- /dev/null +++ b/docs/issues/2024-03-18.3183.pr.merged.md @@ -0,0 +1,38 @@ +[\#3183 PR](https://github.com/rear/rear/pull/3183) `merged`: Stale NFS detection in a very early stage \#3109 +============================================================================================================== + +**Labels**: `enhancement`, `fixed / solved / done` + +#### [gdha](https://github.com/gdha) opened issue at [2024-03-18 16:40](https://github.com/rear/rear/pull/3183): + +#### Relax-and-Recover (ReaR) Pull Request Template + +Please fill in the following items before submitting a new pull request: + +##### Pull Request Details: + +- Type: **Enhancement** + +- Impact: **Low** + +- Reference to related issue (URL): \#3109 + +- How was this pull request tested? Manual with a local NAS system + +- Description of the changes in this pull request: See issue \#3109 + for the full story, but in short we want to bail out ReaR if there + is a stale NFS mount point. + +#### [gdha](https://github.com/gdha) commented at [2024-03-22 13:10](https://github.com/rear/rear/pull/3183#issuecomment-2015072057): + +Merging PR \#3183 for issue \#3109 + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-22 13:45](https://github.com/rear/rear/pull/3183#issuecomment-2015144114): + +@gdha +thank you for this enhancement! + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-03-27.3185.issue.open.md b/docs/issues/2024-03-27.3185.issue.open.md new file mode 100644 index 00000000..a8b03082 --- /dev/null +++ b/docs/issues/2024-03-27.3185.issue.open.md @@ -0,0 +1,327 @@ +[\#3185 Issue](https://github.com/rear/rear/issues/3185) `open`: RAID issue: several RAID1 arrays on partitions on disks with 'unknown' partition tables +======================================================================================================================================================== + +**Labels**: `support / question`, `not ReaR / invalid` + +#### [madurani](https://github.com/madurani) opened issue at [2024-03-27 12:34](https://github.com/rear/rear/issues/3185): + +- ReaR version ("/usr/sbin/rear -V"): + Relax-and-Recover 2.7 / 2022-07-13 + +- If your ReaR version is not the current version, explain why you + can't upgrade: + Last in repo + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): + openSUSE Leap 15.5 + +- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat + /etc/rear/local.conf"): + + + + OUTPUT=ISO + BACKUP=NETFS + BACKUP_URL=nfs://tms3028/svc/tms3028/data + NETFS_KEEP_OLD_BACKUP_COPY=yes + BACKUP_PROG_COMPRESS_OPTIONS=( --use-compress-program=pigz ) + REQUIRED_PROGS=("${REQUIRED_PROGS[@]}" 'pigz' 'ifdown' ) + USE_STATIC_NETWORKING=y + USE_RESOLV_CONF=() + ISO_DEFAULT="automatic" + USER_INPUT_TIMEOUT=5 + BACKUP_PROG_EXCLUDE=( '/backup' '/data' '/home' '/srv'  ) + KERNEL_CMDLINE="ip=192.168.0.1 nm=24 netdev=eth1 gw=192.168.0.1" + +- Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM + guest or PowerVM LPAR): + PC + +- System architecture (x86 compatible or PPC64/PPC64LE or what exact + ARM device): + x86\_64 + +- Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or + ELILO or Petitboot): + BIOS + +- Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or + multipath (DM or NVMe): + local ssd disks + +- Storage layout ("lsblk -ipo + NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"): + + + + gpm:/usr/share/rear/layout/prepare/GNU/Linux # lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT + + lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT + NAME KNAME PKNAME TRAN TYPE FSTYPE LABEL SIZE MOUNTPOINT + /dev/sda /dev/sda sata disk 931.5G + |-/dev/sda1 /dev/sda1 /dev/sda part linux_raid_member gpm:0 100G + | `-/dev/md0 /dev/md0 /dev/sda1 raid1 ext4 99.9G / + |-/dev/sda2 /dev/sda2 /dev/sda part linux_raid_member gpm:1 100G + | `-/dev/md1 /dev/md1 /dev/sda2 raid1 ext4 99.9G /data + |-/dev/sda3 /dev/sda3 /dev/sda part linux_raid_member gpm:2 500G + | `-/dev/md2 /dev/md2 /dev/sda3 raid1 ext4 499.9G /home + |-/dev/sda4 /dev/sda4 /dev/sda part 1K + |-/dev/sda5 /dev/sda5 /dev/sda part linux_raid_member gpm:3 100G + | `-/dev/md3 /dev/md3 /dev/sda5 raid1 ext4 99.9G /srv + |-/dev/sda6 /dev/sda6 /dev/sda part linux_raid_member gpm:4 50G + | `-/dev/md4 /dev/md4 /dev/sda6 raid1 ext4 50G /asc + |-/dev/sda7 /dev/sda7 /dev/sda part linux_raid_member gpm:5 50G + | `-/dev/md5 /dev/md5 /dev/sda7 raid1 ext4 50G /tmp + `-/dev/sda8 /dev/sda8 /dev/sda part linux_raid_member gpm:6 31.5G + `-/dev/md6 /dev/md6 /dev/sda8 raid1 swap 31.5G [SWAP] + /dev/sdb /dev/sdb sata disk 931.5G + |-/dev/sdb1 /dev/sdb1 /dev/sdb part linux_raid_member gpm:0 100G + | `-/dev/md0 /dev/md0 /dev/sdb1 raid1 ext4 99.9G / + |-/dev/sdb2 /dev/sdb2 /dev/sdb part linux_raid_member gpm:1 100G + | `-/dev/md1 /dev/md1 /dev/sdb2 raid1 ext4 99.9G /data + |-/dev/sdb3 /dev/sdb3 /dev/sdb part linux_raid_member gpm:2 500G + | `-/dev/md2 /dev/md2 /dev/sdb3 raid1 ext4 499.9G /home + |-/dev/sdb4 /dev/sdb4 /dev/sdb part 1K + |-/dev/sdb5 /dev/sdb5 /dev/sdb part linux_raid_member gpm:3 100G + | `-/dev/md3 /dev/md3 /dev/sdb5 raid1 ext4 99.9G /srv + |-/dev/sdb6 /dev/sdb6 /dev/sdb part linux_raid_member gpm:4 50G + | `-/dev/md4 /dev/md4 /dev/sdb6 raid1 ext4 50G /asc + |-/dev/sdb7 /dev/sdb7 /dev/sdb part linux_raid_member gpm:5 50G + | `-/dev/md5 /dev/md5 /dev/sdb7 raid1 ext4 50G /tmp + `-/dev/sdb8 /dev/sdb8 /dev/sdb part linux_raid_member gpm:6 31.5G + `-/dev/md6 /dev/md6 /dev/sdb8 raid1 swap 31.5G [SWAP] + /dev/sdc /dev/sdc usb disk 0B + /dev/sr0 /dev/sr0 sata rom 1024M + +Rear command after starting crashed with error(ERROR: Unsupported +partition table 'unknown'): + + gpm:/usr/share/rear/layout/prepare/GNU/Linux # rear -v -D mkbackup + + rear -v -D mkbackup + Relax-and-Recover 2.7 / 2022-07-13 + Running rear mkbackup (PID 22360 date 2024-03-27 13:29:18) + Command line options: /usr/sbin/rear -v -D mkbackup + Using log file: /var/log/rear/rear-gpm.log + Using build area: /var/tmp/rear.wXU8C4Um7Bwu257 + Running 'init' stage ====================== + Running workflow mkbackup on the normal/original system + Running 'prep' stage ====================== + Using backup archive '/var/tmp/rear.wXU8C4Um7Bwu257/outputfs/gpm/backup.tar.gz' + Using '/usr/bin/xorrisofs' to create ISO filesystem images + Using autodetected kernel '/boot/vmlinuz-5.14.21-150500.55.39-default' as kernel in the recovery system + Running 'layout/save' stage ====================== + Creating disk layout + Overwriting existing disk layout file /var/lib/rear/layout/disklayout.conf + ERROR: Unsupported partition table 'unknown' (must be one of 'msdos' 'gpt' 'gpt_sync_mbr' 'dasd') + Some latest log messages since the last called script 200_partition_layout.sh: + 2024-03-27 13:29:21.258329016 Entering debugscript mode via 'set -x'. + 2024-03-27 13:29:21.269645756 Saving disks and their partitions + Error: Can't have a partition outside the disk! + Error: Can't have a partition outside the disk! + Error exit of rear mkbackup (PID 22360) and its descendant processes + Exiting subshell 1 (where the actual error happened) + Aborting due to an error, check /var/log/rear/rear-gpm.log for details + Exiting rear mkbackup (PID 22360) and its descendant processes ... + Running exit tasks + To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.wXU8C4Um7Bwu257 + Terminated + +Reason is unknown partition on sda disk which is member of raid1: + + gpm:/usr/share/rear/layout/prepare/GNU/Linux # grep -v '^#' /var/lib/rear/layout/disklayout.conf + + grep --color=auto -v '^#' /var/lib/rear/layout/disklayout.conf + disk /dev/sda 1000204886016 unknown + +Raid configuration: + + gpm:/usr/share/rear/layout/prepare/GNU/Linux # mdadm --detail --scan --config=partitions + + mdadm --detail --scan --config=partitions + ARRAY /dev/md0 metadata=1.2 name=gpm:0 UUID=7ce553bc:c228e1f8:9f571002:80fc9282 + ARRAY /dev/md6 metadata=1.2 name=gpm:6 UUID=027c4b60:bc0428f3:459b9d16:9899e2d6 + ARRAY /dev/md5 metadata=1.2 name=gpm:5 UUID=3dabeffa:fe73dce2:e80e9c13:ee782d5b + ARRAY /dev/md2 metadata=1.2 name=gpm:2 UUID=562cde3d:ea14ada5:bbe75715:81f659bd + ARRAY /dev/md4 metadata=1.2 name=gpm:4 UUID=3507a2dd:923d63f2:a747f34f:63126f9e + ARRAY /dev/md1 metadata=1.2 name=gpm:1 UUID=1d118b9b:e52e2188:70c75680:580fda55 + ARRAY /dev/md3 metadata=1.2 name=gpm:3 UUID=1ca1aba5:29b38b47:567991c9:1a665cac + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-27 13:19](https://github.com/rear/rear/issues/3185#issuecomment-2022748345): + +@madurani +as far as I see +you made several RAID1 arrays (/dev/md0 to /dev/md6) +each one for matching partitions on sda and sdb. + +What I had tested was RAID1 made of whole disks, cf. +[https://github.com/rear/rear/wiki/Test-Matrix-ReaR-2.7](https://github.com/rear/rear/wiki/Test-Matrix-ReaR-2.7) + +What do the commands + + # parted -s /dev/sda unit GiB print + + # parted -s /dev/sdb unit GiB print + +show on your system? + +#### [madurani](https://github.com/madurani) commented at [2024-03-27 13:21](https://github.com/rear/rear/issues/3185#issuecomment-2022751374): + + gpm:/opt/rear # parted -s /dev/sda unit GiB print + + parted -s /dev/sda unit GiB print + Error: Can't have a partition outside the disk! + Model: ATA WDC WD10EZEX-08M (scsi) + Disk /dev/sda: 932GiB + Sector size (logical/physical): 512B/4096B + Partition Table: unknown + Disk Flags: + + gpm:/opt/rear # parted -s /dev/sdb unit GiB print + + parted -s /dev/sdb unit GiB print + Error: Can't have a partition outside the disk! + Model: ATA WDC WD10EZEX-60M (scsi) + Disk /dev/sdb: 932GiB + Sector size (logical/physical): 512B/4096B + Partition Table: unknown + Disk Flags: + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-27 13:26](https://github.com/rear/rear/issues/3185#issuecomment-2022766051): + +Huh! +The 'parted' output seems to indicate that +your disk setup is somehow just broken +(but I am not a RAID expert), +cf. what I had on +[https://github.com/rear/rear/wiki/Test-Matrix-ReaR-2.7](https://github.com/rear/rear/wiki/Test-Matrix-ReaR-2.7) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-27 13:31](https://github.com/rear/rear/issues/3185#issuecomment-2022779701): + +@madurani +can you describe what you did to setup your RAID1? +E.g. what tool you used or what commands you called? + +What I did during my tests on +[https://github.com/rear/rear/wiki/Test-Matrix-ReaR-2.7](https://github.com/rear/rear/wiki/Test-Matrix-ReaR-2.7) +is using the partitioning tool in YaST. + +With that I made a single RAID1 array of my two +whole disks /dev/sda and /dev/sdb which results +the single RAID1 array /dev/md127 +(with size of the smaller disk). + +That RAID1 array /dev/md127 behaves like a whole disk +and therein I created partitions as shown by 'lsblk' + + `-/dev/md127 raid1 9G + |-/dev/md127p1 part 102M + |-/dev/md127p2 part swap 1G + `-/dev/md127p3 part btrfs 7.5G + +Finally on the /dev/md127p3 partition +a btrfs filesystem is created. + +In the end there is +only one btrfs filesystem +on one /dev/md127p3 partition +on one /dev/md127 RAID1 array +and only that one RAID1 array +is on two disks /dev/sda and /dev/sdb + +#### [madurani](https://github.com/madurani) commented at [2024-03-27 13:35](https://github.com/rear/rear/issues/3185#issuecomment-2022789491): + +I didn't configure mentioned raid. It was done long time ago. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-27 13:36](https://github.com/rear/rear/issues/3185#issuecomment-2022791147): + +@madurani +does your system work OK with your current RAID1 setup? + +Again: I am not a RAID expert. +So your current RAID1 setup could be even correct, +but at least to me it looks "unexpected". + +And with your current 'parted' output +your current RAID1 setup is not supported in ReaR +because ReaR depends on a normal 'parted' output. + +#### [madurani](https://github.com/madurani) commented at [2024-03-27 13:39](https://github.com/rear/rear/issues/3185#issuecomment-2022797076): + +Yes system work properly, without problem. I wanted use rear as backup +before os update. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-27 13:44](https://github.com/rear/rear/issues/3185#issuecomment-2022808512): + +@rear/contributors +could you please have a look here (as time permits). +Perhaps I misunderstand something and +I would like to avoid causing false alarm +about a possibly broken disk setup when +the current system works without problems. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-27 15:35](https://github.com/rear/rear/issues/3185#issuecomment-2023080960): + +I think we need more information about the partitions. Since `parted` +refuses to show us any detail, please print the partition information +using another tool: `fdisk -x /dev/sda` or `gdisk -l /dev/sda`. + +#### [madurani](https://github.com/madurani) commented at [2024-03-27 15:40](https://github.com/rear/rear/issues/3185#issuecomment-2023091490): + + gpm:~ # fdisk -x /dev/sda + Disk /dev/sda: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors + Disk model: WDC WD10EZEX-08M + Units: sectors of 1 * 512 = 512 bytes + Sector size (logical/physical): 512 bytes / 4096 bytes + I/O size (minimum/optimal): 4096 bytes / 4096 bytes + Disklabel type: dos + Disk identifier: 0x8c20fcea + + Device Boot Start End Sectors Id Type Start-C/H/S End-C/H/S Attrs + /dev/sda1 * 2048 209717247 209715200 fd Linux raid autodetect 0/32/33 1023/254/63 80 + /dev/sda2 209717248 419432447 209715200 fd Linux raid autodetect 1023/254/63 1023/254/63 + /dev/sda3 419432448 1468008447 1048576000 fd Linux raid autodetect 1023/254/63 1023/254/63 + /dev/sda4 1468008448 1953525759 485517312 f W95 Ext'd (LBA) 1023/254/63 1023/254/63 + /dev/sda5 1468010496 1677725695 209715200 fd Linux raid autodetect 1023/254/63 1023/254/63 + /dev/sda6 1677727744 1782585343 104857600 fd Linux raid autodetect 1023/254/63 1023/254/63 + /dev/sda7 1782587392 1887444991 104857600 fd Linux raid autodetect 1023/254/63 1023/254/63 + /dev/sda8 1887447040 1953525759 66078720 fd Linux raid autodetect 1023/254/63 1023/254/63 + + + gpm:~ # fdisk -x /dev/sdb + Disk /dev/sdb: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors + Disk model: WDC WD10EZEX-60M + Units: sectors of 1 * 512 = 512 bytes + Sector size (logical/physical): 512 bytes / 4096 bytes + I/O size (minimum/optimal): 4096 bytes / 4096 bytes + Disklabel type: dos + Disk identifier: 0x00000000 + + Device Boot Start End Sectors Id Type Start-C/H/S End-C/H/S Attrs + /dev/sdb1 * 2048 209717247 209715200 fd Linux raid autodetect 0/32/33 1023/254/63 80 + /dev/sdb2 209717248 419432447 209715200 fd Linux raid autodetect 1023/254/63 1023/254/63 + /dev/sdb3 419432448 1468008447 1048576000 fd Linux raid autodetect 1023/254/63 1023/254/63 + /dev/sdb4 1468008448 1953525759 485517312 f W95 Ext'd (LBA) 1023/254/63 1023/254/63 + /dev/sdb5 1468010496 1677725695 209715200 fd Linux raid autodetect 1023/254/63 1023/254/63 + /dev/sdb6 1677727744 1782585343 104857600 fd Linux raid autodetect 1023/254/63 1023/254/63 + /dev/sdb7 1782587392 1887444991 104857600 fd Linux raid autodetect 1023/254/63 1023/254/63 + /dev/sdb8 1887447040 1953525759 66078720 fd Linux raid autodetect 1023/254/63 1023/254/63 + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-27 15:56](https://github.com/rear/rear/issues/3185#issuecomment-2023127276): + +> gpm:~ \# fdisk -x /dev/sda +> Disk /dev/sda: 931.51 GiB, 1000204886016 bytes, **1953525168** +> sectors +> ... +> /dev/sda8 1887447040 **1953525759** 66078720 fd Linux raid autodetect +> 1023/254/63 1023/254/63 + +This is not right. Doesn't the kernel complain about this on boot? What +is the size of /dev/sda8 as seen by the kernel +`blockdev --getsz /dev/sda8` ? What is the size of `/dev/md6` - +`blockdev --getsz /dev/md6` and `mdadm --detail /dev/md6` ? + +Anyway, not a ReaR bug, nor a parted bug, although I would prefer +`parted` to say what exactly it dislikes instead of just complaining +that it dislikes something. What does `parted /dev/sda unit GiB print` +show? + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-03-27.3186.issue.open.md b/docs/issues/2024-03-27.3186.issue.open.md new file mode 100644 index 00000000..ea953ebc --- /dev/null +++ b/docs/issues/2024-03-27.3186.issue.open.md @@ -0,0 +1,67 @@ +[\#3186 Issue](https://github.com/rear/rear/issues/3186) `open`: error: ../../grub-core/loader/i386/efi/linux.c:158:can't allocate initrd +========================================================================================================================================= + +**Labels**: `support / question`, `external tool`, `not ReaR / invalid`, +`special hardware or VM` + +#### [sathyane007](https://github.com/sathyane007) opened issue at [2024-03-27 15:08](https://github.com/rear/rear/issues/3186): + +Hello ; +i have an issue when i try to use rear.iso file to recever linux REH8.7 +on a dell idrac BMR. +On a topic , someone suggest to exclude from rear iso firmware in order +to reduce intrd but i need also firmware updatation on image file . +is there any updation of this kind of issue please ? +Thanks for your help + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-27 15:18](https://github.com/rear/rear/issues/3186#issuecomment-2023037435): + +Hi, this was reported as \#2681 . I know two solutions (besides reducing +the size of the initrd): + +- downgrading grub to a version between 2.02-121.el8 and + grub2-1:2.02-141.el8 : + [https://github.com/rear/rear/issues/2681\#issuecomment-1697073572](https://github.com/rear/rear/issues/2681#issuecomment-1697073572) +- firmware update (helped on HP hardware, don't know if it will help + on Dell hardware though) + +#### [sathyane007](https://github.com/sathyane007) commented at [2024-03-28 12:18](https://github.com/rear/rear/issues/3186#issuecomment-2025052087): + +Hi pcahyna , thanks for your reply +we are in version grub2-2.02-142.el8\_7.1.x86\_64 +Unfortunatly , we can not downgrade to version 2.02-121.el8 . i ask to +upgrade the firmware of dell hardware in case .. +but is there an other solution please to charge rear-iso file ? +in fact we are using rubrik backup as third party backup tools +thanks + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-03-28 12:25](https://github.com/rear/rear/issues/3186#issuecomment-2025065328): + +What Dell hardware is it, exactly? + +#### [sathyane007](https://github.com/sathyane007) commented at [2024-03-29 08:53](https://github.com/rear/rear/issues/3186#issuecomment-2026899680): + +Dell PowerEdge R750 +Bios Version 1.9.2 +Integrated Dell Remote Access Controller +9........................................................................... +Version 6.10.30.00(Build 29) + +#### [sathyane007](https://github.com/sathyane007) commented at [2024-04-03 07:31](https://github.com/rear/rear/issues/3186#issuecomment-2033764225): + +Hello +i have tested on a cisco platform , and i have this version: +grub2-efi-x64-2.02-150.el8.x86\_64 +And i had no problem with rear iso file to load it . + +Someone has any idea why it is not working on Dell Platform , ? + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-04-05 10:35](https://github.com/rear/rear/issues/3186#issuecomment-2039457804): + +@sathyane007 this GRUB problem is apparently firmware (both manufacturer +and version) dependent, that's why. Have you tried updating it? + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-03-28.3187.issue.open.md b/docs/issues/2024-03-28.3187.issue.open.md new file mode 100644 index 00000000..281d4506 --- /dev/null +++ b/docs/issues/2024-03-28.3187.issue.open.md @@ -0,0 +1,130 @@ +[\#3187 Issue](https://github.com/rear/rear/issues/3187) `open`: Bareos+ReaR: Problem booting the system after successful recovery +================================================================================================================================== + +**Labels**: `support / question`, `external tool` + +#### [Ram9998](https://github.com/Ram9998) opened issue at [2024-03-28 10:08](https://github.com/rear/rear/issues/3187): + + + +- ReaR version ("/usr/sbin/rear -V"): 2.6 + +- If your ReaR version is not the current version, explain why you + can't upgrade: + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): + OS\_VENDOR=OracleServer + OS\_VERSION=8 + +- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat + /etc/rear/local.conf"): + OUTPUT=ISO + ISO\_DIR=/backups + BACKUP=BAREOS + BAREOS\_CLIENT=VL-BPCACSAPP01 + BAREOS\_FILESET=LinuxAll + BAREOS\_RESTORE\_JOB=RestoreOLVM + USE\_STATIC\_NETWORKING=y + +- Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM + guest or PowerVM LPAR): VM (OLVM) + +- System architecture (x86 compatible or PPC64/PPC64LE or what exact + ARM device): x86\_64 + +- Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or + ELILO or Petitboot): UEFI GRUB + +- Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or + multipath (DM or NVMe): NVME + +- Storage layout ("lsblk -ipo + NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"): + +
+ Screenshot + +
+ +- Description of the issue (ideally so that others can reproduce + it): + I'm using Bareos+ReaR. Can't boot from a restored system. Problem + installing EFI Boot Manager: + +
+ Screenshot + +
+ +- Workaround, if any: + +- Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" + debug log files): + [rear-VL-BPCACSAPP01.log](https://github.com/rear/rear/files/14787463/rear-VL-BPCACSAPP01.log) + +You can drag-drop log files into this editor to create an attachment +or paste verbatim text like command output or file content +by including it between a leading and a closing line of +three backticks like this: + + verbatim content + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-03-28 10:46](https://github.com/rear/rear/issues/3187#issuecomment-2024893155): + +I am not a Bareos user so I cannot actually help +with Bareos specific issues. + +From what I notice at first glance in your +[https://github.com/rear/rear/files/14787463/rear-VL-BPCACSAPP01.log](https://github.com/rear/rear/files/14787463/rear-VL-BPCACSAPP01.log) +(excerpts) + + + source /usr/share/rear/restore/BAREOS/default/400_restore_backup.sh + ... + ++ mkdir /mnt/local/var/lib/bareos + mkdir: cannot create directory '/mnt/local/var/lib/bareos': No such file or directory + + ... + + + source /usr/share/rear/restore/default/900_create_missing_directories.sh + ... + mkdir: created directory 'etc' + ... + ++ chroot /mnt/local /bin/bash --login -c 'chown -v root:root etc' + chroot: failed to run command '/bin/bash': No such file or directory + + ... + + + source /usr/share/rear/finalize/Linux-i386/670_run_efibootmgr.sh + ... + ++ test -f /mnt/local//boot/efi/EFI/redhat/grubx64.efi + ++ LogPrintError 'Failed to create EFI Boot Manager entries (UEFI bootloader '\''/boot/efi/EFI/redhat/grubx64.efi'\'' not found under target /mnt/local)' + +it seems the Bareos backup or the restore was incomplete +because it looks as if basic directories are missing +in the recreated system under /mnt/local +like /mnt/local/var/lib and /mnt/local/etc +(and many more that 900\_create\_missing\_directories.sh shows) +and also /bin/bash seems to be missing +in the recreated system under /mnt/local +and finally also /mnt/local/boot/efi/EFI/redhat/grubx64.efi +seems to be missing in the recreated system. + +Did you actually do what the Bareos restore script +/usr/share/rear/restore/BAREOS/default/400\_restore\_backup.sh +tells you to do + + Please verify that the backup has been restored correctly to '/mnt/local' + in the provided shell. When finished, type exit in the shell to continue + recovery. + +? + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-01.3188.pr.merged.md b/docs/issues/2024-04-01.3188.pr.merged.md new file mode 100644 index 00000000..5d483580 --- /dev/null +++ b/docs/issues/2024-04-01.3188.pr.merged.md @@ -0,0 +1,92 @@ +[\#3188 PR](https://github.com/rear/rear/pull/3188) `merged`: Bump redhat-plumbers-in-action/differential-shellcheck from 5.1.0 to 5.1.1 +======================================================================================================================================== + +**Labels**: `dependencies` + +#### [dependabot](https://github.com/apps/dependabot) opened issue at [2024-04-01 11:52](https://github.com/rear/rear/pull/3188): + +Bumps +[redhat-plumbers-in-action/differential-shellcheck](https://github.com/redhat-plumbers-in-action/differential-shellcheck) +from 5.1.0 to 5.1.1. + +
+Release notes +

Sourced from redhat-plumbers-in-action/differential-shellcheck's releases.

+
+

v5.1.1

+

What's Changed

+

Bug Fixes

+
    +
  • Fix regex for emacs style annotations- also match -*- shell script -*- (#365) @​jamacku
    • +
+

Dependency Updates

+ +

Full Changelog: https://github.com/redhat-plumbers-in-action/differential-shellcheck/compare/v5.1.0...v5.1.1

+
+
+
+Commits +
    +
  • c150708 v5.1.1
    • +
  • a10f9db fix: fix regex for emacs style annotations
    • +
  • bc4d40a build(deps): bump fedora from 06df381 to 61864fd
    • +
  • 947739d build(deps): bump github/codeql-action from 3.23.2 to 3.24.6
    • +
  • 73a8683 build(deps): bump release-drafter/release-drafter from 5.25.0 to 6.0.0
    • +
  • b76ede8 build(deps): bump actions/upload-artifact from 4.3.0 to 4.3.1
    • +
  • 486a403 build(deps): bump test/bats from 990d8e2 to 2d905aa
    • +
  • See full diff in compare view
    • +
+
+
+ +[![Dependabot compatibility +score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=redhat-plumbers-in-action/differential-shellcheck&package-manager=github_actions&previous-version=5.1.0&new-version=5.1.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) + +Dependabot will resolve any conflicts with this PR as long as you don't +alter it yourself. You can also trigger a rebase manually by commenting +`@dependabot rebase`. + +------------------------------------------------------------------------ + +
+Dependabot commands and options +
+ +You can trigger Dependabot actions by commenting on this PR: + +- `@dependabot rebase` will rebase this PR +- `@dependabot recreate` will recreate this PR, overwriting any edits + that have been made to it +- `@dependabot merge` will merge this PR after your CI passes on it +- `@dependabot squash and merge` will squash and merge this PR after + your CI passes on it +- `@dependabot cancel merge` will cancel a previously requested merge + and block automerging +- `@dependabot reopen` will reopen this PR if it is closed +- `@dependabot close` will close this PR and stop Dependabot + recreating it. You can achieve the same result by closing it + manually +- `@dependabot show ignore conditions` will show all + of the ignore conditions of the specified dependency +- `@dependabot ignore this major version` will close this PR and stop + Dependabot creating any more for this major version (unless you + reopen the PR or upgrade to it yourself) +- `@dependabot ignore this minor version` will close this PR and stop + Dependabot creating any more for this minor version (unless you + reopen the PR or upgrade to it yourself) +- `@dependabot ignore this dependency` will close this PR and stop + Dependabot creating any more for this dependency (unless you reopen + the PR or upgrade to it yourself) + +
+ +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-02.3189.issue.open.md b/docs/issues/2024-04-02.3189.issue.open.md new file mode 100644 index 00000000..a8c9ad9b --- /dev/null +++ b/docs/issues/2024-04-02.3189.issue.open.md @@ -0,0 +1,634 @@ +[\#3189 Issue](https://github.com/rear/rear/issues/3189) `open`: booting rescue system (ISO) of SLES on POWER fails +=================================================================================================================== + +**Labels**: `support / question` + +#### [kai-uwe-rommel](https://github.com/kai-uwe-rommel) opened issue at [2024-04-02 13:17](https://github.com/rear/rear/issues/3189): + +- ReaR version ("/usr/sbin/rear -V"): + 2.7 + +- If your ReaR version is not the current version, explain why you + can't upgrade: + comes from SLES repository + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): + NAME="SLES" + VERSION="15-SP5" + VERSION\_ID="15.5" + PRETTY\_NAME="SUSE Linux Enterprise Server 15 SP5" + ID="sles" + ID\_LIKE="suse" + ANSI\_COLOR="0;32" + CPE\_NAME="cpe:/o:suse:sles:15:sp5" + +- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat + /etc/rear/local.conf"): + OUTPUT=ISO + OUTPUT\_URL=nfs://prometheus/nim/rear + BACKUP=TSM + +(The ISO file is then copied from the NFS server into the VIOS media +library and mounted into the new PowerVM LPAR as a virtual optical +drive.) + +- Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM + guest or PowerVM LPAR): + IBM PowerVM LPAR + +- System architecture (x86 compatible or PPC64/PPC64LE or what exact + ARM device): + PPC64LE + +- Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or + multipath (DM or NVMe): + SAN w/multipath + +- Storage layout ("lsblk -ipo + NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"): + [lsblk.txt](https://github.com/rear/rear/files/14837097/lsblk.txt) + +- Description of the issue (ideally so that others can reproduce + it): + The rear invocation creates an ISO file and that looks fine to me. + When booting it in a new/blank PowerVM LPAR, it ends up with a + kernel panic. + To me it looks like it does not load the initrd. + But the initrd file is in the ISO and seems to be referenced + properly in the grub.cfg file. + [boot.log](https://github.com/rear/rear/files/14837101/boot.log) + Please see content of the boot log file. + +- Workaround, if any: + unfortunately, none - need help + +- Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" + debug log files): + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-02 14:46](https://github.com/rear/rear/issues/3189#issuecomment-2032246470): + +@kai-uwe-rommel +please attach a "rear -D mkrescue" (or "rear -D mkbackup") +debug log file. + +In general: +Caution with possible secrets in a debug log file: +When 'rear' is run via '-D' in debugscript mode +it logs executed commands via the bash command 'set -x' +that print commands and their arguments as they are executed. +So in particular when arguments contain secret values +(e.g. something like a password or whatever else) +such secret values may appear in the log file. +Also secrets may be stored in some other files +like /var/lib/rear/layout/disklayout.conf +or /var/lib/rear/layout/diskrestore.sh +cf. `[password=]` in the section +"Disk layout file syntax" in +doc/user-guide/06-layout-configuration.adoc +online at +[https://github.com/rear/rear/blob/rear-2.7/doc/user-guide/06-layout-configuration.adoc](https://github.com/rear/rear/blob/rear-2.7/doc/user-guide/06-layout-configuration.adoc) +So before you attach your debug log file or other files here +(GitHub is a public accessible place), inspect your files +and verify that they do not accidentally cointain secrets. + +#### [kai-uwe-rommel](https://github.com/kai-uwe-rommel) commented at [2024-04-03 16:31](https://github.com/rear/rear/issues/3189#issuecomment-2035067568): + +The mkrescue debug log: +[rear-mkrescue-debug.log](https://github.com/rear/rear/files/14855435/rear-mkrescue-debug.log) + +#### [kai-uwe-rommel](https://github.com/kai-uwe-rommel) commented at [2024-04-03 16:32](https://github.com/rear/rear/issues/3189#issuecomment-2035069769): + +Also, let me know if you need access to the generated ISO file. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-08 07:34](https://github.com/rear/rear/issues/3189#issuecomment-2042056222): + +@kai-uwe-rommel +it seems your +[https://github.com/rear/rear/files/14855435/rear-mkrescue-debug.log](https://github.com/rear/rear/files/14855435/rear-mkrescue-debug.log) +is only the terminal output of "rear -D mkrescue" (excerpts) + + hermes:~ # rear -D mkrescue + Relax-and-Recover 2.7 / 2022-07-13 + Running rear mkrescue (PID 11791 date 2024-04-03 18:22:27) + Command line options: /usr/sbin/rear -D mkrescue + Using log file: /var/log/rear/rear-hermes.log + ... + Running 'output' stage ====================== + Making ISO image + Making ISO image + Wrote ISO image: /var/lib/rear/output/rear-hermes.iso (319M) + +but I liked to get the debug log **file** i.e. the +`log file: /var/log/rear/rear-hermes.log` +(except secret values if any). + +Additionally please attach the terminal output of + + # rear -s mkrescue + +because in your +[https://github.com/rear/rear/files/14855435/rear-mkrescue-debug.log](https://github.com/rear/rear/files/14855435/rear-mkrescue-debug.log) +I am missing messages that tell a bootloader gets installed +before the ISO is made. + +For comparison on my x86\_64 UEFI homeoffice workstation +(excerpts): + + # grep -v '^#' etc/rear/local.conf + OUTPUT=ISO + BACKUP=NETFS + BACKUP_URL=file:///other/ + ... + + # usr/sbin/rear -s mkrescue + ... + Source output/ISO/Linux-i386/250_populate_efibootimg.sh + Source output/ISO/Linux-i386/260_EFISTUB_populate.sh + Source output/ISO/Linux-i386/300_create_isolinux.sh + Source output/default/400_copy_disk_struct_files.sh + Source output/ISO/Linux-i386/700_create_efibootimg.sh + Source output/ISO/Linux-i386/800_create_isofs.sh + Source output/ISO/Linux-i386/810_prepare_multiple_iso.sh + Source output/ISO/Linux-i386/820_create_iso_image.sh + Source output/ISO/Linux-i386/830_create_iso_image_EFISTUB.sh + Source output/ISO/Linux-i386/850_check_for_errors.sh + +For PPC64LE we have + + # ls -1 usr/share/rear/output/ISO/Linux-ppc64le + 300_create_grub2.sh + 800_create_isofs.sh + 810_prepare_multiple_iso.sh + 820_create_iso_image.sh + +but usr/share/rear/output/ISO/Linux-ppc64le/300\_create\_grub2.sh +does no output on the terminal and it does not error out when + + grub2-mkimage -O powerpc-ieee1275 ... + +fails - at least this needs to be fixed, cf. +[https://github.com/rear/rear/wiki/Coding-Style\#try-hard-to-care-about-possible-errors](https://github.com/rear/rear/wiki/Coding-Style#try-hard-to-care-about-possible-errors) + +So I need your debug log file /var/log/rear/rear-hermes.log +to see in particular what goes on on your particular system +while the GRUB2 bootloader is installed for the iso image. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-08 07:43](https://github.com/rear/rear/issues/3189#issuecomment-2042071677): + +@kai-uwe-rommel +in your initial description you wrote + + But the initrd file is in the ISO and + seems to be referenced properly in the grub.cfg file. + https://github.com/rear/rear/files/14837101/boot.log + Please see content of the boot log file. + +but I fail to see in your +[https://github.com/rear/rear/files/14837101/boot.log](https://github.com/rear/rear/files/14837101/boot.log) +that the initrd file is in the ISO and +that it is referenced properly in the grub.cfg file, +in particular I fail to see the grub.cfg file? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-08 08:05](https://github.com/rear/rear/issues/3189#issuecomment-2042114550): + +@kai-uwe-rommel +see my "FYI (2)" part in +[https://github.com/rear/rear/discussions/3184\#discussioncomment-8902863](https://github.com/rear/rear/discussions/3184#discussioncomment-8902863) +how you may use `KEEP_BUILD_DIR="yes"` +to inspect the recovery system contents after "rear mkrescue" +so you could inspect in particular your grub.cfg file +in your ReaR recovery system. + +#### [kai-uwe-rommel](https://github.com/kai-uwe-rommel) commented at [2024-04-08 12:41](https://github.com/rear/rear/issues/3189#issuecomment-2042647866): + +This is the output of "rear -s mkrescue": + + hermes:~ # rear -s mkrescue + Relax-and-Recover 2.7 / 2022-07-13 + Running rear mkrescue (PID 8073 date 2024-04-08 14:39:53) + Using log file: /var/log/rear/rear-hermes.log + Simulation mode activated, Relax-and-Recover base directory: /usr/share/rear + Source conf/Linux-ppc64le.conf + Source conf/GNU/Linux.conf + Source conf/SUSE_LINUX.conf + Source init/default/005_verify_os_conf.sh + Source init/default/010_EFISTUB_check.sh + Source init/default/010_set_drlm_env.sh + Source init/default/030_update_recovery_system.sh + Source init/default/050_check_rear_recover_mode.sh + Source init/default/950_check_missing_programs.sh + Source prep/default/005_remove_workflow_conf.sh + Source prep/default/020_translate_url.sh + Source prep/default/030_translate_tape.sh + Source prep/default/035_valid_backup_methods.sh + Source prep/default/036_valid_output_methods.sh + Source prep/default/040_check_backup_and_output_scheme.sh + Source prep/default/050_check_keep_old_output_copy_var.sh + Source prep/default/100_init_workflow_conf.sh + Source prep/GNU/Linux/200_include_getty.sh + Source prep/GNU/Linux/200_include_serial_console.sh + Source prep/GNU/Linux/210_include_dhclient.sh + Source prep/GNU/Linux/220_include_lvm_tools.sh + Source prep/GNU/Linux/230_include_md_tools.sh + Source prep/GNU/Linux/240_include_multipath_tools.sh + Source prep/GNU/Linux/280_include_systemd.sh + Source prep/GNU/Linux/280_include_virtualbox.sh + Source prep/GNU/Linux/280_include_vmware_tools.sh + Source prep/GNU/Linux/290_include_drbd.sh + Source prep/GNU/Linux/300_check_backup_and_output_url.sh + Source prep/ISO/default/300_check_iso_dir.sh + Source prep/GNU/Linux/300_include_grub_tools.sh + Source prep/GNU/Linux/310_include_cap_utils.sh + Source prep/ISO/default/320_check_cdrom_size.sh + Source prep/default/320_include_uefi_env.sh + Source prep/ISO/GNU/Linux/320_verify_mkisofs.sh + Source prep/default/321_EFISTUB_check_uefi_env.sh + Source prep/default/330_include_uefi_tools.sh + Source prep/ISO/GNU/Linux/340_add_isofs_module.sh + Source prep/default/340_include_password_tools.sh + Source prep/ISO/GNU/Linux/360_EFISTUB_prechecks.sh + Source prep/default/380_include_opal_tools.sh + Source prep/GNU/Linux/400_guess_kernel.sh + Source prep/TSM/default/400_prep_tsm.sh + Source prep/default/400_save_directories.sh + Source prep/default/490_store_write_protect_settings.sh + Source prep/GNU/Linux/500_EFISTUB_check_kernel.sh + Source layout/save/GNU/Linux/100_create_layout_file.sh + Source layout/save/GNU/Linux/150_save_diskbyid_mappings.sh + Source layout/save/GNU/Linux/190_opaldisk_layout.sh + Source layout/save/GNU/Linux/200_partition_layout.sh + Source layout/save/GNU/Linux/210_raid_layout.sh + Source layout/save/GNU/Linux/220_lvm_layout.sh + Source layout/save/GNU/Linux/230_filesystem_layout.sh + Source layout/save/GNU/Linux/240_swaps_layout.sh + Source layout/save/GNU/Linux/250_drbd_layout.sh + Source layout/save/GNU/Linux/260_crypt_layout.sh + Source layout/save/GNU/Linux/270_hpraid_layout.sh + Source layout/save/GNU/Linux/280_multipath_layout.sh + Source layout/save/default/300_list_dependencies.sh + Source layout/save/default/310_autoexclude_usb.sh + Source layout/save/default/310_include_exclude.sh + Source layout/save/default/320_autoexclude.sh + Source layout/save/default/330_remove_exclusions.sh + Source layout/save/default/335_remove_excluded_multipath_vgs.sh + Source layout/save/GNU/Linux/340_false_blacklisted.sh + Source layout/save/default/340_generate_mountpoint_device.sh + Source layout/save/GNU/Linux/350_copy_drbdtab.sh + Source layout/save/default/350_save_partitions.sh + Source layout/save/default/400_check_backup_special_files.sh + Source layout/save/default/445_guess_bootloader.sh + Source layout/save/default/450_check_bootloader_files.sh + Source layout/save/default/450_check_network_files.sh + Source layout/save/default/490_check_files_to_patch.sh + Source layout/save/GNU/Linux/500_extract_vgcfg.sh + Source layout/save/GNU/Linux/510_current_disk_usage.sh + Source layout/save/default/600_snapshot_files.sh + Source layout/save/default/950_verify_disklayout_file.sh + Source rescue/default/010_merge_skeletons.sh + Source rescue/default/100_hostname.sh + Source rescue/default/200_etc_issue.sh + Source rescue/GNU/Linux/220_load_modules_from_initrd.sh + Source rescue/GNU/Linux/230_storage_and_network_modules.sh + Source rescue/GNU/Linux/240_kernel_modules.sh + Source rescue/GNU/Linux/250_udev.sh + Source rescue/GNU/Linux/260_collect_initrd_modules.sh + Source rescue/GNU/Linux/260_storage_drivers.sh + Source rescue/GNU/Linux/290_kernel_cmdline.sh + Source rescue/GNU/Linux/300_dns.sh + Source rescue/default/300_patch_root_home.sh + Source rescue/GNU/Linux/310_network_devices.sh + Source rescue/GNU/Linux/320_inet6.sh + Source rescue/GNU/Linux/350_routing.sh + Source rescue/GNU/Linux/390_check_usb_modules.sh + Source rescue/GNU/Linux/400_use_serial_console.sh + Source rescue/GNU/Linux/410_use_xen_console.sh + Source rescue/default/430_prepare_timesync.sh + Source rescue/GNU/Linux/500_clone_keyboard_mappings.sh + Source rescue/default/500_ssh.sh + Source rescue/GNU/Linux/550_copy_ldconfig.sh + Source rescue/default/550_vagrant.sh + Source rescue/default/850_save_sysfs_uefi_vars.sh + Source rescue/default/860_set_uefi_vars.sh + Source rescue/default/900_clone_users_and_groups.sh + Source rescue/default/910_copy_logfile.sh + Source rescue/GNU/Linux/950_cfg2html.sh + Source rescue/GNU/Linux/960_collect_MC_serviceguard_infos.sh + Source rescue/GNU/Linux/990_sysreqs.sh + Source build/GNU/Linux/005_create_symlinks.sh + Source build/GNU/Linux/090_create_lib_directories_and_symlinks.sh + Source build/GNU/Linux/100_copy_as_is.sh + Source build/GNU/Linux/110_touch_empty_files.sh + Source build/GNU/Linux/130_create_dotfiles.sh + Source build/GNU/Linux/150_adjust_permissions.sh + Source build/GNU/Linux/390_copy_binaries_libraries.sh + Source build/GNU/Linux/400_copy_modules.sh + Source build/GNU/Linux/420_copy_firmware_files.sh + Source build/GNU/Linux/450_symlink_mingetty.sh + Source build/default/490_fix_broken_links.sh + Source build/default/500_ssh_setup.sh + Source build/default/501_check_ssh_keys.sh + Source build/default/502_include_mdadm_conf.sh + Source build/default/503_store_tty_root_password.sh + Source build/GNU/Linux/600_verify_and_adjust_udev.sh + Source build/SUSE_LINUX/610_link_systemd_lib.sh + Source build/GNU/Linux/610_verify_and_adjust_udev_systemd.sh + Source build/GNU/Linux/620_verify_os_release_file.sh + Source build/GNU/Linux/630_simplify_systemd_reboot_halt_poweroff_shutdown.sh + Source build/GNU/Linux/630_verify_resolv_conf_file.sh + Source build/GNU/Linux/640_verify_lvm_conf.sh + Source build/default/950_check_missing_programs.sh + Source build/default/960_remove_encryption_keys.sh + Source build/default/970_add_rear_release.sh + Source build/default/975_update_os_conf.sh + Source build/default/990_verify_rootfs.sh + Source build/default/995_md5sums_rootfs.sh + Source pack/GNU/Linux/900_create_initramfs.sh + Source output/default/010_set_umask.sh + Source output/default/100_mount_output_path.sh + Source output/default/150_save_copy_of_prefix_dir.sh + Source output/default/200_make_boot_dir.sh + Source output/default/200_make_prefix_dir.sh + Source output/default/250_create_lock.sh + Source output/ISO/Linux-ppc64le/300_create_grub2.sh + Source output/default/400_copy_disk_struct_files.sh + Source output/ISO/Linux-ppc64le/800_create_isofs.sh + Source output/ISO/Linux-ppc64le/810_prepare_multiple_iso.sh + Source output/ISO/Linux-ppc64le/820_create_iso_image.sh + Source output/default/940_grub2_rescue.sh + Source output/default/940_grub_rescue.sh + Source output/default/950_copy_result_files.sh + Source output/TSM/default/950_dsmc_save_result_files.sh + Source output/default/950_email_result_files.sh + Source output/TSM/default/960_dsmc_verify_isofile.sh + Source output/default/970_remove_lock.sh + Source output/default/980_umount_output_dir.sh + Exiting rear mkrescue (PID 8073) and its descendant processes ... + Running exit tasks + +#### [kai-uwe-rommel](https://github.com/kai-uwe-rommel) commented at [2024-04-08 12:42](https://github.com/rear/rear/issues/3189#issuecomment-2042650472): + +And this is the actual debug log from /var/log/rear: +[rear-mkrescue-debug.log](https://github.com/rear/rear/files/14905510/rear-mkrescue-debug.log) + +#### [kai-uwe-rommel](https://github.com/kai-uwe-rommel) commented at [2024-04-08 12:44](https://github.com/rear/rear/issues/3189#issuecomment-2042658547): + +After the "rear mkrescue" and after the failed boot I inspected the ISO +file: + + C:\IBM\Temp>7z l rear-hermes.iso + 7-Zip 22.00 (x64) : Copyright (c) 1999-2022 Igor Pavlov : 2022-06-15 + Scanning the drive for archives: + 1 file, 333985792 bytes (319 MiB) + Listing archive: rear-hermes.iso + -- + Path = rear-hermes.iso + Type = Iso + Physical Size = 333985792 + Created = 2024-04-06 14:01:04.00 + Modified = 2024-04-06 14:01:04.00 + Date Time Attr Size Compressed Name + ------------------- ----- ------------ ------------ ------------------------ + 2024-04-06 14:01:04 D.... boot + 2024-04-06 14:01:03 D.... boot\grub + 2024-04-06 14:01:03 ..... 115 115 boot\grub\grub.cfg + 2024-04-06 14:01:03 ..... 383756 383756 boot\grub\powerpc.elf + 2024-04-06 14:01:00 ..... 285125611 285125611 initrd.cgz + 2024-02-12 13:14:40 ..... 48086576 48086576 kernel + 2024-04-06 14:01:04 D.... ppc + 2024-04-06 14:01:03 ..... 160 160 ppc\bootinfo.txt + ------------------- ----- ------------ ------------ ------------------------ + 2024-04-06 14:01:04 333596218 333596218 5 files, 3 folders + +So there is both the initrd.cgz and the grub.cfg files. + +#### [kai-uwe-rommel](https://github.com/kai-uwe-rommel) commented at [2024-04-08 12:45](https://github.com/rear/rear/issues/3189#issuecomment-2042660324): + +And the grub.cfg file in the ISO contains: + + set timeout=100 + + menuentry "Relax-and-Recover" { + linux /kernel root=/dev/ram0 selinux=0 + initrd /initrd.cgz + } + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-08 14:27](https://github.com/rear/rear/issues/3189#issuecomment-2042901448): + +@kai-uwe-rommel +thank you for your prompt replies! + +Now I am stuck because everything looks OK +(as far as I can see) but things are not OK. + +I am not a booting expert and I know basically nothing +about POWER architecure specific things - I only know +that low-level things when booting on POWER architecture +are rather different compared to booting on x86 architecture. + +As far as I know currently we do not have +an active ReaR upstream maintainer who is an +expert in POWER architecure specific things. + +The /etc/os-release in your initial comment shows +you have "SUSE Linux Enterprise Server 15 SP5". + +Is this exactly what you actually have or do you perhaps have +a SUSE product/extension where SUSE officially supports ReaR? + +Because in your initial comment you wrote (excerpts): + + ReaR version 2.7 + ... + comes from SLES repository + +it indicates that you have a SUSE product/extension +where SUSE officially supports ReaR because otherwise +I assume you won't have ReaR from a SLES repository +(I mean from an official SUSE SLES repository). + +See the section "SUSE support for Relax-and-Recover" in +[https://en.opensuse.org/SDB:Disaster\_Recovery](https://en.opensuse.org/SDB:Disaster_Recovery) + +For example - as far as I know (no warranty) - the +SUSE Linux Enterprise High Availability Extension +is included in +SUSE Linux Enterprise Server for SAP Applications. + +If you have a SUSE product/extension +where SUSE officially supports ReaR +I would recommend that you file +an official support request at SUSE +via your official support contact at SUSE. +If you do that please provide an URL to this issue +in your SUSE support request. + +#### [kai-uwe-rommel](https://github.com/kai-uwe-rommel) commented at [2024-04-09 14:36](https://github.com/rear/rear/issues/3189#issuecomment-2045343704): + +It is indeed a SLES 15 and might even be a SLES 15 for SAP, I will have +to check. +When I started this, I just looked and found that SUSE made ReaR +available through their SLES 15 repository so this made things easier as +I did not have to add some other repo or install it manually. +When it comes from the standard SLES 15 repo, will SUSE then support +it? +I will open a ticket and try to get support ... + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-09 15:19](https://github.com/rear/rear/issues/3189#issuecomment-2045457648): + +As far as I know - at least in the usual cases - +SUSE provides Relax-and-Recover (ReaR) only via the +SUSE Linux Enterprise High Availability Extension (SLE-HA) +which is the only SUSE product/extension +where SUSE officially supports ReaR. + +But what matters in the end is what your particular +support contract states. + +It cannot be wrong to open a ticket at SUSE. +In the worst case you get notified that ReaR is not +supported with your current support contract. + +Your +[https://github.com/rear/rear/files/14837097/lsblk.txt](https://github.com/rear/rear/files/14837097/lsblk.txt) +shows "/hana/" mountpoints which indicate you run SAP HANA +which further indicates you have SLES for SAP Applications +which is rather common for SAP HANA on POWER architecture. + +#### [kai-uwe-rommel](https://github.com/kai-uwe-rommel) commented at [2024-04-09 15:20](https://github.com/rear/rear/issues/3189#issuecomment-2045459753): + +I have checked and we do have + +- High Availability Extension +- SLES for SAP Applications + So we should be covered. + +#### [kai-uwe-rommel](https://github.com/kai-uwe-rommel) commented at [2024-04-09 15:28](https://github.com/rear/rear/issues/3189#issuecomment-2045485364): + +The strange thing is that the SUSE customer portal does not let me +create a support case and insists that "You are not currently entitled +to use the support system" - we will have te check what is missing +there. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-12 12:38](https://github.com/rear/rear/issues/3189#issuecomment-2051683257): + +@kai-uwe-rommel + +an offhanded idea based on what I vaguely remember +from the past about booting issues which we had +here at ReaR upstream with POWER architecture: + +You use 'BACKUP=TSM' which means +all what TSM needs to restore a TSM backup +gets included in the ReaR recovery system. + +As far as I vaguely remember those TSM restore parts +are relatively big so the initrd which contains the +ReaR recovery system becomes relatively big. +In your case according to your +[https://github.com/rear/rear/issues/3189\#issuecomment-2042658547](https://github.com/rear/rear/issues/3189#issuecomment-2042658547) + + Date Time Attr Size Compressed Name + ... + 2024-04-06 14:01:00 ..... 285125611 285125611 initrd.cgz + +your initrd is about 272 MiB big +(285125611 / 1024 / 1024 = 271.9). + +As far as I vaguely remember there can be +weird (inexplicable) booting issues on POWER architecture +when the initrd is "somehow too big". +As far as I vaguely remember in such cases +there are no helpful (error) messages during booting +but is only "just somehow does not work". + +To find out if the initrd size causes this issue here +I would like to suggest that you replace +'BACKUP=TSM' with 'BACKUP=REQUESTRESTORE' i.e. + + OUTPUT=ISO + OUTPUT_URL=nfs://prometheus/nim/rear + BACKUP=REQUESTRESTORE + +then make a new ReaR recovery system ISO image with + + # rear -D mkrescue + +and inspected the new ISO same as what you did for your +[https://github.com/rear/rear/issues/3189\#issuecomment-2042658547](https://github.com/rear/rear/issues/3189#issuecomment-2042658547) +and post the numbers for the new ISO here. + +Also try out if that boots the ReaR recovery system +i.e. if it works that you can log in as 'root' +in the booted ReaR recovery system. + +You cannot use that new ISO to recreate a system +because with 'BACKUP=REQUESTRESTORE' +no backup restore software is included +in the ReaR recovery system, see the +'BACKUP=REQUESTRESTORE' description in "man rear" +e.g. online at +[https://github.com/rear/rear/blob/master/doc/rear.8.adoc](https://github.com/rear/rear/blob/master/doc/rear.8.adoc) + +That new ISO is only used here to find out +if the initrd size causes this issue here. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-12 12:50](https://github.com/rear/rear/issues/3189#issuecomment-2051701776): + +Only as a side note FYI: +I found an older issue about initrd size on POWER architecture +[https://github.com/rear/rear/issues/1142](https://github.com/rear/rear/issues/1142) +but that one is about booting on ppc64 via yaboot +while here GRUB2 is used as bootloader. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-12 13:08](https://github.com/rear/rear/issues/3189#issuecomment-2051730662): + +@kai-uwe-rommel +another offhanded idea what might help: + +In your +[https://github.com/rear/rear/files/14855435/rear-mkrescue-debug.log](https://github.com/rear/rear/files/14855435/rear-mkrescue-debug.log) +I see + + Using '/usr/bin/mkisofs' to create ISO filesystem images + +Nowadays 'xorrisofs' is in general considered "better" +than the traditional 'mkisofs' (I know basically nothing +about the details of making ISO images so I cannot explain +why 'xorrisofs' is considered "better" than 'mkisofs'). +Using 'xorrisofs' preferred was added via +[https://github.com/rear/rear/commit/fb5bebca167ce5e4ca151292c00958f41f4d9ba4](https://github.com/rear/rear/commit/fb5bebca167ce5e4ca151292c00958f41f4d9ba4) +that points to +[https://github.com/rear/rear/pull/962](https://github.com/rear/rear/pull/962) +but there is no explanation WHY 'xorrisofs' is preferred. +I guess at that time it was preferred just because +'xorrisofs' worked for Debian where the other ones +had issues. +But at least +[https://github.com/rear/rear/issues/3084](https://github.com/rear/rear/issues/3084) +shows a case where 'xorrisofs' is better on UEFI systems +where otherwise 'ebiso' was needed as a workaround, cf. +[https://github.com/rear/rear/issues/3084\#issuecomment-1835840904](https://github.com/rear/rear/issues/3084#issuecomment-1835840904) + +By default on SLES 15 you get '/usr/bin/mkisofs' installed +but 'xorrisofs' (in RPM package 'xorriso') should be also +available for SLES 15, cf. +[https://github.com/rear/rear/issues/3084\#issuecomment-1833496190](https://github.com/rear/rear/issues/3084#issuecomment-1833496190) + +When you install the RPM package 'xorriso' (this is possible +in addition to an already installed RPM package 'mkisofs') +then ReaR will automatically use 'xorrisofs' via how +ISO\_MKISOFS\_BIN is set in usr/share/rear/conf/default.conf +see +[https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/default.conf\#L893](https://github.com/rear/rear/blob/rear-2.7/usr/share/rear/conf/default.conf#L893) +and you will see that during "rear -D mkrescue" as + + Using '/usr/bin/xorrisofs' to create ISO filesystem images + +So please also try out if it helps in your case +when you use 'xorrisofs' to make the ISO image. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-03.3190.issue.closed.md b/docs/issues/2024-04-03.3190.issue.closed.md new file mode 100644 index 00000000..0ea97902 --- /dev/null +++ b/docs/issues/2024-04-03.3190.issue.closed.md @@ -0,0 +1,330 @@ +[\#3190 Issue](https://github.com/rear/rear/issues/3190) `closed`: Feature Idea: Create portable rear recover solution OUTPUT=PORTABLE +====================================================================================================================================== + +**Labels**: `enhancement`, `sponsored` + +#### [schlomo](https://github.com/schlomo) opened issue at [2024-04-03 13:36](https://github.com/rear/rear/issues/3190): + +DeaR @rear/contributors I'd like to collect your thoughts on the +following proposal to create a portable rear recovery solution. + +Problem +------- + +The problem to solve is to use `rear recover` based on another +(non-ReaR) rescue system instead of the ReaR rescue system. The reasons +are due to corporate requirements of a customer environment or due to +software compatibility problems between the ReaR rescue system and +proprietary backup software. + +Solution Approach +----------------- + +The idea is to create a new `OUTPUT=PORTABLE` method that will produce +an "ReaR Portable Recovery" archive containing + +- `/etc/rear` +- `/usr/share/rear` +- `/usr/sbin/rear` +- `/var/lib/rear` + +For recovery the user would to the following: + +1. start the recovery system from their rescue solution, e.g. the + [System Rescue CD](https://www.system-rescue.org/) +2. configure the system to work in the network and have the same + identity as the original system (`$HOSTNAME` etc.) +3. install - if needed - the proprietary backup agent or other software + required for the recovery +4. unpack the ReaR portable bundle to `/` +5. run `rear recover` as usual + +I expect users to potentially run into trouble with the automated +migration of hardware IDs for network, but for simple cases I expect +this to "just work". + +What do you think? What did I forget? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-03 13:57](https://github.com/rear/rear/issues/3190#issuecomment-2034694849): + +@schlomo +I like the generic idea very much. + +My reason (as far as I understand it) is that +with OUTPUT=PORTABLE it is possible to separate what belongs to +the ReaR recovery installer (i.e. what "rear recover" needs) +from what belongs to ReaR's bootable rescue system +so implementing it will help to get our code cleaner +via Keep Separated Items Separated ("KSIS"). + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-03 14:09](https://github.com/rear/rear/issues/3190#issuecomment-2034726801): + +Only an offhanded thought FYI +regarding + + 4. unpack the ReaR portable bundle to / + +I wonder if this may overwrite things +in the non-ReaR rescue system? + +Therefore I am wondering if it is possible +that also "rear recover" could run from within any directory +as "rear mkbackup" can, e.g. in a git checkout directory like + + git clone https://github.com/rear/rear.git + cd rear + vi etc/rear/local.conf + usr/sbin/rear mkbackup + +If this is possible, then with OUTPUT=PORTABLE +it could be possible to + + 4. unpack the ReaR portable bundle to any directory + and change into that directory + 5. in that directory run "usr/sbin/rear recover" + +What I mean is to get what belongs to +ReaR's recovery installer (i.e. what "rear recover" needs) +separated from what belongs to the non-ReaR rescue system. + +The idea behind is that what "rear recover" needs +could be different what there is in the non-ReaR rescue system, +e.g. "rear recover" may need certain specific binaries +like the exact binaries from the original system +and not whatever binaries in the non-ReaR rescue system. + +Or in other words: +How to ensure that what there is in the non-ReaR rescue system +is what "rear recover" needs? + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-05 17:15](https://github.com/rear/rear/issues/3190#issuecomment-2040286738): + +Update: Started to work on this in `portable-recovery` branch. After +some twiddling I got this far with a SystemRescueCD. + +In this test it failed to install the boot loaded because ReaR thinks it +is running on Arch Linux (SysRescueCD) and not the restored OS. + +Looks promising, but I need to make the system read the `os.conf` stuff +from the recovery info and not from the rescue system. Good opportunity +to finally get rid of the `os.conf` in `/etc/rear` 😃 + +Your thoughts are welcome. + + # hostnamectl hostname rear-ol8u7 + # REAR_VAR=/tmp/rear_var ./usr/sbin/rear -p -d recover + Relax-and-Recover 2.7 / Git + Running rear recover (PID 28794 date 2024-04-05 17:06:59) + Command line options: ./usr/sbin/rear -p -d recover + Using log file: /tmp/rear_var/log/rear/rear-rear-ol8u7.log + Using build area: /var/tmp/rear.cKm2s52s4lNQrlg + Setting TMPDIR to '/var/tmp/rear.cKm2s52s4lNQrlg/tmp' (was unset when ReaR was launched) + Running 'init' stage ====================== + Running 'setup' stage ====================== + Running 'verify' stage ====================== + Starting required daemons for NFS: RPC portmapper (portmap or rpcbind) and rpc.statd if available. + Started RPC portmapper 'rpcbind'. + RPC portmapper 'rpcbind' available. + RPC status rpc.statd available. + Started rpc.idmapd. + Using backup archive '/var/tmp/rear.cKm2s52s4lNQrlg/outputfs/rear-ol8u7/backup.tar.gz' + Will do driver migration (recreating initramfs/initrd) + Calculating backup archive size + Backup archive size is 3.2G /var/tmp/rear.cKm2s52s4lNQrlg/outputfs/rear-ol8u7/backup.tar.gz (compressed) + Running 'layout/prepare' stage ====================== + Comparing disks + Device sda has size 13958643712 bytes but 12884901888 bytes is expected (needs manual configuration) + Switching to manual disk layout configuration (GiB sizes rounded down to integer) + /dev/sda had size 12884901888 (12 GiB) but is now 13958643712 (13 GiB) + /dev/loop0 was not used on the original system and has now 717123584 (0 GiB) + Could not automap /dev/sda (no disk with same size 12884901888 found) + Original disk /dev/sda does not exist (with same size) in the target system + Using /dev/sda (the only available of the disks) for recreating /dev/sda + Current disk mapping table (source => target): + /dev/sda => /dev/sda + + UserInput -I LAYOUT_MIGRATION_CONFIRM_MAPPINGS needed in /home/schlomo/src/rear/usr/share/rear/layout/prepare/default/300_map_disks.sh line 334 + Confirm or edit the disk mapping + 1) Confirm disk mapping and continue 'rear recover' + 2) Confirm identical disk mapping and proceed without manual configuration + 3) Edit disk mapping (/tmp/rear_var/lib/rear/layout/disk_mappings) + 4) Use Relax-and-Recover shell and return back to here + 5) Abort 'rear recover' + (default '1' timeout 300 seconds) + 1 + UserInput: Valid choice number result 'Confirm disk mapping and continue 'rear recover'' + User confirmed disk mapping + Disabling excluded components in /tmp/rear_var/lib/rear/layout/disklayout.conf + Applied disk layout mappings to /tmp/rear_var/lib/rear/layout/disklayout.conf + Applied disk layout mappings to /tmp/rear_var/lib/rear/layout/config/df.txt + Trying to automatically resize last partition when disk size changed + Examining msdos device /dev/sda to automatically resize its last active partition + New /dev/sda is 1073741824 bytes bigger than old device + Checking /dev/sda1 if it is the last partition on /dev/sda + Checking /dev/sda2 if it is the last partition on /dev/sda + Found 'primary' partition /dev/sda2 as last partition on /dev/sda + Determining if last partition /dev/sda2 is resizeable + Determining new size for last partition /dev/sda2 + Determining if last partition /dev/sda2 actually needs to be increased or shrinked + Skip increasing last partition /dev/sda2 (new device less than 10% bigger) + UserInput -I LAYOUT_FILE_CONFIRMATION needed in /home/schlomo/src/rear/usr/share/rear/layout/prepare/default/500_confirm_layout_file.sh line 26 + Confirm or edit the disk layout file + 1) Confirm disk layout and continue 'rear recover' + 2) Edit disk layout (/tmp/rear_var/lib/rear/layout/disklayout.conf) + 3) View disk layout (/tmp/rear_var/lib/rear/layout/disklayout.conf) + 4) View original disk space usage (/tmp/rear_var/lib/rear/layout/config/df.txt) + 5) Use Relax-and-Recover shell and return back to here + 6) Abort 'rear recover' + (default '1' timeout 300 seconds) + 1 + UserInput: Valid choice number result 'Confirm disk layout and continue 'rear recover'' + User confirmed disk layout file + Marking component '/dev/sda' as done in /tmp/rear_var/lib/rear/layout/disktodo.conf + Marking component '/dev/sda1' as done in /tmp/rear_var/lib/rear/layout/disktodo.conf + Marking component '/dev/sda2' as done in /tmp/rear_var/lib/rear/layout/disktodo.conf + Marking component 'pv:/dev/sda2' as done in /tmp/rear_var/lib/rear/layout/disktodo.conf + Marking component '/dev/ol_rear-ol8u7' as done in /tmp/rear_var/lib/rear/layout/disktodo.conf + Marking component '/dev/mapper/ol_rear--ol8u7-swap' as done in /tmp/rear_var/lib/rear/layout/disktodo.conf + Marking component '/dev/mapper/ol_rear--ol8u7-root' as done in /tmp/rear_var/lib/rear/layout/disktodo.conf + Marking component 'fs:/' as done in /tmp/rear_var/lib/rear/layout/disktodo.conf + Marking component 'fs:/boot' as done in /tmp/rear_var/lib/rear/layout/disktodo.conf + Marking component 'swap:/dev/mapper/ol_rear--ol8u7-swap' as done in /tmp/rear_var/lib/rear/layout/disktodo.conf + Running 'layout/recreate' stage ====================== + UserInput -I LAYOUT_CODE_CONFIRMATION needed in /home/schlomo/src/rear/usr/share/rear/layout/recreate/default/100_confirm_layout_code.sh line 26 + Confirm or edit the disk recreation script + 1) Confirm disk recreation script and continue 'rear recover' + 2) Edit disk recreation script (/tmp/rear_var/lib/rear/layout/diskrestore.sh) + 3) View disk recreation script (/tmp/rear_var/lib/rear/layout/diskrestore.sh) + 4) View original disk space usage (/tmp/rear_var/lib/rear/layout/config/df.txt) + 5) Use Relax-and-Recover shell and return back to here + 6) Abort 'rear recover' + (default '1' timeout 300 seconds) + 1 + UserInput: Valid choice number result 'Confirm disk recreation script and continue 'rear recover'' + User confirmed disk recreation script + Disks to be completely overwritten and recreated by /tmp/rear_var/lib/rear/layout/diskrestore.sh: + /dev/sda + Determining disks to be wiped ... + UserInput -I WIPE_DISKS_CONFIRMATION needed in /home/schlomo/src/rear/usr/share/rear/layout/recreate/default/120_confirm_wipedisk_disks.sh line 168 + Disks to be wiped: /dev/sda + 1) Confirm disks to be wiped and continue 'rear recover' + 2) Skip wiping disks and continue 'rear recover' + 3) Use Relax-and-Recover shell and return back to here + 4) Abort 'rear recover' + (default '1' timeout 300 seconds) + 1 + UserInput: Valid choice number result 'Confirm disks to be wiped and continue 'rear recover'' + User confirmed disks to be wiped + Removed LVM physical volume /dev/sda3 + Wiping child devices of /dev/sda in reverse ordering: /dev/sda3 /dev/sda2 /dev/sda1 /dev/sda + Wiped first 16777216 bytes of /dev/sda3 + Wiped last 16777216 bytes of /dev/sda3 + Wiped first 16777216 bytes of /dev/sda2 + Wiped last 16777216 bytes of /dev/sda2 + Wiped first 16777216 bytes of /dev/sda1 + Wiped last 16777216 bytes of /dev/sda1 + Wiped first 16777216 bytes of /dev/sda + Wiped last 16777216 bytes of /dev/sda + Start system layout restoration. + Disk '/dev/sda': creating 'msdos' partition table + Disk '/dev/sda': creating partition number 1 with name 'primary' + Disk '/dev/sda': creating partition number 2 with name 'primary' + Creating LVM PV /dev/sda2 + Creating LVM VG 'ol_rear-ol8u7' (some properties may not be preserved) + Creating LVM volume 'ol_rear-ol8u7/swap' (some properties may not be preserved) + Creating LVM volume 'ol_rear-ol8u7/root' (some properties may not be preserved) + Creating filesystem of type xfs with mount point / on /dev/mapper/ol_rear--ol8u7-root. + Mounting filesystem / + Creating filesystem of type xfs with mount point /boot on /dev/sda1. + Mounting filesystem /boot + Creating swap on /dev/mapper/ol_rear--ol8u7-swap + Disk layout created. + Recreated storage layout: + NAME KNAME TRAN TYPE FSTYPE LABEL SIZE MOUNTPOINTS + /dev/loop0 /dev/loop0 loop squashfs 683.9M /run/archiso/sfs/airootfs + /dev/sda /dev/sda sata disk 13G + |-/dev/sda1 /dev/sda1 part xfs 1G /mnt/local/boot + `-/dev/sda2 /dev/sda2 part LVM2_member 11G + |-/dev/mapper/ol_rear--ol8u7-swap /dev/dm-0 lvm swap 1.2G + `-/dev/mapper/ol_rear--ol8u7-root /dev/dm-1 lvm xfs 9.8G /mnt/local + /dev/sr0 /dev/sr0 ata rom iso9660 RESCUE1002 763M /run/archiso/bootmnt + UserInput -I LAYOUT_MIGRATED_CONFIRMATION needed in /home/schlomo/src/rear/usr/share/rear/layout/recreate/default/200_run_layout_code.sh line 168 + Confirm the recreated disk layout or go back one step + 1) Confirm recreated disk layout and continue 'rear recover' + 2) Go back one step to redo disk layout recreation + 3) Use Relax-and-Recover shell and return back to here + 4) Abort 'rear recover' + (default '1' timeout 300 seconds) + 1 + UserInput: Valid choice number result 'Confirm recreated disk layout and continue 'rear recover'' + User confirmed recreated disk layout + Running 'restore' stage ====================== + Restoring from '/var/tmp/rear.cKm2s52s4lNQrlg/outputfs/rear-ol8u7/backup.tar.gz' (restore log in /tmp/rear_var/lib/rear/restore/recover.backup.tar.gz.28794.restore.log) ... + Restored 6817 MiB [avg. 86180 KiB/sec] OK + Restored 6893 MiB in 82 seconds [avg. 86089 KiB/sec] + Restoring finished (verify backup restore log messages in /tmp/rear_var/lib/rear/restore/recover.backup.tar.gz.28794.restore.log) + Created SELinux /mnt/local/.autorelabel file : after reboot SELinux will relabel all files + Recreating directories (with permissions) from /tmp/rear_var/lib/rear/recovery/directories_permissions_owner_group + Running 'finalize' stage ====================== + Checking if certain restored files are consistent with the recreated system + See /tmp/rear_var/lib/rear/layout/config/files.md5sum what files are checked + Restored files in /mnt/local do not fully match the recreated system + (files in the backup are not same as when the ReaR rescue/recovery system was made) + /mnt/local/src/rear/etc/rear/os.conf: FAILED open or read + /mnt/local/src/rear/etc/rear/site.conf: FAILED open or read + /mnt/local/src/rear/etc/rear/local.conf: FAILED open or read + Manually check if those changed files cause issues in your recreated system + Migrating disk-by-id mappings in certain restored files in /mnt/local to current disk-by-id mappings ... + /etc/multipath.conf not available in target system, creating it... + Copied /etc/multipath/bindings to target system /mnt/local + UserInput -I RESTORED_FILES_CONFIRMATION needed in /home/schlomo/src/rear/usr/share/rear/finalize/default/520_confirm_finalize.sh line 41 + Confirm restored config files are OK or adapt them as needed + 1) Confirm it is OK to recreate initrd and reinstall bootloader and continue 'rear recover' + 2) Edit restored etc/fstab (/mnt/local/etc/fstab) + 3) View restored etc/fstab (/mnt/local/etc/fstab) + 4) Use Relax-and-Recover shell and return back to here + 5) Abort 'rear recover' + (default '1' timeout 300 seconds) + 1 + UserInput: Valid choice number result 'Confirm it is OK to recreate initrd and reinstall bootloader and continue 'rear recover'' + User confirmed restored files + Installing GRUB2 boot loader... + Failed to generate boot/grub2/grub.cfg in /mnt/local - trying to install GRUB2 nevertheless + Determining where to install GRUB2 (no GRUB2_INSTALL_DEVICES specified) + Found possible boot disk /dev/sda - installing GRUB2 there + Failed to install GRUB2 on /dev/sda + Failed to install GRUB2 - you may have to manually install it + WARNING: + For this system + Arch/10.02 on Linux-i386 (based on 10.02/i386) + there is no code to install a boot loader on the recovered system + or the code that we have failed to install the boot loader correctly. + Please contribute appropriate code to the Relax-and-Recover project, + see http://relax-and-recover.org/development/ + Take a look at the scripts in /home/schlomo/src/rear/usr/share/rear/finalize - for example + for PC architectures like x86 and x86_64 see the script + /home/schlomo/src/rear/usr/share/rear/finalize/Linux-i386/660_install_grub2.sh + and for POWER architectures like ppc64le see the script + /home/schlomo/src/rear/usr/share/rear/finalize/Linux-ppc64le/660_install_grub2.sh + --------------------------------------------------- + | IF YOU DO NOT INSTALL A BOOT LOADER MANUALLY, | + | THEN YOUR SYSTEM WILL NOT BE ABLE TO BOOT. | + --------------------------------------------------- + You can use 'chroot /mnt/local bash --login' + to change into the recovered system and + manually install a boot loader therein. + + Running 'wrapup' stage ====================== + Finished 'recover'. The target system is mounted at '/mnt/local'. + Exiting rear recover (PID 28794) and its descendant processes ... + Running exit tasks + To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.cKm2s52s4lNQrlg + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-12 17:28](https://github.com/rear/rear/issues/3190#issuecomment-2052182530): + +Implemented via a9594b08339e32669d29f555b392606425da977f + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-03.3191.issue.closed.md b/docs/issues/2024-04-03.3191.issue.closed.md new file mode 100644 index 00000000..dd4a6b52 --- /dev/null +++ b/docs/issues/2024-04-03.3191.issue.closed.md @@ -0,0 +1,186 @@ +[\#3191 Issue](https://github.com/rear/rear/issues/3191) `closed`: x86\_64 UEFI system needs GRUB2\_IMAGE\_FORMAT=x86\_64-efi +============================================================================================================================= + +**Labels**: `bug`, `fixed / solved / done` + +#### [jsmeix](https://github.com/jsmeix) opened issue at [2024-04-03 15:48](https://github.com/rear/rear/issues/3191): + +This is related to the changes in +[https://github.com/rear/rear/pull/3157](https://github.com/rear/rear/pull/3157) + +On my openSUSE Leap 15.5 x86\_64 UEFI system +with current GitHub master code with +[https://github.com/rear/rear/commit/3db2724c7860e38fad96ba4d35c8b174616c1496](https://github.com/rear/rear/commit/3db2724c7860e38fad96ba4d35c8b174616c1496) +that makes "rear mkrescue" fail with + + ++ test + ++ LogPrintError 'grub2-mkstandalone may fail to make a bootable EFI image of GRUB2 (no /usr/*/grub*/i386-efi/moddep.lst file)' + ... + ++ grub2-mkstandalone -v '--modules=cryptodisk ext2 fat gcry_rijndael gcry_sha256 luks part_gpt part_msdos' -O i386-efi -o /var/tmp/rear.t0eFjv7bA3CjENy/tmp/mnt/EFI/BOOT/BOOTX64.efi /boot/ + grub/grub.cfg=/var/tmp/rear.t0eFjv7bA3CjENy/tmp/mnt/EFI/BOOT/grub.cfg + grub2-mkstandalone: error: /usr/share/grub2/i386-efi/modinfo.sh doesn't exist. Please specify --target or --directory. + +because I have + + # uname -m + x86_64 + + # find /usr -type f | grep 'moddep.lst' + /usr/share/grub2/x86_64-efi/moddep.lst + /usr/share/grub2/i386-pc/moddep.lst + + # find /usr -type f | grep 'modinfo.sh' + /usr/share/grub2/x86_64-efi/modinfo.sh + /usr/share/grub2/i386-pc/modinfo.sh + +so I need `GRUB2_IMAGE_FORMAT=x86_64-efi` in +usr/share/rear/prep/Linux-i386/330\_set\_efi\_arch.sh + + --- a/usr/share/rear/prep/Linux-i386/330_set_efi_arch.sh + +++ b/usr/share/rear/prep/Linux-i386/330_set_efi_arch.sh + @@ -16,7 +16,8 @@ case "$REAL_MACHINE" in + ;; + (x86_64) + EFI_ARCH=x64 + - GRUB2_IMAGE_FORMAT=i386-efi + + # GRUB2_IMAGE_FORMAT=i386-efi + + GRUB2_IMAGE_FORMAT=x86_64-efi + ;; + (*) + BugError "Unknown architecture $REAL_MACHINE" + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-03 15:55](https://github.com/rear/rear/issues/3191#issuecomment-2034993218): + +@pcahyna +for now this is just a quick issue report +so that it is not forgotten. +I will have a closer look tomorrow. + +At first glance in +usr/share/rear/prep/Linux-i386/330\_set\_efi\_arch.sh +[https://github.com/rear/rear/blob/master/usr/share/rear/prep/Linux-i386/330\_set\_efi\_arch.sh](https://github.com/rear/rear/blob/master/usr/share/rear/prep/Linux-i386/330_set_efi_arch.sh) +it looks like a typo + + (i686|i586|i386) + ... + GRUB2_IMAGE_FORMAT=x86_64-efi + (x86_64) + ... + GRUB2_IMAGE_FORMAT=i386-efi + +to set GRUB2\_IMAGE\_FORMAT +to `x86_64...` in case of `i686|i586|i386` +and to `i386...` in case of `x86_64`. + +@pcahyna +is this a typo or even intentional? + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-04-03 16:08](https://github.com/rear/rear/issues/3191#issuecomment-2035019760): + +@jsmeix Oops, I must have swapped the two! Really sorry. It is curious +how even a simplest change can break things. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-04 09:11](https://github.com/rear/rear/issues/3191#issuecomment-2036618057): + +I am confused when looking at +usr/share/rear/prep/Linux-i386/330\_set\_efi\_arch.sh +because it sets the GRUB2\_IMAGE\_FORMAT variable +only to `grub-mkstandalone --format` values +for EFI ('x86\_64-efi' and 'i386-efi') +regardless that prep/Linux-i386/330\_set\_efi\_arch.sh +is run for EFI and BIOS systems. + +So on a BIOS system GRUB2\_IMAGE\_FORMAT +is set to a value for EFI systems +which is a contradiction. + +The prep/Linux-\*/330\_set\_efi\_arch.sh scripts tell \[sic!\] + + # Se the variables even if USING_UEFI_BOOTLOADER empty or no explicit 'true' value + +so that is intentional but it does not tell why. + +As far as I see it works currently only because +the generic GRUB2\_IMAGE\_FORMAT variable +is only used in case of EFI + + # find usr/sbin/rear usr/share/rear/ -type f | xargs grep -l 'GRUB2_IMAGE_FORMAT' + + usr/share/rear/prep/Linux-i386/330_set_efi_arch.sh + usr/share/rear/prep/Linux-arm/330_set_efi_arch.sh + usr/share/rear/prep/Linux-ia64/330_set_efi_arch.sh + usr/share/rear/lib/uefi-functions.sh + usr/share/rear/output/RAWDISK/Linux-i386/270_create_grub2_efi_bootloader.sh + +but things will fail if GRUB2\_IMAGE\_FORMAT +will be also used for BIOS systems in the future. + +@pcahyna +could you explain the intent behind why on a BIOS system +GRUB2\_IMAGE\_FORMAT is set to a value for EFI systems? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-05 14:16](https://github.com/rear/rear/issues/3191#issuecomment-2039921140): + +This issue does not depend on openSUSE Leap 15.5 +but (probably) happens on all x86\_64 UEFI systems. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-04-05 19:43](https://github.com/rear/rear/issues/3191#issuecomment-2040521205): + +> I am confused when looking at +> usr/share/rear/prep/Linux-i386/330\_set\_efi\_arch.sh because it sets +> the GRUB2\_IMAGE\_FORMAT variable only to `grub-mkstandalone --format` +> values for EFI ('x86\_64-efi' and 'i386-efi') regardless that +> prep/Linux-i386/330\_set\_efi\_arch.sh is run for EFI and BIOS +> systems. +> +> So on a BIOS system GRUB2\_IMAGE\_FORMAT is set to a value for EFI +> systems which is a contradiction. + +The intent of the variable is to hold the image format for EFI GRUB on +the platform, so the variable is simply misnamed. I should have called +it `GRUB2_EFI_IMAGE_FORMAT`. + +> The prep/Linux-\*/330\_set\_efi\_arch.sh scripts tell \[sic!\] +> +> # Se the variables even if USING_UEFI_BOOTLOADER empty or no explicit 'true' value +> +> so that is intentional but it does not tell why. + +My bad, I should have indicated why and I don't even recall exactly, but +I believe it has referred to the EFI\_ARCH\* variables and not to +`GRUB2_IMAGE_FORMAT` originally. I think I did not want to audit all +uses of EFI\_ARCH\* whether they are protected by a test of for +`USING_UEFI_BOOTLOADER` (we don't want to refer to an undefined variable +and to prevent it the easiest way is to define it always). Actually I +see at least some use of EFI\_ARCH\* that is not conditional on +`USING_UEFI_BOOTLOADER`: +usr/share/rear/output/RAWDISK/Linux-i386/{270\_create\_grub2\_efi\_bootloader.sh,260\_create\_syslinux\_efi\_bootloader.sh}, +so setting the variable always is the right thing already. + +> As far as I see it works currently only because the generic +> GRUB2\_IMAGE\_FORMAT variable is only used in case of EFI +> +> # find usr/sbin/rear usr/share/rear/ -type f | xargs grep -l 'GRUB2_IMAGE_FORMAT' +> +> usr/share/rear/prep/Linux-i386/330_set_efi_arch.sh +> usr/share/rear/prep/Linux-arm/330_set_efi_arch.sh +> usr/share/rear/prep/Linux-ia64/330_set_efi_arch.sh +> usr/share/rear/lib/uefi-functions.sh +> usr/share/rear/output/RAWDISK/Linux-i386/270_create_grub2_efi_bootloader.sh +> +> but things will fail if GRUB2\_IMAGE\_FORMAT will be also used for +> BIOS systems in the future. + +Another variable will have to be introduced then. You may wonder why not +to have just one variable that holds the correct image format for the +given situation, regardless of whether it is UEFI or BIOS. I believe it +is better to keep it separated for the case when we need to handle a +hybrid boot setup. I think we are already creating bootable media with +ESP even on BIOS systems and we may be able to create hybrid boot media +in the future (maybe we are already able to do that in some situations). +We are also already capable of restoring hybrid bootloader setups. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-04.3192.pr.merged.md b/docs/issues/2024-04-04.3192.pr.merged.md new file mode 100644 index 00000000..e6481b67 --- /dev/null +++ b/docs/issues/2024-04-04.3192.pr.merged.md @@ -0,0 +1,72 @@ +[\#3192 PR](https://github.com/rear/rear/pull/3192) `merged`: Set GRUB2\_IMAGE\_FORMAT correctly in prep/Linux-i386/330\_set\_efi\_arch.sh +========================================================================================================================================== + +**Labels**: `bug`, `fixed / solved / done` + +#### [jsmeix](https://github.com/jsmeix) opened issue at [2024-04-04 09:33](https://github.com/rear/rear/pull/3192): + +- Type: **Bug Fix** + +- Impact: **High** + I assume all SUSE x86\_64 UEFI systems have + [https://github.com/rear/rear/issues/3191](https://github.com/rear/rear/issues/3191) + +- Reference to related issues (URLs): + [https://github.com/rear/rear/issues/3191](https://github.com/rear/rear/issues/3191) + [https://github.com/rear/rear/issues/3195](https://github.com/rear/rear/issues/3195) + +- How was this pull request tested? + [https://github.com/rear/rear/issues/3191\#issue-2223294278](https://github.com/rear/rear/issues/3191#issue-2223294278) + [https://github.com/rear/rear/issues/3195\#issuecomment-2039733438](https://github.com/rear/rear/issues/3195#issuecomment-2039733438) + +- Description of the changes in this pull request: + +In prep/Linux-i386/330\_set\_efi\_arch.sh +set GRUB2\_IMAGE\_FORMAT=i386-efi +in case of i686|i586|i386 +and +set GRUB2\_IMAGE\_FORMAT=x86\_64-efi +in case of x86\_64 + +Additionally describe current GRUB2\_IMAGE\_FORMAT usage +only in case of EFI so that currently it does not matter +when GRUB2\_IMAGE\_FORMAT is set to a value for EFI systems +also on BIOS systems, cf. +[https://github.com/rear/rear/issues/3191\#issuecomment-2036618057](https://github.com/rear/rear/issues/3191#issuecomment-2036618057) + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-04-04 20:11](https://github.com/rear/rear/pull/3192#issuecomment-2038124104): + +Add `Fixes #3191` to the description and the issue will be closed +automatically by GitHub after merge. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-05 12:36](https://github.com/rear/rear/pull/3192#issuecomment-2039698786): + +Because of +[https://github.com/rear/rear/issues/3195](https://github.com/rear/rear/issues/3195) +I will merge this one right now +so @edmcman can test it more easily. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-05 12:44](https://github.com/rear/rear/pull/3192#issuecomment-2039713956): + +I will wait for feedback, see +[https://github.com/rear/rear/issues/3195\#issuecomment-2039712157](https://github.com/rear/rear/issues/3195#issuecomment-2039712157) +until I set this one to "fixed/solved/done". + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-05 13:00](https://github.com/rear/rear/pull/3192#issuecomment-2039747769): + +Got positive feedback: +[https://github.com/rear/rear/issues/3195\#issuecomment-2039733438](https://github.com/rear/rear/issues/3195#issuecomment-2039733438) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-05 13:03](https://github.com/rear/rear/pull/3192#issuecomment-2039755797): + +@pcahyna +my pleasure to correcting your mistake! + +You corrected or avoided much more mistakes of me. + +Have a nice weekend! + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-04.3193.issue.open.md b/docs/issues/2024-04-04.3193.issue.open.md new file mode 100644 index 00000000..a6608fb9 --- /dev/null +++ b/docs/issues/2024-04-04.3193.issue.open.md @@ -0,0 +1,93 @@ +[\#3193 Issue](https://github.com/rear/rear/issues/3193) `open`: recover only disk layout (partitions, logical volumes and filesystems)? +======================================================================================================================================== + +**Labels**: `support / question` + +#### [Framsfex](https://github.com/Framsfex) opened issue at [2024-04-04 17:09](https://github.com/rear/rear/issues/3193): + +I have tested ReaR only once with Redhat (recovery was a desaster, I +will write another message to this topic). +Maybe I have overlooked it in the documentation: + +(How) Is it possible to backup+restore ONLY the disk layout with +partitions, logical volumes and filesystems but without content? + +Background of my question: I use linuxclone +([https://fex.belwue.de/linuxtools/linuxclone.html](https://fex.belwue.de/linuxtools/linuxclone.html)) +to backup my files and need only a disk layout recovery., because +linuxclone cannot handle LVM. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-05 12:29](https://github.com/rear/rear/issues/3193#issuecomment-2039673357): + +@Framsfex +I assume that in your + + backup+restore ONLY the disk layout + with partitions, logical volumes and filesystems + but without content + +"without content" means without a backup of the files. + +Use "rear mkrescue" to only create a bootable +ReaR recovery system that contains info about the +disk layout with partitions, logical volumes and filesystems +but without making a backup of the files. + +To only recreate the disk layout with partitions, +logical volumes and filesystems +but to not restore a backup of the files +there is the **experimental** workflow 'layoutonly' +so instead of "rear recover" (that also restores a backup) +you can try out how far "rear layoutonly" does what you need. +Because 'layoutonly' is **experimental** it may not (yet) do +what you need for your particular intended use case. + +Note that you are rather outside of what ReaR is usually +meant to do - which is to recreate a (destroyed) system +as it was before (as much as possible). + +#### [Framsfex](https://github.com/Framsfex) commented at [2024-04-06 08:12](https://github.com/rear/rear/issues/3193#issuecomment-2041012776): + +On Fri 2024-04-05 (05:30), Johannes Meixner wrote: + +> "without content" means without a backup of the files. + +Yes. I should have written it more precise. + +> Use "rear mkrescue" to only create a bootable +> ReaR recovery system that contains info about the +> disk layout with partitions, logical volumes and filesystems +> but no backup of the files. +> To only recreate the disk layout with partitions, +> logical volumes and filesystems +> but to not restore a backup of the files + +This is what I need! Great. + +> there is the **experimental** workflow 'layoutonly' +> so instead of "rear recover" (that also restores a backup) +> you can try out how far "rear layoutonly" does what you need. +> Because 'layoutonly' is **experimental** it may not (yet) do +> what you need for your particular intended use case. + +Thanks, I will test it! + +> Note that you are rather outside of what ReaR is usually +> meant to do - which is to recreate a (destroyed) system +> as it was before (as much as possible). + +Yes, I know, my use case is very special. + +-- +Ullrich Horlacher Server und Virtualisierung +Rechenzentrum TIK +Universitaet Stuttgart E-Mail: \*\*\*@\*\*\*.\*\*\* +Allmandring 30a Tel: ++49-711-68565868 +70569 Stuttgart (Germany) WWW: +[https://www.tik.uni-stuttgart.de/](https://www.tik.uni-stuttgart.de/) +\*\*\*@\*\*\*.\*\*\*> + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-04.3194.issue.open.md b/docs/issues/2024-04-04.3194.issue.open.md new file mode 100644 index 00000000..4a69271e --- /dev/null +++ b/docs/issues/2024-04-04.3194.issue.open.md @@ -0,0 +1,885 @@ +[\#3194 Issue](https://github.com/rear/rear/issues/3194) `open`: PBA environment is stuck in Linux kernel log messages +====================================================================================================================== + +**Labels**: `support / question`, `special hardware or VM` + +#### [edmcman](https://github.com/edmcman) opened issue at [2024-04-04 19:44](https://github.com/rear/rear/issues/3194): + + + +- ReaR version ("/usr/sbin/rear -V"): + +git eadcc683 (March 4th) + +- If your ReaR version is not the current version, explain why you + can't upgrade: + +I'm getting an error on `rear mkopalpba` with the latest version. I'll +file an issue for that shortly. + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): + + + + No LSB modules are available. + Distributor ID: Ubuntu + Description: Ubuntu 22.04.3 LTS + Release: 22.04 + Codename: jammy + +- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat + /etc/rear/local.conf"): + + + + OUTPUT=RAWDISK + OUTPUT_URL="file:///var/lib/rear/output" + SECURE_BOOT_BOOTLOADER="/boot/efi/EFI/ubuntu/shimx64.efi" + +- Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM + guest or PowerVM LPAR): + +PC System 76 + +- System architecture (x86 compatible or PPC64/PPC64LE or what exact + ARM device): + +x86 + +- Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or + ELILO or Petitboot): + +Open Firmware + +Unsure about bootloader, whatever PBA is using + +- Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or + multipath (DM or NVMe): + +local NVMe + +- Storage layout ("lsblk -ipo + NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"): + + + + /dev/nvme1n1 /dev/nvme1n1 nvme disk 1.9T + `-/dev/nvme1n1p1 /dev/nvme1n1p1 /dev/nvme1n1 nvme part vfat OPAL PBA 99M + /dev/nvme0n1 /dev/nvme0n1 nvme disk 465.8G + |-/dev/nvme0n1p1 /dev/nvme0n1p1 /dev/nvme0n1 nvme part vfat EFI 512M /boot/efi + |-/dev/nvme0n1p2 /dev/nvme0n1p2 /dev/nvme0n1 nvme part ext4 1.7G /boot + `-/dev/nvme0n1p3 /dev/nvme0n1p3 /dev/nvme0n1 nvme part crypto_LUKS 463.6G + `-/dev/mapper/nvme0n1p3_crypt + /dev/dm-0 /dev/nvme0n1p3 crypt LVM2_member 463.6G + |-/dev/mapper/vgubuntu-root + | /dev/dm-1 /dev/dm-0 lvm ext4 461.7G / + `-/dev/mapper/vgubuntu-swap_1 + /dev/dm-2 /dev/dm-0 lvm swap 1.9G [SWAP] + +- Description of the issue (ideally so that others can reproduce it): + +When I produce and install the PBA, it gets stuck while displaying some +Linux kernel status messages. + +![image](https://github.com/rear/rear/assets/1017189/605aff75-7a19-41e3-a4d5-b5bb9e60ecf8) + +- Workaround, if any: + +- Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" + debug log files): + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-05 14:33](https://github.com/rear/rear/issues/3194#issuecomment-2039956806): + +@edmcman +I am not a "rear mkopalpba" user. +I don't have a TCG Opal 2-compliant self-encrypting disk. +So I cannot reproduce anything with such hardware. + +From what your kernel messages show, inparticular + + ... firmware load for ... failed ... + ... no suitable firmware found! + +my blind guess is that the PBA image which is made +by "rear mkopalpba" does not contain the firmware +which the kernel (usualy via some udev mechanism) +tries to upload into the 'iwlwifi' hardware. + +In particular network hardware (and certain GPUs) +is known to need firmware uploaded into it to make it work. +So without such firmware the 'iwlwifi' hardware +won't work which means networking won't work +when this is the only networking hardware. +What is unexpected in this case is that it seems +the startup of the whole kernel is stuck when it fails +to upload firmware into the 'iwlwifi' hardware +regardless that I would assume a system should work +even without networking hardware. + +I don't know it is possible to inspect what there is +inside the PBA image? + +On my openSUSE Leap 15.5 workstation +I have lots of `/lib/firmware/iwlwifi-*` files. + +Do you have `/lib/firmware/` files in your PBA image? + +Perhaps "rear -D mkopalpba" tells something about +whether or not firmware is included in the PBA image? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-05 14:39](https://github.com/rear/rear/issues/3194#issuecomment-2039968870): + +I found the following documentation in +usr/share/rear/conf/default.conf + + # The following variable, if non-empty, overrides FIRMWARE_FILES for the PBA system. + OPAL_PBA_FIRMWARE_FILES=() + ... + # The by default empty FIRMWARE_FILES array means that + # usually all files in the /lib*/firmware/ directories + # get included in the rescue/recovery system but on certain + # architectures like ppc64 or ppc64le the default could be different + # cf. the conf/Linux-ppc64.conf and conf/Linux-ppc64le.conf scripts: + FIRMWARE_FILES=() + +so by default you should get all files +in the /lib\*/firmware/ directories included in the PBA image +(as far as I understand what is told in default.conf) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-05 14:50](https://github.com/rear/rear/issues/3194#issuecomment-2039997864): + +The only ReaR script that deals with OPAL\_PBA\_FIRMWARE\_FILES is +usr/share/rear/prep/OPALPBA/Linux-i386/001\_configure\_workflow.sh +which does what default.conf tells about OPAL\_PBA\_FIRMWARE\_FILES +so by default only FIRMWARE\_FILES matters. + +The only ReaR script that deals with FIRMWARE\_FILES is +usr/share/rear/build/GNU/Linux/420\_copy\_firmware\_files.sh +which should also be run for "rear mkopalpba" according to + + # usr/sbin/rear -s mkopalpba | grep firmware + Source build/GNU/Linux/420_copy_firmware_files.sh + +So I would suggest to run "rear -D mkopalpba" +and check the ReaR log file what happens after + + + source ...usr/share/rear/build/GNU/Linux/420_copy_firmware_files.sh + +and - if possible - also check what firmware files +got actually included in the PBA image. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-05 15:01](https://github.com/rear/rear/issues/3194#issuecomment-2040031507): + +Hmm... +perhaps the 'iwlwifi' firmware issue is not the reason +why the startup of the whole kernel is stuck because +before that there are kernel messages that look like +problems with the 'nvme' disk and such issues may let +the startup of the whole kernel stuck +and +before the 'nvme' messages there is another firmware issue +regarding 'regulatory.db' but that is also WIFI related +so probably not severe enough to let the kernel stuck? + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-05 15:07](https://github.com/rear/rear/issues/3194#issuecomment-2040044584): + +> perhaps the 'iwlwifi' firmware issue is not the reason +> why the startup of the whole kernel is stuck because +> before that there are kernel messages that look like +> problems with the 'nvme' disk and such issues may let +> the startup of the whole kernel stuck + +I think this is just a warning. This message is present during normal +Linux boots as well. + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-05 15:09](https://github.com/rear/rear/issues/3194#issuecomment-2040048104): + +I will take a look at the firmware. I initially ignored that because I +didn't think that missing firmware should cause the machine not to boot. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-05 15:16](https://github.com/rear/rear/issues/3194#issuecomment-2040064647): + +Another blind guess: + +Does the image with the kernel messages in your initial posting +show where the kernel is stuck? +Are there no further kernel (error or panic) messages? +I.e. all of a sudden there are no further kernel messages? + +In this case the kernel may just proceed rather well +but you only don't get further messages on your console. + +See the various 'CONSOLE' variables descriptions +in usr/share/rear/conf/default.conf for + + OPAL_PBA_USE_SERIAL_CONSOLE + USE_SERIAL_CONSOLE + SERIAL_CONSOLE_DEVICES + SERIAL_CONSOLE_DEVICES_KERNEL + SERIAL_CONSOLE_DEVICE_SYSLINUX + SERIAL_CONSOLE_DEVICE_GRUB + +what you may have to manually specify to ensure that +all kernel messages appear on your system console. + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-05 15:27](https://github.com/rear/rear/issues/3194#issuecomment-2040088856): + +There are very few files in the PBA image itself. In the initrd.cgz +image inside of the PBA, there are more, but I don't see any firmware +files. I will continue to investigate why that happens. I do not have +any firmware-related configuration options, so I would expect that the +default would be to include all the firmware, even in the PBA image. I +believe that the rescue image includes the firmware, but I will double +check that. + +Nothing appears on the screen beyond what is in the photo I posted. I +waited for a while (about an hour) and there are no other messages. + +I will investigate the console configs. Thanks for the idea. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-05 15:35](https://github.com/rear/rear/issues/3194#issuecomment-2040111954): + +Normally (i.e. for "rear mkrescue") the initrd contains +all the ReaR recovery system files. +What is booted when one boots for example from an ISO image +that was made by "rear mkrescue" is the kernel of the +original system where "rear mkrescue" was run +(I mean the original system's kernel is copied to be +used as kernel for what is booted e.g. from an ISO image) +plus ReaR's special initrd that contains all the files +of the ReaR recovery system (which runs in a ramdisk). + +I assume that with "rear mkopalpba" it is basically the same +i.e. that the initrd in the PBA image ontains all the files +of the system that is booted when one boot from a PBA image. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-05 15:40](https://github.com/rear/rear/issues/3194#issuecomment-2040122157): + +Regarding no further messages on the console see +[https://github.com/rear/rear/issues/2843](https://github.com/rear/rear/issues/2843) +in particular see +[https://github.com/rear/rear/issues/2843\#issuecomment-1196588256](https://github.com/rear/rear/issues/2843#issuecomment-1196588256) +what usually helps when you do not have a serial console. + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-05 17:08](https://github.com/rear/rear/issues/3194#issuecomment-2040278879): + +I can confirm that the mkrescue's initrd contains /lib/firmware, but +mkopalpba's initrd does not. + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-05 17:16](https://github.com/rear/rear/issues/3194#issuecomment-2040289231): + +From `rear -D mkopalpba` log file: + + + source /usr/share/rear/build/GNU/Linux/420_copy_firmware_files.sh + ++ is_false no + ++ case "$1" in + ++ return 0 + ++ LogPrint 'Omit copying files in /lib*/firmware/ (FIRMWARE_FILES='\''no'\'')' + 2024-04-05 13:14:48.813546125 Omit copying files in /lib*/firmware/ (FIRMWARE_FILES='no') + ++ return + + source_return_code=0 + + test 0 -eq 0 + + cd /home/ed + + test 1 + + Debug 'Leaving debugscript mode (back to previous bash flags and options settings).' + 2024-04-05 13:14:48.815543501 Leaving debugscript mode (back to previous bash flags and options settings). + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-05 17:22](https://github.com/rear/rear/issues/3194#issuecomment-2040296434): + +It seems that the PBA intentionally does not include firmware by default +to conform to the small 100 MiB size requirement: +[https://github.com/rear/rear/blob/master/usr/share/rear/prep/OPALPBA/Linux-i386/001\_configure\_workflow.sh\#L16](https://github.com/rear/rear/blob/master/usr/share/rear/prep/OPALPBA/Linux-i386/001_configure_workflow.sh#L16) + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-05 18:36](https://github.com/rear/rear/issues/3194#issuecomment-2040410425): + +Here's what I put in /etc/rear/local.conf: + + OUTPUT=RAWDISK + OUTPUT_URL="file:///var/lib/rear/output" + SECURE_BOOT_BOOTLOADER="/boot/efi/EFI/ubuntu/shimx64.efi" + OPAL_PBA_FIRMWARE_FILES+=( '*/iwlwifi-so-a0-gf-a0-83.ucode*' ) + USE_SERIAL_CONSOLE='no' + +Here is what I get: +![image](https://github.com/rear/rear/assets/1017189/62724ecb-4744-4a5d-8eb5-84e54a13dea5) + +On a normal boot, here are the iwlwifi messages: + + ed@banana:~$ sudo dmesg | fgrep iwlwifi + [ 12.822755] iwlwifi 0000:00:14.3: Detected crf-id 0x400410, cnv-id 0x80401 wfpm id 0x80000020 + [ 12.822780] iwlwifi 0000:00:14.3: PCI dev 7a70/0094, rev=0x430, rfid=0x2010d000 + [ 12.825493] iwlwifi 0000:00:14.3: api flags index 2 larger than supported by driver + [ 12.825508] iwlwifi 0000:00:14.3: TLV_FW_FSEQ_VERSION: FSEQ Version: 0.0.2.41 + [ 12.826166] iwlwifi 0000:00:14.3: loaded firmware version 83.e8f84e98.0 so-a0-gf-a0-83.ucode op_mode iwlmvm + [ 12.963263] iwlwifi 0000:00:14.3: Detected Intel(R) Wi-Fi 6E AX211 160MHz, REV=0x430 + [ 12.971359] iwlwifi 0000:00:14.3: WRT: Invalid buffer destination + [ 13.146853] iwlwifi 0000:00:14.3: WFPM_UMAC_PD_NOTIFICATION: 0x20 + [ 13.146862] iwlwifi 0000:00:14.3: WFPM_LMAC2_PD_NOTIFICATION: 0x1f + [ 13.146870] iwlwifi 0000:00:14.3: WFPM_AUTH_KEY_0: 0x90 + [ 13.146879] iwlwifi 0000:00:14.3: CNVI_SCU_SEQ_DATA_DW9: 0x0 + [ 13.147210] iwlwifi 0000:00:14.3: loaded PNVM version e28bb9d7 + [ 13.147279] iwlwifi 0000:00:14.3: RFIm is deactivated, reason = 5 + [ 13.163651] iwlwifi 0000:00:14.3: Detected RF GF, rfid=0x2010d000 + [ 13.232694] iwlwifi 0000:00:14.3: base HW address: ac:19:8e:9d:c8:1c + [ 13.279489] iwlwifi 0000:00:14.3 wlp0s20f3: renamed from wlan0 + [ 14.405180] iwlwifi 0000:00:14.3: WRT: Invalid buffer destination + [ 14.564884] iwlwifi 0000:00:14.3: WFPM_UMAC_PD_NOTIFICATION: 0x20 + [ 14.564914] iwlwifi 0000:00:14.3: WFPM_LMAC2_PD_NOTIFICATION: 0x1f + [ 14.564922] iwlwifi 0000:00:14.3: WFPM_AUTH_KEY_0: 0x90 + [ 14.564930] iwlwifi 0000:00:14.3: CNVI_SCU_SEQ_DATA_DW9: 0x0 + [ 14.566155] iwlwifi 0000:00:14.3: RFIm is deactivated, reason = 5 + [ 14.656388] iwlwifi 0000:00:14.3: Registered PHC clock: iwlwifi-PTP, with index: 0 + +So the +`[ 12.825493] iwlwifi 0000:00:14.3: api flags index 2 larger than supported by driver` +message is "normal". + +I'll try to add /lib/firmware/regulatory.db\* to the firmware files, but +I don't really think that the wifi is the problem. + +@jsmeix Setting USE\_SERIAL\_CONSOLE='no' should have provided more +console output, right? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-09 07:43](https://github.com/rear/rear/issues/3194#issuecomment-2044355011): + +Regarding +"PBA ... small 100 MiB size requirement": + +To me a 100 MiB size maximum for a bootable image +looks rather arbitrary and needlessly small +with nowadays disk and memory sizes. + +By googling for "PBA image size limit" I found +[https://github.com/Drive-Trust-Alliance/sedutil/issues/293](https://github.com/Drive-Trust-Alliance/sedutil/issues/293) +that tells +(I copy whole text parts to have them here saved): + + Technically on Opal TCG 2.0 compliant drives, + the PBA image can use the entire "Shadow MBR" + dedicated space which must be 128MB. + + In reality it's not a question of how big + the PBA image can be but how big it should be. + It should be (for most folks) a minimum size + that boots the system enough to authenticate + a credential set and unlock the drive, + then causes a reboot into the "real" MBR space + (or EFI space for newer drives). + + WinMagic (which has a commercial product + "SecureDoc" that can manage TCG drives) + has a pretty good explanation of it here: + https://www.winmagic.com/blog/how-seds-really-work/ + +Therein the link with link text + + https://www.winmagic.com/blog/how-seds-really-work/ + +has a wrong URL behind the link text +so one must copy&paste the link text +and then one gets (excerpts): + + Seagate DriveTrust and TCG Opal SEDs + have an MBR (Master Boot Record) Shadow. + ... + The "host" computer (i.e. the laptop) doesn't get + direct access to physical storage media ... + the drive presents a virtual view of the storage. + In the virtual view the storage space is presented + in logical blocks of 512 bytes to the host. + Each logical block gets a number assigned to it, + a Logical Block address (LBA). + The LBA starts at zero and goes up to a really big number + depending on the size of the drive. + The highest one is called Max LBA. + + When the drive ships from the factory it is already + encrypting everything written to it. + If you buy a laptop from an OEM (Lenovo, HP, etc) + with say Windows 7 pre-loaded on it the entire drive + is encrypted, even Windows. + This encryption is completely transparent to the OS + and at this point provides no protection of the data + because there is no authentication required to boot. + + When our SecureDoc software executes in Windows + it first checks if the drive is a SED. + If it isn't then we encrypt the drive using software + but if it is a SED we use TCG commands to create the + MBR shadow and write our pre-boot authentication (PBA) + software to it. + The MBR shadow is a 128 MB area that is totally "off the map". + If the OS or a virus were to read each LBA on the drive + from LBA 0 to MAX LBA it would still not to be able + to see it or modify it. + + Once MBR shadow is written by SecureDoc the drive is + configured to "lock" in the powered off state. + When the laptop is next turned on it doesn't see + the normal virtual space. + Rather it sees the MBR shadow in the 1st 128 MB + of the LBA's and above 128 MB to MAX LBA + it sees only 512 byte blocks of zeros. + That is, the MBR shadow is mapped into the first 128 MB + of virtual LBA space instead of the normal Windows MBR. + + The host boots into the SecureDoc PBA code in the MBR shadow + instead of Windows, authenticates the user (e.g. they type in + a password), and then unlocks the drive. + At this point SecureDoc maps the MBR shadow + out of the LBA virtual space + and the normal Windows MBR and the rest of the + original drive is mapped in. + The host is rebooted this time into Windows and + it boots up as it did before the machine was + protected with MBR shadow. + +Accordingly it seems the PBA image size limit +is the size of the MBR shadow and that seems to be +a hardware limit of the Self Encrypting Drive. +Because the text above tells +"we use TCG commands to create the MBR shadow" +so it could be possible to use TCG commands +to create a MBR shadow of arbitrary size? +But the MBR shadow could be different hardware storage +than the normal disk storage (totally "off the map") +so the size of the MBR shadow storage cannot be changed. + +At least the usual SED factory default seems to be +that the MBR shadow size is "128 MB". +I assume "128 MB" is actually 128 MiB. + +128 MB = 128 \* 1000 \* 1000 Bytes = 128000000 Bytes + +128 MiB = 128 \* 1024 \* 1024 Bytes = 134217728 Bytes + +128 MB is less than 6 MiB but more than 6 MB smaller +than 128 MiB so at least 120 MiB should be possible +for a PBA image. + +20 MiB more could make a noticeable difference +when a PBA image size is near the size limit. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-09 07:47](https://github.com/rear/rear/issues/3194#issuecomment-2044361097): + +By the way: +Regarding Self Encrypting Drives in general I found +[https://www.ibm.com/docs/en/psfa/7.2.1?topic=administration-about-self-encrypting-drives](https://www.ibm.com/docs/en/psfa/7.2.1?topic=administration-about-self-encrypting-drives) +that reads (excerpt): + + Self-encrypting drives (SEDs) encrypt data as it is written + to the disk. Each disk has a disk encryption key (DEK) + that is set at the factory and stored on the disk. + +So the whole security of a SED depends on +how secure that key is "stored on the disk". + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-09 08:31](https://github.com/rear/rear/issues/3194#issuecomment-2044442451): + +@edmcman +yes, USE\_SERIAL\_CONSOLE='no' *should* have provided +more console output +BUT +what actually matters is what 'console' kernel +command line options are set when booting the kernel. + +Normally (i.e. when booting the ReaR recovery system) +there is a ReaR recovery system bootloader screen +where you can see and if needed modify the +kernel command line options. +For example on a BIOS system +when booting a ReaR recovery system ISO image +with SYSLINUX/ISOLINUX as bootloader one can get +the ReaR recovery system kernel command line arguments +by selecting in the ReaR recovery system boot menue +the topmost enty of the form "Recover HOSTNAME" and then +pressing the \[Tab\] key which shows the kernel command line. + +I don't know if something like that is also available +when booting ReaR's PBA image. + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-10 13:10](https://github.com/rear/rear/issues/3194#issuecomment-2047506692): + +I can't seem to access any type of menu, but I was able to extract the +grub configuration file from the PBA image: + + insmod all_video + set timeout=0 + set default=0 + menuentry "TCG Opal pre-boot authentication" { + linux /vmlinuz-6.5.0-26-generic selinux=0 quiet splash systemd.volatile=yes systemd.unit=sysinit-opalpba.target vt.handoff=7 libata.allow_tpm=1 + initrd /initrd.cgz + } + +I'll try to remove `quiet splash` and see if that helps... + +#### [sei-eschwartz](https://github.com/sei-eschwartz) commented at [2024-04-10 13:43](https://github.com/rear/rear/issues/3194#issuecomment-2047583634): + +I was able to modify the grub config and set the timeout to 10, which +let me change the kernel arguments to whatever I wanted. I tried a few +things. First, I disabled iwlwifi. This removed the iwlwifi warnings, +but the boot did not seem to progress any further. Second, I added the +`ignore_loglevel` argument which causes more kernel logging: + +![image](https://github.com/rear/rear/assets/50171988/7a1b7dfb-c7ed-42ab-961a-eb599bc656c7) + +My suspicion, which could be wrong, is that the kernel is "booted" in +some sense, but we're either stuck in systemd, or maybe plymouth isn't +working as expected. + +I just saw that passing `debug` can make systemd log some more, so I'll +try that. And maybe `rd.plymouth=0 plymouth.enable=0`. + +#### [sei-eschwartz](https://github.com/sei-eschwartz) commented at [2024-04-10 14:06](https://github.com/rear/rear/issues/3194#issuecomment-2047646496): + +Good news! If I pass `nomodeset`, then I am prompted for the opal +password, and then the machine reboots. I don't currently have an OS +installed on the Opal drive, so I got an error about there not being an +OS installed, but that seems expected :-) + +So now we know that the problem has something to do with modesetting. I +wonder if it is missing firmware for the i915 driver? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-10 14:47](https://github.com/rear/rear/issues/3194#issuecomment-2047760674): + +Perhaps 'nomodeset' should be set by default +because according to what I understand how things work in +[https://github.com/rear/rear/issues/3194\#issuecomment-2044355011](https://github.com/rear/rear/issues/3194#issuecomment-2044355011) +the only purpose of booting what there is in the MBR shadow +is to unlock the drive and for that task the simple +kernel framebuffer driver with traditional VGA resolution +should be sufficient. + +Perhaps simple traditional VGA output to unlock the drive +(I assume what you get on the screen in this case +is only traditional text console output) +could be even an advantage compared to having that +e.g. in Full HD resolution only in the upper left quarter +of a 4K display (tiny text which is really hard to read). + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-10 14:48](https://github.com/rear/rear/issues/3194#issuecomment-2047763523): + +I found a bunch of PBA related settings in +[Ubuntu.conf](https://github.com/rear/rear/blob/master/usr/share/rear/conf/Ubuntu.conf) +and tried them: +[local.conf.txt](https://github.com/rear/rear/files/14933759/local.conf.txt) + +This still got stuck, which suggests the problem is perhaps the i915 +firmware. It's pretty big, though, so not sure if including it all will +work: + + ed@banana:~/rear$ du -ch /lib/firmware/i915/ + 27M /lib/firmware/i915/ + 27M total + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-10 14:52](https://github.com/rear/rear/issues/3194#issuecomment-2047772221): + +I would try to avoid special GPU stuff if possible. +When things look OK on the screen with 'nomodeset' +for the only purpose to unlock the drive +then I think 'nomodeset' is the simplest and +most straightforward way (i.e. KISS principle). + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-10 14:54](https://github.com/rear/rear/issues/3194#issuecomment-2047779009): + +Yes, I think if there is any general take away from my problems, it is +that `nomodeset` should probably be the default. + +One of the developers the OPAL functionality clearly was using Plymouth +for a fancier way of doing things: +[https://github.com/rear/rear/blob/master/usr/share/rear/conf/Ubuntu.conf](https://github.com/rear/rear/blob/master/usr/share/rear/conf/Ubuntu.conf) +[https://github.com/rear/rear/blob/b838a352136811900511a209d06c809ce552e636/usr/share/rear/skel/default/etc/scripts/unlock-opal-disks](https://github.com/rear/rear/blob/b838a352136811900511a209d06c809ce552e636/usr/share/rear/skel/default/etc/scripts/unlock-opal-disks) + +But I think most people would be happy with a working text prompt! + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-10 15:00](https://github.com/rear/rear/issues/3194#issuecomment-2047794207): + +Success! With the i915 firmware, I get the GUI prompt, which is pretty +slick I have to admit: + +![image](https://github.com/rear/rear/assets/1017189/5d175f00-2187-4f4b-9559-2af401e569a4) + +But IMHO: + +1. `nomodeset` should be the default option +2. There should be an optional way to disable this, and perhaps a + section added to + [https://github.com/rear/rear/blob/master/doc/user-guide/13-tcg-opal-support.adoc](https://github.com/rear/rear/blob/master/doc/user-guide/13-tcg-opal-support.adoc) + to discuss how to enable the GUI. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-10 15:01](https://github.com/rear/rear/issues/3194#issuecomment-2047796551): + +FYI: +For the bootloader setup of the ReaR recovery system +we set `vga=normal` + + usr/share/rear/conf/templates/RESULT_usage_RAMDISK.txt + --command-line='root=/dev/ram0 vga=normal' + + usr/share/rear/lib/bootloader-functions.sh + echo "append initrd=$REAR_INITRD_FILENAME root=/dev/ram0 vga=normal rw $KERNEL_CMDLINE" + echo "append initrd=$REAR_INITRD_FILENAME root=/dev/ram0 vga=normal rw $KERNEL_CMDLINE auto_recover $ISO_RECOVER_MODE" + echo " append initrd=$PXE_INITRD root=/dev/ram0 vga=normal rw $KERNEL_CMDLINE $PXE_RECOVER_MODE" + echo " append initrd=$PXE_HTTP_DOWNLOAD_URL/$PXE_INITRD root=/dev/ram0 vga=normal rw $KERNEL_CMDLINE $PXE_RECOVER_MODE" + echo "linux (tftp)/$PXE_KERNEL root=/dev/ram0 vga=normal rw $KERNEL_CMDLINE $PXE_RECOVER_MODE" + + usr/share/rear/output/USB/Linux-i386/300_create_extlinux.sh + append initrd=/$USB_PREFIX/$REAR_INITRD_FILENAME root=/dev/ram0 vga=normal rw $KERNEL_CMDLINE + append initrd=/$USB_PREFIX/$REAR_INITRD_FILENAME root=/dev/ram0 vga=normal rw $KERNEL_CMDLINE auto_recover + + usr/share/rear/output/default/940_grub2_rescue.sh + echo " linux $grub_boot_dir/$boot_kernel_name root=/dev/ram0 vga=normal rw $KERNEL_CMDLINE" + + usr/share/rear/output/PXE/default/810_create_pxelinux_cfg.sh + append initrd=$OUTPUT_PREFIX_PXE/$PXE_INITRD root=/dev/ram0 vga=normal rw $KERNEL_CMDLINE $PXE_RECOVER_MODE + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-10 15:06](https://github.com/rear/rear/issues/3194#issuecomment-2047808817): + +That's interesting. Those parameters do *not* make it into the PBA +image. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-10 15:10](https://github.com/rear/rear/issues/3194#issuecomment-2047817897): + +I am not a SED user but from what I see at +[https://github.com/rear/rear/issues/3194\#issuecomment-2047794207](https://github.com/rear/rear/issues/3194#issuecomment-2047794207) +I even think the "system76 ... Ubuntu" GUI prompt +is misleading because (as far as I understand what goes on) +what you run there is not your actual "system76 ... Ubuntu" +but ReaR's Pre-Boot Authorization (PBA) system from the MBR shadow +which was made by ReaR from parts of your "system76 ... Ubuntu" +but that PBA system is not your actual "system76 ... Ubuntu" +so it should not show up as if it was your "system76 ... Ubuntu". + +At least when I look at the screenshot in +[https://github.com/rear/rear/issues/3194\#issuecomment-2047794207](https://github.com/rear/rear/issues/3194#issuecomment-2047794207) +I would never ever think this is a ReaR PBA system +so it would be at least misleading to others +who fail to imagine what actaully goes on there. + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-10 15:17](https://github.com/rear/rear/issues/3194#issuecomment-2047835101): + +Yes, that is from the PBA image. System 76 is the PC manufacturer and +their logo shows in other Plymouth prompts as well. + +FWIW, as a sed user, I don't really care too much about the Ubuntu logo +appearing there. Perhaps it would be more appropriate to have a ReaR +logo, but I think anyone using a ReaR OPAL PBA probably knows what is +going on. A very simple mitigation could be to include ReaR in the text +prompt, e.g., "ReaR PBA: Enter password to unlock TCG OPAL disks". 🤷 + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-10 15:17](https://github.com/rear/rear/issues/3194#issuecomment-2047835236): + +I think 'vga=normal' alone cannot work +when the GPU is used but without firmware +because the GPU without firmware cannot show +any screen output (also none in 'vga=normal' mode). + +So I think 'nomodeset' is crucial when a GPU is used +that needs firmware (which many nowadays GPUs need) +to get screen output in simple console text mode +when one does not want to bother with firmware stuff +under the rather low size limits of the MBR shadow. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-10 15:20](https://github.com/rear/rear/issues/3194#issuecomment-2047841441): + +@edmcman +could you please also post a screenshot how it looks +with 'nomodeset' in simple console text mode i.e. for +[https://github.com/rear/rear/issues/3194\#issuecomment-2047646496](https://github.com/rear/rear/issues/3194#issuecomment-2047646496) +so I could get a better understanding how that things look +"in real world out there". + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-10 15:44](https://github.com/rear/rear/issues/3194#issuecomment-2047895860): + +So I just added `nomodeset` but left the i915 firmware and plymouth +specific config settings, and the GUI still appeared. So apparently KMS +isn't needed for the GUI. + +Still working on getting the CLI interface for you. + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-10 15:50](https://github.com/rear/rear/issues/3194#issuecomment-2047911097): + +![image](https://github.com/rear/rear/assets/1017189/8c95b174-cd16-4e30-8224-9836dbcb1f27) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-11 07:55](https://github.com/rear/rear/issues/3194#issuecomment-2049136226): + +This looks rather terse. + +Perhaps the `password_prompt` text in +usr/share/rear/skel/default/etc/scripts/unlock-opal-disks +could be a bit more descriptive to make it more clear +what it is about (but must still fit on a 80 columns +traditional console) like + + Enter password to unlock TCG Opal 2 disks + +or less tecnically accurate + + Enter password to unlock self-encrypting disks + +or both + + Enter password to unlock TCG Opal 2 self-encrypting disks + +where the latter results on the screen +via the ask\_for\_password() function + + Enter password to unlock TCG Opal 2 self-encrypting disks: + +with a trailing blank which is 59 characters long. + +Or in simpler wording - for the user it does not matter +how ('self' or via another method) it is encrypted +(this is a technical implementation detail) +what matters is that there are encrypted disks +which need to be unlocked: + + Enter password to unlock TCG Opal 2 encrypted disks: + +It matters for the user that the disks are TCG Opal 2 compliant +(i.e. the SEDs are encrypted in a TCG Opal 2 compliant way) +because other kind of SEDs are not supported according to +[https://github.com/rear/rear/blob/master/doc/user-guide/13-tcg-opal-support.adoc](https://github.com/rear/rear/blob/master/doc/user-guide/13-tcg-opal-support.adoc) + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-11 12:54](https://github.com/rear/rear/issues/3194#issuecomment-2049632675): + +I think the consumer of the message can be divided in two cases: (1) the +person who configured the drives and PBA, (2) someone else. + +The person who configured the drives really only needs the message to +tell them when the PBA is ready for the password. They already know the +drives are encrypted. + +Other people probably need slightly more context. I think a slightly +longer explanation would be more useful. Something like: + +"This computer is configured to boot from an encrypted disk. Please +enter the encryption password to unlock the disks:" + +That is longer than 80 characters, but is that really a problem? We +could put a line-break before the prompt sentence. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-11 13:41](https://github.com/rear/rear/issues/3194#issuecomment-2049722900): + +Some general understanding questions: + +I just don't understand +[https://github.com/rear/rear/blob/master/doc/user-guide/13-tcg-opal-support.adoc](https://github.com/rear/rear/blob/master/doc/user-guide/13-tcg-opal-support.adoc) +I confuses me what is where (USB stick vs. disk vs. SED). + +I thought you get the prompt + + Enter password to unlock disks: + +only when booting the ReaR PBA system +(from the shadow MBR of a SED). + +Now I am wondering what the purpose is +of that SED with the ReaR PBA system on it? + +Will that SED become your normal operating system disk +or will that SED become your ReaR recovery medium that +you only use to recreate your operating system? + +When that SED with the ReaR PBA system +becomes your normal operating system disk +it means (as far as I imagine things) that +when that SED is unlocked, +it will boot the normal operating system +and +when that SED is locked, +it will boot the ReaR PBA system. + +So any user of the computer with that SED +may get the ReaR PBA system booted +(when that SED is locked). + +If my above guess how all that belongs together is right, +I understand now why there was so much effort taken +to make booting the ReaR PBA system look nice +with splash screen and whatnot: +To make booting the computer with that SED +look mostly same as booting with a normal disk. + +In this case 'nomodeset' text-only booting by default +would make booting from a TCG Opal 2-compliant SED +look rather different for normal users compared to +booting from a normal (unencrypted) disk. + +Is my above guess how all that belongs together right +or do I misunderstand and confuse things? + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-11 14:39](https://github.com/rear/rear/issues/3194#issuecomment-2049849413): + +Overall I think you have a good understanding of how things will work. + +> Will that SED become your normal operating system disk +> or will that SED become your ReaR recovery medium that +> you only use to recreate your operating system? + +The former. It's kind of weird, but I came to the project for the PBA +support only :-) + +> In this case 'nomodeset' text-only booting by default +> would make booting from a TCG Opal 2-compliant SED +> look rather different for normal users compared to +> booting from a normal (unencrypted) disk. + +I could be wrong, but I think that `nomodeset` was still able to use +Plymouth for the nice GUI prompt and it didn't noticeably look different +to me from Plymouth outside of PBA. + +Also, one detail that you may not be thinking of is that after the PBA +unlocks the disks, the computer actually reboots. So I don't think the +PBA will ever be a completely seamless experience. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-11 14:56](https://github.com/rear/rear/issues/3194#issuecomment-2049888437): + +@edmcman +yes - of course - 'nomodeset' does not mean text-only. +Thank you for correcting me - I got confused. + +Cf. +[https://askubuntu.com/questions/716957/what-do-the-nomodeset-quiet-and-splash-kernel-parameters-mean](https://askubuntu.com/questions/716957/what-do-the-nomodeset-quiet-and-splash-kernel-parameters-mean) + + The newest kernels have moved the video mode setting + into the kernel. So all the programming of the hardware + specific clock rates and registers on the video card + happen in the kernel rather than in the X driver when + the X server starts.. This makes it possible to have + high resolution nice looking splash (boot) screens + and flicker free transitions from boot splash + to login screen. Unfortunately, on some cards this + doesn't work properly and you end up with a black screen. + Adding the nomodeset parameter instructs the kernel + to not load video drivers and use BIOS modes instead + until X is loaded. + +i.e. with 'nomodeset' one basically gets +a simple kernel video (framebuffer?) driver +with some traditional VGA resolution and likely +some lower frames per second rate (e.g. 60 Hz) +as I wrote above (where I was not yet confused) +[https://github.com/rear/rear/issues/3194\#issuecomment-2047760674](https://github.com/rear/rear/issues/3194#issuecomment-2047760674) +and that should be well sufficient for the only task +to enter the password to unlock the SED. + +Also thank you that you explicitly described that +after the PBA unlocked the disks, the computer reboots. +I assumed that from what I had read but now I know it +actually behaves this way. + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-11 15:36](https://github.com/rear/rear/issues/3194#issuecomment-2049981533): + +I was expecting `nomodeset` to disable the GUI as well. It was a +pleasant surprise. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-04.3195.issue.closed.md b/docs/issues/2024-04-04.3195.issue.closed.md new file mode 100644 index 00000000..bc7ad95f --- /dev/null +++ b/docs/issues/2024-04-04.3195.issue.closed.md @@ -0,0 +1,206 @@ +[\#3195 Issue](https://github.com/rear/rear/issues/3195) `closed`: `rear mkopalpba` fails +========================================================================================= + +**Labels**: `bug`, `fixed / solved / done`, +`won't fix / can't fix / obsolete` + +#### [edmcman](https://github.com/edmcman) opened issue at [2024-04-04 19:53](https://github.com/rear/rear/issues/3195): + + + +- ReaR version ("/usr/sbin/rear -V"): + +Relax-and-Recover 2.7 / Git + +ffdea7e7 + +- If your ReaR version is not the current version, explain why you + can't upgrade: + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): + + + + No LSB modules are available. + Distributor ID: Ubuntu + Description: Ubuntu 22.04.3 LTS + Release: 22.04 + Codename: jammy + +- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat + /etc/rear/local.conf"): + + + + OUTPUT=RAWDISK + OUTPUT_URL="file:///var/lib/rear/output" + SECURE_BOOT_BOOTLOADER="/boot/efi/EFI/ubuntu/shimx64.efi" + +- Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM + guest or PowerVM LPAR): + +System 76 PC + +- System architecture (x86 compatible or PPC64/PPC64LE or what exact + ARM device): + +x86 + +- Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or + ELILO or Petitboot): + +Open Firmware, GRUB + +- Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or + multipath (DM or NVMe): + +NVMe + +- Storage layout ("lsblk -ipo + NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"): + + + + /dev/nvme1n1 /dev/nvme1n1 nvme disk 1.9T + `-/dev/nvme1n1p1 /dev/nvme1n1p1 /dev/nvme1n1 nvme part vfat OPAL PBA 99M + /dev/nvme0n1 /dev/nvme0n1 nvme disk 465.8G + |-/dev/nvme0n1p1 /dev/nvme0n1p1 /dev/nvme0n1 nvme part vfat EFI 512M /boot/efi + |-/dev/nvme0n1p2 /dev/nvme0n1p2 /dev/nvme0n1 nvme part ext4 1.7G /boot + `-/dev/nvme0n1p3 /dev/nvme0n1p3 /dev/nvme0n1 nvme part crypto_LUKS 463.6G + `-/dev/mapper/nvme0n1p3_crypt + /dev/dm-0 /dev/nvme0n1p3 crypt LVM2_member 463.6G + |-/dev/mapper/vgubuntu-root + | /dev/dm-1 /dev/dm-0 lvm ext4 461.7G / + `-/dev/mapper/vgubuntu-swap_1 + /dev/dm-2 /dev/dm-0 lvm swap 1.9G [SWAP] + +- Description of the issue (ideally so that others can reproduce it): + +After updating to the latest git head, I get the following error when +running `rear mkopalpba`. Prior to updating, I did not receive this +error on eadcc683. + + root@banana:~# rear -d mkopalpba + Relax-and-Recover 2.7 / Git + Running rear mkopalpba (PID 110956 date 2024-04-04 15:48:18) + Command line options: /usr/sbin/rear -d mkopalpba + Using log file: /var/log/rear/rear-banana.log + Using build area: /var/tmp/rear.i338ZwpVcBBJ9za + Setting TMPDIR to '/var/tmp/rear.i338ZwpVcBBJ9za/tmp' (was unset when ReaR was launched) + Running 'init' stage ====================== + Running workflow mkopalpba on the normal/original system + Running 'prep' stage ====================== + Re-configuring Relax-and-Recover to create a TCG Opal pre-boot authentication (PBA) image + Found EFI system partition /dev/nvme0n1p1 on /boot/efi type vfat + Using UEFI Boot Loader for Linux (USING_UEFI_BOOTLOADER=1) + Using autodetected kernel '/boot/vmlinuz-6.5.0-26-generic' as kernel in the recovery system + Modified ReaR recovery system area after 'prep' stage (/var/tmp/rear.i338ZwpVcBBJ9za/rootfs contains regular files) + Running 'rescue' stage ====================== + Creating recovery system root filesystem skeleton layout + Skipping 'docker0': not bound to any physical interface. + Skipping 'lo': not bound to any physical interface. + Handling network interface 'wlp0s20f3' + wlp0s20f3 is a physical device + Handled network interface 'wlp0s20f3' + Skipping 'zcctun0': not bound to any physical interface. + Included current keyboard mapping (via 'dumpkeys -f') + No default US keyboard mapping included (no KEYMAPS_DEFAULT_DIRECTORY specified) + No support for different keyboard layouts (neither KEYMAPS_DEFAULT_DIRECTORY nor KEYMAPS_DIRECTORIES specified) + Using '/boot/efi/EFI/ubuntu/shimx64.efi' as UEFI Secure Boot bootloader file + Running 'build' stage ====================== + Copying files and directories + Copying binaries and libraries + A list of executables with their dependencies has been stored in /var/tmp/rear.i338ZwpVcBBJ9za/tmp/executable-dependencies + Copying only currently loaded kernel modules (MODULES contains 'loaded_modules') and those in MODULES_LOAD + Omit copying files in /lib*/firmware/ (FIRMWARE_FILES='no') + Skip copying broken symlink '/etc/mtab' target '/proc/126971/mounts' on /proc/ /sys/ /dev/ or /run/ + Skip copying broken symlink '/etc/resolv.conf' target '/run/systemd/resolve/stub-resolv.conf' on /proc/ /sys/ /dev/ or /run/ + Testing that the recovery system in /var/tmp/rear.i338ZwpVcBBJ9za/rootfs contains a usable system + Testing each binary with 'ldd' and look for 'not found' libraries within the recovery system + Testing that the existing programs in the PROGS array can be found as executable command within the recovery system + Testing that each program in the REQUIRED_PROGS array can be found as executable command within the recovery system + Running 'pack' stage ====================== + Creating recovery/rescue system initramfs/initrd initrd.cgz with gzip default compression + Created initrd.cgz with gzip default compression (68896025 bytes) in 5 seconds + Running 'output' stage ====================== + ERROR: Creating a raw disk image requires an EFI bootloader or syslinux + You may use debugscript mode '-D' for full debug messages with 'set -x' output + Aborting due to an error, check /var/log/rear/rear-banana.log for details + Exiting rear mkopalpba (PID 110956) and its descendant processes ... + Running exit tasks + To remove the build area you may use (with caution): rm -Rf --one-file-system /var/tmp/rear.i338ZwpVcBBJ9za + Terminated + +- Workaround, if any: + +- Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" + debug log files): + +[rear-banana.log](https://github.com/rear/rear/files/14876514/rear-banana.log) + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-04 20:01](https://github.com/rear/rear/issues/3195#issuecomment-2038105688): + +git bisect results + + 3db2724c7860e38fad96ba4d35c8b174616c1496 is the first bad commit + commit 3db2724c7860e38fad96ba4d35c8b174616c1496 + Author: Pavel Cahyna + Date: Mon Mar 4 15:56:53 2024 +0100 + + Introduce a variable for GRUB image format + + Use it as the argument of the -O option to + grub-mkstandalone/grub-mkimage instead of the hardcoded x86_64-efi. + + For easier porting to non-x86_64 EFI platforms. + + usr/share/rear/lib/uefi-functions.sh | 18 +++++++++--------- + .../Linux-i386/270_create_grub2_efi_bootloader.sh | 6 +++--- + usr/share/rear/prep/Linux-arm/330_set_efi_arch.sh | 2 ++ + usr/share/rear/prep/Linux-i386/330_set_efi_arch.sh | 3 +++ + 4 files changed, 17 insertions(+), 12 deletions(-) + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-04 20:03](https://github.com/rear/rear/issues/3195#issuecomment-2038110093): + +Looks like this is the problem: +[https://github.com/rear/rear/commit/3db2724c7860e38fad96ba4d35c8b174616c1496\#r140627966](https://github.com/rear/rear/commit/3db2724c7860e38fad96ba4d35c8b174616c1496#r140627966) + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-04 20:13](https://github.com/rear/rear/issues/3195#issuecomment-2038125809): + +Duplicate of \#3191 + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-05 12:43](https://github.com/rear/rear/issues/3195#issuecomment-2039712157): + +@edmcman +right now I merged +[https://github.com/rear/rear/pull/3192](https://github.com/rear/rear/pull/3192) + +Please test our current ReaR upstream GitHub master code +as described in the section +[https://en.opensuse.org/SDB:Disaster\_Recovery\#Testing\_current\_ReaR\_upstream\_GitHub\_master\_code](https://en.opensuse.org/SDB:Disaster_Recovery#Testing_current_ReaR_upstream_GitHub_master_code) +and +please provide feedback here whether or not your issue is then fixed. +I depend on your feedback because I am not a "rear mkopalpba" user. +I don't have a TCG Opal 2-compliant self-encrypting disk. + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-05 12:54](https://github.com/rear/rear/issues/3195#issuecomment-2039733438): + +@jsmeix I can confirm this fixed the error. I'll just note here for +posterity because of \#3194 I can't say that the generated PBA works, +but there is one generated now :-) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-05 12:58](https://github.com/rear/rear/issues/3195#issuecomment-2039745246): + +@edmcman +thank you for your prompt feedback - much appreciated! +Have a nice weekend! + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-04.3196.pr.closed.md b/docs/issues/2024-04-04.3196.pr.closed.md new file mode 100644 index 00000000..036aaff4 --- /dev/null +++ b/docs/issues/2024-04-04.3196.pr.closed.md @@ -0,0 +1,41 @@ +[\#3196 PR](https://github.com/rear/rear/pull/3196) `closed`: Fix swapped image formats in 330\_set\_efi\_arch.sh +================================================================================================================= + +**Labels**: `bug`, `won't fix / can't fix / obsolete` + +#### [edmcman](https://github.com/edmcman) opened issue at [2024-04-04 20:09](https://github.com/rear/rear/pull/3196): + +#### Relax-and-Recover (ReaR) Pull Request Template + +Please fill in the following items before submitting a new pull request: + +##### Pull Request Details: + +- Type: **Bug Fix** + +- Impact: **Normal** + +- Reference to related issue (URL): \#3195 + +- How was this pull request tested? I tested it locally. It fixed the + error I was receiving in \#3195. + +- Description of the changes in this pull request: It looks like the + programmer just mixed up the two grub2 image types. This commit + swaps them. + +#### [edmcman](https://github.com/edmcman) commented at [2024-04-04 20:11](https://github.com/rear/rear/pull/3196#issuecomment-2038123809): + +Oops, this is a duplicate of \#3192. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-05 12:56](https://github.com/rear/rear/pull/3196#issuecomment-2039738570): + +@edmcman +no need to "oops". +Thank you for your efforts to find the root cause on your own +regardless that I was a little bit faster this time :-) + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-05.3197.issue.closed.md b/docs/issues/2024-04-05.3197.issue.closed.md new file mode 100644 index 00000000..2371631c --- /dev/null +++ b/docs/issues/2024-04-05.3197.issue.closed.md @@ -0,0 +1,80 @@ +[\#3197 Issue](https://github.com/rear/rear/issues/3197) `closed`: Bug: /var/run should be a symlink to /run, not a directory +============================================================================================================================= + +#### [schlomo](https://github.com/schlomo) opened issue at [2024-04-05 14:31](https://github.com/rear/rear/issues/3197): + +We found a - I think legacy - bug in current ReaR: + +`/var/run` and `/run` are both directories, whereas on all modern Linux +distros one is a symlink to the other and systemd actually expects all +PID files to be under `/run`: + +My test shows that *only CentOS 6* comes with a dedicated `/var/run` +directory: + + $ tools/run-in-docker -- ls -ldr /var/run + ********** ubuntu:18.04 ********** + lrwxrwxrwx 1 root root 4 May 30 2023 /var/run -> /run + ********** ubuntu:20.04 ********** + lrwxrwxrwx 1 root root 4 Oct 3 2023 /var/run -> /run + ********** ubuntu:22.04 ********** + lrwxrwxrwx 1 root root 4 Jan 11 14:03 /var/run -> /run + ********** ubuntu:23.04 ********** + lrwxrwxrwx 1 root root 4 Oct 4 2023 /var/run -> /run + ********** ubuntu:devel ********** + lrwxrwxrwx 1 root root 4 Sep 26 2023 /var/run -> /run + ********** opensuse/leap:42 ********** + lrwxrwxrwx 1 root root 4 Aug 13 2019 /var/run -> /run + ********** opensuse/leap:15 ********** + lrwxrwxrwx 1 root root 4 Oct 31 18:18 /var/run -> /run + ********** registry.suse.com/suse/sle15 ********** + lrwxrwxrwx 1 root root 4 Oct 31 14:37 /var/run -> /run + ********** centos:6 ********** + drwxr-xr-x 1 root root 4096 Nov 13 16:53 /var/run + ********** centos:7 ********** + lrwxrwxrwx 1 root root 6 Nov 13 2020 /var/run -> ../run + ********** centos:8 ********** + lrwxrwxrwx 1 root root 6 Sep 15 2021 /var/run -> ../run + ********** sl:7 ********** + lrwxrwxrwx 1 root root 6 Nov 1 17:57 /var/run -> ../run + ********** quay.io/centos/centos:stream8 ********** + lrwxrwxrwx 1 root root 6 Nov 6 04:51 /var/run -> ../run + ********** quay.io/centos/centos:stream9 ********** + lrwxrwxrwx 1 root root 6 Nov 6 04:09 /var/run -> ../run + ********** fedora:29 ********** + lrwxrwxrwx 1 root root 6 Mar 7 2019 /var/run -> ../run + ********** fedora:31 ********** + lrwxrwxrwx 1 root root 6 Jun 7 2020 /var/run -> ../run + ********** fedora:34 ********** + lrwxrwxrwx 1 root root 6 Feb 21 2022 /var/run -> ../run + ********** fedora:37 ********** + lrwxrwxrwx 1 root root 6 Nov 4 05:50 /var/run -> ../run + ********** fedora:38 ********** + lrwxrwxrwx 1 root root 6 Nov 5 06:48 /var/run -> ../run + ********** archlinux ********** + lrwxrwxrwx 1 root root 6 Sep 18 2023 /var/run -> ../run + ********** manjarolinux/base ********** + lrwxrwxrwx 1 root root 6 Sep 22 2023 /var/run -> ../run + ** SCRIPT RUN TIME 13 SECONDS ** + +In the specific example a systemd service fails to start because systemd +simply rewrites the `PIDFile` setting from `/var/run` to `/run` but the +unit definition creates the PID file under `/var/run`. + +On the original system this is not a problem because `/var/run` is a +symlink to `/run`, but in our ReaR Rescue system this creates a failure +as the service creates the PID file in `/var/run` while systemd then +looks for it in `/run`. + +I propose updating ReaR to also use a symlink for `/var/run` as +suggested by +[FHS](https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s13.html) + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-05 14:56](https://github.com/rear/rear/issues/3197#issuecomment-2040022950): + +Thanks to @idna38 for reporting this bug to me + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-05.3198.issue.closed.md b/docs/issues/2024-04-05.3198.issue.closed.md new file mode 100644 index 00000000..dc68f036 --- /dev/null +++ b/docs/issues/2024-04-05.3198.issue.closed.md @@ -0,0 +1,92 @@ +[\#3198 Issue](https://github.com/rear/rear/issues/3198) `closed`: Bug: ReaR should abort as early as possible if rear recover is run outside rescue system +=========================================================================================================================================================== + +#### [schlomo](https://github.com/schlomo) opened issue at [2024-04-05 16:32](https://github.com/rear/rear/issues/3198): + +Problem +------- + +While working on the \#3190 story I noticed that we first apply an +update downloaded from `$RECOVERY_UPDATE_URL` and *then* check if we are +actually in the recovery system :face\_with\_head\_bandage:. This could +lead to an update getting applied to an origin system if somebody would +accidentally (or maliciously) run `rear recover` there. + +![image](https://github.com/rear/rear/assets/101384/791e843a-8b08-4416-aaaa-ce83f1c8e229) + +Proposed Solution +----------------- + +Check very early for recovery mode + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-10 07:06](https://github.com/rear/rear/issues/3198#issuecomment-2046675591): + +init/default/030\_update\_recovery\_system.sh contains + + test "$WORKFLOW" != "recover" && return + +so I wonder how it could apply an update downloaded +into the original system? + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-10 13:09](https://github.com/rear/rear/issues/3198#issuecomment-2047503447): + +Yes, still felt wrong to check so late for recovery mode 🤷 + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-10 13:27](https://github.com/rear/rear/issues/3198#issuecomment-2047546343): + +Of course the check for recovery mode was too late +but I asked because I do not understand your +initial description of this issue which looks +as if you got update downloaded appiled onto +your original system by 030\_update\_recovery\_system.sh +i.e. as if I made a severe bug in that script. + +FYI about its history: + + # git log -p --follow usr/share/rear/init/default/050_check_rear_recover_mode.sh + +shows at its end i.e. at the very beginning of time +(excerpts): + + commit d8f1571a213a9df272327bb070e8a87f78fc14c3 + Author: Johannes Meixner + Date: Thu Oct 27 12:44:16 2016 +0200 + + renumbered by ading a trailing 0 so that 12 becomes 120 + except 00 which becomes 005 and adapted symlinks to point + again to the right re-numbered scripts + ... + rename from usr/share/rear/init/default/05_check_rear-recover_mode.sh + rename to usr/share/rear/init/default/050_check_rear-recover_mode.sh + + commit ad2283c402736253e4f76d36659f353695aeceea + Author: Gratien D'haese + Date: Fri Dec 11 18:14:31 2015 +0100 + + prevent any other workflow in REAR rescue mode + then recover - see issue #719 + ... + --- /dev/null + +++ b/usr/share/rear/init/default/05_check_rear-recover_mode.sh + +so 05\_check\_rear-recover\_mode.sh +(relatively early in 00...99) +became 050\_check\_rear-recover\_mode.sh +(relatively same early in 000...999). + +So my fault was to add 030\_update\_recovery\_system.sh +before 050\_check\_rear-recover\_mode.sh +so I needed the extra check +instead of after it and then leave out the check. + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-10 13:56](https://github.com/rear/rear/issues/3198#issuecomment-2047621768): + +Ah, so I sort of fixed what happened at that old XX to XXX change? Nice. + +I didn't have a problem with unplanned system updates, it was more of a +thought. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-05.3199.issue.open.md b/docs/issues/2024-04-05.3199.issue.open.md new file mode 100644 index 00000000..b19ca4ce --- /dev/null +++ b/docs/issues/2024-04-05.3199.issue.open.md @@ -0,0 +1,183 @@ +[\#3199 Issue](https://github.com/rear/rear/issues/3199) `open`: Proceed regardless that 'mkrescue' could be destructive with a 'iso' BACKUP\_URL? +================================================================================================================================================== + +**Labels**: `support / question` + +#### [iringor](https://github.com/iringor) opened issue at [2024-04-05 20:42](https://github.com/rear/rear/issues/3199): + + + +- ReaR version ("/usr/sbin/rear -V"): + Relax-and-Recover 2.6 / 2020-06-17 + This also happens to Relax-and-Recover 2.4 / Git for RHEL 7.9 + +- If your ReaR version is not the current version, explain why you + can't upgrade: + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): + NAME="Red Hat Enterprise Linux" + VERSION="8.6 (Ootpa)" + +- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat + /etc/rear/local.conf"): + For details, please check it from + [https://github.com/iringor/rear.git](https://github.com/iringor/rear.git) + \[root@dvsdsas1 ~\]\# cat /etc/rear/site.conf | grep . | grep -v + ^\# + export TMPDIR="/rear" + OUTPUT=ISO + ISO\_DIR="/rear" + OUTPUT\_URL=null + BACKUP=NETFS + BACKUP\_URL=iso:///backup + REL=`cat /etc/redhat-release |awk -F"release" '{print $2}' |awk -F. '{print $1}'` + if \[ $REL -le 6 \]; then + LISTOFBACKUP="$(df -T -x tmpfs -x devtmpfs -x devpts -x sysfs -x + proc | grep -oE '/\\S+' | grep -v /dev/ | awk '! /^/$|^/boot$/ + {print}'| tr '\\n' ' ')" + else + LISTOFBACKUP="$(df -T -x tmpfs -x devtmpfs -x devpts -x sysfs -x + proc | awk 'NR>1{print $7}' | awk '! /^/$|^/boot$/ {print}'| tr + '\\n' ' ')" + fi + LISTOFBACKUP="${LISTOFBACKUP} /var/log/cups /var/log/httpd " + if \[ -f /usr/local/sbin/mondoexclude.txt \]; then + LISTOFBACKUP="${LISTOFBACKUP}`cat /usr/local/sbin/mondoexclude.txt`" + fi + BACKUP\_PROG\_EXCLUDE=("${BACKUP\_PROG\_EXCLUDE\[@\]}" + ${LISTOFBACKUP}) + ISO\_MAX\_SIZE=4400 + MIGRATION\_MODE=false + +- Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM + guest or PowerVM LPAR): + VMware + +- System architecture (x86 compatible or PPC64/PPC64LE or what exact + ARM device): + x86 + +- Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or + ELILO or Petitboot): + BIOS GRUB + +- Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or + multipath (DM or NVMe): + local disk + +- Storage layout ("lsblk -ipo + NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"): + \[root@dvsdsas1 ~\]\# lsblk -ipo + NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT + NAME KNAME PKNAME TRAN TYPE FSTYPE LABEL SIZE MOUNTPOINT + /dev/sda /dev/sda disk 20G + |-/dev/sda1 /dev/sda1 /dev/sda part xfs 1G /boot + `-/dev/sda2 /dev/sda2 /dev/sda part LVM2_member 19G |-/dev/mapper/VolGroup01-root /dev/dm-0 /dev/sda2 lvm xfs 17G / `-/dev/mapper/VolGroup01-swap + /dev/dm-1 /dev/sda2 lvm swap 2G \[SWAP\] + /dev/sdb /dev/sdb disk 200G + `-/dev/sdb1 /dev/sdb1 /dev/sdb part LVM2_member 200G |-/dev/mapper/VolGroup02-LogVol01 /dev/dm-2 /dev/sdb1 lvm xfs 100G /usr/sap/sapmnt |-/dev/mapper/VolGroup02-LogVol02 /dev/dm-3 /dev/sdb1 lvm xfs 40G /install |-/dev/mapper/VolGroup02-LogVol03 /dev/dm-4 /dev/sdb1 lvm xfs 10G /u01 |-/dev/mapper/VolGroup02-lv_swap2 /dev/dm-5 /dev/sdb1 lvm swap 16G [SWAP] `-/dev/mapper/VolGroup02-LogVol04 + /dev/dm-6 /dev/sdb1 lvm xfs 10G /rear + /dev/sr0 /dev/sr0 sata rom 1024M + +- Description of the issue (ideally so that others can reproduce + it): + I am getting this email notification everyday even if the mkbackup + script is only running once a month. + +From: (Cron Daemon) +Sent: Thursday, April 4, 2024 1:30 AM +To: +Subject: Cron test -f /var/lib/rear/layout/disklayout.conf +&& /usr/sbin/rear checklayout || /usr/sbin/rear mkrescue + +Proceed regardless that 'mkrescue' could be destructive with a 'iso' +BACKUP\_URL ? +(default 'No' timeout 300 seconds) +ERROR: The mkrescue workflow does not work for the BACKUP\_URL scheme +'iso' +Some latest log messages since the last called script +040\_check\_backup\_and\_output\_scheme.sh: +2024-04-04 01:30:24.256120130 Including +prep/default/040\_check\_backup\_and\_output\_scheme.sh +2024-04-04 01:30:24.314857703 UserInput: called in +/usr/share/rear/prep/default/040\_check\_backup\_and\_output\_scheme.sh +line 30 +2024-04-04 01:30:24.350941007 UserInput: No choices specified +2024-04-04 01:30:24.368811100 Proceed regardless that 'mkrescue' could +be destructive with a 'iso' BACKUP\_URL ? +2024-04-04 01:30:24.386439123 (default 'No' timeout 300 seconds) +2024-04-04 01:30:24.422376325 UserInput: 'read' timed out with non-zero +exit code +2024-04-04 01:30:24.477753667 Aborting 'mkrescue' with a 'iso' +BACKUP\_URL by default Aborting due to an error, check +/var/log/rear/rear-dvsdsas1.log for details + +- Workaround, if any: + +- Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" + debug log files): + +You can drag-drop log files into this editor to create an attachment +or paste verbatim text like command output or file content +by including it between a leading and a closing line of +three backticks like this: + + verbatim content + +#### [iringor](https://github.com/iringor) commented at [2024-04-09 23:17](https://github.com/rear/rear/issues/3199#issuecomment-2046187080): + +I think the issue is resolved. I was not aware that when installing +rear, it automatically creates a cron for rear mkrescue that runs +everyday at 1:30 AM and it looks like this is a default setting. I will +comment out this cron for now and if there is no more recurring issue +related to this in a week I will close this case. Thanks. + +\[root@server ~\]\# grep -R rear /etc/cron\* +/etc/cron.d/rear:\#30 1 \* \* \* root /usr/sbin/rear checklayout || +/usr/sbin/rear mkrescue + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-11 10:56](https://github.com/rear/rear/issues/3199#issuecomment-2049430339): + +@iringor +it seems your issue is similar like this one +[https://github.com/rear/rear/issues/2787](https://github.com/rear/rear/issues/2787) +where also a cron job broke things. + +The /etc/cron.d/rear/ related things +were removed for ReaR 2.5 see +[https://github.com/rear/rear/issues/2139](https://github.com/rear/rear/issues/2139) +and +[https://github.com/rear/rear/issues/1892](https://github.com/rear/rear/issues/1892) + +See the comments therein why that cron thing is broken +and why I and others are against such cron things. + +It seems your issue here is just another case +that shows why that cron thing is broken. + +I guess your cron job is a leftover from ReaR 2.4 + +#### [iringor](https://github.com/iringor) commented at [2024-04-11 20:25](https://github.com/rear/rear/issues/3199#issuecomment-2050470048): + +Hi Johannes, + +Yes, it looks similar. But my issue is now fixed. I could not access +[https://relax-and-recover.org/](https://relax-and-recover.org/) before +when I started researching about ReaR. The site was blocked by our IT +admin for some reason. I didn't find this in the man pages, but it is +mentioned in +[https://relax-and-recover.org/usage/](https://relax-and-recover.org/usage/) +that a cron job can be created (or automatically created?) to sync with +the rescue image. It is actually a good tool, this "rear checklayout," +to alert me if there is a need to create a new image. Thank you. +-Don Ringor + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-06.3200.issue.open.md b/docs/issues/2024-04-06.3200.issue.open.md new file mode 100644 index 00000000..55d13450 --- /dev/null +++ b/docs/issues/2024-04-06.3200.issue.open.md @@ -0,0 +1,230 @@ +[\#3200 Issue](https://github.com/rear/rear/issues/3200) `open`: Redhat recover failed: / cannot be remounted rw +================================================================================================================ + +**Labels**: `support / question` + +#### [Framsfex](https://github.com/Framsfex) opened issue at [2024-04-06 08:48](https://github.com/rear/rear/issues/3200): + +- ReaR version ("/usr/sbin/rear -V"): + +Relax-and-Recover 2.6 / 2020-06-17 +(Installed on RHEL7 with "yum install rear") + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): + +NAME="Red Hat Enterprise Linux Server" +VERSION="7.9 (Maipo)" +ID="rhel" +ID\_LIKE="fedora" +VARIANT="Server" +VARIANT\_ID="server" +VERSION\_ID="7.9" + +- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat + /etc/rear/local.conf"): + +OUTPUT=ISO +OUTPUT\_URL=nfs://129.69.2.11/spnfs\_data0/sp\_i/Rear +BACKUP=NETFS +BACKUP\_URL=nfs://129.69.2.11/spnfs\_data0/sp\_i/Rear +BACKUP\_PROG\_EXCLUDE=("${BACKUP\_PROG\_EXCLUDE\[@\]}" '/media' +'/var/tmp' '/var/crash' '/mnt') +NETFS\_KEEP\_OLD\_BACKUP\_COPY= + +- Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM + guest or PowerVM LPAR): + +Cisco Systems Inc UCSC-C240-M5S A0 +Intel(R) Xeon(R) Gold 6132 CPU @ 2.60GHz, 56 x 3700 MHz + +- System architecture (x86 compatible or PPC64/PPC64LE or what exact + ARM device): + +x86 + +- Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or + ELILO or Petitboot): + +GRUB BIOS + +- Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or + multipath (DM or NVMe): + +Local harddisk, Hardware RAID + +- Storage layout ("lsblk -ipo + NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"): + + + + NAME KNAME PKNAME TRAN TYPE FSTYPE LABEL SIZE MOUNTPOINT + /dev/sda /dev/sda disk 557.9G + |-/dev/sda1 /dev/sda1 /dev/sda part xfs boot 1G /boot + `-/dev/sda2 /dev/sda2 /dev/sda part LVM2_member 556.9G + |-/dev/mapper/rhel-root /dev/dm-0 /dev/sda2 lvm xfs 32G / + |-/dev/mapper/rhel-home /dev/dm-5 /dev/sda2 lvm xfs 256G /local + |-/dev/mapper/rhel-swap /dev/dm-6 /dev/sda2 lvm swap 16G [SWAP] + +- Description of the issue (ideally so that others can reproduce it): + +sp-i is a Cisco UCS server with RHEL7. +I did a backup with ReaR which seems successful. +Then I did an upgrade to RHEL8, which is a bit complicated with Redhat, +but it worked. The server bootet successfully RHEL8. +(1 week pause) +To reproduce and test the upgrade process I wanted to go back to RHEL7. +For this I booted the ReaR Rescue CD (rear-sp-i.iso) and selected +"recover", which seems successfully, too. But after rebooting the +recovered RHEL7 system it was not able to remount the / filesystem from +ro to rw state: + +[https://fex.rus.uni-stuttgart.de/fop/NtoxIEXY/X-20240404231512.jpg](https://fex.rus.uni-stuttgart.de/fop/NtoxIEXY/X-20240404231512.jpg) + +I looked in the NFS rear directory and found: + + -RW- 15,938,002 2024-03-25 14:07 backup.log + -RW- 4,321,746,328 2024-03-25 14:06 backup.tar.gz + -RW- 202 2024-04-04 01:31 README + -RW- 603,398,144 2024-04-04 01:31 rear-sp-i.iso + -RW- 102,804 2024-04-04 01:31 rear-sp-i.log + -RW- 0 2024-03-25 14:07 selinux.autorelabel + -RW- 268 2024-04-04 01:31 VERSION + +rear-sp-i.iso is much younger than backup! And it was created at 1:30, a +time where I am not working! +My assumption: Some unknown (cron?) job has recreated rear-sp-i.iso with +RHEL8 and its newer xfs is incompatible with RHEL7 xfs. + +Any idea what I can do in this situation? + +I have second server (sp-j) with identical configuration, still running +with RHEL7 +Shall I run ReaR there and use this rear-sp-j.iso? + +#### [Framsfex](https://github.com/Framsfex) commented at [2024-04-06 08:51](https://github.com/rear/rear/issues/3200#issuecomment-2041021046): + +[rear-sp-i.log](https://github.com/rear/rear/files/14893001/rear-sp-i.log) + +#### [abbbi](https://github.com/abbbi) commented at [2024-04-07 07:25](https://github.com/rear/rear/issues/3200#issuecomment-2041348936): + +> rear-sp-i.iso is much younger than backup! And it was created at 1:30, +> a time where I am not working! +> My assumption: Some unknown (cron?) job has recreated rear-sp-i.iso +> with RHEL8 and its newer xfs is incompatible with RHEL7 xfs. + +attempting to restore a RHEL7 system with an recovery iso that contains +already an RHEL8 system will not work and is not supported. REAR itself +does not setup any cronjobs, you should clarify how the recovery iso was +created after update to RHEL8. I dont see how REAR is at fault here. + +One possible "dirty" solution would be: + +1. create bootable Recovery iso image on the identical left over RHEL7 + system +2. boot iso, extract the REAR disk layout config from the overwritten + ISO Image based on RHEL8 and copy it to the running instance +3. run recovery + +It should then re-create disk partitions based on the information from +the original system using the tools compatible to the RHEL7 backup. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-11 10:59](https://github.com/rear/rear/issues/3200#issuecomment-2049434516): + +Only a guess but regarding "unknown (cron?) job" see +[https://github.com/rear/rear/issues/3199\#issuecomment-2049430339](https://github.com/rear/rear/issues/3199#issuecomment-2049430339) + +@Framsfex +perhaps you also have an old cron job +as a leftover from ReaR 2.4 or before? + +#### [Framsfex](https://github.com/Framsfex) commented at [2024-04-11 11:49](https://github.com/rear/rear/issues/3200#issuecomment-2049522843): + +On Thu 2024-04-11 (04:00), Johannes Meixner wrote: + +> Only a guess but regarding "unknown (cron?) job" see +> [https://github.com/rear/rear/issues/3199\#issuecomment-2049430339](https://github.com/rear/rear/issues/3199#issuecomment-2049430339) + +That's it! +We have the same time stamp of the new created ISO image! +This is what I have already assumed. +Too bad, our Redhat installation now is broken, we have had to reinstall +it. + +> perhaps you also have an old cron job +> as a leftover from ReaR 2.4 or before? + +We have installed ReaR 2.4 from the RHEL 7.9 repo and then made the +upgrade to RHEL 8.3 (where the hidden cronjob run) which we tried to +recover with ReaR back to RHEL7 +I do not know which ReaR version comes with RHEL 8.3 + +-- +Ullrich Horlacher Server und Virtualisierung +Rechenzentrum TIK +Universitaet Stuttgart E-Mail: \*\*\*@\*\*\*.\*\*\* +Allmandring 30a Tel: ++49-711-68565868 +70569 Stuttgart (Germany) WWW: +[https://www.tik.uni-stuttgart.de/](https://www.tik.uni-stuttgart.de/) +\*\*\*@\*\*\*.\*\*\*> + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-04-11 16:45](https://github.com/rear/rear/issues/3200#issuecomment-2050102657): + +Hi @Framsfex sorry to hear that. In RHEL 8.3 there is also ReaR 2.4, but +the iso made on a different system may be incompatible. + +> I have second server (sp-j) with identical configuration, still +> running with RHEL7 +> Shall I run ReaR there and use this rear-sp-j.iso? + +I think this may work, if the disk layout is identical (therefore you +don't need to worry about copying the disklayout from the original +server). But there may be some issues when bringing up the network. The +rescue system will ask you to map MAC addresses of the second system to +the MAC addresses of the first system, and then use the IP addresses of +the second server (unless your IP addresses are configured by DHCP). + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-04-11 16:46](https://github.com/rear/rear/issues/3200#issuecomment-2050103646): + +In RHEL 9 we deleted the cron job btw, thus users won't face this +problem anymore. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-12 06:13](https://github.com/rear/rear/issues/3200#issuecomment-2051055408): + +@pcahyna +I don't know how ReaR's cron file was installed by Red Hat's RPM +but when Red Hat did it as we had in packaging/rpm/rear.spec via + + %config(noreplace) %{_sysconfdir}/cron.d/rear + +cf. +[https://github.com/rear/rear/commit/89a8f18ec402b439caf4800421644f5bf5d174e5](https://github.com/rear/rear/commit/89a8f18ec402b439caf4800421644f5bf5d174e5) +then according to the table in +[https://www.cl.cam.ac.uk/~jw35/docs/rpm\_config.html](https://www.cl.cam.ac.uk/~jw35/docs/rpm_config.html) + + The following table shows what we ended up with + after installing an RPM, + optionally editing the resulting files, + and then upgrading the RPM. + + File marked as | Changed in | On-disk file | On-disk file + | update RPM? | untouched | edited + ... + | No | File from update | Edited file + %config(noreplace) | | + | Yes | File from update | Edited file, + | | | file from the + | | | update in .rpmnew + +there might be the cron file left even when an RPM package update +does no longer contain that cron file? +The case when a config file is no longer provided by an +RPM package update is not described there so I don't know +what actually happens in this case. +I only liked to note that the RPM package update might not +have removed the unwanted cron file. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-07.3201.issue.open.md b/docs/issues/2024-04-07.3201.issue.open.md new file mode 100644 index 00000000..497679f6 --- /dev/null +++ b/docs/issues/2024-04-07.3201.issue.open.md @@ -0,0 +1,176 @@ +[\#3201 Issue](https://github.com/rear/rear/issues/3201) `open`: are these errors a problem? +============================================================================================ + +**Labels**: `support / question` + +#### [hspindel](https://github.com/hspindel) opened issue at [2024-04-07 03:27](https://github.com/rear/rear/issues/3201): + + + +- ReaR version ("/usr/sbin/rear -V"): + +2.7 + +- If your ReaR version is not the current version, explain why you + can't upgrade: + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): + + + + NAME="Rocky Linux" + VERSION="9.3 (Blue Onyx)" + ID="rocky" + ID_LIKE="rhel centos fedora" + VERSION_ID="9.3" + PLATFORM_ID="platform:el9" + PRETTY_NAME="Rocky Linux 9.3 (Blue Onyx)" + ANSI_COLOR="0;32" + LOGO="fedora-logo-icon" + CPE_NAME="cpe:/o:rocky:rocky:9::baseos" + HOME_URL="https://rockylinux.org/" + BUG_REPORT_URL="https://bugs.rockylinux.org/" + SUPPORT_END="2032-05-31" + ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9" + ROCKY_SUPPORT_PRODUCT_VERSION="9.3" + REDHAT_SUPPORT_PRODUCT="Rocky Linux" + REDHAT_SUPPORT_PRODUCT_VERSION="9.3" + +- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat + /etc/rear/local.conf"): + + + + OUTPUT=ISO + BACKUP=NETFS + BACKUP_URL=cifs://192.1.1.12/rocky-dd + BACKUP_OPTIONS="cred=/etc/rear/creds,vers=2.1" + USE_RAMDISK=0 + BACKUP_PROG_EXCLUDE=( "${BACKUP_PROG_EXCLUDE[@]}" '/home/timeshift' ) + +- Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM + guest or PowerVM LPAR): + +Dell T7910 + +- System architecture (x86 compatible or PPC64/PPC64LE or what exact + ARM device): + +x86 + +- Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or + ELILO or Petitboot): + +BIOS. GRUB. + +- Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or + multipath (DM or NVMe): + +Local mdadm RAID 5, 4 4TB HDD + +- Storage layout ("lsblk -ipo + NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"): + + + + [root@server2 howard]# lsblk -ipo NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT + NAME KNAME PKNAME TRAN TYPE FSTYPE LABEL SIZE MOUNTPOI + /dev/sda /dev/sda usb disk 0B + /dev/sdb /dev/sdb usb disk 0B + /dev/sdc /dev/sdc usb disk 0B + /dev/sdd /dev/sdd usb disk 0B + /dev/sde /dev/sde sas disk 3.6T + |-/dev/sde1 /dev/sde1 /dev/sde part 1M + |-/dev/sde2 /dev/sde2 /dev/sde part xfs 1G /boot + |-/dev/sde3 /dev/sde3 /dev/sde part LVM2_member 21.4G + | |-/dev/mapper/rl_server2-root + | | /dev/dm-0 /dev/sde3 lvm xfs 70G / + | `-/dev/mapper/rl_server2-swap + | /dev/dm-1 /dev/sde3 lvm swap 15.7G [SWAP] + `-/dev/sde4 /dev/sde4 /dev/sde part linux_raid_member server2:home 3.6T + `-/dev/md127 /dev/md127 /dev/sde4 raid5 xfs 10.8T /home + /dev/sdf /dev/sdf sas disk 3.6T + |-/dev/sdf1 /dev/sdf1 /dev/sdf part LVM2_member 21.4G + | `-/dev/mapper/rl_server2-root + | /dev/dm-0 /dev/sdf1 lvm xfs 70G / + `-/dev/sdf2 /dev/sdf2 /dev/sdf part linux_raid_member server2:home 3.6T + `-/dev/md127 /dev/md127 /dev/sdf2 raid5 xfs 10.8T /home + /dev/sdg /dev/sdg sas disk 3.6T + |-/dev/sdg1 /dev/sdg1 /dev/sdg part LVM2_member 21.4G + | `-/dev/mapper/rl_server2-root + | /dev/dm-0 /dev/sdg1 lvm xfs 70G / + `-/dev/sdg2 /dev/sdg2 /dev/sdg part linux_raid_member server2:home 3.6T + `-/dev/md127 /dev/md127 /dev/sdg2 raid5 xfs 10.8T /home + /dev/sdh /dev/sdh sas disk 3.6T + |-/dev/sdh1 /dev/sdh1 /dev/sdh part LVM2_member 21.4G + | `-/dev/mapper/rl_server2-root + | /dev/dm-0 /dev/sdh1 lvm xfs 70G / + `-/dev/sdh2 /dev/sdh2 /dev/sdh part linux_raid_member server2:home 3.6T + `-/dev/md127 /dev/md127 /dev/sdh2 raid5 xfs 10.8T /home + /dev/sr0 /dev/sr0 sata rom 1024M + [root@server2 howard]# + +- Description of the issue (ideally so that others can reproduce it): + +rear runs to completion. Expected backup files are created. But error +messages are reported as follows: + + Archived 63353 MiB in 5364 seconds [avg 12094 KiB/sec] + ERROR: Unmounting '/var/tmp/rear.h64Kn9vXbQZsRIa/outputfs' failed. + Some latest log messages since the last called script 980_umount_NETFS_dir.sh: + + 2024-04-06 03:36:22.294409388 Archived 63353 MiB in 5364 seconds [avg 12094 KiB/sec] + 2024-04-06 03:36:22.714317169 Including backup/NETFS/GNU/Linux/600_start_selinux.sh + 2024-04-06 03:36:22.759003585 Including backup/NETFS/GNU/Linux/620_force_autorelabel.sh + 2024-04-06 03:36:22.785127124 Including backup/NETFS/default/970_remove_lock.sh + 2024-04-06 03:36:22.853179624 Unmounting '/var/tmp/rear.h64Kn9vXbQZsRIa/outputfs' + 2024-04-06 03:36:22.918956518 Forced unmount of '/var/tmp/rear.h64Kn9vXbQZsRIa/outputfs' + 2024-04-06 03:36:22.929532470 Unmounting '/var/tmp/rear.h64Kn9vXbQZsRIa/outputfs' failed. + Some messages from /var/tmp/rear.h64Kn9vXbQZsRIa/tmp/rear.mkbackup.stdout_stderr since the last cal + ed script 980_umount_NETFS_dir.sh: + Including backup/NETFS/GNU/Linux/310_stop_selinux.sh + Including backup/NETFS/default/400_create_include_exclude_files.sh + Including backup/NETFS/default/500_make_backup.sh + Including backup/NETFS/GNU/Linux/600_start_selinux.sh + Including backup/NETFS/GNU/Linux/620_force_autorelabel.sh + Including backup/NETFS/default/970_remove_lock.sh + umount: /var/tmp/rear.h64Kn9vXbQZsRIa/outputfs: target is busy. + umount: /var/tmp/rear.h64Kn9vXbQZsRIa/outputfs: target is busy. + Use debug mode '-d' for some debug messages or debugscript mode '-D' for full debug messages with ' + et -x' output + Aborting due to an error, check /var/log/rear/rear-server2.log for details + Exiting rear mkbackup (PID 21223) and its descendant processes ... + Running exit tasks + Directory /var/tmp/rear.h64Kn9vXbQZsRIa/outputfs still mounted - trying lazy umount + Terminated + +- Workaround, if any: + +rear seems to work, but I am concerned whether the error messages from +umount are something to worry about + +- Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" + debug log files): + +You can drag-drop log files into this editor to create an attachment +or paste verbatim text like command output or file content +by including it between a leading and a closing line of +three backticks like this: + + verbatim content + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-04-08 17:24](https://github.com/rear/rear/issues/3201#issuecomment-2043285591): + +Does `/var/tmp/rear.h64Kn9vXbQZsRIa/outputfs` remain mounted after ReaR +exits? If so, is there any process using it? +(`fuser -m /var/tmp/rear.h64Kn9vXbQZsRIa/outputfs` should display it.) + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-08.3202.issue.open.md b/docs/issues/2024-04-08.3202.issue.open.md new file mode 100644 index 00000000..a5675c67 --- /dev/null +++ b/docs/issues/2024-04-08.3202.issue.open.md @@ -0,0 +1,240 @@ +[\#3202 Issue](https://github.com/rear/rear/issues/3202) `open`: What to improve in ReaR project as lesson learned from XZ Attack? +================================================================================================================================== + +**Labels**: `discuss / RFC`, `ReaR Project` + +#### [jsmeix](https://github.com/jsmeix) opened issue at [2024-04-08 12:39](https://github.com/rear/rear/issues/3202): + +For background information about the XZ Attack see +[https://en.wikipedia.org/wiki/XZ\_Utils\_backdoor](https://en.wikipedia.org/wiki/XZ_Utils_backdoor) +and follow the links therein, in particular see +[https://tukaani.org/xz-backdoor/](https://tukaani.org/xz-backdoor/) +and +[https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27](https://gist.github.com/thesamesam/223949d5a074ebc3dce9ee78baad9e27) + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-08 12:54](https://github.com/rear/rear/issues/3202#issuecomment-2042685076): + +I find in particular this one interesting: + +[https://mastodon.social/@AndresFreundTec/112208281684931915](https://mastodon.social/@AndresFreundTec/112208281684931915) + +From my point of view the most important outcome in that discussion is +this +posting (excerpt): + + Elias Probst @eliasp + + @jejb so it all comes down to being able + to actually understand and review every contribution, + so it doesn't really matter in the end who contributed, + but what was contributed instead. + +I.e. take out the "who" part from finding a solution +how to avoid malicious code changes and +focus on the code changes itself. + +Another posting in that discussion also indicates +that the "who" part likely does not help +to avoid malicious code changes (excerpt): + + Christoph Petrausch @hikhvar@norden.social + + @eliasp @AndresFreundTec But even with the web of trust. + How should that scale in a global way and also + not exclude less privileged people? + + A student in a remote country side + wants to start contributing to code. + Welp, nobody there to establish web of trust. + +I.e. trying to solve the "who" part via a "web of trust" +does not work in practice. +In contrast understanding what was contributed +(regardless who contributed it) +should work in practice. + +From my personal point of view this means: + +New code that is not easy to understand should be +rejected at upstream by those upstream maintainers +who review contributions from others +regardless if some other one is an unknown newcomer +or a "well known expert in sophisticated coding ingenuity". + +Old code that is known to be OK since ages can stay as is. +But when old code is changed the changes must be easy to understand +(which may mean that bigger parts of old code must be overhauled). + +Simply put: +Code that is hard for others to understand must be rejected +because it would be unmaintainable by others in the future. + +So in the end from my personal point of view +it means to more enforce what is meant with + +[https://github.com/rear/rear/wiki/Coding-Style](https://github.com/rear/rear/wiki/Coding-Style) + +i.e. not some nitpicking coding syntax style "rules" +(which cause more annoyance than being actually helpful) +but to more enforce the overall idea behind: + + Make yourself understood + to make your code maintainable which means + at any time later others still understand your code + so they can properly fix and enhance your code as needed. + +#### [gdha](https://github.com/gdha) commented at [2024-04-08 12:59](https://github.com/rear/rear/issues/3202#issuecomment-2042697184): + +An important aspect is to meet the active contributors once personally. +A "web of trust" should be a "people of trust" (in our case). All ReaR +contributors have met each other in person - that helps. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-08 13:08](https://github.com/rear/rear/issues/3202#issuecomment-2042718569): + +In addition to enforce that code must be easy to understand +I think we should also enforce to review all code changes. + +This means that all code changes +(also code changes by us, the upstream maintainers) +must be approved by at least one other upstream maintainer +even if that makes merging code changes into 'master' slower. + +But I think that a certain kind of slowness is helpful +because it results from "verify" in "Trust, but verify" +[https://en.wikipedia.org/wiki/Trust,\_but\_verify](https://en.wikipedia.org/wiki/Trust,_but_verify) + +In practice enforced code change reviews means +that all code changes must happen via a pull request. + +I mean only actual code changes (at least for now). +I.e. all what changes what ReaR does on the user's system. + +E.g. typo fixes in comments can still be directly merged +regardless that even a "typo fix" in a comment may +(unintentionally or by evil intent) be a false change +of the meaning of a comment so others get a false +understanding what (some weird looking) code +actually does which could be a first step to do +later a malicious change in the actual code. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-08 13:13](https://github.com/rear/rear/issues/3202#issuecomment-2042728967): + +@gdha +yes, that all ReaR contributors have met each other in person +does help very much. + +But: + +This would exclude people in far away countries. +Currently I have no good idea how to solve that properly. + +And it does not help when the account +of a ReaR upstream maintainer got hijacked +or when a ReaR upstream maintainer was tricked +by a malicious external person. +I think enforced code change reviews +by at least one other upstream maintainer help. +And at least two active upstream maintainers help +against social engineering attrition of single +upstream maintainers (as it happened to the +single upstream maintainer during the XZ Attack). + +#### [gdha](https://github.com/gdha) commented at [2024-04-09 06:34](https://github.com/rear/rear/issues/3202#issuecomment-2044246020): + +As mentioned already mentioned before all Pull Requests must be reviewed +by another person then the author, and I would like to add an additional +third person - the co-called "approver" of the PR. This way there always +3 persons involved before a PR is getting merged. + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-09 06:50](https://github.com/rear/rear/issues/3202#issuecomment-2044265183): + +We can enable [branch +protection](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches) +to help us "remember" the PR rule, although maintainers of course can +override that. + +I'm against a 6-eyes approval, at least now, because I think that going +from "just do it" to "everything via PR" is already a big and probably +sometimes inconvenient step for us. As we don't have any actual problem, +I'd like to take this slowly, step by step. And inspect and adapt after +every change. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-09 08:26](https://github.com/rear/rear/issues/3202#issuecomment-2044433154): + +I also think that 6-eyes approval is too much for us for now. +Often I don't get a single approval for my pull requests and +when I get one I think it is often not after a thorough review. +I totally understand this because in most cases I don't do +a thorough review but only have a look if I spot something +and if not I just trust you. + +I always prefer to change things slowly, step by step. + +In think for now it is enough to have some kind of "enforced" +approval by at least one other upstream maintainer +for pull requests from an upstream maintainer. + +Perhaps we should "enforce" approval by at least +two upstream maintainers for pull requests +from external contributors? + +With "enforce" I mean that we set up something in GitHub +which we can of course override but that would at least +remind us each time to not "just merge" something without +approval by at least one other upstream maintainer. + +FWIW: +I remember well how @pcahyna asked me +again and again questions during +[https://github.com/rear/rear/pull/2961](https://github.com/rear/rear/pull/2961) +because he did not understand my description in default.conf. +I even felt somewhat annoyed by his insisting questions +("why the heck doesn't he understand my nice description?") +until finally I saw it that what I described +did not match what I had implemented, cf. +[https://github.com/rear/rear/pull/2961\#discussion\_r1363679930](https://github.com/rear/rear/pull/2961#discussion_r1363679930) +I was blind regarding my own faulty description +(from 09. Oct. 2023 until 18. Oct. 2023). +I still cannot understand why I was so blind. +So @pcahyna helped me to finally see my fault and +he helped our users to not get my faulty description. + +#### [gdha](https://github.com/gdha) commented at [2024-04-09 08:32](https://github.com/rear/rear/issues/3202#issuecomment-2044443881): + +Perhaps, the biggest danger is not in the source code itself, but in the +produced ISO images stored on a NAS location where everybody (within a +company) can replace it with a malicious code (or whatever evil). + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-09 08:39](https://github.com/rear/rear/issues/3202#issuecomment-2044456334): + +Yes, that is however by design and we shouldn't worry about it. + +Maybe Secure Boot with custom keys can be used to create end-end secured +DR boot media + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-09 09:42](https://github.com/rear/rear/issues/3202#issuecomment-2044587483): + +With this issue here I meant only our source code, +in particular how to better ensure that our users +actually get what we intended to provide them. + +Regarding secure ReaR recovery system images: +I don't remember there had been a user request in the past +for a secured ReaR boot medium so at least currently +this seems to be not an issue for our users. +I assume our users know how to keep e.g. ReaR ISOs secure. +I assume they handle ReaR boot media same as their backups. +One does not want to restore a maliciously modified backup +(e.g. where /usr/sbin/sshd got "additional functionality"). + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-12 13:43](https://github.com/rear/rear/issues/3202#issuecomment-2051790940): + +FYI: +an openSUSE report about the XZ Backdoor issue: + +[https://news.opensuse.org/2024/04/12/learn-from-the-xz-backdoor/](https://news.opensuse.org/2024/04/12/learn-from-the-xz-backdoor/) + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-09.3203.pr.open.md b/docs/issues/2024-04-09.3203.pr.open.md new file mode 100644 index 00000000..7384eb2d --- /dev/null +++ b/docs/issues/2024-04-09.3203.pr.open.md @@ -0,0 +1,825 @@ +[\#3203 PR](https://github.com/rear/rear/pull/3203) `open`: On hold: New source\_variable\_from\_file() +======================================================================================================= + +**Labels**: `enhancement`, `discuss / RFC` + +#### [jsmeix](https://github.com/jsmeix) opened issue at [2024-04-09 11:44](https://github.com/rear/rear/pull/3203): + +- Type: **Enhancement** + +- Impact: **Normal** + +- Reference to related issue (URL): + +[https://github.com/rear/rear/pull/3171](https://github.com/rear/rear/pull/3171) + +- How was this pull request tested? + +Currently mostly tested only on command line, see +[https://github.com/rear/rear/pull/3171](https://github.com/rear/rear/pull/3171) +and +[https://github.com/rear/rear/pull/3203\#issuecomment-2045058844](https://github.com/rear/rear/pull/3203#issuecomment-2045058844) + +- Description of the changes in this pull request: + +In lib/global-functions.sh added +new function get\_shell\_file\_config\_variable() + +For details see the comment +at get\_shell\_file\_config\_variable() +in lib/global-functions.sh + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-09 11:51](https://github.com/rear/rear/pull/3203#issuecomment-2044925890): + +@lzaoral +in your +[https://github.com/rear/rear/pull/3171\#issuecomment-2012100925](https://github.com/rear/rear/pull/3171#issuecomment-2012100925) +you wrote + + the final '|| return 1' is necessary because + bash returns 127 on unbound variables + +Why does it matter what non-zero return code +the get\_var\_from\_file() function returns? + +Or in other words: +Why not have the get\_var\_from\_file() implementation +simpler by omitting the final '|| return 1' like + + function get_var_from_file() { + bash -c 'unset $1 || exit 1 ; source "$0" >/dev/null ; set -u ; echo "${!1}"' "$1" "$2" + } + +? + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-04-09 12:04](https://github.com/rear/rear/pull/3203#issuecomment-2044979689): + +> Why does it matter what non-zero return code the +> get\_var\_from\_file() function returns? + +@jsmeix I suppose it does not. I just did that to be consistent with +surrounding functions which return either `0` or `1`. It should be safe +to drop this part. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-09 12:10](https://github.com/rear/rear/pull/3203#issuecomment-2045003239): + +@lzaoral +thank you for your prompt reply. +I dropped it via +[https://github.com/rear/rear/pull/3203/commits/ef14f5f382147d67af76210a8b0d8f957df0f037](https://github.com/rear/rear/pull/3203/commits/ef14f5f382147d67af76210a8b0d8f957df0f037) +but I did not test it with the dropped final '|| return 1' + +#### [lzaoral](https://github.com/lzaoral) commented at [2024-04-09 12:25](https://github.com/rear/rear/pull/3203#issuecomment-2045058844): + +@jsmeix Thank you! It seems to work as advertised: + + $ get_var_from_file /etc/os-release VERSION; echo $? + 39 (Thirty Nine) + 0 + $ get_var_from_file /etc/os-release VERSIO; echo $? + /etc/os-release: line 1: !1: unbound variable + 127 + $ get_var_from_file /etc/os-releas VERSIO; echo $? + /etc/os-releas: line 1: /etc/os-releas: No such file or directory + /etc/os-releas: line 1: !1: unbound variable + 127 + $ get_var_from_file /etc/os-releas VERSION; echo $? + /etc/os-releas: line 1: /etc/os-releas: No such file or directory + /etc/os-releas: line 1: !1: unbound variable + 127 + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-04-11 19:29](https://github.com/rear/rear/pull/3203#issuecomment-2050376112): + +@schlomo I really hope we are not going to implement a parser for shell +assignments when this parser already exists - in the shell itself, just +because of vague security justifications. The `os-release` file is +documented to be readable this way (see the manual page), as are +`/etc/sysconfig` files - guess how they are read by `ifup`/`ifdown` (See +`source_config()` in `/etc/sysconfig/network-scripts/network-functions`) +? What attack vectors are you speaking about when the files that we +intend to source this way are already sourced at every boot? + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-11 19:51](https://github.com/rear/rear/pull/3203#issuecomment-2050411936): + +For such shell files I agree with your logic, but the function claims to +be more generic than shell config files. + +For shell config files one can source them and echo, even in a subshell. +But then I'd call the function differently, e.g. +`get_shell_file_config_variable` or such. + +#### [pcahyna](https://github.com/pcahyna) commented at [2024-04-11 20:06](https://github.com/rear/rear/pull/3203#issuecomment-2050433590): + +@schlomo + +> For such shell files I agree with your logic + +That's the intent of the function, as documented in the comment: + + # The get_var_from_file function is meant to be used with files + # like /etc/os-release or certain files in /etc/sysconfig or /etc/default + # that basically only do shell-compatible variable assignments (VAR="value") + +> but the function claims to be more generic than shell config files. + +> But then I'd call the function differently, e.g. +> get\_shell\_file\_config\_variable or such. + +If you feel that the function name promises more than the function is +able to do, I agree with renaming it. + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-11 20:36](https://github.com/rear/rear/pull/3203#issuecomment-2050494028): + +Yes, I think that would be beneficial + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-12 06:26](https://github.com/rear/rear/pull/3203#issuecomment-2051071868): + +I prefer names that tell what a function actually is about. +`get_var_from_file` is too unspecific. + +Regarding `get_shell_file_config_variable`: +I wonder why it is limited to *config* variables? +I.e. why not only `get_shell_file_variable` +because it can get the value of any variable +that is set in a shell syntax file? +Or is `get_shell_file_variable` confusing +because one does not know if it is about +a 'shell file' or a 'file variable' +so `get_shell_file_config_variable` +is perhaps really what tells best what it is about? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-12 06:45](https://github.com/rear/rear/pull/3203#issuecomment-2051095105): + +Via +[https://github.com/rear/rear/pull/3203/commits/12c3ccb8f00affa7d3a36e915602abee12328fc5](https://github.com/rear/rear/pull/3203/commits/12c3ccb8f00affa7d3a36e915602abee12328fc5) +I renamed get\_var\_from\_file into get\_shell\_file\_config\_variable +to tell what that function actually is about +plus adapted and enhanced/fixed comments. + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-12 07:41](https://github.com/rear/rear/pull/3203#issuecomment-2051195543): + +@jsmeix may I suggest an alternative implementation that doesn't call +the `bash` binary and provides a little bit more error reporting +compared to +[https://github.com/rear/rear/pull/3203\#issuecomment-2045058844](https://github.com/rear/rear/pull/3203#issuecomment-2045058844)? + +This implementation also uses [Bash Restricted +mode](https://www.gnu.org/software/bash/manual/html_node/The-Restricted-Shell.html) +to add a little bit of security protection. + +The error message also makes it clear that the config file was executed +by Bash, hopefully this helps understanding errors better + +About the name: Yes, I believe that the limit to shell config files is +important because it doesn't read other key-value config files that use +`:` as separator or that allow whitespace around the `=` + +**code s.sh** + + function Error() { + echo -e "ERROR: $*" + exit 1 + } + + function get_shell_file_config_variable() { + [[ -r "$1" && "$2" =~ [a-zA-Z_][a-zA-Z_0-9]* ]] || Error "Arg 1 '$1' is not a readable file or arg 2 '$2' is not valid variable name" + local file="$1" + local key="$2" + + ( + function die() { + echo "ERROR: $*" + exit 1 + } + unset $key + set -r -e -u -o pipefail + eval "$(< "$file")" || die "Bash errors while executing '$file'" + [[ -v $key ]] || die "$key is not set in $file" + echo ${!key} + ) + } + + get_shell_file_config_variable /etc/os-release VERSION + echo $? + + get_shell_file_config_variable /etc/os-release SCHLOMO + echo $? + + get_shell_file_config_variable /etc/passwd VERSION + echo $? + + get_shell_file_config_variable /etc/cron.daily/dpkg foo + echo $? + + get_shell_file_config_variable /etc/nothing VERSION + echo $? + +**example run s.sh** + + root@84cc40add750:/rear/test# bash s.sh + 22.04.4 LTS (Jammy Jellyfish) + 0 + ERROR: SCHLOMO is not set in /etc/os-release + 1 + s.sh: line 18: root:x:0:0:root:/root:/bin/bash: restricted: cannot specify `/' in command names + s.sh: line 19: daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 20: bin:x:2:2:bin:/bin:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 21: sys:x:3:3:sys:/dev:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 22: sync:x:4:65534:sync:/bin:/bin/sync: restricted: cannot specify `/' in command names + s.sh: line 23: games:x:5:60:games:/usr/games:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 24: man:x:6:12:man:/var/cache/man:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 25: lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 26: mail:x:8:8:mail:/var/mail:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 27: news:x:9:9:news:/var/spool/news:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 28: uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 29: proxy:x:13:13:proxy:/bin:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 30: www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 31: backup:x:34:34:backup:/var/backups:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 32: list:x:38:38:Mailing: command not found + s.sh: line 33: irc:x:39:39:ircd:/run/ircd:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: eval: line 34: syntax error near unexpected token `(' + s.sh: eval: line 34: `gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin' + ERROR: Bash errors while executing '/etc/passwd' + 1 + s.sh: line 25: /usr/libexec/dpkg/dpkg-db-backup: restricted: cannot specify `/' in command names + ERROR: Bash errors while executing '/etc/cron.daily/dpkg' + 1 + ERROR: Arg 1 '/etc/nothing' is not a readable file or arg 2 'VERSION' is not valid variable name + root@84cc40add750:/rear/test# + +WDYT? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-12 07:56](https://github.com/rear/rear/pull/3203#issuecomment-2051218758): + +I think I better withdraw my pull request here +and leave it to you and @pcahyna and @lzaoral +how to get the value of a variable out of a file +because I am neither a sufficient expert in this area +nor will I find the needed time to get this task done. + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-12 08:32](https://github.com/rear/rear/pull/3203#issuecomment-2051287162): + +@jsmeix why? I think that you identified a relevant problem that we have +again and again. Please don't be offended by the fact that somebody +might have a different perspective. + +You proposed a solution, got feedback and a even suggestion for a +different solution. We even already aligned on a suitable name. + +Besides that you gave us the opportunity for an interesting technical +challenge (secure parsing of shell configs) which is always a good +thing. + +I think my context is simply slightly different (and I see it as an +asset for ReaR that we all are so different), for example I already +implemented a `set_variable_from_commvault_status` function: +[https://github.com/rear/rear/blob/09579ae652cf3475a94c1ab79030dfee64cff51c/usr/share/rear/prep/GALAXY11/default/400\_prep\_galaxy.sh\#L5-L23](https://github.com/rear/rear/blob/09579ae652cf3475a94c1ab79030dfee64cff51c/usr/share/rear/prep/GALAXY11/default/400_prep_galaxy.sh#L5-L23) + +Maybe you just collect all the feedback and proceed with that? I at +least greatly appreciate the love to the details that you bring to ReaR! + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-12 12:03](https://github.com/rear/rear/pull/3203#issuecomment-2051629427): + +@schlomo +at least for now I would like in this particular case +to relax from this particular Relax-and-Recover issue. +I spent already way too many hours on this issue +which seems to have basically unlimited depth +so at least for now I feel a bit "fed up" with it. +Let's wait and see how things look to me +after the weekend (but no promises). + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-12 12:09](https://github.com/rear/rear/pull/3203#issuecomment-2051638115): + +@pcahyna @lzaoral +feel free to use what I further developed here so far +which is all based on your foundations during +[https://github.com/rear/rear/pull/3171](https://github.com/rear/rear/pull/3171) +to make whatever +"get the value of a variable out of a file" +functionality as you like most for your needs. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-12 13:33](https://github.com/rear/rear/pull/3203#issuecomment-2051773594): + +Since +[https://github.com/rear/rear/pull/3177](https://github.com/rear/rear/pull/3177) +is merged there is a merge conflict here +which I solved right now: + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-15 08:12](https://github.com/rear/rear/pull/3203#issuecomment-2056083708): + +Via +[https://github.com/rear/rear/pull/3203/commits/f79454fe2101b50180ba5abf79e68d581a7cf317](https://github.com/rear/rear/pull/3203/commits/f79454fe2101b50180ba5abf79e68d581a7cf317) +I renamed the function again, +now from get\_shell\_file\_config\_variable +into source\_variable\_from\_file +to make it explicit that the file is sourced +and +to make the name less generic (no longer generic 'get') +so we can - as needed - implement alternative functions +how to get a variable value out of a file +with different methods. + +Some "forensics" why we use 'source' at all: + +Using 'source' was introduced via +[https://github.com/rear/rear/pull/3165/files](https://github.com/rear/rear/pull/3165/files) +therein see +[https://github.com/rear/rear/pull/3165\#pullrequestreview-1903010946](https://github.com/rear/rear/pull/3165#pullrequestreview-1903010946) +which shows the following outdated code diff + + 28 - local version_id=$(grep "^VERSION_ID=" /etc/os-release | cut -d\" -f2 ) # 7 + + 28 + + 29 + local version_id + 30 + version_id="$(. /etc/os-release && echo "$VERSION_ID")" + +where 'grep' was replaced by '.' +There I only liked to have '.' replaced by explicit 'source' +but I did not question why 'grep' was replaced by 'source'. + +Because 'source' fails for readonly variables, see +[https://github.com/rear/rear/pull/3165\#discussion\_r1505473662](https://github.com/rear/rear/pull/3165#discussion_r1505473662) +@lzaoral "reworked the parsing completely", cf. +[https://github.com/rear/rear/pull/3165\#discussion\_r1504116328](https://github.com/rear/rear/pull/3165#discussion_r1504116328) +so that 'source' was no longer used and currently +we use 'grep' again but now together with 'eval', see +[https://github.com/rear/rear/pull/3165/files](https://github.com/rear/rear/pull/3165/files) + +Then during +[https://github.com/rear/rear/pull/3171](https://github.com/rear/rear/pull/3171) +in +[https://github.com/rear/rear/pull/3171\#pullrequestreview-1931528461](https://github.com/rear/rear/pull/3171#pullrequestreview-1931528461) +@pcahyna proposed again to use 'source' +(now with a separated "bash in between") +because + + I think this is simpler and more robust than using eval and grep + (think of possible multi-line strings, for example). + +see also +[https://github.com/rear/rear/pull/3171\#issuecomment-2015082283](https://github.com/rear/rear/pull/3171#issuecomment-2015082283) + + ... the "bash in between" + which was up to now more a workaround for source + that has a problem with readonly variables + +(End of "forensics" why we use 'source'.) + +From there we further developed +how to get a variable value out of a file +with the 'source' method. + +For me the generic advantage of the 'source' method is +that we let 'bash' interpret and execute the file +so we get the value that 'bash' actually sets +(think of possible 'if' conditions, for example). + +Of course the generic disadvantage of the 'source' method is +that we let 'bash' interpret and execute the file +so we execute a (shell-syntax) file only +to get a variable value out of a file. + +I think when we restrict the sourcing shell +or when we let it abort at any errors, then +the generic advantage of the 'source' method +does no longer hold, see my experiments in +[https://github.com/rear/rear/pull/3171](https://github.com/rear/rear/pull/3171) + +I think we either use 'source' as it normally works +and then the sourced file must be trusted +or we better use another method +when the sourced file cannot be trusted. + +But even then we would need additional security things +to avoid unintended bash interpretation/execution +of the variable value when it is e.g. something like + + $( COMMAND ) + +or whatever kind of bash code that could trigger +interpretation/execution when used later in bash. + +This all looks rather hopeless to get it secure. +Perhaps it actually is hopeless to securely +process unknown files or use data from unknown files? + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-15 12:41](https://github.com/rear/rear/pull/3203#issuecomment-2056760487): + +An offhanded generic idea how to avoid problems +with +"safely processing/interpreting untrustworthy files" +or with +"safely using data from untrustworthy files" +or generically with +"safely using untrustworthy files": + +My idea is to "simply" not use untrustworthy files. +(wow - what a great ingenious new idea ;-) + +Actually my idea is that a file is untrustworthy +for a particular user account, +if other users could have modified that file. + +Or in other words: +Only those files are trustworthy for a particular user account, +where only that particular user could have written the file. + +For ReaR this means: +Only those files are trustworthy to be used by ReaR +where only 'root' could have written the file. + +To check if only 'root' could have written a file +the only possible way in practice is +to check the current file owner, group, and permissions. + +So I suggest another new helper function: + + function is_trustworthy_for_root () { + local resolved_file + resolved_file="$( readlink -e "$1" )" || return 1 + # Owner name and group name must be 'root root': + test "$( stat -c '%U %G' $resolved_file )" = "root root" || return 1 + # Neither group nor others must have write permission + # so the human readable permission string must be '-' + # for group and others for example like "-rwxr-xr-x" + [[ "$( stat -c '%A' $resolved_file )" == ?????-??-? ]] + } + +This helper function can then be called where needed +e.g. in source\_variable\_from\_file() like + + is_trustworthy_for_root "$1" || return 1 + bash -c ... + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-15 13:49](https://github.com/rear/rear/pull/3203#issuecomment-2056911103): + +My curent `is_trustworthy_for_root` implementation +is likely on the one hand too sceptical because +e.g. on my openSUSE Leap 15.5 system: + + # stat -c '%U %G %A' /etc/chrony.keys + root chrony -rw-r----- + + # ( set -x ; is_trustworthy_for_root /etc/chrony.keys && echo Y || echo N ) + + is_trustworthy_for_root /etc/chrony.keys + + local resolved_file + ++ readlink -e /etc/chrony.keys + + resolved_file=/etc/chrony.keys + ++ stat -c '%U %G' /etc/chrony.keys + + test 'root chrony' = 'root root' + + return 1 + + echo N + N + +but on the other hand it is likely too careless +because it considers only traditional +owner, group, and permissions +but not newer things like ACLs. + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-15 14:14](https://github.com/rear/rear/pull/3203#issuecomment-2056966072): + +@jsmeix I like your idea to reduce to attack surface by checking for +trustworthy files (and you could use `stat --format "%A %u %g"` to check +for all requirements in one comparison, if you like). + +I'm wondering if we really need that as we actually don't have that +check in the `Source()` function of ReaR and don't care about the +ownership of the ReaR files themselves. + +So maybe your initial thought was correct and we don't need to care so +much about that? + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-15 14:25](https://github.com/rear/rear/pull/3203#issuecomment-2056992029): + +@jsmeix I was thinking about further along the lines of "preventing +command execution" and came up with this approach: Setting the `PATH` in +the restricted shell to something without binaries in it effectively +prevents running anything. + +Here is the amended version of my suggestion embedded in a shell script +to test the behavior: + + function Error() { + echo -e "ERROR: $*" + exit 1 + } + + function get_shell_file_config_variable() { + [[ -r "$1" && "$2" =~ [a-zA-Z_][a-zA-Z_0-9]* ]] || Error "Arg 1 '$1' is not a readable file or arg 2 '$2' is not valid variable name" + local file="$1" + local key="$2" + + ( + function die { + echo "ERROR: $*" + exit 1 + } + set -e -u -o pipefail + unset $key + declare -rx PATH=/dev/null + set -r + eval "$(< "$file")" || die "Bash errors while executing '$file'" + [[ -v $key ]] || die "$key is not set in $file" + echo ${!key} + ) + } + + get_shell_file_config_variable /etc/os-release VERSION + echo $? + + get_shell_file_config_variable /etc/os-release SCHLOMO + echo $? + + get_shell_file_config_variable /etc/passwd VERSION + echo $? + + get_shell_file_config_variable /etc/cron.daily/dpkg foo + echo $? + + bad_file=$(mktemp) + echo -e "date\nFOO='bar in $bad_file'" >$bad_file + get_shell_file_config_variable $bad_file FOO + echo $? + + get_shell_file_config_variable /etc/nothing VERSION + echo $? + +Is there a reason why you prefer to run an extra `bash` binary? This +version also has cleaner error handling and doesn't show internal errors +about undefined variables or such: + + root@9487ecb740f3:/rear/test# bash s.sh + 22.04.4 LTS (Jammy Jellyfish) + 0 + ERROR: SCHLOMO is not set in /etc/os-release + 1 + s.sh: line 20: root:x:0:0:root:/root:/bin/bash: restricted: cannot specify `/' in command names + s.sh: line 21: daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 22: bin:x:2:2:bin:/bin:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 23: sys:x:3:3:sys:/dev:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 24: sync:x:4:65534:sync:/bin:/bin/sync: restricted: cannot specify `/' in command names + s.sh: line 25: games:x:5:60:games:/usr/games:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 26: man:x:6:12:man:/var/cache/man:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 27: lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 28: mail:x:8:8:mail:/var/mail:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 29: news:x:9:9:news:/var/spool/news:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 30: uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 31: proxy:x:13:13:proxy:/bin:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 32: www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 33: backup:x:34:34:backup:/var/backups:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: line 34: list:x:38:38:Mailing: command not found + s.sh: line 35: irc:x:39:39:ircd:/run/ircd:/usr/sbin/nologin: restricted: cannot specify `/' in command names + s.sh: eval: line 36: syntax error near unexpected token `(' + s.sh: eval: line 36: `gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin' + ERROR: Bash errors while executing '/etc/passwd' + 1 + s.sh: line 27: /usr/libexec/dpkg/dpkg-db-backup: restricted: cannot specify `/' in command names + ERROR: Bash errors while executing '/etc/cron.daily/dpkg' + 1 + s.sh: line 20: date: command not found + bar in /tmp/tmp.5UswwxhrrT + 0 + ERROR: Arg 1 '/etc/nothing' is not a readable file or arg 2 'VERSION' is not valid variable name + root@9487ecb740f3:/rear/test# + +ATM there is only one scenario that I don't like which is that I didn't +manage to abort reading the config file if there is anything wrong with +it. The example with `line 20: date: command not found` illustrates +that: The line with `date` in the "config file" is ignored but the 2nd +line with `FOO=` is then read. Strictly speaking I'd prefer the code to +abort on that with an error. + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-15 14:27](https://github.com/rear/rear/pull/3203#issuecomment-2056996752): + +Since we talked about the example of reading `/etc/os-release` I read +again +[https://www.freedesktop.org/software/systemd/man/latest/os-release.html](https://www.freedesktop.org/software/systemd/man/latest/os-release.html) +and realised that we should also read `/usr/lib/os-release` if it +doesn't exist... More complexity and probably not part of the function +under discussion, but of the use case for which we bring it in. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-15 14:44](https://github.com/rear/rear/pull/3203#issuecomment-2057037332): + +My current is\_trustworthy\_for\_root is meant +for files that are not ReaR files, +like some shell-syntax config files. + +We use our Source() function only for ReaR files +(hopefully - as far as I assumed up to now). + +We don't care about the ownership of the ReaR files +and currently I cannot tell if this is a problem. +It could be a problem when we e.g. Source() files +as ReaR scripts that could have been "injected" +by others (i.e. new scripts in writable directories) +or modified by others (i.e. existing scripts). +Perhaps we may better call is\_trustworthy\_for\_root +for each file that we Source() to be on the safe side? +Hopefully at least usr/share/rear/lib/global-functions.sh +is safe so that others cannot modify is\_trustworthy\_for\_root(). +Up to now I did not at all think about that. +Perhaps too carelessly I blindly assumed that +in any case only root can install ReaR +(perhaps because only root can install RPMs) +so I assumed ReaR files are owned by root and all is safe. +But I never verified that others cannot modify ReaR files. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-15 14:53](https://github.com/rear/rear/pull/3203#issuecomment-2057057375): + +Regarding "why calling 'source'" +and "why having another bash in between" +see my explanations above and in particular +see the history of all that in +[https://github.com/rear/rear/pull/3165](https://github.com/rear/rear/pull/3165) +and +[https://github.com/rear/rear/pull/3171](https://github.com/rear/rear/pull/3171) + +Neither "calling 'source'" +nor "having another bash in between" +was my idea so I cannot properly answer questions +about all the reasoning behind. + +I only recognized that things worked better +(i.e. less false failures from too simple implementations) +when using 'source' from within a separated bash +but I did not compare that with possible other ways +how one could get a variable value out of a file. + +I would much appreciate if if @pcahyna and @lzaoral +could comment here to help to get things sorted out +so that we all better understand each other. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-15 15:03](https://github.com/rear/rear/pull/3203#issuecomment-2057079972): + +@schlomo +regarding your suggested get\_shell\_file\_config\_variable(): + +I think your goal is different than the goal +behind the 'source' method, cf. my +[https://github.com/rear/rear/pull/3203\#issuecomment-2056083708](https://github.com/rear/rear/pull/3203#issuecomment-2056083708) +therein in particular what I think what +the generic advantage of the 'source' method is +and what +the generic disadvantage of the 'source' method is. + +So it makes sense to also have your function in ReaR +for cases where we cannot 'source' a file. + +Perhaps over time we learn that in practice we never +actually needed to 'source' a file to get a variable +value out of it and then we could "just drop" +that source\_variable\_from\_file function +because it is only a ReaR internal function +so we are free to do with it what we want. + +Perhaps in the end only is\_trustworthy\_for\_root +and its consequences "survive" from here +which could be a rather good end result. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-15 15:20](https://github.com/rear/rear/pull/3203#issuecomment-2057119098): + +Regarding /etc/os-release versus /usr/lib/os-release: + +At least on my openSUSE Leap 15.5 I have: + + # ls -l /etc/os-release + lrwxrwxrwx 1 root root ... /etc/os-release -> ../usr/lib/os-release + +so at least on openSUSE Leap 15.5 +/etc/os-release exists + + # rpm -qf /etc/os-release + openSUSE-release-15.5-lp155.286.1.x86_64 + +but I don't know if /etc/os-release exists +as file or as symlink to /usr/lib/os-release +on other Linux distributions. + +By the way: +This link was the reason why I do + + resolved_file="$( readlink -e "$filename" )" || return 1 + +in is\_trustworthy\_for\_root() because otherwise +the link would be reported as untrustworthy because + + # stat -c %A /etc/os-release + lrwxrwxrwx + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-18 09:32](https://github.com/rear/rear/pull/3203#issuecomment-2063439858): + +FYI +where we call 'source' in our scripts: + +I used the output of + + # find usr/sbin/rear usr/share/rear/ -type f \ + | xargs egrep '^source |[^$_(]source ' \ + | egrep -v ': *#|resource|while read source' + +and removed the false positives: + + usr/sbin/rear: + if ! source $SHARE_DIR/conf/default.conf ; then + if ! source $SHARE_DIR/lib/_input-output-functions.sh ; then + source $script || BugError "Failed to source $script" + source $SHARE_DIR/lib/progresssubsystem.nosh || BugError "Failed to source $SHARE_DIR/lib/progresssubsystem.nosh" + + usr/share/rear/restore/NBKDC/default/400_restore_backup.sh: + source $VAR_DIR/recovery/nbkdc_settings + + usr/share/rear/layout/recreate/default/200_run_layout_code.sh: + ( source $LAYOUT_CODE ) + + usr/share/rear/layout/prepare/default/200_recreate_hpraid.sh: + ( source $LAYOUT_CODE ) + + usr/share/rear/layout/prepare/Linux-s390/400_run_dasd_format_code.sh: + ( source $DASD_FORMAT_CODE ) + + usr/share/rear/layout/save/GNU/Linux/190_opaldisk_layout.sh: + source "$(opal_device_attributes "$device" attributes)" + + usr/share/rear/skel/default/bin/dhclient-script: + source /etc/scripts/dhcp-setup-functions.sh + + usr/share/rear/skel/default/etc/profile: + source "$script" + + usr/share/rear/skel/default/etc/syslog-ng-v3.conf: + source src { + + usr/share/rear/skel/default/etc/scripts/system-setup: + source /usr/share/rear/conf/default.conf || echo -e "\n'source /usr/share/rear/conf/default.conf' failed with exit code $?" + source /etc/rear/$conf.conf || echo -e "\n'source /etc/rear/$conf.conf' failed with exit code $?" + source $system_setup_script || echo -e "\n'source $system_setup_script' results exit code $?\n" + if ! source $system_setup_script ; then + + usr/share/rear/skel/default/etc/scripts/system-setup.d/58-start-dhclient.sh: + source /etc/scripts/dhcp-setup-functions.sh + + usr/share/rear/skel/default/etc/scripts/unlock-opal-disks: + source /usr/share/rear/lib/opal-functions.sh + [[ -f /.OPAL_PBA_SETTINGS.sh ]] && source /.OPAL_PBA_SETTINGS.sh + [[ ${#OPAL_PBA_TKNPATH[@]} -gt 0 ]] && source /usr/share/rear/lib/authtoken-functions.sh + source "/etc/scripts/system-setup.d/$system_setup_script" + + usr/share/rear/skel/default/etc/syslog-ng.conf: + source src { + + usr/share/rear/skel/NBKDC/etc/scripts/system-setup.d/90-start-nbkdc.sh: + source $VAR_DIR/recovery/nbkdc_settings + + usr/share/rear/skel/SESAM/etc/scripts/system-setup.d/59-start-sesam-client.sh: + source $SHARE_DIR/lib/sesam-functions.sh + source $SESAM_VAR_DIR/var/ini/sesam2000.profile + + usr/share/rear/prep/DUPLICITY/default/200_find_duply_profile.sh: + source $CONF # is a normal shell configuration file + source $DUPLY_PROFILE_FILE + + usr/share/rear/prep/SESAM/default/400_prep_sesam.sh: + source $SHARE_DIR/lib/sesam-functions.sh + + usr/share/rear/rescue/GNU/Linux/220_load_modules_from_initrd.sh: + source /etc/sysconfig/kernel + source /etc/dracut.conf + source $s + + usr/share/rear/lib/rear-shell.bashrc: + for script in $SHARE_DIR/lib/*functions.sh ; do source $script ; done + source $SHARE_DIR/lib/progresssubsystem.nosh + + usr/share/rear/lib/drlm-functions.sh: + source $DRLM_CFG + source $DRLM_CFG + + usr/share/rear/lib/opaladmin-workflow.sh: + source "$(opal_device_attributes "$device" attributes)" + + usr/share/rear/lib/sesam-functions.sh: + source $sesam2000ini_file + + usr/share/rear/lib/opal-functions.sh: + source "$(opal_device_attributes "$device" attributes)" + source "$(opal_device_attributes "$device" attributes)" + + usr/share/rear/lib/framework-functions.sh: + source "$source_file" + + usr/share/rear/lib/global-functions.sh: + source $CONF # is a normal shell configuration file + + usr/share/rear/build/OPALPBA/Linux-i386/810_deduplicate_files.sh: + source "$deduplication_script" + + usr/share/rear/build/USB/default/800_enforce_usb_output.sh: + local_conf_output=$( source $ROOTFS_DIR/etc/rear/local.conf ; echo $OUTPUT ) + +i.e. 43 places where we call 'source' in our scripts. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-09.3204.pr.merged.md b/docs/issues/2024-04-09.3204.pr.merged.md new file mode 100644 index 00000000..d8f59389 --- /dev/null +++ b/docs/issues/2024-04-09.3204.pr.merged.md @@ -0,0 +1,81 @@ +[\#3204 PR](https://github.com/rear/rear/pull/3204) `merged`: Better describe OUTPUT\_PREFIX and NETFS\_PREFIX +============================================================================================================== + +**Labels**: `documentation`, `fixed / solved / done` + +#### [jsmeix](https://github.com/jsmeix) opened issue at [2024-04-09 14:51](https://github.com/rear/rear/pull/3204): + +- Type: **Enhancement** + +- Impact: **None** + +No default should change. + +- Reference to related issue (URL): + +No GitHub issue. + +A colleague asked me how to get backup.tar.gz +and the ISO and all other RESULT\_FILES +in the same directory that is not $HOSTNAME. + +I had to somewhat reverse engineer things to find out how +because this is basically not described in default.conf. + +- How was this pull request tested? + +I did not yet test that this pull request +does not change anything. + +I had only tested that with + + OUTPUT=ISO + OUTPUT_PREFIX="my_prefix" + NETFS_PREFIX="$OUTPUT_PREFIX" + BACKUP=NETFS + BACKUP_URL=file:///other/ + +backup.tar.gz and the ISO and all other RESULT\_FILES +get stored in the same "my\_prefix" directory. + +- Description of the changes in this pull request: + +In default.conf better descriptions +for OUTPUT\_PREFIX and NETFS\_PREFIX +(OUTPUT\_PREFIX was not at all described) +plus explanation how OUTPUT\_PREFIX and NETFS\_PREFIX +belong to each other for the usual case +OUTPUT=ISO and BACKUP=NETFS +and made it explicit that by default + + NETFS_PREFIX="$OUTPUT_PREFIX" + +to get backup.tar.gz and ISO image stored at the same place. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-18 09:46](https://github.com/rear/rear/pull/3204#issuecomment-2063466534): + +@rear/contributors +my changes here are only comments in default.conf +so they cannot change what ReaR does when it runs +but wrong or misleading comments in default.conf +could make things go wrong for our users +when they do not get correct information from us +so I would appreciate at least a quick review +and an explicit approval from one of you. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-19 06:45](https://github.com/rear/rear/pull/3204#issuecomment-2065866266): + +@gdha +thank you for having a look here! + +I also think "all bits help", cf. + + I think better try to improve things than do nothing. + +in my +[https://github.com/rear/rear/pull/3182\#issuecomment-2065861644](https://github.com/rear/rear/pull/3182#issuecomment-2065861644) + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-09.3205.pr.merged.md b/docs/issues/2024-04-09.3205.pr.merged.md new file mode 100644 index 00000000..b9b0153f --- /dev/null +++ b/docs/issues/2024-04-09.3205.pr.merged.md @@ -0,0 +1,30 @@ +[\#3205 PR](https://github.com/rear/rear/pull/3205) `merged`: Some misc fixes +============================================================================= + +#### [schlomo](https://github.com/schlomo) opened issue at [2024-04-09 21:48](https://github.com/rear/rear/pull/3205): + +This is stuff that I fixed while working on the \#3190: + +- Obsolete old and ancient validations +- Support RHEL 8 and RHEL 9 for OS detection +- abort earlier if `rear recover` runs on non-rescue system + +Please have a look, I'll merge them soon otherwise. + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-10 09:57](https://github.com/rear/rear/pull/3205#issuecomment-2047085448): + +FYI, the [koji build +job](https://koji.fedoraproject.org/koji/taskinfo?taskID=116151447) +failed with + + BuildError: Unknown origin for rust-srpm-macros-epel-26.2-1.el9.noarch: file:///var/tmp/koji/tasks/7264/116147264/repo_5998851_premerge/ + +which seems to be unrelated to ReaR because our own package build was +successful. + +I'll be merging this now. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-09.3206.pr.merged.md b/docs/issues/2024-04-09.3206.pr.merged.md new file mode 100644 index 00000000..35787902 --- /dev/null +++ b/docs/issues/2024-04-09.3206.pr.merged.md @@ -0,0 +1,115 @@ +[\#3206 PR](https://github.com/rear/rear/pull/3206) `merged`: Portable recovery +=============================================================================== + +#### [schlomo](https://github.com/schlomo) opened issue at [2024-04-09 21:52](https://github.com/rear/rear/pull/3206): + +Add `OUTPUT=PORTABLE` and `--portable` command line option to faciliate +using ReaR in truly portable mode. + +The portable archive contains **only** ReaR, nothing else. + +Tested with an OL9 restore via SystemRescueCD + +I'll do some more testing both of portable usage and regular usage to +ensure that this change doesn't hurt us. + +Implements \#3190 and should be merged after \#3205, where I extracted +the unrelated fixes. To review you can simply look at the last commit +here. + +#### [gdha](https://github.com/gdha) commented at [2024-04-10 06:39](https://github.com/rear/rear/pull/3206#issuecomment-2046636269): + +I miss some documentation here. Is it possible to add it? + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-10 10:35](https://github.com/rear/rear/pull/3206#issuecomment-2047173476): + +@gdha I added a short manual + +@jsmeix the help workflow had bad indentation which I of course fixed, +besides that I reduced the whitespace changes. + +My goal at the moment is to get feedback on this and do some more tests. +There will be a second phase of development with further optimisations +(for example, I want to see if I can skip the `build` stage entirely to +speed things up). + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-10 11:23](https://github.com/rear/rear/pull/3206#issuecomment-2047273428): + +Aaargh, now I get also +[https://github.com/rear/rear/pull/3168\#issuecomment-1983377528](https://github.com/rear/rear/pull/3168#issuecomment-1983377528) +because the `TMPDIR` is set for portable mode. Which means that I need +to extend \#3168 to not touch the temp dir for portable mode. + +About dracut not showing errors in the ReaR log, I could redirect the +output to properly capture it. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-10 12:14](https://github.com/rear/rear/pull/3206#issuecomment-2047387337): + +I did not have a look here (no time yet) +so the following is only an offhanded thought: + +Regarding +[https://github.com/rear/rear/pull/3206\#issuecomment-2047273428](https://github.com/rear/rear/pull/3206#issuecomment-2047273428) +dracut and portable mode: + +Currently we have in sbin/rear + + test -e "/etc/rear-release" && RECOVERY_MODE='y' || RECOVERY_MODE='' + readonly RECOVERY_MODE + +and we do not change TMPDIR in RECOVERY\_MODE + + if ! test "$RECOVERY_MODE" ; then + # We set TMPDIR to ReaR's TMP_DIR only when we are not in RECOVERY_MODE + +so I think you need to enhance how RECOVERY\_MODE is set +so that RECOVERY\_MODE is also set for "rear recover" +in "portable mode"? + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-10 13:28](https://github.com/rear/rear/pull/3206#issuecomment-2047547566): + +@jsmeix about the dracut not showing errors problem I figured out what +is going on: + +In normal (non-debug) ReaR mode stderr goes to `$STDOUT_STDERR_FILE` and +the `Error` function can then quote from that. However, our code +surrounding dracut doesn't use the `Error` function but instead only +does a `LogPrint` suggesting to look into the `$RUNTIME_LOGFILE`, and +that is the reason that the logfile doesn't contain useful infos. + +How should we solve this? Maybe extract the "pull last lines from +stderr" into a function that can then be used? Or add a `LogPrintError` +function that will also show last errors if exist? + +I'm not solving this here but I think it is important to keep in mind +and fix so that users can get an info about the problem for errors +without running ReaR again in debug mode. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-10 13:39](https://github.com/rear/rear/pull/3206#issuecomment-2047575814): + +I made the separated +[https://github.com/rear/rear/issues/3207](https://github.com/rear/rear/issues/3207) +regarding +[https://github.com/rear/rear/pull/3206\#issuecomment-2047547566](https://github.com/rear/rear/pull/3206#issuecomment-2047547566) + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-11 18:03](https://github.com/rear/rear/pull/3206#issuecomment-2050228898): + +I found a bug in this code, it doesn't work when used from a dist +install :-( + +I'll keep debugging then. + +#### [schlomo](https://github.com/schlomo) commented at [2024-04-12 14:53](https://github.com/rear/rear/pull/3206#issuecomment-2051915587): + +> I found a bug in this code, it doesn't work when used from a dist +> install :-( +> +> I'll keep debugging then. + +It's the `Makefile` patching the main script, so I'll improve the +Makefile for this, too. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-10.3207.issue.open.md b/docs/issues/2024-04-10.3207.issue.open.md new file mode 100644 index 00000000..fe8c2b94 --- /dev/null +++ b/docs/issues/2024-04-10.3207.issue.open.md @@ -0,0 +1,82 @@ +[\#3207 Issue](https://github.com/rear/rear/issues/3207) `open`: Recreating initrd failure shows misleading message when not in debug mode +========================================================================================================================================== + +**Labels**: `enhancement`, `minor bug` + +#### [jsmeix](https://github.com/jsmeix) opened issue at [2024-04-10 13:38](https://github.com/rear/rear/issues/3207): + +See +[https://github.com/rear/rear/pull/3206](https://github.com/rear/rear/pull/3206) +therein in particular (excerpts): +[https://github.com/rear/rear/pull/3206\#issuecomment-2047273428](https://github.com/rear/rear/pull/3206#issuecomment-2047273428) + + dracut not showing errors in the ReaR log + +and +[https://github.com/rear/rear/pull/3206\#issuecomment-2047547566](https://github.com/rear/rear/pull/3206#issuecomment-2047547566) + + about the dracut not showing errors problem I figured out what is going on: + + In normal (non-debug) ReaR mode stderr goes to $STDOUT_STDERR_FILE + and the Error function can then quote from that. + However, our code surrounding dracut doesn't use the Error function + but instead only does a LogPrint suggesting to look into + the $RUNTIME_LOGFILE, and that is the reason that the + logfile doesn't contain useful infos. + + How should we solve this? + Maybe extract the "pull last lines from stderr" into a function + that can then be used? Or add a LogPrintError function + that will also show last errors if exist? + + ... I think it is important to keep in mind and fix + so that users can get an info about the problem + for errors without running ReaR again in debug mode. + +#### [jsmeix](https://github.com/jsmeix) commented at [2024-04-10 14:19](https://github.com/rear/rear/issues/3207#issuecomment-2047685054): + +I will have to think about it... + +We already have a LogPrintError function which is +currently used differently, see its description in +lib/\_input-output-functions.sh + +For specific usage examples see for example +finalize/default/060\_compare\_files.sh + + LogPrintError "Restored files in $TARGET_FS_ROOT do not fully match the recreated system" + LogPrintError "(files in the backup are not same as when the ReaR rescue/recovery system was made)" + ... + LogPrintError "$( sed -e "s|^/|$TARGET_FS_ROOT/|" -e 's/^/ /' <<< "$md5sum_stdout" )" + LogPrintError "Manually check if those changed files cause issues in your recreated system" + +and also +usr/share/rear/prep/PXE/default/010\_PXE\_check.sh +usr/share/rear/lib/opaladmin-workflow.sh +usr/share/rear/lib/\_input-output-functions.sh +usr/share/rear/lib/opal-functions.sh +usr/share/rear/build/default/990\_verify\_rootfs.sh +usr/share/rear/restore/FDRUPSTREAM/default/260\_copy\_log\_and\_report.sh +that use several subsequent LogPrintError calls +where ReaR log file output is unwanted. +In general the current LogPrintError function +is not meant to provide ReaR log file output. +In contrast the Error function can provide ReaR log file output +because the Error function is the last function that is called +so ReaR log file output will be shown only once. + +My current offhanded guess is that a separated function +(extract the 'pull last lines from stderr' into a function) +to show ReaR log file output could be a possible solution +so that we have control when ReaR log file output is shown. + +The current `# Extract lines ...` code +in the Error function depends on that this happens +within the `[Bug]*Error` function so some adaptions +and enhancements are needed to make that code a +generic `LastSourcedScriptStdoutStderr` function. + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\] diff --git a/docs/issues/2024-04-19.3208.issue.open.md b/docs/issues/2024-04-19.3208.issue.open.md new file mode 100644 index 00000000..dbd07fdc --- /dev/null +++ b/docs/issues/2024-04-19.3208.issue.open.md @@ -0,0 +1,78 @@ +[\#3208 Issue](https://github.com/rear/rear/issues/3208) `open`: Recovery from Rubrik Replica does not work +=========================================================================================================== + +#### [rmccrack](https://github.com/rmccrack) opened issue at [2024-04-19 18:51](https://github.com/rear/rear/issues/3208): + + + +- ReaR version ("/usr/sbin/rear -V"):rear-2.7-1.git.5366 + +- If your ReaR version is not the current version, explain why you + can't upgrade: + +- OS version ("cat /etc/os-release" or "lsb\_release -a" or "cat + /etc/rear/os.conf"): RHEL 7.9 + +- ReaR configuration files ("cat /etc/rear/site.conf" and/or "cat + /etc/rear/local.conf"): + +- BACKUP=CDM + +- OUTPUT=ISO + +- export + LD\_LIBRARY\_PATH=$LD\_LIBRARY\_PATH:/usr/lib64/python2.7/site-packages/;/usr/lib64/bind9-export/:/usr/lib64/R/lib/" + +- Hardware vendor/product (PC or PowerNV BareMetal or ARM) or VM (KVM + guest or PowerVM LPAR): Vmware VM + +- System architecture (x86 compatible or PPC64/PPC64LE or what exact + ARM device): + +- Firmware (BIOS or UEFI or Open Firmware) and bootloader (GRUB or + ELILO or Petitboot): + +- Storage (local disk or SSD) and/or SAN (FC or iSCSI or FCoE) and/or + multipath (DM or NVMe): + +- Storage layout ("lsblk -ipo + NAME,KNAME,PKNAME,TRAN,TYPE,FSTYPE,LABEL,SIZE,MOUNTPOINT"): + +- Description of the issue (ideally so that others can reproduce + it): + When I attempt a restore from an iso created using the rear -v + mkrescue command, and select the option to restore from a replica + Rubrik cluster, the agent will not register on the replica cluster + when I attempt to register it using the IP address. I get an error + indicating the agent is not running (refusal to connect on port + 12801). When I check on the partially restored system, the Rubrik + agents are not running. Attempts to start the agents using + "systemctl start rubrikagents" result in an error + "rubrikagents.service not found" + +- Workaround, if any: + None I can find so far. + +- Attachments, as applicable ("rear -D mkrescue/mkbackup/recover" + debug log files): + +- The backup log files did not reflect any errors. + The system I was trying to recover (a Vmware VM) no longer exists + and I have no option left to restore except to use the Rubrik + replica copy of the backup. + +You can drag-drop log files into this editor to create an attachment +or paste verbatim text like command output or file content +by including it between a leading and a closing line of +three backticks like this: + + verbatim content + +------------------------------------------------------------------------ + +\[Export of Github issue for +[rear/rear](https://github.com/rear/rear).\]