Backup restore finishes with error: failed to decrypt [...]: b'scrypt: Passphrase is incorrect

[pl853]

Qubes OS version:

R4.0.1-RC1

Affected component(s):

Qubes Backup manager

---

Steps to reproduce the behavior:

1. Select restore qube from backup in the QubesManager
2. Select Qube in which backup is found
3. Select path to backup file
4. Check verify backup integrity
   6 Check encrypted backup box
5. Fill in decryption password

Expected behavior:

Restore selected backup

Actual behavior:

Display's message: failed to decrypt var/tmp/{the file name}/qubes.xml.000.enc: b'scrypt: Passphrase is incorrect\n'

General notes:

I've used R4.0.1-RC1 for a few day's now and i have not encountered a problem up till now. I tried every option that's available:

• Backed up all the Qubes at the same time. (On 5 different qubes and an USB, which was encrypted at first, which didn't work. afterwards i removed the encryption and it still couldn't restore from the usb)
• Tried a couple of different encryption passwords for the backup. Which included simple letters and numbers such as 1 or a etc.
• Tried checking all the boxes at restore options at the same time and separate.
• I also tried an option mentioned in another issue [R4.0 RC2] qvm-backup-restore or qvm-backup error #3211 which had a similar problem:
  At line 134 in file /usr/lib/python3.5/site-packages/qubesadmin/backup/core3.py:
  vm.label = vm.properties.pop('label')
  try:
  vm.label = self.labels[vm.label]
  except KeyError:
  pass

This also didn't work.

Afterwards i re-installed Qubes 4.0 and retried the steps and it worked the first try. So the problem is in R4.0.1-RC1. I also tried to re-install R4.0.1-RC1 and tried it again but with no luck.

Thanks in advance for the help!

---

Related issues:

#3321
#3219

[marmarek]

I've reproduced the problem already, looking for solution.

[pl853]

Thankyou, if i can help in any way, ill be there.

…

On Sat, Nov 10, 2018, 5:22 PM Marek Marczykowski-Górecki < ***@***.***> wrote: I've reproduced the problem already, looking for solution. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#4493 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AXUmK_oST329LIHtWKsgzkXKZv_W5QPNks5utv0qgaJpZM4YU3nd> .

[marmarek]

Does it happen also with backups created on earlier release (4.0, not 4.0.1-rc1)?

[pl853]

I didnt try that. Ill try it when i get home and get back to you immediatly.

…

On Sat, Nov 10, 2018, 5:25 PM Marek Marczykowski-Górecki < ***@***.***> wrote: Does it happen also with backups created on earlier release (4.0, not 4.0.1-rc1)? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#4493 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AXUmK9pBmpx2Kmt7MS9A8WeoPdShMA2Kks5utv36gaJpZM4YU3nd> .

[marmarek]

No need, it's the backup code broken, not the restore one.
Specifically, this change broke it: QubesOS/qubes-core-admin@4e49b95#diff-d5cd0937e32eff778591ab56cd19526eR260

[pl853]

Okay, should i try to reverse it or will it compremise security?

…

On Sat, Nov 10, 2018, 5:48 PM Marek Marczykowski-Górecki < ***@***.***> wrote: No need, it's the backup code broken, not the restore one. Specifically, this change broke it: ***@***.*** #diff-d5cd0937e32eff778591ab56cd19526eR260 <QubesOS/qubes-core-admin@4e49b95#diff-d5cd0937e32eff778591ab56cd19526eR260> — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#4493 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AXUmK-AwvYOLrFn10bTkUWPjaLAYzpBxks5utwNGgaJpZM4YU3nd> .

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-dom0-4.0.35-1.fc25 has been pushed to the r4.0 testing repository for dom0.
To test this update, please install it with the following command:

sudo qubes-dom0-update --enablerepo=qubes-dom0-current-testing

Changes included in this update

[qubesos-bot]

Automated announcement from builder-github

The package qubes-core-dom0-4.0.37-1.fc25 has been pushed to the r4.0 stable repository for dom0.
To install this update, please use the standard update command:

sudo qubes-dom0-update

Or update dom0 via Qubes Manager.

Changes included in this update

[akkuladezeit]

Is there any way to Backup without using the Tool in Qubes3 it was easy but on 4.1 it seams impossible? I just wont to migrate to a New Hardware but every Restore fails ... and i dont think it will be fixed in near Future

[akkuladezeit]

@joetretter:

it seems that there is an issue that vanishes when runing one of the steps a second time without changing any of the inputs.

That smells like a hardware error, e.g. faulty RAM and/or overheating.

I dont observed any System Crash.. so i dont thinks That it is a ram problem.

Its more likely a Problem of amd ryzen series 1 ?

[joetretter]

I have been able to drill down this problem to scrypt not working properly on my AMD Ryzen 7 1700 Eight-Core Processor.
Even downloading the code and running the tests, I could see that it fails on this computer - but it didn't always fail at the same test. To be able to move my VMS to another computer, I worked around this issue by temporarily replacing the scrypt executable on the source and target machine with this script:

#!/bin/bash
# 2023-04-30 Joe Tretter, fake replacement for scrypt to work around backup restore issue in qubes-os
# THIS IS UNSAFE BECAUSE IT DOES NOT ENCRYPT!!!
if [[ "$1" == "enc" ]]
then 
>&2 echo -en "Please enter passphrase: "
>&2 echo -en "Please confirm passphrase: "
else
>&2 echo -en "Please enter passphrase: "
fi
lastArgNo=$#
secondLastArgNo=$((lastArgNo-1))
in=${!secondLastArgNo}
out=${!lastArgNo}
if [[ "-" = "${in}" ]]; then 
cat > $out
else
cat $in > $out
fi

[rustybird]

Hmm scrypt recently released v.1.3.2, with a zillion changes since v.1.3.1 from 3 years ago. If there's some hardware incompatibility maybe the new version fixed it along the way?

[joetretter]

Hmm scrypt recently released v.1.3.2, with a zillion changes since v.1.3.1 from 3 years ago. If there's some hardware incompatibility maybe the new version fixed it along the way?

Possible - but I forgot to mention in my previous post that I have not been able to reproduce the issue with various other (Non-QubesOS) XEN kernels I have tried on the same hardware. It might be a problem in one of the kernel modules or the specific combination of kernel parameters used in QubesOS.

[DemiMarie]

Hmm scrypt recently released v.1.3.2, with a zillion changes since v.1.3.1 from 3 years ago. If there's some hardware incompatibility maybe the new version fixed it along the way?

Possible - but I forgot to mention in my previous post that I have not been able to reproduce the issue with various other (Non-QubesOS) XEN kernels I have tried on the same hardware. It might be a problem in one of the kernel modules or the specific combination of kernel parameters used in QubesOS.

That smells like either:

1. Memory corruption.
2. Xen mishandling some instruction.

[joetretter]

Hmm scrypt recently released v.1.3.2, with a zillion changes since v.1.3.1 from 3 years ago. If there's some hardware incompatibility maybe the new version fixed it along the way?

Possible - but I forgot to mention in my previous post that I have not been able to reproduce the issue with various other (Non-QubesOS) XEN kernels I have tried on the same hardware. It might be a problem in one of the kernel modules or the specific combination of kernel parameters used in QubesOS.

That smells like either:

1. Memory corruption.
2. Xen mishandling some instruction.

Memory corruption is unlikely; the machine has 2x16GB, and I have reproduced the problem with every permutation of 1- and 2- RAM sticks across both RAM slots on the mainboard. I have also repeatedly executed the latest memtest86+.
My specific machine is a Dell Inspiron 5675.

[DemiMarie]

Hmm scrypt recently released v.1.3.2, with a zillion changes since v.1.3.1 from 3 years ago. If there's some hardware incompatibility maybe the new version fixed it along the way?

Possible - but I forgot to mention in my previous post that I have not been able to reproduce the issue with various other (Non-QubesOS) XEN kernels I have tried on the same hardware. It might be a problem in one of the kernel modules or the specific combination of kernel parameters used in QubesOS.

That smells like either:

1. Memory corruption.
2. Xen mishandling some instruction.

Memory corruption is unlikely; the machine has 2x16GB, and I have reproduced the problem with every permutation of 1- and 2- RAM sticks across both RAM slots on the mainboard. I have also repeatedly executed the latest memtest86+. My specific machine is a Dell Inspiron 5675.

Can you reproduce this problem without Xen using the Qubes-provided scrypt and Linux kernel? Qubes should boot without Xen, but no VMs will start.

[joetretter]

Hmm scrypt recently released v.1.3.2, with a zillion changes since v.1.3.1 from 3 years ago. If there's some hardware incompatibility maybe the new version fixed it along the way?

Possible - but I forgot to mention in my previous post that I have not been able to reproduce the issue with various other (Non-QubesOS) XEN kernels I have tried on the same hardware. It might be a problem in one of the kernel modules or the specific combination of kernel parameters used in QubesOS.

That smells like either:

1. Memory corruption.
2. Xen mishandling some instruction.

Memory corruption is unlikely; the machine has 2x16GB, and I have reproduced the problem with every permutation of 1- and 2- RAM sticks across both RAM slots on the mainboard. I have also repeatedly executed the latest memtest86+. My specific machine is a Dell Inspiron 5675.

Can you reproduce this problem without Xen using the Qubes-provided scrypt and Linux kernel? Qubes should boot without Xen, but no VMs will start.

I don't have Qubes installed on that computer at the moment, I can try that out when I have some time... What do I need to do to boot without Xen?

[rustybird]

What do I need to do to boot without Xen?

#8574 (comment)

[akkuladezeit]

I have been able to drill down this problem to scrypt not working properly on my AMD Ryzen 7 1700 Eight-Core Processor.

Even downloading the code and running the tests, I could see that it fails on this computer - but it didn't always fail at the same test. To be able to move my VMS to another computer, I worked around this issue by temporarily replacing the scrypt executable on the source and target machine with this script:

#!/bin/bash

# 2023-04-30 Joe Tretter, fake replacement for scrypt to work around backup restore issue in qubes-os

# THIS IS UNSAFE BECAUSE IT DOES NOT ENCRYPT!!!

if [[ "$1" == "enc" ]]

then 

>&2 echo -en "Please enter passphrase: "

>&2 echo -en "Please confirm passphrase: "

else

>&2 echo -en "Please enter passphrase: "

fi

lastArgNo=$#

secondLastArgNo=$((lastArgNo-1))

in=${!secondLastArgNo}

out=${!lastArgNo}

if [[ "-" = "${in}" ]]; then 

cat > $out

else

cat $in > $out

fi

Can you give some instruction how to repace it ? That would be nice 🥹

Maybe an Backup Option without any encrypt would be nice for such Problems..

[joetretter]

Can you give some instruction how to repace it ? That would be nice 🥹

1) Open a terminal in your personal vm
2) run command 
cat > scrypt
3) paste the script and push Ctrl+D
4) verify the integrity with md5sum scrypt (the hash is supposed to be 1b3ca107905f023a65960ce0829a0287)
5) open a dom0 terminal
6) enter the following commands:
sudo su
cd /usr/bin
mv scrypt scrypt.original
qvm-run --pass-io personal 'cat /home/user/scrypt' > scrypt
chmod 777 scrypt

Maybe an Backup Option without any encrypt would be nice for such Problems..
Yes, I agree - funny enough the restore has an unencrypted option even though there is no such option in the backup ;)

[joetretter]

Can you reproduce this problem without Xen using the Qubes-provided scrypt and Linux kernel? Qubes should boot without Xen, but no VMs will start.

I have tried it. Without Xen the issue does not happen (see screenshots).

[DemiMarie]

@joetretter I think you found a genuine bug in Xen, then. Please report this to xen-devel@lists.xenproject.org.

[akkuladezeit]

Can you give some instruction how to repace it ? That would be nice 🥹

1) Open a terminal in your personal vm

2) run command 

cat > scrypt

3) paste the script and push Ctrl+D

4) verify the integrity with md5sum scrypt (the hash is supposed to be 1b3ca107905f023a65960ce0829a0287)

5) open a dom0 terminal

6) enter the following commands:

sudo su

cd /usr/bin

mv scrypt scrypt.original

qvm-run --pass-io personal 'cat /home/user/scrypt' > scrypt

chmod 777 scrypt

Maybe an Backup Option without any encrypt would be nice for such Problems..

Yes, I agree - funny enough the restore has an unencrypted option even though there is no such option in the backup ;)

Thanks Works Great

[Eric678]

From above closed issue that included 4.2 in affected: add Ryzen 9 to affected hardware - this does seem to be AMD specific, so is different from the original issue in 2018.

The very odd thing is, I know I tested this in 4.2.0-rc1 and there was no problem. I was having problems writing USB storage and so did quite a bit of testing. Going back, I have a full backup from early October that is ~80GB compressed that verifies OK, while one a week later after doing an upgrade fails with "scrypt:Input is not valid scrypt-encrypted block" maybe half way through. Not sure what versions were then. Now a backup of a fresh 4.2.0 instal reliably fails quickly, so it looks like quite a recent regression. Add that to the sudden appearance of app qube kernel panics in 6.1.62 and stable 4.2.0 is looking bad.

[DemiMarie]

@joetretter Does the test suite pass in a Qubes VM? If so, the problem is specific to Xen paravirtualization.

[joetretter]

@joetretter Does the test suite pass in a Qubes VM? If so, the problem is specific to Xen paravirtualization.

The test is not passing in a Qubes-VM either. I am in contact with the XEN folks and they asked me to try with the latest microcode. I have spent about three days and I'm afraid I am not able to get the latest microcode working in Qubes, I can't seem to figure out the right way to make it known to the kernel. As it's been released in parallel to my testing, I have upgraded to 4.2 in the hope that the latest firmware would be included or the problem be solved. Unfortunately it isn't. Can the Qubes project get the latest AMD firmware into 4.2, or can you point me to a resource how I can get it working? I am running Manjaro (archlinux) on the same machine and there I have the latest firmware (see attachments)

[marmarek]

@joetretter do you have most recent dracut package installed (version 059-5) in dom0?

[joetretter]

@joetretter do you have most recent dracut package installed (version 059-5) in dom0?

Yes, I have dracut 059-5.fc37

[DemiMarie]

@joetretter You’ll need to get a firmware update from your hardware vendor. R4.2 has support for applying such updates, so long as the vendor makes them available via fwupd.

@marmarek I’m starting to wonder if we should offer late loading of firmware on AMD systems too, so long as we can find the firmware somewhere. AMD doesn’t test it on most client parts so it isn’t great, but on systems like what @joetretter has it might be the difference between being able to use Qubes OS and not being able to.

[brendanhoar]

@joetretter do you have most recent dracut package installed (version 059-5) in dom0?

I just wanted to note that on my intel Thinkpad P52 with the release version of R4.2 and kernel latest, I also have that most recent version of dracut.

However, no microcode is being loaded (according to xen dmesg and Linux dmesg). I've even performed a sudo dracut --regenerate-all --force and rebooted with no change.

B

[marmarek]

However, no microcode is being loaded (according to xen dmesg and Linux dmesg). I've even performed a sudo dracut --regenerate-all --force and rebooted with no change.

It gets loaded only if there is a newer version available. Most likely your firmware already loads latest version (as it should!) so OS doesn't need to load any updates.