Script still fails after attribute list step

[norristh]

Thanks for fixing #78 !

However, now the script version 1.1.3 fails with:

INFO:root:Finding partition geometry
INFO:root:Finalizing MFT reconstruction of partition at offset 0
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
Traceback (most recent call last):
  File "main.py", line 374, in <module>
    main()
  File "main.py", line 357, in main
    parts.update(scanner.get_partitions())
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 836, in get_partitions
    self.finalize_reconstruction(part)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 672, in finalize_reconstruction
    self.add_from_attribute_list(parsed, part, node.offset)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 627, in add_from_attribute_list
    _integrate_attribute_list(parsed, part, image)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 248, in _integrate_attribute_list
    child_parsed = parse_file_record(dump)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 139, in parse_file_record
    if (header['size_alloc'] > len(entry) or
TypeError: '>' not supported between instances of 'NoneType' and 'int'

[Lazza]

Thank you for reporting this issue.

OK, this is a bit weird. It seems there is a file record which is not complete (truncated).

This should almost never occur (in fact, this is the first time it does). Anyhow I added an extra check. Could you please try with the latest code?

Thank you.

[norristh]

Now it fails with:

INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
Traceback (most recent call last):
  File "main.py", line 374, in <module>
    main()
  File "main.py", line 357, in main
    parts.update(scanner.get_partitions())
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 837, in get_partitions
    self.finalize_reconstruction(part)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 671, in finalize_reconstruction
    for node in many_attributes_it:
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 666, in <genexpr>
    node for node in part.files.values()
RuntimeError: dictionary changed size during iteration

[Lazza]

Could you please try to change this line:

node for node in part.files.values()

To this one?

node for node in iter(part.files.values())

Please note that it occurs twice in ntfs.py.

Thank you.

[norristh]

Same failure:

INFO:root:Finding partition geometry
INFO:root:Finalizing MFT reconstruction of partition at offset 0
INFO:root:Adding extra attributes from $ATTRIBUTE_LIST
Traceback (most recent call last):
  File "main.py", line 374, in <module>
    main()
  File "main.py", line 357, in main
    parts.update(scanner.get_partitions())
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 837, in get_partitions
    self.finalize_reconstruction(part)
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 671, in finalize_reconstruction
    for node in many_attributes_it:
  File "/root/RecuperaBit/recuperabit/fs/ntfs.py", line 666, in <genexpr>
    node for node in iter(part.files.values())
RuntimeError: dictionary changed size during iteration

[Lazza]

Can you try like this?

node for node in list(part.files.values())

[norristh]

Progress! It completed indexing. Running "recoverable" printed almost 3300 entries, but almost all with no size given, and the number of files as 16 or 32 or other low multiples of 8. I don't know whether that's normal. Partition # 23 showed with nearly the full size of the hard drive, and more than 100,000 files.

Running "restore 23 5" recovered many files, yay! It worked through the \Users directories. I don't think it restored all the files which were originally on the hard drive. Again, I don't know whether that's normal. (Any tips to possibly recover more files from \Users ?)

The script did eventually fail again with:

INFO:root:Restoring #35925 Root/Windows/System32/config/COMPONENTS
INFO:root:Restoring #36618 Root/Windows/System32/config/DEFAULT
INFO:root:Restoring #47412 Root/Windows/System32/config/DRIVERS
WARNING:root:Missing part for File(#47412, ^^3113^^, DRIVERS, offset = 6386280 sectors), 2124 clusters skipped
Traceback (most recent call last):
  File "main.py", line 374, in <module>
    main()
  File "main.py", line 371, in main
    interpret(cmd, arguments, parts, shorthands, args.outputdir)
  File "main.py", line 181, in interpret
    logic.recursive_restore(myfile, part, partition_dir)
  File "/root/RecuperaBit/recuperabit/logic.py", line 259, in recursive_restore
    recursive_restore(child, part, outputdir, make_dirs=False)
  File "/root/RecuperaBit/recuperabit/logic.py", line 259, in recursive_restore
    recursive_restore(child, part, outputdir, make_dirs=False)
  File "/root/RecuperaBit/recuperabit/logic.py", line 259, in recursive_restore
    recursive_restore(child, part, outputdir, make_dirs=False)
  [Previous line repeated 1 more time]
  File "/root/RecuperaBit/recuperabit/logic.py", line 246, in recursive_restore
    outfile.write(piece)
TypeError: a bytes-like object is required, not str

I don't care about any files other than in \Users, but if you want me to test anything else to debug that crash, or send the list of >100 partitions if that's unusual behavior, I'm happy to do so. (Though I'm doing this recovery for a friend on a large external HD of hers, so will only have the image file to test with for a few more days.)

For background info: I used ddrescue to 99.99% image a hard drive which no longer booted or mounted due to read errors in early sectors. I'm running recuperabit on the image.

And thank you for such a useful tool, and for your quick responses here! I greatly appreciate it.

[Lazza]

@norristh thank you very much for your feedback, it is very helpful!

Running "recoverable" printed almost 3300 entries, but almost all with no size given, and the number of files as 16 or 32 or other low multiples of 8. I don't know whether that's normal.

This is due to heavy disk fragmentation. The more a drive is used, the more it happens. RecuperaBit does not attempt to blindly merge partitions, thus sometimes many different partitions are detected.

(Any tips to possibly recover more files from \Users ?)

Possibly they could be in the LostFiles tree (-1 instead of 5) or in a different one. You should mainly check those with the same starting offset.

TypeError: a bytes-like object is required, not str

Ouch. This is left from the migration to Python 3. It seems the smaller sample NTFS on which I tested it did not reach this part of the code. It should be fixed now.

but if you want me to test anything else to debug that crash [...] I'm happy to do so

Thank you. It would be great if you could try the latest commit.

Though I'm doing this recovery for a friend

Please be aware that if there are compressed files, those will not be recovered. I mean NTFS-compressed files, not archives. RecuperaBit does print a warning in this case though.

[norristh]

Success all around! The restore completed with all seven partitions I tried, both Root and LostFiles for each. And by restoring from all partitions with more than 20,000 files, between Root and LostFiles I may have restored all the files my friend was worried about.

Thank you again for working on and sharing this tool!

[Lazza]

I am glad it worked for you. 😊 Yesterday I had a very bad day and this news is cheering me up a bit.

Per the testing you performed, I can now tag a new release on GitHub with all these fixes. Thanks!