Lots of metadata errors when trying to export with --exiftool --keyword-template "{label}"

[gabeklavans]

Describe the bug
The app reports lots of errors on my photo library when trying to export all metadata using exiftool. Most of them are complaints about the file having the wrong filetype, but there are others too (see "screenshots").

To Reproduce

1. Delete entire system photo library
2. Create a new system library from scratch using the Photos app
3. Connect the new system library with iCloud
4. mkdir osxphotos-gabe
5. Run the command: osxphotos export --cleanup --update --download-missing --use-photokit --preview-if-missing --exiftool --keyword-template "{label}" --add-missing-to-album 'osxphotos missing' --report 'export_{today.date}.csv' osxphotos-gabe

Expected behavior
All photos are downloaded from the cloud with their proper metadata

Screenshots
Here's a selection of the kinds of errors I get:

❌️  Error exporting photo (16C6E421-064D-4B7A-B372-07EB1CC49721: IMG_5953.PNG) as /Users/gabe/Pictures/osxphotos-gabe/IMG_5953.PNG: Error: Not a valid PNG (looks more like a JPEG) -
/var/folders/c6/gjhzls_n4gb8twdhljq6bhs00000gn/T/osxphotos_export_0azdpv19/IMG_5953_exiftool.png

❌️  Error exporting photo (FB13365A-AB09-4B10-A14A-68873207523D: IMG_8411.HEIC) as /Users/gabe/Pictures/osxphotos-gabe/IMG_8411.HEIC: Error: Not a valid HEIC (looks more like a JPEG) -
/var/folders/c6/gjhzls_n4gb8twdhljq6bhs00000gn/T/osxphotos_export_3cz6km5k/IMG_8411_exiftool.heic

❌️  Error exporting photo (FACB38BC-0246-462F-B489-944CC44DD255: IMG_3907.MOV) as /Users/gabe/Pictures/osxphotos-gabe/IMG_3907.MOV: Error: '0xa70503cb' atom is too large for rewriting
(3969 MB) - /var/folders/c6/gjhzls_n4gb8twdhljq6bhs00000gn/T/osxphotos_export_n1nlab32/IMG_3907_exiftool.mov

❌️  Error exporting photo (27B5E16B-8FCD-4DF8-8C1A-82555A9A66FE: IMG_0194_Original.HEIC) as /Users/gabe/Pictures/osxphotos-gabe/IMG_0194_Original.mov: Error: Truncated '0x7f000000'
atom - /var/folders/c6/gjhzls_n4gb8twdhljq6bhs00000gn/T/osxphotos_export_lp0kbh2y/IMG_0194_Original_exiftool (1).mov

and potentially others.

Desktop (please complete the following information):

osxphotos, version 0.67.0
Python 3.12.0 (main, Oct  2 2023, 12:03:24) [Clang 15.0.0 (clang-1500.0.40.1)]
macOS 14.2.1, arm64

Additional context
Running the script mentioned here does in fact report lots of files with the "wrong" extension; way too many for me to hope to deal with by hand. Likely over 100. I'd have to script something up to deal with it. I was hoping that fully re-downloading my library from iCloud would fix all these, but it does not. I guess Apple just has a bunch of these with the wrong extension, potentially from when I imported photos from other services.

I'm not sure if it's worth fixing all these, since a normal export seems to function fine. I may just be missing my juicy complete metadata...

Anyway, I thought I'd report this in case anyone has some advice or if this is a sign of some backend change breaking things. It could very well be my fault, somehow, though.

[RhetTbull]

These are probably actual errors where the photo in the database has the wrong UTI (uniform type identifier) associated with it. I've seen this happen before, particularly when external apps are used to edit photos. If the photo type is wrong, exiftool will not corrupt the image but also likely won't export the metadata to the image. But just to make sure there's not a bug in osxphotos, please run the following commands and send me the output to osxphotos.py@gmail.com

osxphotos debug-dump --dump photos --uuid 16C6E421-064D-4B7A-B372-07EB1CC49721 --uuid FB13365A-AB09-4B10-A14A-68873207523D > debug.txt

[gabeklavans]

For some further output, here's the result of that command listed in my Reproduce section with a fresh library:

Exporting 8636 photos ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Processed: 8636 photos, exported: 4301, updated: 2, skipped: 0, updated EXIF data: 4301, missing: 8714, error: 114

So looks like 114 files had errors of some variety. Not sure why 8000 reported as missing, maybe some downloads failed.

[oPromessa]

@gabeklavans for one of your errors "atom is too large for rewriting
(3969 MB)" it looks the file is too big 4GB so you may have to use exiftool option. -api LargeFileSupport=1. Y providing it to osxphotos via --exiftool-option. Do osxphotos export -h for more info.

See: exiftool option largeFileSupport

As @RhetTbull points out the UTI info may be incorrect. But please note not all image types support EXIF and XMP data in the same way. PNG for instance does not. I normally convert then into JPG with sips command and try to move as much metadata over as exiftool can.

I also do it for TIF files.

Not the smartest code I've written ;) but I do that with the following bash functions:

# Command which convert .PNG files into .JPG and copies EXIF fields ...

exifpng2jpg () 
{ 
    for a in "$@";
    do
        fname=`basename "$a" .png`;
        xtension=png;
        echo 1st - ${fname}.${xtension};
        if [ \( -f "${fname}".png -o -f "${fname}".PNG \) -a ! -f "${fname}".jpg ]; then
            echo ok - ${fname}.${xtension};
            sips -s format jpeg -s formatOptions 90 """${fname}".${xtension}"" --out """${fname}".jpg"";
            exiftool -overwrite_original_in_place -TagsFromFile "${fname}".${xtension} "-all:all>all:all" "${fname}".jpg;
        else
            fname=`basename "$a" .PNG`;
            xtension=PNG;
            echo 2nd - ${fname}.${xtension};
            if [ \( -f "${fname}".png -o -f "${fname}".PNG \) -a ! -f "${fname}".jpg ]; then
                echo ok - ${fname}.${xtension};
                sips -s format jpeg -s formatOptions 90 """${fname}".${xtension}"" --out """${fname}".jpg"";
                exiftool -overwrite_original_in_place -TagsFromFile "${fname}".${xtension} "-all:all>all:all" "${fname}".jpg;
            fi;
        fi;
    done
}
export -f exifpng2jpg

# Command which convert .TIF files into .JPG and copies EXIF fields ... Uses ModifyDate as the source for DateTimeOriginal, File    ModifyDate, CreateDate fields

exiftif2jpg () 
{ 
    for a in "$@";
    do
        fname=`basename "$a" .tif`;
        dname=`dirname "$a" `;
        if [ -f "${dname}/${fname}".tif -a ! -f "${dname}/${fname}".jpg ]; then
            echo ok - ${fname};
            sips -s format jpeg -s formatOptions 100 """${dname}/${fname}".tif"" --out """${dname}/${fname}".jpg"";
            exiftool -overwrite_original_in_place -TagsFromFile "${dname}/${fname}".tif "-all:all>all:all" "${dname}/${fname}".jpg;
            echo Performing... ""-DateTimeOriginal\<FileModifyDate"" ""-CreateDate\<FileModifyDate"" on "${dname}/${fname}".jpg;
            exiftool -overwrite_original_in_place """-DateTimeOriginal<ModifyDate""" """-CreateDate<ModifyDate""" """-FileModifyDate<ModifyDate""" -P -v "${dname}/${fname}".jpg;
        fi;
    done
}
export -f exiftif2jpg

[gabeklavans]

@gabeklavans for one of your errors "atom is too large for rewriting
(3969 MB)" it looks the file is too big 4GB so you may have to use exiftool option. -api LargeFileSupport=1. Y providing it to osxphotos via --exiftool-option. Do osxphotos export -h for more info.

Thanks for the tip on this. However, it doesn't seem to work, in my case. When I add the following to my command: --exiftool-option "-api LargeFileSupport=1" I still get errors like:

❌️  Error exporting photo (24C469FA-6FD6-42F2-91EC-B3E5B32A1AF7: IMG_4593(1).MOV) as /Users/gabe/Pictures/osxphotos-gabe/IMG_4593(1).MOV: Error: 'etTy' atom is too large for rewriting
(742 MB) - /var/folders/c6/gjhzls_n4gb8twdhljq6bhs00000gn/T/osxphotos_export_6an33s1q/24C469FA-6FD6-42F2-91EC-B3E5B32A1AF7_exiftool.mov

I will probably still keep exif on as it works for most of my exports. I just saw a bunch of errors and worried that it was messing up my export. It may be messing with the ability to download missing files, as when I ran without the exiftool option it seemed to successfully download way more photos, but maybe now that they're actually downloaded I can continue to use the exiftool option with minimal worry.

Man, I really wish Apple would just provide a proper export tool...

[RhetTbull]

Another option is to export sidecars using --sidecar XMP as that preserves the metadata but doesn't run exiftool. You can always use exiftool later to apply the sidecars to the images or use another digital asset manager that reads the sidecars.

[gabeklavans]

Another option is to export sidecars using --sidecar XMP as that preserves the metadata but doesn't run exiftool. You can always use exiftool later to apply the sidecars to the images or use another digital asset manager that reads the sidecars.

Yeah, I considered that. A pretty neat option, but I was striving for maximum convenience, at maybe the cost of some lost metadata. As long as I can see all the photos, I think I'm happy.

Anyway, it sounds like all my questions have been answered, or have been given paths forward, and that it's not a problem with the application. Thanks for the help!

[RhetTbull]

From the data you sent, the PNG file is really a JPEG. It looks like it may have been adjusted by Google Photos. I've seen this in the past with images touched by an external app having the wrong UTI.

    `originalFilename': 'IMG_5953.PNG',
    'filename': '16C6E421-064D-4B7A-B372-07EB1CC49721.png',
    'adjustmentFormatID': 'com.google.photos.editing.filtering.nondestructive',
    'UTI': 'public.jpeg',

For the HEIC image, Photos thinks this is a HEIC:

    'originalFilename': 'IMG_8411.HEIC',
    'filename': 'FB13365A-AB09-4B10-A14A-68873207523D.heic',
    'UTI': 'public.heic',
    'hasAdjustments': 0,
    'adjustmentFormatID': None,

And I can see no indication an external app touched this image so that's a mystery.

I have seen other cases where the UTI is wrong in Photos (always associated with an external app). There's an issue (#382) to add an option to fix these upon export. One of these days I'll get around to implementing that; it's not a trivial change to the export code because I'll need to maintain the mapping between "reported" and "actual" extension for every photo exported and will need to figure out a way to verify the actual UTI (ideally without calling exiftool).