Export: some edited photos are not converted to jpeg, and stay HEIC instead

[jor63]

Hi,

The export --convert-to-jpeg function seems to sometimes silently keep HEIC file format, despite being renamed as .jpeg
(and thus falsely showing up as jpg file type in the finder: I had to use the unix "file" command to identify the problem).
While exporting an album with 257 photos, this happened on 39 edited pictures (out of 83), and 0 unedited pictures (out of 174).

I was running this command
osxphotos export ./ToPrint --exiftool --report "export.csv" --album "toprint-2009-?" --skip-original-if-edited --verbose --download-missing --use-photokit --convert-to-jpeg --touch-file --only-photos
with osxphotos 0.55.5 on a Photos v6 system library (Big Sur, 11.6.5), with optimised iCloud storage.

Please find below:

• a typical verbose result for a problematic picture
• the csv file of the export (which shows all problematic files had flag "converted_to_jpeg=0")
• detailed info on another HEIC picture generating the problem
• an archive of the 2 corresponding files from the library: original .heic, and edited ".jpeg"

My conclusion:
Despite its name, the edited file 1C7F58A5-CD65-4AFD-8F64-54745BD0EE63_1_201_a.jpeg is NOT a jpeg file, but rather an heic file. I suppose that osxphotos, fooled by the name, just copies it, instead of converting it to a real jpeg. I also suppose Apple is doing this to save disk space: the true jpeg file obtained by exporting the modified picture from Photos.app is 4 times bigger (4.4Mb vs. 1.1Mb)...
I didn't start looking into why certain edited photos DO have the announced jpeg format, nor why certain others were correctly converted by osxphotos (with flag "converted_to_jpeg=1", but I could try and do that if useful for you.

Before closing, let me state how bloody grateful I am that you created this project. I'm an old school geek (eg. at CERN while Tim Berners Lee was implementing the 1st http protocol with my pianist friend...), and I am completely despaired by apple's "It just works" philosophy, which more and more means "It only just works, and don't even try to make it work better".
Discovering and using your project gives me back hope!...

Best,

Jean

Exporting IMG_4471.HEIC (160A0B72-4CFA-47AA-A496-E741065AE70C.heic) (143/256)
Skipping original version of IMG_4471.HEIC
Exporting edited version of IMG_4471.HEIC (160A0B72-4CFA-47AA-A496-E741065AE70C.heic)
Writing metadata with exiftool for IMG_4471_edited.jpeg
Exported IMG_4471.HEIC to /Users/orloff/tmpnobackup/ToPrint/IMG_4471_edited.jpeg
Writing metadata with exiftool for IMG_4471_edited.mov
Exported IMG_4471.HEIC to /Users/orloff/tmpnobackup/ToPrint/IMG_4471_edited.mov
❌️ exiftool error for file /Users/orloff/tmpnobackup/ToPrint/IMG_4471_edited.jpeg: Error: Not a valid JPEG (looks more like a MOV) - /var/folders/n8/nbd7t7fd7sj1hcjf7cnxkpb00000gq/T/osxphotos_export_uh7ryquu/160A0B72-4CFA-47AA-A496-E741065AE70C_1_201_a_exiftool.jpeg
❌️ Error exporting photo (160A0B72-4CFA-47AA-A496-E741065AE70C: IMG_4471.HEIC) as /Users/orloff/tmpnobackup/ToPrint/IMG_4471_edited.jpeg: Error: Not a valid JPEG (looks more like a MOV) - /var/folders/n8/nbd7t7fd7sj1hcjf7cnxkpb00000gq/T/osxphotos_export_uh7ryquu/160A0B72-4CFA-47AA-A496-E741065AE70C_1_201_a_exiftool.jpeg
Exported new file /Users/orloff/tmpnobackup/ToPrint/IMG_4471_edited.jpeg
Exported new file /Users/orloff/tmpnobackup/ToPrint/IMG_4471_edited.mov
Touched date on file /Users/orloff/tmpnobackup/ToPrint/IMG_4471_edited.jpeg
Touched date on file /Users/orloff/tmpnobackup/ToPrint/IMG_4471_edited.mov

export.csv

~ osxphotos inspect
Using last opened Photos library: /Users/orloff/Pictures/iPhoto Library.photoslibrary

╭──────────────────────────────────────────────────────────────────── IMG_6477.HEIC ─────────────────────────────────────────────────────────────────────╮
│ Filename: IMG_6477.HEIC │
│ Type: photo live │
│ UUID: 1C7F58A5-CD65-4AFD-8F64-54745BD0EE63 │──╮╮
│ Date: 2021-06-12T20:25:00.405186+02:00 │ ││
│ Date added: 2021-06-12T20:25:02.825840+02:00 │ ││
│ Date modified: 2022-01-12T18:26:57.367532+02:00 │ ││
│ Dimensions: 3153 x 2365 Orientation: 1 │ ││
│ File size: 2.41 MB | 3.36 MB (Live video) │ ││
│ Title: - │ ││
│ Description: - │ ││
│ Edited: ✔ External edits: - │ ││
│ Keywords: famileo, ToPrint, Printed │ ││
│ Persons: - │ ││
│ Location: │ ││
│ Place: │ ││
│ Categories/Labels: Adult, Outdoor, People, Plant, Shrub │ ││
│ Flags: favorite, visible, cloud asset, in cloud │ ││
│ Albums: Famileo, Printed/toprint-2009-07 │ ││
│ Moment: Parc Montjoly │ ││
│ EXIF: Apple iPhone SE (2nd generation) 3.99mm ISO 20 0 ev ƒ1.8 1/287s │ ││
│ Score: 0.4658203125 │ ││
│ Path original: /Users/orloff/Pictures/iPhoto Library.photoslibrary/originals/1/1C7F58A5-CD65-4AFD-8F64-54745BD0EE63.heic │ ││
│ Path live video: /Users/orloff/Pictures/iPhoto Library.photoslibrary/originals/1/1C7F58A5-CD65-4AFD-8F64-54745BD0EE63_3.mov │ ││
│ Path edited: /Users/orloff/Pictures/iPhoto Library.photoslibrary/resources/renders/1/1C7F58A5-CD65-4AFD-8F64-54745BD0EE63_1_201_a.jpeg │ ││
│ Path edited live video: /Users/orloff/Pictures/iPhoto Library.photoslibrary/resources/renders/1/1C7F58A5-CD65-4AFD-8F64-54745BD0EE63_2_100_a.mov │ ││
│ Path preview: /Users/orloff/Pictures/iPhoto Library.photoslibrary/resources/derivatives/1/1C7F58A5-CD65-4AFD-8F64-54745BD0EE63_1_105_c.jpeg │ ││
Inspecting uuid=1C7F58A5-CD65-4AFD-8F64-54745BD0EE63
Press Ctrl+C to quit

InLibrary.zip

[RhetTbull]

Hi @jor63 I'm glad that you've found osxphotos useful. Thanks for the debug data -- it looks like a couple different things could be going on. First, as you've surmised, some of the files labeled .jpeg in Photos are actually .HEIC. Other users have reported similar issues, usually with edited images, (see #306, #382). I built a script that can be run with osxphotos to find all such photos (see here) and have a rough design for an osxphotos command to fix these in the database. I'm not sure what the root cause is though.

What I do know: Apple's developer documentation states that external editors should save the edited file in JPEG but some editors ignore this (I've seen TIFF, labeled as JPEG, for example); Apple sometimes preserves the edited file as HEIC when edited on your phone or the Mac (but it labels the file as HEIC when it does this).

osxphotos examines the UTI (universal type identifier) in the database which is something like public.jpeg or public.heic to determine whether the file should be converted (that is, osxphotos doesn't look at the file's extension).

Your first example is troubling as it looks like osxphotos switched the video and photo component of a. "live photo":

exiftool error for file /Users/orloff/tmpnobackup/ToPrint/IMG_4471_edited.jpeg: Error: Not a valid JPEG (looks more like a MOV) - /var/folders/n8/nbd7t7fd7sj1hcjf7cnxkpb00000gq/T/osxphotos_export_uh7ryquu/160A0B72-4CFA-47AA-A496-E741065AE70C_1_201_a_exiftool.jpeg

osxphotos determines if a particular asset is a photo or a movie by examining the type listed in the database so I'm concerned that osxphotos either got this wrong (a bug) or the database is corrupt.

with osxphotos 0.55.5 on a Photos v6 system library (Big Sur, 11.6.5), with optimised iCloud storage

Another possibility is the code that downloads the missing files could be causing this (though it's been well tested on large libraries). I've not done extended testing on Big Sur though so there could be a problem specific to this OS.

In the attached zip is a python file that, when run with osxphotos, will output additional debug data. (osxphotos has a built in hidden debug-dump command but your situation requires more info than this will provide).

896.zip

Once you download and unzip the file, it will contain issue_896.py. Change into the directory that contains this file then run the following command:

osxphotos run issue_896.py --uuid 160A0B72-4CFA-47AA-A496-E741065AE70C --uuid 1C7F58A5-CD65-4AFD-8F64-54745BD0EE63 --uuid 8378DF0C-C6DF-487F-85D6-E10FC5AE90F4 --uuid CB4E2AB5-127C-449F-89EC-718A650FB879 > 896_debug.txt

Then send me (rturnbull+git@gmail.com) or post here the 896_debug.txt

Note: the script will strip location information out of the debug data but it may contain keywords, titles, descriptions you deem sensitive. If so, you can edit the .txt file to remove this before sending.

[jor63]

Hi,
and thanx 4 the superfast reply! I ran the debug code: there were errors (not sure why), but I guess the output still is OK.
Both included below.

896_debug.txt

896_command_and_errors.txt

[RhetTbull]

Hi @jor63 Sorry about the error. I think it's fixed -- please re-run with this version.

896.zip

[RhetTbull]

The additional data from the second run will help but there are some clues here for the one photo that did end up in the output (before the error):

'UTI': 'public.jpeg',
'UTI_original': 'public.heic',

path: /Users/orloff/Pictures/iPhoto Library.photoslibrary/originals/1/160A0B72-4CFA-47AA-A496-E741065AE70C.heic
file type: ISO Media, HEIF Image HEVC Main or Main Still Picture Profile
path_edited: /Users/orloff/Pictures/iPhoto Library.photoslibrary/resources/renders/1/160A0B72-4CFA-47AA-A496- 
E741065AE70C_1_201_a.jpeg
file type: ISO Media, HEIF Image HEVC Main or Main Still Picture Profile

This data shows Photos thinks that:

1. The original photo is a HEIC (it is)
2. The photo is edited and the edited version is a JPEG (it is actually a HEIC though)

Thus, it appears the Photos library is corrupted. I'll take a look at the additional data you sent. If it's consistent with this, it might be best to start with repairing the Photos library.

Do you know how these photos were edited? (This first example is IMG_4471.HEIC? Were the edits done in a 3rd party app? On the iPhone in Photos? On the Mac in Photos?

[jor63]

Hi again,

and thanks for the modified code, which runs without error!
From what I understand of the output below, it confirms that Photos thinks all problematic edited files are jpeg, while they are in fact heic.
osxphotos thus copies them instead of converting them, and exiftool complains they look more like videos than jpeg...

I hesitate launching repair before backing up, and am confused about what to do for backing up as some files are on icloud (because of the "Optimise Mac storage" option). Is it enough to just copy the (partially complete) .photoslibrary package on an external disk? (I will try the safer solution proposed by kimcha you mentioned on reddit)

Concerning the editing of the photos, I never used anything else than Photos-v6 for editing on the mac, but also made edits on iphone and ipad, running ios-photos-v7, and recently v8. Maybe I also opened the library with PowerPhotos, but I'm not sure.

Thanks for your help!

Jean

896-1_debug.txt

[RhetTbull]

I hesitate launching repair before backing up,

As you are using iCloud, the "original" copy is always in the cloud. If the repair corrupted your database, you could simply delete the library and have iCloud re-download.

However, if you want to play it safe, I recommend creating a second user account on the Mac, signing in to iCloud in the new account and having it re-create your library. Then test osxphotos again on the new library.

[RhetTbull]

From what I understand of the output below, it confirms that Photos thinks all problematic edited files are jpeg, while they are in fact heic.

This appears to be the case. Photos database reports that the UTI of the edited images is public.jpeg and is even storing the images with .jpeg extension on disk but they are in fact HEIC images.

[jor63]

Dear @RhetTbull,
Sorry for being so long to get back at you ( many issues with backup on a NAS via afp... sorted out, now it seems!)

I did try a repair on the problematic library, and it didn't change anything:

~ file Pictures/iPhoto\ Library.photoslibrary/resources/renders/1/160A0B72-4CFA-47AA-A496-E741065AE70C_1_201_a.jpeg 
Pictures/iPhoto Library.photoslibrary/resources/renders/1/160A0B72-4CFA-47AA-A496-E741065AE70C_1_201_a.jpeg: ISO Media, HEIF Image HEVC Main or Main Still Picture Profile

So it seems I still got the same edited heic files that are wrongly taken to be jpeg (for more detail, see the output of your script below).

I recreated an new from iCloud, using another account on my machine with the same iCloud user and full local copies of the iCloud pics. That newly recreated library appears to be sane, or at least, the problematic files got corrected and are true jpegs.

What would you recommend? Copying that complete new library and trash the other one?

Thanks a thousand for your help!

Jean

896_debug_2.txt

[RhetTbull]

Hi Jean. I don't know how well copying the library from the other account will work but you could try that. I would trash the existing library: close Photos, delete the library, then copy the library from the secondary account over. Then I'd fix the permissions (Cmd + I in Finder, Sharing & Permissions, unlock, make yourself the owner) then open Photos while holding down Option and select the new library. If that doesn't work (and permissions could cause some trouble) then I would delete the library and just download a new version from iCloud.