DNG being exported as TIFF

[narensankar0529]

I have some DNG RAW files in my library and when using --convert-to-jpeg --exiftool, I found errors in the logs - photo_181016_4116163723_edited.jpeg: Error: Not a valid JPEG (looks more like a TIFF)

And looking at the file it looks like for some reason osxphotos outputted TIFFs with jpeg extensions. Files are all around 70MB is size. Any ideas why?

[RhetTbull]

This is a bug. I think it's actually the DNG file (which exiftool reports as TIFF if it's labeled JPG) that's being output. I suspect there was an error in the file conversion code and it left the DNG file marked as JPG. I'll have to look into the error handling of the conversion code.

[narensankar0529]

The DNG is only 9.8MB. So there is actual conversion from DNG to TIFF happening here. And preview also opens the file as TIFF.

[RhetTbull]

Is the DNG file one you're willing to share with me? If so, email to rturnbull+git@gmail.com and I'll take a look in a test library. I just tried --convert-to-jpeg with my library and it worked as expected on DNG files.

[narensankar0529]

Emailed.

Here is my command line - /usr/local/bin/osxphotos export --update --cleanup --not-hidden --not-shared --not-screenshot
--skip-original-if-edited --skip-raw --skip-live --skip-bursts
--touch-file
--exiftool-path "/usr/local/bin/exiftool"
--convert-to-jpeg
--exiftool
--exiftool-merge-keywords
--exiftool-merge-persons \

[RhetTbull]

Got it -- will take a look later this weekend.

[RhetTbull]

I imported your test image into Photos then exported with your command and the export succeeds as expected. The DNG file is converted to a 12M jpeg file and the exiftool attributes are correctly written.

I did see in the code that I don't check the result of the conversion method for success so I'll add code to capture any errors during conversion and report those which might help debug the issue you're having.

[RhetTbull]

v0.39.6 adds error handling for --convert-to-jpeg. If you get any error output during export, please post it here and I'll investigate further. It will look something like this:

Error exporting photo DSC03584.dng (D05A5FE3-15FB-49A1-A15D-AB3DA6F8B068.dng) as DSC03584.dng: Could not create CIImage for /Users/rhet/Pictures/Test-10.15.7.photoslibrary/originals/D/D05A5FE3-15FB-49A1-A15D-AB3DA6F8B068.dng

[narensankar0529]

So I have figured out the issue. The issue is that for some reason for a number of images that were taken as RAW DNG, there is an additional FullSizeRender.jpg available. I think this might have been generated by some software, but I cannot find enough data in the exif to figure that out. This file is stored as a raw TIFF. Not sure how Photos allows that, but it is clear it is there. So osxphotos is exporting this FullSizeRender.jpg as a .jpeg file - but without actually doing the conversion. I guess because you assume that if the extension is .jpg then it is already in JPEG format. But it seems that this is a case where there is a TIFF stored with jpg extension.
So I think you would need to add verification code to determine if the file is actually JPEG and if not convert it before you output it.

Another minor issue - for some reason if there are files that are already .jpg - then the output is with extension .jpg. For DNG and HEIC the output is with extension .jpeg. Is this intentional?

[RhetTbull]

This is strange as osxphotos doesn't export the FullSizeRender.jpg files. Can you post the path to the FullSizeRender.jpg within your library?

Have the DNG files in question been edited? If so, as you are exporting --skip-original-if-edited then only the edit would be exported but only if there's a render available. (The render is in a path similar to Photos Library.photoslibrary/resources/renders/7/7ABE2A5A-DBFA-4750-9AC9-C3C3CF8AE57D_1_201_a.jpeg. If the edits were done in Photos and on Big Sur, a rendered image is not always available and if not, the edits won't get exported (known bug #225).

If you can get the UUID of one of the DNG files in question, I can walk you through some additional debugging commands in osxphotos that would provide helpful data. There's an applescript in the osxphotos repo that you can run to get the UUID. Select the photo in Photos then run the applescript and a pop-up dialog will show some info about the photo. The UUID is a 36-character string that looks similar to this: 7ABE2A5A-DBFA-4750-9AC9-C3C3CF8AE57D

On the subject on JPG vs jpeg, yes, this is known and it's annoying but it's due to macOS behavior--other users have found this annoying as well. Here's how osxphotos handles those:

• If exporting original photo, osxphotos uses whatever extension the photo was imported with
• If photo has been edited, Photos changes extension to .jpeg (Catalina and below) or .heic (Big Sur) and osxphotos uses that
• If using --convert-to-jpeg, osxphotos uses .jpeg because that's what the OS conversion function uses
• There are a few cases (e.g. --use-photokit) where the extension is not known ahead of time and osxphotos asks the OS for the correct extension based on the Photo's UTI. macOS always returns 'jpeg' for public.jpeg UTI.

I have thought about adding a --jpg-ext option that would allow you to specify what extension to use for jpg files. e.g. --jpg-ext JPG or --jpg-ext jpeg. I'll open a new issue for this.

[narensankar0529]

The DNG has been edited. There is a render available and the render is actually a TIFF file with jpeg extension. Not sure why this was written this way and by what app. Since I made this edit some time ago I don't have the memory of what app did this.

The AppleScript returns - D66A6077-B847-448A-8939-DEA12F070019/L0/001, IMG_1379.HEIC, missing value, missing value, missing value

In this case this is a HEIC original with a TIFF in jpeg render. And it is output as TIFF in jpeg and obviously exiftool dies on this file as well. The edits were done on iPhone and synced over iCloud. On the iPhone this shows up as FullSizeRender.jpg whereas on the Mac this is named as D66A6077-B847-448A-8939-DEA12F070019_1_201_a.jpeg. In both cases this is actually a TIFF file but somehow saved with jpeg extension.

So not sure what is the right way to handle this by osxphotos. Are you actually doing the conversion to jpeg or are you asking photos to do the conversion? When I do the export from Photos it does output this as jpeg correctly.

[RhetTbull]

OK, this is helpful. Here's what I think is going on: however this was edited on the iPhone is creating a TIFF but named as a .jpeg as you've noted. Because you are exporting with --skip-original-if-edited and --convert-to-jpeg, osxphotos sees that the image has been edited, sees the jpeg file and so exports that but does not convert it because it already appears to be a jpeg. However, when running exiftool, exiftool does generate an error, because it's not a jpeg, and osxphotos reports this.

On conversion -- osxphotos does the conversion using macOS native CoreImage system calls so any image that Photos can convert, osxphotos should be able to. But in this example, I don't think that's ever getting called as the file appears to be a jpeg to osxphotos.

Try the following:
osxphotos debug-dump --dump photos --uuid D66A6077-B847-448A-8939-DEA12F070019 > debug.txt and email me the text file (rturnbull+git@gmail.com). Note: this may dump private information about the photo such as persons, keywords, locations so you might want to look at the output first and remove anything you don't want to share (of course, I'll treat any log data you email as confidential).

[RhetTbull]

Got the debug dump -- thanks. Interesting observations from the data:

1. Photo is originally a .heic :'UTI_original': 'public.heic',
2. Photo has been edited by something called luminarflex 'adjustmentFormatID': 'com.macphun.luminarflex.photoextension',
3. Photos thinks the edited photos is actually a jpeg: 'UTI': 'public.jpeg',

However, #3 is incorrect as you've noted the edited files are really TIFF files. But because of #3, osxphotos also thinks this is a jpeg and thus doesn't attempt to convert it and exiftool thus fails. I don't think this is a problem osxphotos can solve as looking at every single file to determine what type it is (to verify it's really a jpeg) would be very slow and difficult to get 100% right.

I've noticed that the Photos assigns the UTI for the image based on file extension when importing, even if this is incorrect. Photos however displays the image with no problem. The code in Photos that displays must look at image header to determine type and this is independent of the database code that tracks metadata. So it's expected that Photos would display your image with no problem even though it's really a TIFF.

I think this is a bug in either "luminar flex" or Photos. Much more likely it's in luminar. My guess is the issue is that luminar is writing TIFF data in their Photos plugin but the Photos editing extension protocol expects JPEG. From Apple's Photo Extension documentation:

Read this property to find a URL for writing edited asset content. Then, if editing a photo asset, write the altered photo image to a file in JPEG format at this URL. If editing a video asset, export the video to a QuickTime (.mov) file at this URL.

If luminar is an important part of your workflow, I suggest you contact the developer to look into this.

You can verify if this is the issue for the other photos causing the same error by doing the same debug-dump command for each UUID and looking for 'adjustmentFormatID': 'com.macphun.luminarflex.photoextension', in the debug output.

[narensankar0529]

Thanks for digging through this. I am going to manually convert all the images to jpeg and rewrite and will file a bug with Luminar.