Lay the groundwork to be able to re-import everything later on to rebuild the library

[SebAgain]

Hello,

First of all, I would like to thank you for this library, which perfectly meets a crucial need for me.

My goal is to export enough information from Photos to be able to recreate the Photos library if necessary:

• Retrieve albums;
• Retrieve keywords associated with photos and videos;
  • exiftool = true
  • favorite_rating = true
  • person_keyword = true
  • album_keyword = true
  • finder_tag_keywords = true
  • keyword_template = ["{keyword}", "{label}"]
  • exiftool_merge_keywords = true
  • exiftool_merge_persons = true
• Retrieve geolocation data;
  • exiftool = true
• Retrieve Live Photos;
  • skip_live = false
• Retrieve the original files as well as any edits made;
  • skip_original_if_edited = false
  • export_aae = true
• Retrieve the intelligent keywords generated by Photos.
  • keyword_template = ["{label}"]

I've spent the past week browsing the osxphotos GitHub and forum to build my .toml configuration file.
Here is the .toml config export file I have developed:

[export]
cleanup = true
db = "/Users/XXX/Pictures/Photos Library.photoslibrary"
directory = "{folder_album}"
folder = ["2024 - Test",]
download_missing = true
#filename = "{created.strftime,%Y-%m-%d}" not working
export_aae = true
exiftool = true
ignore_date_modified = true
favorite_rating = true
person_keyword = true
album_keyword = true
finder_tag_keywords = true
keyword_template = ["{keyword}", "{label}"]
exiftool_merge_keywords = true
exiftool_merge_persons = true
sidecar = [ "xmp", "exiftool",]
sidecar_drop_ext = true
skip_live = false
skip_original_if_edited = false
update = true
verbose = 1
#ramdb = true
#report = "/Users/XXX/Pictures/osxphotos/export_{today}.csv"
#append = true

Here are my questions :

1. Skip edited videos :
   In my export, I have an edited version of the videos, but I don't want to include the edited version of the videos—only the photos. How can I do this?
   There is no "skip_edited_videos = true" options...

2. Naming of the files :

• I would like to rename the files as follows:
  • IMG_20180412_230123.heic/jpg/png/etc (original extension). for photos
  • IMG_20180412_230123_edited.jpg for edited photos
  • VID_20180412_230123.mp4/mov/etc (original extension). for videos

filename = "{created.strftime,%Y_%m_%d}" is not working, so I couldn't try to put this in place.
How can I differentiate between images and videos?

Do I need to create one configuration file for the originals and another one for the edited files? I don't see how to do the equivalent of an "if" statement.

3. To be able to retrieve the Live Photos videos :

• I saw that to recreate Live Photos, the file names need to be as they are in Photos (something like this - I can't find the associated discussion):

• IMG_5062.heic (original file);

• IMG_5062.mov (original file);

• IMG_E5062.heic (edited photo).
  If we proceed with the naming in point 2, does that mean I won't be able to reconstruct the Live Photos ?

Would the following proposal work?

• IMG_20180412_230123.heic (original file)
• IMG_20180412_230123.mov (original file)
• IMG_E20180412_230123.mov (edited photo)

4. Keywords/metadata for videos :
   It seems that I can't write the keyword into videos metadata. What is wrong with my config file ?

5. How can I add an "edited" keyword to edited images?
   I haven't been able to apply it only to edited files using the keyword template.

6. Export on NAS/External devices :
   For now, I'm running tests on my local computer. When I do it on a NAS or an external drive, is it a good practice to set ramdb = true?

7. How to know if you are using the "copy on write" feature of APFS?

8. When importing, will we be able to find the location of the faces along with the associated names?

I'm open to all advice for completing this task. Thank you very much for your help. I'll be happy to share my configuration file to assist newcomers like myself.

👍

[RhetTbull]

Glad you found osxphotos and that it's useful for you!

Your TOML looks ok with exception to comments below. However, if the goal is to be able to recreate a library, then osxphotos import --exportdb can do this by reading all the metadata from the .osxphotos_export.db created by osxphotos export to re-import the photos and reconstruct all the metadata. See osxphotos help import for more information.

In my export, I have an edited version of the videos, but I don't want to include the edited version of the videos--only the photos. How can I do this?

There is no option to skip only edited videos. You can skip all edited assets with --skip-edited but you can't skip edited videos while exporting edited photos.

You could run a script following export to clean up any video file with the _edited suffix but if you re-exported with --update, these would be re-exported.

• I would like to rename the files as follows:
  • IMG_20180412_230123.heic/jpg/png/etc (original extension). for photos
• IMG_20180412_230123_edited.jpg for edited photos
  • VID_20180412_230123.mp4/mov/etc (original extension). for videos

filename = "{created.strftime,%Y_%m_%d}" is not working, so I couldn't try to put this in place. How can I differentiate between images and videos?

The "{created.strftime,%Y_%m_%d}" template is valid and I tested it works. I think you meant to say it doesn't work to produce the output format you wanted which of course it doesn't do because it doesn't match the format you specified. It would produce files named "2024_08_19.jpeg" and so on.

The following would work for your format:

filename = "{photo_or_video,photo=IMG;video=VID}_{created.strftime,%Y%m%d_%H%M%S}"

• If we proceed with the naming in point 2, does that mean I won't be able to reconstruct the Live Photos ?

If you use osxphotos import then it will properly reconstruct the names when it encounters an "_edited.jpeg" type name. Of course, if you use osxphotos import many of the steps you are taking aren't needed as it can read from the export database to reconstruct everything.

If you want to proceed with the "correct" naming scheme, this would work:

filename = "{photo_or_video,photo=IMG;video=VID}_{edited_version?E,}{created.strftime,%Y%m%d_%H%M%S}"

This inserts the E before the date if photo is edited and I believe this will work for reconstructing the edited images (assuming the .AAE file is present). In this case you should also use --edited-suffix = "" to avoid adding _edited to the images.

Would the following proposal work?

Yes, and that's what the template above does.

Keywords/metadata for videos :

This is a limitation of certain video files and exiftool. For normal Quicktime .mov files, the keywords are written to XMP:TagsList. QuickTime has different metadata fields than photos do.

How can I add an "edited" keyword to edited images?

-keyword-template "{edited?edited,}"

This tests to see if image is edited and if so uses value edited otherwise uses the null value (no keyword). However, this won't do exactly what I think you're asking...if a photo has any edits, then the edited keyword would be written to both the original image and the edited image. There is currently no way to write separate metadata to just the edited version (though this is a good idea -- please open a new issue to suggest this).

For now, I'm running tests on my local computer. When I do it on a NAS or an external drive, is it a good practice to set ramdb = true?

Yes! You must do this to avoid issues with the sqlite export database used by OSXPhotos. There is an open issue to automatically check for this (#1419)

How to know if you are using the "copy on write" feature of APFS?

If you do not use --alt-copy and your source and target are both on the same APFS volume then it will automatically be used.

When importing, will we be able to find the location of the faces along with the associated names?

No. This data is written to the image when using --exiftool (and sidecar if using --sidecar) but Photos will not read these nor is it possible to programmatically set this data. This is a deliberate limitation by Apple and OSXPhotos is not able to work around this. You can use --person-keyword or a custom keyword template to preserve this info as a keyword but that's the best OSXPhotos can do at the moment.

[SebAgain]

Hi @RhetTbull,

Thank you for taking the time to respond and for your promptness.

Your TOML looks ok with exception to comments below. However, if the goal is to be able to recreate a library, then osxphotos import --exportdb can do this by reading all the metadata from the .osxphotos_export.db created by osxphotos export to re-import the photos and reconstruct all the metadata. See osxphotos help import for more information.

I didn't realize that the ".osxphotos_export.db" file would be useful for reimporting the library. So, the sidecar files "XMP" and "exiftool" are not necessary for this purpose.

They are useful if we want to use other software, if I understand correctly. Am I correct ?
Does it means that if we have only the edited images/videos + the associated AAE file, we can retrieve the original file + edited file in Photos after re-importing the database ?

There is no option to skip only edited videos. You can skip all edited assets with --skip-edited but you can't skip edited videos while exporting edited photos.
You could run a script following export to clean up any video file with the _edited suffix but if you re-exported with --update, these would be re-exported.

I understand the limitation.

I would like to rename the files as follows:

• IMG_20180412_230123.heic/jpg/png/etc (original extension). for photos
• IMG_20180412_230123_edited.jpg for edited photos
• VID_20180412_230123.mp4/mov/etc (original extension). for videos

filename = "{created.strftime,%Y_%m_%d}" is not working, so I couldn't try to put this in place. How can I differentiate between >>images and videos?

The "{created.strftime,%Y_%m_%d}" template is valid and I tested it works. I think you meant to say it doesn't work to produce the output format you wanted which of course it doesn't do because it doesn't match the format you specified. It would produce files named "2024_08_19.jpeg" and so on.

The following would work for your format:

filename = "{photo_or_video,photo=IMG;video=VID}{created.strftime,%Y%m%d%H%M%S}"

Here is the command in my .toml file filename = "{photo_or_video,photo=IMG;video=VID}_{created.strftime,%Y%m%d_%H%M%S}" and I still have the same error : Error parsing osxphotos_all.toml config file: Unknown option: filename = {photo_or_video,photo=IMG;video=VID}{created.strftime,%Y%m%d%H%M%S}

I just thought of something. Does this mean that the videos from Live Photos will have the "VID" prefix? In that case, my renaming isn't ideal. To me, a Live Photo video is still considered an image.

• If we proceed with the naming in point 2, does that mean I won't be able to reconstruct the Live Photos ?

If you use osxphotos import then it will properly reconstruct the names when it encounters an "_edited.jpeg" type name. Of course, if you use osxphotos import many of the steps you are taking aren't needed as it can read from the export database to reconstruct everything.

If you want to proceed with the "correct" naming scheme, this would work:

filename = "{photo_or_video,photo=IMG;video=VID}_{edited_version?E,}{created.strftime,%Y%m%d_%H%M%S}"

This inserts the E before the date if photo is edited and I believe this will work for reconstructing the edited images (assuming the .AAE file is present). In this case you should also use --edited-suffix = "" to avoid adding _edited to the images.

I didn't try as I have issue with the filename command. However, if I understand correctly, it's not important to follow the naming of Photos for Live Photos if we want to rebuild them.

I noticed that for a Live Photo, there is the original image and Live video, as well as the edited image and the edited Live video. But I imagine you can't change this behavior (except if we skip_edited=truebut see my question at the beginning).

Keywords/metadata for videos :

This is a limitation of certain video files and exiftool. For normal Quicktime .mov files, the keywords are written to XMP:TagsList. QuickTime has different metadata fields than photos do.

I fully understand.

How can I add an "edited" keyword to edited images?

-keyword-template "{edited?edited,}"

This tests to see if image is edited and if so uses value edited otherwise uses the null value (no keyword). However, this won't do exactly what I think you're asking...if a photo has any edits, then the edited keyword would be written to both the original image and the edited image. There is currently no way to write separate metadata to just the edited version (though this is a good idea -- please open a new issue to suggest this).

Yes it doesn't do what I expected. I'll open a new issue to suggest this.

I've continued to explore the possibilities, and I've noticed some interesting things that raise a few questions and remarks :

1. Strange behavior :
   I noticed a cloud icon on one of my videos, indicating that the video was not available. I'm not using iCloud to store my library—all the photos and videos are on my Mac.

However, osxphotos was able to export this video.
How do you explain this behavior? (It's good news, by the way).

2. Update even though there are no changes in Photos.
   I noticed that in an album where I had 2 photos and a video, if I performed a new import without making any changes in Photos, the .AAE files were updated.

Is this normal?

3. Edited photos quality
   Does this apply only to edited photos, or does it include original photos as well?
   In Photos, you can either export the original (without conversion) or export the photo with edits, choosing the quality.

How can I export the original with its original extension and export the edited version by converting it to JPG and choosing the quality?

4. Exported video quality
   I noticed that videos that have been edited are in .mov format, whereas the original was in .mp4.
   I don't think there's a command to choose the export quality of the video, like the --jpeg-quality command for images.
   What are the default settings for this export?

Once I have a better grasp of this configuration, I'll consider developing my script using the API. I think it will be more adaptable to my needs.

Thank you for your support,

[RhetTbull]

I didn't realize that the ".osxphotos_export.db" file would be useful for reimporting the library. So, the sidecar files "XMP" and "exiftool" are not necessary for this purpose.

I'm a "belt and suspenders" kind of guy so I export the XMP sidecars as these also provide portability to other apps. I do not use --exiftool for my personal backups because I want to preserve the original files.

Does it means that if we have only the edited images/videos + the associated AAE file, we can retrieve the original file + edited file in Photos after re-importing the database ?

No, you would need the original + edited + AAE files. In most cases, Photos will reconstruct the edited image from the original + AAE so you could probably skip the edited but there are cases where this doesn't work. See this discussion by @oPromessa for more on this. I export the edited images just to be safe.

I just thought of something. Does this mean that the videos from Live Photos will have the "VID" prefix? In that case, my renaming isn't ideal. To me, a Live Photo video is still considered an image.

No. Live Photos are "photos" with a video component but the asset in the library is a photo. The {photo_or_video} template evaluates the asset in the library and in this case, the asset type for Live Photo is "photo".

osxphotos export ~/Desktop/export --live --filename "{photo_or_video,photo=IMG;video=VID}_{created.strftime,%Y%m%d_%H%M%S}" --verbose
Exporting 1 photo to /Users/rhet/Desktop/export...
Exporting IMG_1853.HEIC (E537B989-1A00-4C87-91BD-73EE5684C56F.heic) (1/1)
Exported IMG_1853.HEIC to /Users/rhet/Desktop/export/IMG_20230323_101938.HEIC
Exported IMG_1853.HEIC to /Users/rhet/Desktop/export/IMG_20230323_101938.mov
Exported /Users/rhet/Desktop/export/IMG_20230323_101938.HEIC
Exported /Users/rhet/Desktop/export/IMG_20230323_101938.mov
Exporting edited version of IMG_1853.HEIC (E537B989-1A00-4C87-91BD-73EE5684C56F.heic)
Exported IMG_1853.HEIC to /Users/rhet/Desktop/export/IMG_20230323_101938_edited.heic
Exported IMG_1853.HEIC to /Users/rhet/Desktop/export/IMG_20230323_101938_edited.mov
Exported /Users/rhet/Desktop/export/IMG_20230323_101938_edited.heic
Exported /Users/rhet/Desktop/export/IMG_20230323_101938_edited.mov

I didn't try as I have issue with the filename command. H

Ah, I see the issue. In the TOML, it's not filename but filename_template:

[export]
filename_template = "{photo_or_video,photo=IMG;video=VID}_{created.strftime,%Y%m%d_%H%M%S}"

This is confusing (and not documented) but is there for historical reasons that would have caused issues in the code if I stuck with filename. The TOML format isn't documented unfortunately (happy to accept a PR) but I view this as "acceptable" because the command provides --save-config TOML_FILE to let OSXPhotos write the TOML. I don't edit the TOML by hand even for my own use -- I use the --save-config option. There are a few more options like this where the TOML name is different than the option but off the top of my head I don't recall what they are.

However, osxphotos was able to export this video.
How do you explain this behavior? (It's good news, by the way).

I'm not sure what's going on in this case. It may be a referenced file (not copied to the library) and Photos couldn't access it but OSXPhotos could. Photos stores paths to reference files via "security scoped bookmarks" to comply with App Sandboxing. OSXPhotos happily ignores the sandboxing.

I noticed that in an album where I had 2 photos and a video, if I performed a new import without making any changes in Photos, the .AAE files were updated.

I assume you meant export not import? I'm not sure what's going on in this case. In my tests, AAE files are not re-exported if there was no change.

Edited photos quality
Does this apply only to edited photos, or does it include original photos as well?

The original is the same file that was imported to Photos with no quality loss. OSXPhotos works by copying the edited image out of the Photos library to the export location so it will be at whatever resolution stored by Photos. OSXPhotos does not provide access to the fine-grained export controls available through the GUI. If you use --download-missing, the edited image is exported by Photos, not copying the file, so it is rendered at time of export but OSXPhotos still cannot control the export size/resolution. This is useful for videos as sometimes Photos does not create an edited version of a video but instead renders it at run time using the AAE. It's a bit of misnomer because --download-missing can be used in this case even when the library is not an iCloud library. What it really does is force OSXPhotos to use AppleScript to tell Photos to do the export. See this discussion for more information on this issue. It's something I'm working on but have not made progress.

Exported video quality
I noticed that videos that have been edited are in .mov format, whereas the original was in .mp4.
I don't think there's a command to choose the export quality of the video, like the --jpeg-quality command for images

Photos creates the edited version in .mov format. See previous response. There is no video equivalent to --jpeg-quality. --jpeg-quality only applies to --convert-to-jpeg which causes OSXPhotos to convert all non-jpeg photos (not videos) to jpeg prior to export. OSXPhotos uses hardware accelerated native macOS APIs to do the conversion from the image exported from the Photos library so this is a different case than the previous question about controlling export quality as per the Photos export GUI.