RAW files (RAF, CR2) have wrong face preview due to wrong dimensions

[skylatian]

The bug

This bug differs from #12440 in that the preview box correctly surrounds the person's face, but the preview image that shows in the People tab (or in file details) is not zoomed/cropped to the correct location in a photo.

This issue is fixed when manually selecting "Select Featured Photo." This cases the profile picture/preview for that particular person to be cropped correctly.

(note - the image shown vs. source for "featured thumbnail" are slightly different photos (taken a few seconds apart), but the effect happens with that photo as well and I didn't want to re-blur for anonymity)

The overall "People" tab. Every one of these is a validly recognized person, the thumbnail ("Featured Photo") is just misaligned.

The OS that Immich Server is running on

Windows 10 Server

Version of Immich Server

v1.116.2

Version of Immich Mobile App

n/a

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - "D:/nas/Media:/home/user/photos1:ro"
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false
      
  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:2d1463258f2764328496376f5d965f20c6a67f66ea2b06dc42af351f75248792
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c", "shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

volumes:
  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION="D:/immich-library"
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
# TZ=America/[]

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=[redacted]

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

The issue:

1. Open Photos tab
2. Witness faces not being centered

The manual fix:

1. Click on a person in the People tab (or from elsewhere)
2. Top right menu->Select Featured Photo
3. (Choose a photo)
4. See that Featured Photo now appears correctly system-wide

Relevant log output

No response

Additional information

No response

[C-Otto]

If I understand you correctly, each person's face photo (as shown on the "/people" URL, i.e. the "People" page) is showing a wrong part of the correct image (e.g. a plant behind the person instead of the actual head)?

[skylatian]

Correct, yes! That’s a much better way of putting it.

[C-Otto]

Thank you for the clarification! Do you know if there's anything special about the people/faces/photos? Did you modify them? Can you reproduce this (with new people/photos)?

[skylatian]

I didn't realize this until now, but it seems like it might only be happening with Fuji Raw (.RAF) files! The same behavior occurs if I select a .RAF photo as the preview manually. I just checked with Sony's Raw files (ARW) and they select the face correctly.

[C-Otto]

That's great (for debugging) :) Could you please provide a sample .RAF picture with a face on it?

[skylatian]

I will get back to you on that. For the life of me, I can't find one that uploads and causes the same issue. I appreciate your help!

[C-Otto]

It's possible that this issue was fixed in the meantime...

[skylatian]

I just updated from 116 to 117 earlier today, that might be it. All the scans had been done on 116. I’m re-running the face detection+recognition jobs now, I will report back once they’re done.

[stjeanp]

I just rebuilt all thumbnails in my library (~40k images), the majority of which are CR2 raw files. I'm seeing the exact same issue. The box around the face is correct, but the part of the photo used for the person's picture is wrong. I picked one person, opened an image with a (to me) good face, confirmed that the box was actually around his face, then went back to the person and set that as his featured photo. His featured photo is now a section of wall, not his face. I'm happy to share the file, if you need more data.

I'd rather not redo all of the face detection and grouping, because that was a lot of work, but if y'all think it'd be worthwhile, I'll kick it off tonight (US/Central) and it should be done by morning and will report back.

Edit: I just kicked off a full face detection/recognition job. Will report back when it's done.

I'm also running 1.117.0.

[C-Otto]

@stjeanp yes, please share the file and let us know of any pattern you might suspect (only CR2 raw files, for example?)

[stjeanp]

@C-Otto The redo of the face detection/grouping completed and I've found a good example, since this person only appears in five files, all of which are CR2 files. I can't attach the zip of them because it's too big, but if you message me, I can give you a URL to download it. Sorry. Here's some explanatory screenshots though. Please let me know what else you'd like me to do.

Before any changes were made:

Here's the person's face box:

And here's one where her feature photo is now a piece of the wall instead of her face, after selecting the topmost image to use as her feature photo:

[stjeanp]

Sorry, here's the face box on the image I used as the new featured photo:

[C-Otto]

Thanks, that looks helpful. I just sent a mail regarding the link to the raw file(s).

[stjeanp]

Got it, replied with the link. Let me know when you're good and I'll take it down.

[C-Otto]

Got it, thanks - feel free to delete the file.

[skylatian]

Looks like you got a sample file, thanks for taking care of that @stjeanp

I can confirm it's still happening after rebuilding all faces and people on 1.117. Thanks for your help @C-Otto

[C-Otto]

The image dimensions are off, for the images provided by @stjeanp I see 1728x1152 (2MP), while Gimp shows 3522x2348 (8MP). Using exiftool I get:

Image Width                     : 1728
Image Height                    : 1152
Canon Image Width               : 3504
Canon Image Height              : 2336
AF Image Width                  : 3504
AF Image Height                 : 2336
Exif Image Width                : 3504
Exif Image Height               : 2336
Image Size                      : 3504x2336

Any idea what might be the cause of having three different sizes?

This wrong image size does not only mess up the "generate face preview" bit, but also shows in the details pane on the screenshots above.

[C-Otto]

@skylatian do you notice any weirdness related to reported and actual image dimensions? You should be able to see what Immich thinks in the detail view (which you included in your screenshots). Maybe the actual images don't match these dimensions?

[skylatian]

@C-Otto I do, good catch.

• The immich-reported dimensions for the .RAF file are 2944 x 4416.
• All of my .RAF files show either 2944x4416 or 4416x2944 (depending on orientation) so it’s not random.
• The corresponding jpegs for each raw are 4160 x 6240 (as expected)
• If I download and check the .RAF in file explorer or finder, the dimensions correctly show as 4160 x 6240.

It seems immich is listing the resolution as scaled down by a factor of ~0.707 for mine compared to ~0.49 for @stjeanp’s. Sony RAWs do not have this problem. I have no idea why exiftool would report three sizes, maybe it’s something related to the baked-jpeg preview?

[mertalev]

There is an "Exif Image Width/Height" field in the CR2 sample above that's accurate. Can you check with exiftool if the RAF file also has this? We could prefer this field when it exists during metadata extraction.

[C-Otto]

Are you able to see/extract the JPG preview image that's part of the raw image? If so, is this preview smaller than the actual image? In https://exiftool.org/forum/index.php?topic=4709.0 this is mentioned as a possible cause.

With exiftool -b -previewImage -w .jpg IMG_8119.CR2 I get a JPG file that has size 1728x1152, confirming this idea. If this also holds true for RAF files, we might need to tweak exiftool or exiftool-vendored, or work around this in Immich itself.

[stjeanp]

I'm not sure where those Image Width values are coming from. Looking at the original in Lightroom, it agrees with the others that the dimensions are 3504x2336. I thought it might be one of the thumbnails, but the standard view one is 2160x1440. I'm hanging on to that hunch, but can't see the other thumbnails. I'm looking on the filesystem now.

[stjeanp]

Interestingly enough, the CR2 format allows for embedded, smaller JPEG versions of the image. I'm looking for a tool to do a deep dive into that CR2 file to see if it has any of that in it. Sort of a structure dump.

[skylatian]

from exiftool -'*height*' -'*width* -'*size*' filename.RAF I get (organized):

Exif Image Height               : 2944
Exif Image Width                : 4416

Image Height                    : 2944
Image Width                     : 4416

Raw Image Height                : 1008
Raw Image Width                 : 6276

Raw Image Full Height           : 4182
Raw Image Full Width            : 6384

File Size                       : 59 MB
AF Area Point Size              : 4
AF Area Zone Size               : n/a
Grain Effect Size               : Small
Raw Image Full Size             : 6384x4182
Raw Image Cropped Size          : 6240x4160
Raw Zoom Size                   : 6240x4160
Image Size                      : 6240x4160

I have no idea where that RawImageHeight term comes from.

Output of exiftool -b -previewImage -w .jpg confirms that my built-in jpeg preview is 2944×4416, seems like it's from Exif Image Height and Exif Image Width.

[mertalev]

Of course this one doesn't use Exif Image Width/Height for the original... that would be too simple. Image Size looks a bit more promising as it's correct for both images.

[C-Otto]

How many faces/people are affected? Even with the fix, you'd need to somehow fix the preview images (manually is one way, but there are others).

[stjeanp]

For me, I've got 900+ detected faces over ~40k images. I don't mind doing a rebuild once the fix is in. That might be onerous for some users though.

[skylatian]

@skylatian can you provide a RAF file where neither "ImageWidth" nor "ExifImageWidth" show the correct width?

Happy to, I'll open a PR later today.

Unfortunately, I looked at a raw (.DNG) from a family member's Ricoh GR3, and neither ImageWidth nor Image Size lists the correct resolution.

Image Height                    : 4064
Image Width                     : 6112
File Size                       : 31 MB
Default Crop Size               : 6000 4000
Preview Image Size              : 720x480
Image Size                      : 6112x4064

Image Size and Image Width are the same, but the correct resolution is Default Crop Size as confirmed by Lightroom and Finder. If I'm not mistaken, #13377 wouldn't address this. Thoughts?

[mertalev]

@skylatian can you provide a RAF file where neither "ImageWidth" nor "ExifImageWidth" show the correct width?

Happy to, I'll open a PR later today.

Unfortunately, I looked at a raw (.DNG) from a family member's Ricoh GR3, and neither ImageWidth nor Image Size lists the correct resolution.

Image Height                    : 4064

Image Width                     : 6112

File Size                       : 31 MB

Default Crop Size               : 6000 4000

Preview Image Size              : 720x480

Image Size                      : 6112x4064

Image Size and Image Width are the same, but the correct resolution is Default Crop Size as confirmed by Lightroom and Finder. If I'm not mistaken, #13377 wouldn't address this. Thoughts?

This is truly incredible.

At least using Image Size wouldn't be a regression in this case. Maybe we can consider Default Crop Size as well if it wouldn't cause issues elsewhere.

[stjeanp]

It's not going to bother the three Canon bodies I've got RAWs from, none of them use that tag.

stjeanp@gepetto:~$ ls Canon*
CanonEOS30D.CR2  CanonEOS70D.CR2  CanonPowerShotG12.CR2
stjeanp@gepetto:~$ fgrep -i 'default crop size' Canon*
stjeanp@gepetto:~$ 

[skylatian]

This is truly incredible.

Isn't it? Update on the Ricoh, though. I hadn't run exiftool with the -a flag previously, and running it with that reveals:

exiftool -'*height*' -'*width*' -'*size*' -'*jpg*' -a -g1 file.DNG

---- File ----
File Size                       : 31 MB
---- EXIF ----
Image Width                     : 160
Image Height                    : 120
Image Width                     : 6112
Image Height                    : 4064
Default Crop Size               : 6000 4000
Image Width                     : 6000
Image Height                    : 4000
---- MakerNotes ----
Preview Image Size              : 720x480
---- Composite ----
Image Size                      : 6112x4064

So... there are THREE Image Width tags? This is unique to the Ricoh, as far as I can tell. I agree Default Crop Size is probably the right move for this camera. That tag isn't present in the Fufi RAFs, though.

[C-Otto]

Please provide interesting example images so that I can tune the PR and retain that knowledge in automated tests. It doesn't make a lot of sense to break metadata extraction for other cameras, nor does it feel right to ignore information already noted in this issue. It's a mess, but if we're able to hide that mess in somewhat clean code, I'd say that's an improvement.

[C-Otto]

@skylatian could you provide a problematic DNG file, please?

[stjeanp]

I've had no luck tracking down a DNG from one of those cameras. @skylatian - have you been able to upload one?

[skylatian]

Apologies, I missed the previous comment! I have not, the only file I have is 31.2mb, which is larger than github's 25mb limit. If that would still work the for devs I would be happy to upload it to google drive or something though, just let me know. I won't be able to get a (potentially) smaller file sent to me for another week or so.

[C-Otto]

@skylatian absolutely, that would help! Thank you

[skylatian]

Of course! There are GPS coordinates included but it's not a privacy concern, I will have to delete it from my drive at some point though so let me know when I can take it down. Thanks for looking into this! https://drive.google.com/file/d/1XSicJPWW-3HDeBaV9wcty0SZJJrb53p4/

[C-Otto]

I got it, thanks

[C-Otto]

@skylatian that DNG file doesn't expose any issue in the current version of Immich. Could you please check?

[stjeanp]

@skylatian - sorry to be a pest, but have you had any luck finding a DNG that displays the issue or is it a red herring?

[skylatian]

@stjeanp shoot, apologies, I've been absolutely swamped recently. I appreciate the extra ping! it seems like it's a red herring, it seems to perform properly with immich.

[C-Otto]

Alright, #13377 is ready for review then