Changing Tag shortens MakerNote on Olympus JPEG

[tonke90]

I want to change the orientation tag of the original JPEGs of my old Olympus camera without orientation sensor and leave the rest as it is. But exiv2 will also shorten the MakerNote Tag, I don't know if this is related to the warnings.

I'm not sure if this is a bug or intended behavior. In case it is intended and it somehow fixes the files to be standard conform, I miss a option to not do that.

$ exiv2 -M"set Exif.Image.Orientation Short 8" -k exif.JPG   
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
$ diff <(exiv2 -pa orig.JPG) <(exiv2 -pa exif.JPG)
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
Warning: Directory OlympusCs, entry 0x0101: Strip 0 is outside of the data area; ignored.
4c4
< Exif.Image.Orientation                       Short       1  top, left
---
> Exif.Image.Orientation                       Short       1  left, bottom
26c26
< Exif.Photo.MakerNote                         Undefined 11388  (Binary value suppressed)
---
> Exif.Photo.MakerNote                         Undefined 9270  (Binary value suppressed)

[boardhead]

What model camera is this?

[tonke90]

Exif.Image.ImageDescription                  Ascii      32  OLYMPUS DIGITAL CAMERA         
Exif.Image.Make                              Ascii      24  OLYMPUS IMAGING CORP.  
Exif.Image.Model                             Ascii      17  E-PL1
Exif.Image.Software                          Ascii      32  Version 1.2  

[boardhead]

This model has maker notes that are actually much better behaved than other Olympus models, but the PreviewImage pointer seems to be invalid (in my sample at least), and points past the end of the file. It is possible that my file is missing the PreviewImage trailer, but it seems your file has the same problem.

The MakerNotes are reduced in size because there is a lot of empty space (filled with garbage bytes) in the maker notes of this model. This empty space is removed when the maker notes are rewritten. Re-writing of the maker notes is necessary for some other Olympus models where the values of some MakerNote tags run outside the size of the MakerNote data block. Without this feature, these tags would be lost, so it wouldn't be advisable to disable this feature if possible.

[tonke90]

Thanks for the detailed explanation! I was guessing that it somehow fixes the file, but I would expect a warning when something like this is happening, at least on a higher debug level.

Still, I would wish for an option to disable this behavior, as I want to do minimal changes to the files and the changes would be inconsistent to the other non-rotated files.

[tonke90]

Just for future reference, at least it seems like it is only rewriting the size of the MakerNote and is not doing more intrusive changes.

$ diff <(xxd -c 1 orig.JPG) <(xxd -c 1 exif.JPG)
67c67
< 00000042: 01  .
---
> 00000042: 08  .
381,382c381,382
< 0000017c: 7c  |
< 0000017d: 2c  ,
---
> 0000017c: 36  6
> 0000017d: 24  $

[boardhead]

This doesn't make sense to me. In my sample the empty areas are scattered throughout the maker notes, so just changing the size wouldn't work. Could you attach both images so I can take a look?

[tonke90]

Sure, first one is the out-of-camera Jpeg, second one is after modifying the orientation tag.

[boardhead]

Ooops. This appears to be a rather serious Exiv2 bug. Apparently the original MakerNotes have been written to the file, but the size has been changed to that of the reformatted MakerNotes. :(

Much of the MakerNotes may be lost if the edited image is subsequently edited by other software.

[tonke90]

Ops. Glad I could help to point it out!

Can you point me to an alternative option to really only change the orientation tag?

[tonke90]

The same seems to happen for the raw files:

$ exiv2 -M"set Exif.Image.Orientation Short 8" -k exif.ORF                          
Error: Offset of directory OlympusIp, entry 0x1104 is out of bounds: Offset = 0xfffff440; truncating the entry
Error: Offset of directory OlympusIp, entry 0x1104 is out of bounds: Offset = 0xfffff440; truncating the entry
$ diff <(exiv2 -ph orig.ORF) <(exiv2 -ph exif.ORF)
Error: Offset of directory OlympusIp, entry 0x1104 is out of bounds: Offset = 0xfffff440; truncating the entry
32c32
<   0000  01 00                                            ..
---
>   0000  08 00                                            ..
107c107
< 0x927c Photo        MakerNote                   Undefined 1446464 1446464
---
> 0x927c Photo        MakerNote                   Undefined 1212992 1212992
$ diff <(xxd -c 1 orig.ORF) <(xxd -c 1 exif.ORF)      
127c127
< 0000007e: 01  .
---
> 0000007e: 08  .
430,431c430,431
< 000001ad: 12  .
< 000001ae: 16  .
---
> 000001ad: 82  .
> 000001ae: 12  .
6100c6100
< 000017d3: 12  .
---
> 000017d3: 00  .

[boardhead]

Sorry, I don't know of anything that can do this. ExifTool rewrites the MakerNotes and compresses the maker notes to 9270 bytes (the same size the Exiv2 writes), except that the compressed MakerNotes are properly written to the updated file.

I'm not surprised that the same problem exists for raw files.

[tonke90]

Okay, thanks for your input. I noticed the more intrusive change in (your?) ExifTool, that is what got me to Exiv2. Nevertheless, thank you for the great work! It is highly appreciated :)

I though so, that the raw files are handled similar. Let me know if I can help, providing test files or testing myself.

[boardhead]

One more thing: The file you posted does contain the PreviewImage trailer., and the PreviewImageStart 0x0101 tag is valid and correct, although it does point outside the EXIF data.

Yes. I am the ExifTool author.

[tonke90]

Okay, what does that tell me? :) I'm pretty sure these are untouched originals, but I could later take new sample images.

[boardhead]

This means that there is no problem with the PreviewImage in your original JPG. It is more-or-less standard practice for the MakerNotes to reference a large PreviewImage outside the EXIF data. So I'm saying that you should ignore the Exiv2 warning for this.

[tonke90]

Oh, I see. Thanks, good to know!

[clanmills]

Happy New Year, everybody. This will be difficult to fix. I don't believe there's a work-around using exiv2.

Exiv2 decodes the metadata and stores it in memory (with the readMetadata() function). If the metadata is modified and rewritten (with the writeMetadata() function), all metadata, including the makernote, is rebuilt and written to file. The Olympus makernote is illegally "pointing" outside it's data and that's what the warning message is about. It's coming from readMetadata().

It's rather odd that exiv2 should know the new length of the maker note, and write the old data. However exiv2 is very nervous about the PreviewImage. My thoughts? This is a tough puzzle.

[clanmills]

I've pushed this to Exiv2 v0.27.4 scheduled for June 2021. I'm not sure this can be fixed. My priority at present is Exiv2 v0.27.3 scheduled for June 2020.

[clanmills]

This looks like a tough puzzle. I'll set the milestone to v1.00. Setting the milestone means it will be consider for fixing. It doesn't guarantee that a contributor will take on the challenge.